Bench Talk for Design Engineers

Bench Talk


Bench Talk for Design Engineers | The Official Blog of Mouser Electronics

Identify Anomalous Device Behavior Through Logging and Alerting Jeff Fellinge

As more and more embedded devices are added to already crowded networks, spotting attackers becomes more difficult every day. In this blog, we’ll look at a few of the technologies available that give defenders an advantage in detecting attackers and what you as a system designer or implementer can do to help in these efforts. We’ll start by looking at a typical intrusion detection system (IDS), followed by more sophisticated security information event management (SIEM) systems and how you as a device developer can help support these controls.

Logging and Alerting Using a Traditional Intrusion Detection System

A traditional IDS attempts to discover attackers by their actions. An IDS might run on a network or host and compare the traffic that it sees against its own library of signatures to identify events that could be a part of a larger attack. In many cases, these systems can only alert on what they themselves see. Depending on the quality of its signatures and configuration, an IDS might be prone to a higher false positive rate, or events that match a signature but do not denote a real attack. Tuning an IDS for your network is an essential step in reducing the number of false positives. The tuning process is affected by the quality of the signatures, how often they are updated, how much the IDS system knows about its environment, and where the IDS sensors are located. For example, installing a network IDS sensor on the outside of your firewall will be very noisy and not necessarily useful in detecting attackers that have successfully penetrated your network. However, installing a network IDS sensor on the outside of your firewall might yield interesting data from the internet or help create a demographic of where certain attacks originate. Alerts triggered by an IDS sensor installed on a quiet network—say one used for administrative purposes only—should be treated much differently and investigated accordingly.

Evading Detection on a Traditional Intrusion Detection System

Sophisticated attackers understand how an IDS works and will construct attacks that attempt to evade detection. One technique an attacker might use is to simply access your network using legitimate access points. An attacker that logs into a VPN concentrator using a phished (stolen) username and password from an unsuspecting user might look legitimate to the IDS system. However, while the individual login event might not trigger an alert, the subsequent actions taken by the attacker might be detected by another type of monitoring system. For this reason, an IDS alone is generally not enough to detect all types of attacks. Successful attack detection requires sharing good information from a variety of logs and events that can be correlated and filtered to present high-quality alerts that defenders can investigate.

Take for example these two actions:

  • First, an employee who works in the finance group remotely logs into the network and is assigned an IP address.
  • Then, an engineering database server is accessed by that IP address minutes later.

Separately these events might look normal. However, correlating these events together might raise a flag as to why is a finance user accessing an engineering system. Correlation is possible with multiple sources of events combined with strong analytics and rules customized to the specific environment. The more data that is available, the stronger the correlation that can be created.

Log data ingested by correlation engines should be sourced from a variety of systems and across business groups. For example, did a user log into an IT-managed VPN concentrator after the human resources system records for that user were disabled? That answer might require authentication logs from the VPN concentrator as well as HR system data queries. As another example, correlating whether physical access to a building by an individual that occurred within a few hours from a remote login by that same user from another country might require cooperation between a landlord and an IT group.

Logging and Alerting Using SIEM Systems

SIEM systems provide platforms for correlation and event alerting. SIEM platforms offer a powerful defense layer that helps defenders discern true signals and clues left by an attacker from day-to-day noise and activity. Remember though that the out-of-the-box rules are rarely ever enough and determining and building the right use cases from which to build the correlation rules takes time and deep knowledge of the network environment.

Your device must generate appropriate events and share these logs with other monitoring and SIEM systems to be secure. Most modern embedded devices include the capability to forward event logs a remote server. The specific operating system and firmware that powers an embedded device might facilitate logging that you can augment with your own application-specific events. Many devices support syslog, a standard for message logging. As the end user of a device that supports syslog, you will generally be able to configure where to forward any events as well as specify how much information is sent—e.g., a critical event or just a warning.

Managing Events in a SIEM System

As a designer of a device, you will have much more control. You can enable which events should be sent and how the events will be formatted. When designing a new embedded device, consider how your device will reside on the network and how the events it generates might enhance the broader monitoring system in place. Ask yourself what events and alerts you could add to your device to help a defender discover anomalous behavior even if it’s not on your device. Configure your device to log an event whenever an interesting activity occurs on your device. These activities could include when a user logs in, when a user escalates their privilege (such as sudo, or enable), when a user writes a configuration, or when a user reboots the system. Separately these events might not be significant but in conjunction with other events they may corroborate a bigger story. By making this type of information available, defenders can use your device activity along with other data to enhance their detection processes.

In addition to defining and logging specific events, be sure to include enough metadata about the event to be useful to the defender. For example, a user log-in event might come with a TCP/IP address and time-stamp, but it would be much more helpful to include with the event message additional data like the username, the make and model of the device, and whether the login was successful or failed. Machines will read your events and people will use business intelligence tools and slicers to pivot your data, so it is important to create your events such that they are easy to parse.

To help the operations center include your device into their systems, be sure to clearly document the structure and nature of the events you will support. Including a description of the events and their schema will also teach the defenders t how your device operates and what they should expect when watching its behavior on the network. Also, document which network protocols your device will use and with what systems it will generally communicate. This information is a part of the data flow diagram that describes how your device will communicate with other systems.


Logging and alerting are fundamental tools for detecting not only errors and creating maintenance tickets but also helping defenders detect attackers by piecing together individual events into a bigger picture. When working with network-connected embedded devices, no matter how unimportant your device may be, it is still another node on the network. And whether your device is the target of an attacker or a bystander, the events that it creates and logs and then shares with other security systems can be critical to discovering and stopping attackers.

Please see my other security blog entries including Surface Area Management of Embedded Devices through Host and Network Based Firewalls to learn more security best practices.

« Back

Jeff Fellinge has over 25 years’ experience in a variety of disciplines ranging from Mechanical Engineering to Information Security. Jeff led information security programs for a large cloud provider to reduce risk and improve security control effectiveness at some of the world’s largest datacenters. He enjoys researching and evaluating technologies that improve business and infrastructure security and also owns and operates a small metal fabrication workshop. 

All Authors

Show More Show More
View Blogs by Date