In early 2022, Mandiant, in partnership with Schneider Electric, analyzed a set of novel industrial control system (ICS)-oriented attack tools—which we call INCONTROLLER (aka PIPEDREAM)—built to target machine automation devices. The tools can interact with specific industrial equipment embedded in different types of machinery leveraged across multiple industries. While the targeting of any operational environments using this toolset is unclear, the malware poses a critical risk to organizations leveraging the targeted equipment. INCONTROLLER is very likely state sponsored and contains capabilities related to disruption, sabotage, and potentially physical destruction.

INCONTROLLER represents an exceptionally rare and dangerous cyber attack capability. It is comparable to TRITON, which attempted to disable an industrial safety system in 2017; INDUSTROYER, which caused a power outage in Ukraine in 2016; and STUXNET, which sabotaged the Iranian nuclear program around 2010. To help asset owners find and defend against INCONTROLLER, we have included a range of mitigations and discovery methods throughout this report. As future modifications to these tools are likely, we believe behavior-based hunting and detection methods will be most effective.

If you need support responding to related activity, please contact Mandiant Consulting. Further analysis of related threats is available as part of Mandiant Advantage Threat Intelligence. 

This report is related to information shared in CISA Alert (AA22-103A). For more information from Schneider Electric, please see their bulletin. For more information from CODESYS, please see their advisory.

INCONTROLLER is comprised of three main components:

INCONTROLLER Was Built to Manipulate and Disrupt Industrial Processes

Industrial automation networks rely on a variety of equipment that enable operators to translate information and instructions into chains of physical actions. Given the diversity of assets present in industrial networks, industrial automation equipment typically speaks different languages across different portions of the network, which is possible using standardized industrial communication protocols.

INCONTROLLER includes three tools that enable the attacker to send instructions to ICS devices using industrial network protocols, such as OPC UA; Modbus; Codesys, which is used by EcoStruxure Machine Expert and SoMachine; and Omron FINS. While the tool's capabilities could enable the actor to communicate with a variety of products from different original equipment manufacturers (OEMs), the actor developed modules for specific controllers from Schneider Electric and Omron. The targeted equipment consists of machine automation solutions whose use cases span from supporting simple, repetitive machines to complex modular machines in distributed architectures:

• OPC servers
• Schneider Electric Modicon M251, Modicon M258, and Modicon M221 Nano PLCs
  • Other devices leveraging Modbus and Codesys may also be affected
• Omron NX1P2 and NJ501 PLCs and R88D-1SN10F-ECT servo drive
  • Other devices from NJ and NX PLC series may also be affected

We highly doubt that the threat actor would target these devices at random. It is more likely they were chosen because of reconnaissance into specific target environment(s). We note that this would be consistent with previous ICS malware, such as TRITON, which targeted a critical safety system that was almost certainly identified prior to compromising the target's industrial environment.

INCONTROLLER: Tooling Overview

TAGRUN

TAGRUN's capabilities, such as the ability to scan for and enumerate OPC UA servers, suggests a reconnaissance role. OPC acts as a central communications protocol to collect and store data from ICS assets in industrial environments. Access to this data can provide attackers with a detailed overview of production systems and control processes. The tool was likely developed for reconnaissance, but it can also write and change tag values, which could be used to modify data to either support an attack or mask process changes. TAGRUN also verifies whether the target environment is running a Windows operating system and provides different ping commands depending on this check's return value. This suggests that the actor may use non-Windows devices to execute TAGRUN.

TAGRUN’s capabilities include:

• Scanning for OPC UA servers on a network
• Reading the structure of OPC UA servers
• Reading/writing tag values for data on an OPC UA server
• Brute forcing credentials
• Outputting log files

CODECALL

CODECALL communicates with ICS devices using the Modbus protocol, which potentially gives it the ability to interact with devices from different manufacturers. However, the tool contains a specific module to interact with, scan, and attack Schneider Electric's Modicon M251 (TM251MESE) PLC using Codesys, which is used by the company's proprietary EcoStruxure Machine Expert protocol. We have reason to believe the tool also targets Schneider Electric's Modicon M221 Nano PLC and the Modicon M258 PLC, and it potentially affects additional devices leveraging these protocols.

CODECALL’s general capabilities include:

• Identifying Schneider Electric and Modbus-enabled devices on a network
• Connecting to specific devices over Modbus or Codesys
• Reading/writing device registers over Modbus
• Requesting device ID from a session over Modbus
• Defining, dumping, or loading command macro file(s)
• Executing device specific commands over Codesys, such as:
  • Attempting to login using a username/password and by brute forcing credentials with a provided dictionary file
  • Downloading/uploading files to the PLC device
  • Retrieving file/directory listings
  • Deleting files
  • Disconnecting sessions from the PLC device
  • Attempting a DDoS attack
  • Crashing the device with a specifically crafted packet
  • Adding a route if the device gateway IP exists on a different interface
  • Sending custom raw packets

OMSHELL

OMSHELL is designed to obtain shell access to Omron PLCs, including Omron NX1P2, NJ501, R88D-1SN10F-ECT servo drive, and possibly other similar devices from the NJ/NX product lines. The tool primarily operates using the HTTP protocol, however it also utilizes Omron's proprietary FINS over UDP protocol for scanning and device identification. The framework is modular, which means the attacker can develop and deploy additional capabilities into the tool.

OMSHELL’s capabilities include:

• Scanning for and identifying Omron devices on the network
• Wiping the device’s program memory and resetting the device
• Loading backup configuration and backup data from or restoring data to the device
• Activating the telnet daemon on the device
• Connecting to the device via the telnet daemon, uploading and optionally executing an arbitrary payload or command
• Connecting to a backdoor present on a device and providing arbitrary command execution
• Performing a network traffic capture
• Killing arbitrary processes running on the device
• Transferring files to the device
• Connecting and communicating with attached servo drives

We have reason to believe that indicator-based detections would not be effective at detecting INCONTROLLER in victim environments, in part because the attacker would almost certainly modify or customize the tool prior to using it in a specific victim environment. Instead, defenders should focus their efforts on behavior-based hunting and detection methods for these tools.

Potential Supporting Windows Tooling

We are also tracking two additional tools affecting Windows-based systems that may be related to this threat activity. It is possible that these tools could be used to support the overall attack lifecycle in an INCONTROLLER attack by exploiting Windows-based systems in IT or operational technology (OT) environments.

• One of the tools exploits CVE-2020-15368 in the AsrDrv103.sys driver, which would result in installation and exploitation of a vulnerable driver. ASRock motherboards may be leveraged in some human-machine interfaces (HMIs) and engineering workstations in OT environments.
• The other tool, which we track as ICECORE, is a backdoor providing reconnaissance and command and control functionality.

Attack Scenarios

It is feasible that each tool could be used independently, or the actor may use the three tools to attack a single environment. We highlight that the devices targeted by INCONTROLLER are often integrated in automation machinery (e.g., a milling machine or press) and could plausibly be present in a variety of industrial sectors and processes even without the user's explicit knowledge.

We developed three cyber physical attack scenarios that highlight a range of possible outcomes from an attack using INCONTROLLER. In each of the three cases, TAGRUN could have been used at earlier stages to enumerate the victim environment, identify its targets, and learn about the physical process.

The impact of these scenarios would depend on the nature of the victim facility and the extent of the attacker's understanding of and interaction with the controlled physical process. We note that our current understanding of INCONTROLLER is still limited given that it leverages an extensible structure that can support new features implemented by the author.

INCONTROLLER Is Very Likely State-Sponsored Malware

We believe INCONTROLLER is very likely linked to a state-sponsored group given the complexity of the malware, the expertise and resources that would be required to build it, and its limited utility in financially motivated operations. We are unable to associate INCONTROLLER with any previously tracked group at this stage of our analysis, but we note the activity is consistent with Russia's historical interest in ICS. While our evidence connecting INCONTROLLER to Russia is largely circumstantial, we note it given Russia's history of destructive cyber attacks, its current invasion of Ukraine, and related threats against Europe and North America.

• Since at least 2014, Russia-nexus threat actors have targeted ICS assets and data with multiple ICS-tailored malware families (PEACEPIPE, BlackEnergy2, INDUSTROYER, TRITON, and VPNFILTER).

• INCONTROLLER's functionality is consistent with the malware used in Russia's prior cyber physical attacks. For example, the 2015 and 2016 Ukrainian blackouts both involved physical process manipulations combined with disruptive attacks against embedded devices. INCONTROLLER similarly allows the malware operator to manipulate physical processes, while also containing denial-of-service (DoS) capabilities to disrupt the availability of PLCs.

Recommendations

While the nature of any potential intended victims remains uncertain, INCONTROLLER poses a critical risk to organizations with compatible devices. The targeted devices are embedded in multiple types of machinery and could plausibly be present in many different industrial sectors. Given the consistencies with prior Russia-nexus threat activity, we suggest that INCONTROLLER poses the greatest threat to Ukraine, NATO member states, and other states actively responding to Russia's invasion of Ukraine. Organizations should take immediate action to determine if the targeted ICS devices are present in their environments and begin applying vendor-specific countermeasures.

We also recommend that at-risk organizations conduct threat hunts to detect this activity in their networks. Mandiant Advantage Threat Intelligence subscribers have access to additional reporting containing threat hunting guidance and YARA detections.

If you need support responding to related activity, please contact Mandiant Consulting. Further analysis is available as part of Mandiant Advantage Threat Intelligence.

Mitigations

OPC UA

We recommend several steps to mitigate risk and counter malicious activity in environments using this protocol:

• Proper segmentation of IT and OT networks to aid in preventing attackers pivoting from corporate networks into industrial environments.
• Allow listing accepted primary/subordinate devices, behavior patterns, and commands to aid in establishing approved baselines and detecting anomalies with the aid of network monitoring.
• Implementation of an industrial firewall with deep packet inspection to aid in controlling access and approved capabilities.
• Implementation of ICS-aware intrusion protection systems to aid in monitoring for function codes from potentially malicious sources.
• Monitoring and blocking of external traffic to OPC UA ports, when possible, to aid in detecting anomalous traffic and prevent external network traffic directed at OPC UA-associated ports.
• Enabling and aggregating audit logs for OPC servers and clients.
• Periodic reviewing of audit logs for inconsistent or nefarious connections, security options negotiations, configuration changes, and user interaction.

Schneider Electric

To help keep your Schneider Electric products secure and protected, it is in your best interest that you implement the cyber security best practices as indicated in the Cybersecurity Best Practices document provided on the Schneider Electric website: Recommended Cybersecurity Best Practices White paper | Schneider Electric.

Additionally, Cybersecurity Guidelines for EcoStruxure Machine Expert, Modicon and PacDrive Controllers and Associated Equipment User Guide could help you ensure that only legitimate users can access your Schneider Electric product: Cybersecurity Guidelines for EcoStruxure Machine Expert, Modicon and PacDrive Controllers and Associated Equipment, User Guide | Schneider Electric.

You should pay special attention to features and cyber security devices that help to restrict access to authorized users only. This includes examples such as intrusion detection systems, network firewalls, secure remote access, device authentication, device firewall, disabling/filtering unsecure or programming protocols.

Omron

According to public vulnerability notices, Omron has previously identified other vulnerabilities that use the same or similar FIN ports that are used by OMSHELL. Omron's guidance for unpatched vulnerabilities, as noted in their security brief, indicates that external firewall filtering of identified FIN ports can be used as a mitigation. Mandiant believes that the recommended methodology may be a viable mitigation, though this mechanism has not been tested with INCONTROLLER. Additional guidance related to Omron's previous recommendations can be found in the related ICS Advisory for that older vulnerability.

Discovery Methods

TAGRUN

• Search for and investigate irregular connections to OPC UA endpoints and enable robust audit logging for OPC UA applications. Aggregate OPC UA logs and audit records to a central location where applicable.
• Review OPC UA audit records for evidence of credential bruteforcing, nefarious certificate usage, irregular connection attempts, configuration changes, and changes to OPC tags.
• Search for and investigate TAGRUN ping command execution.
• Review OT network traffic for evidence of pingsweep activity.

CODECALL

• Enable robust logging for Schneider Electric PLC devices and aggregate logs to a central location where applicable.
• Review Schneider Electric device logs for evidence of the following activity:
  • Credential bruteforcing
  • Error codes associated with abnormal device crashes/reboots
  • Files uploaded or downloaded
  • File deletion
  • Unauthorized changes in device configuration and execution of commands
  • Connections to devices outside of documented norms for the device and environment
• Search for and investigate evidence of ARP scanning followed by abnormal Modbus/Codesys traffic differing from environment baselines.
• Search for abnormal Modbus and Codesys traffic flows compared to environment baselines.

OMSHELL

• Search for and investigate evidence of the creation/existence of OMSHELL-related host-based indicators on systems with access to OT resources and connectivity (e.g., packet captures).
• Enable robust logging for Omron PLC devices and aggregate logs to a central location where applicable.
• Review Omron device logs for evidence of the following activity:
  • Activation of Telnet daemon on the device.
  • Unauthorized Telnet connection attempts including the use of default credentials.
  • Wiping of PROGRAM memory and device resets.
  • Unauthorized changes in device configuration and execution of commands.
  • Connections to devices outside of documented norms for the device and environment.
  • Files uploaded or downloaded.
• Identify and investigate nefarious pingsweep scanning activity, telnet traffic, and HTTP traffic on systems with access and connectivity to OT resources/devices:
• Search for and investigate evidence of Omron FINS traffic outside of standard norms and environment baselines.

Collect, identify, and investigate nefarious HTTP POST data to Omron devices containing Omron API commands.

Appendix: MITRE ATT&CK for ICS Mapping

Appendix: YARA Rules

Acknowledgements

This research was made possible thanks to the hard work of many people not listed on the byline. A huge thanks to the Schneider Electric Team, Mandiant Advanced Practices, FLARE, Consulting, Managed Defense, and everyone else who supported this effort.

Special thanks to Jared Scott Wilson, Glen Chason, Benjamin Read, Jonathan Leathery, Conor Quigley, and Wesley Mok.