Hacking Operational Technology for Defense: Lessons Learned From OT Red Teaming Smart Meter Control Infrastructure
High-profile security incidents in the past decade have brought increased scrutiny to cyber security for operational technology (OT). However, there is a continued perception across critical infrastructure organizations that OT networks are isolated from public networks—such as the Internet. In Mandiant’s experience, the concept of an ‘air gap’ separating OT assets from external networks rarely holds true in practice.
In 2018, we released a blog post presenting the tools and techniques that TEMP.Veles used during the TRITON incident to traverse from an external compromise of the information technology (IT) network of a critical infrastructure organization to the safety systems located deep in the OT network. We regularly reproduce this approach in our OT-focused red team engagements to expose similar attack paths across client infrastructure and to identify environment specific opportunities to prevent and detect network propagation techniques across intermediary systems.
In this blog post, we share another case study from one of our OT Red Team engagements to illustrate the tactics, techniques, and procedures (TTPs) that can be leveraged by sophisticated threat actors to breach the protected perimeter between an IT network and an OT network. We also examine some of the different types of critical information often found in IT networks that an attacker can leverage during later stages of the Targeted Attack Lifecycle. The goal of this engagement was to access an endpoint meter control infrastructure for a state-wide smart grid environment from the Internet and turn it off.
To hear our experts relay more on this and other OT Red Team lessons learned, join our FireEye Mandiant Virtual Summit session.
Building the Foundation: Information Gathering for IT-OT Network Propagation
Targeted OT attacks attempting to cause physical impacts require planning. A sophisticated actor who is motivated to disrupt or modify an industrial process from a public network will necessarily need to maintain access to the victim environment and remain undetected for enough time to accomplish their objective. Throughout this time, the actor will strive to learn about the control process to formulate the attack, figure out how to pivot to the OT systems and bypass security controls, and sometimes even develop or deploy custom OT malware.
Similar to the techniques used by TEMP.Veles to reach the OT network during the TRITON incident, Mandiant’s experience during red team engagements highlights that collecting information from IT network assets plays a crucial role in targeted OT attacks. As a result, the internal reconnaissance phase for OT targeted attacks begins in the enterprise network, where the actor obtains knowledge and resources to propagate from an initial compromise in the IT network to remote access in the OT network. Detailed information collected about the target, their security operations, and their environment can also support an actor's attempts at remaining undetected while expanding operations.
Figure 1: Targeted OT attack from a public network
Thinking Like an Adversary: How to Turn Off Smart Energy Meters
The ideal scenario for an attacker targeting OT systems is to achieve their objective while remaining undetected. Mandiant’s Red Team works with clients across critical infrastructure industries to simulate attack scenarios in which actors can accomplish this goal by gaining access to OT systems via compromise of external facing IT networks. During these engagements, we emulate real actor behaviors to learn about our target and to determine the best paths for IT/OT network propagation.
For this engagement, we simulated an end-to-end OT-specific attack scenario in which we tested the security controls and response capabilities of an organization to protect smart grid meter control infrastructure from an external attacker. Mandiant leveraged weaknesses in people, process, and technology to gain remote access from the public Internet and to achieve a set of pre-approved objectives in the OT environment.
Establishing a Foothold in the IT Network
Mandiant conducted a spear phishing exercise to gain initial access into the client’s enterprise network from the Internet. We defined a combination of two different phishing scenarios that we deployed to test email filtering and egress monitoring controls:
- Embedded link for a malicious file hosted on a Mandiant owned domain on the Internet
- Email attachment for a Microsoft Office document with auto - executable macro code
This exercise allowed our team to achieve code execution on a user workstation in the enterprise environment and to establish an unattributable egress communication path to a Mandiant hosted Cobalt Strike Command and Control (C&C) server on the Internet. After establishing a stable communication path with workstations in the enterprise environment, we utilized the following publicly available offensive security tools (OST) to escalate privileges and to obtain domain administrator level access:
- ldapsearch to enumerate information in the enterprise domain
- PowerSploit to exploit common security misconfigurations in IT
- WMImplant to move laterally from one system to another in the internal network
- Mimikatz to extract credentials for local user and domain administrator accounts
As domain administrators, we gained unrestricted access to a variety of resources connected to the enterprise domain (e.g. server resources, file shares, IT applications, and administrator consoles for IT systems). During the initial stages of our engagement, our actions were in no way different to other less sophisticated intrusions on industrial organizations, such as financially-motivated compromises.
Defining Our Path to the OT Network
Similar to real world threat actors carrying out targeted OT attacks, Mandiant’s OT Red Team dedicates significant effort for internal reconnaissance in the IT network to develop a logical mapping of the extended network architecture and discover targets of interest (people, processes, or technology). The information we acquire helps us to (a) define paths to propagate from the IT to the OT network and (b) achieve our final objective in the OT network without raising alarms. During OT Red Team engagements across different industries, we follow a real attacker’s cost-benefit analysis to determine which sources or methods are most likely to help us obtain that information.
Figure 2: Information sources and target information from enterprise networks
For this engagement, we leveraged the domain administrator credentials obtained in the previous phase to gain access to Microsoft System Center Configuration Manager (SCCM) in the IT network. Logged into the SCCM console, we leveraged software deployment features for collection to establish C&C over user workstations belonging to pre-selected departments in the client organization.
Mandiant chose the specific groups based on the names of their departments and the description attributes, which suggested a high likelihood of member users with high privilege access for network infrastructure or application management. This included members of the following groups: network management, firewall administration, control engineering, and smart meter operations.
Access to user workstations of target employees in these departments enabled us to:
- Capture keystrokes to obtain remote desktop protocol (RDP) credentials for the OT network by using a Cobalt Strike modified script
- Login to department file shares and extract OT system design documents
- Extract network design documents and backup files for OT firewall configurations found in the firewall management console
- Find plaintext credentials for OT management systems from operation manuals
Internal reconnaissance in the IT network not only allowed us to obtain remote access credentials for the OT network, but to also gain a deeper understanding of the business processes and technical control system operations in the OT environment by reviewing internal OT-specific operational procedures and security documentation such as asset inventories and configurations.
Propagating to the OT Network
During the process of propagation from IT to OT networks, an actor will leverage previously compromised systems, credentials, or applications to access systems in higher security zones—such as OT demilitarized zones (DMZ). Based on our observations during multiple red teaming engagements and research, the most likely attack vectors for propagation are:
Table 1: Most likely attack vectors for IT/OT propagation
For this engagement, we initially analyzed the system architecture to define the best path to follow. Engineers from the target organization were required to use multi-factor-authentication (MFA) to gain remote access to jumpbox servers in the OT network. While not impossible, bypassing this setup would require more time and resources. We instead decided to search for other plausible attack propagation paths.
Figure 3: Formal communication path from enterprise to OT network
Reviewing the firewall configuration files, we identified a dedicated communication path for management access to a Microsoft Windows patch management server in a DMZ between the IT network and the OT network. This patch management server was running on a virtual machine in the DMZ network, while the administration console for the underlying hypervisor software itself was hosted in the IT network.
Mandiant logged into the administration console for the hypervisor software using IT network domain administrator credentials. We then leveraged guest machine administration features via direct console access to execute commands on the patch management server in the DMZ network. The compromise of the patch management server in the DMZ allowed us to pivot via SMB connections to Microsoft Windows-based intermediary systems in the OT network.
Figure 4: Remote attack propagation path from IT network to OT network
Lastly, we compromised Microsoft Windows server systems in the OT network to complete the objectives of the exercise. Using OT credentials retrieved in the previous phases, we authenticated to the SMB service (using single factor authentication) by pivoting through the patch management server in the DMZ network. This enabled us to execute remote console commands on management servers (such as the domain controller) in the OT network.
With access to the domain controller in the core OT network, we extracted credentials for high privilege domain administrator accounts in the OT network using DCSYNC and Mimikatz. Using these accounts, we gained control of management servers, application servers, and operator workstations in the OT network. Mandiant was also able to use compromised credentials to login to the human machine interface (HMI) portal for the meter control infrastructure and issue a disconnect command for a target endpoint meter in the smart grid environment.
Strategic Collection and Detection Opportunities During Reconnaissance and Network Propagation
Although specific capabilities such as malware and tooling vary amongst incidents, internal reconnaissance and network propagation are consistently needed for sophisticated adversaries to expand remote operations from external networks to OT systems. Focusing collection, detection, and hunting efforts on assets or information that are likely to be compromised during these phases presents defenders with strategic opportunities to hunt for and detect targeted adversary activity before it poses a risk to control systems.
- In a previous blog post stating our approach to OT security, we highlighted that IT networks close to production networks and OT intermediary systems remain the best zones to detect OT targeted attacks, a.k.a. “The Funnel of Opportunity”. As actors pivot across systems and networks to gather information and elevate privileges, they leave footprints that can be tracked before they propagate to critical systems.
- An actor who covertly performs internal reconnaissance and propagates to the OT network is already positioned to cause damage on mission critical assets and is unlikely to be discovered. Early detection of adversary activity before reaching critical OT systems will decrease the dwell time and the risk of an incident.
- OT defenders can prioritize collection and detection, alert triage, and incident response efforts by becoming familiar with the types of information and services that OT focused threat actors commonly search for during internal reconnaissance in IT networks and network propagation across OT intermediary systems.
- Understanding where this information resides presents defenders with a catalog of systems and networks to focus collection and detection efforts on. Defenders can create tailored detections to hunt for adversary activity pursuing this information, prioritize alert response efforts, and identify additional security controls to implement. Mandiant red teaming in OT can help organizations identify which data is valuable for attackers to support their network propagation efforts and which systems are most likely to be compromised by attackers targeting OT networks.