Mandiant and Ivanti's investigations into widespread Ivanti zero-day exploitation have continued across a variety of industry verticals, including the U.S. defense industrial base sector. Following the initial publication on Jan. 10, 2024, Mandiant observed mass attempts to exploit these vulnerabilities by a small number of China-nexus threat actors, and development of a mitigation bypass exploit targeting CVE-2024-21893 used by UNC5325, which we introduced in our "Cutting Edge, Part 2" blog post. 

Notably, Mandiant has identified UNC5325 using a combination of living-off-the-land (LotL) techniques to better evade detection, while deploying novel malware such as LITTLELAMB.WOOLTEA in an attempt to persist across system upgrades, patches, and factory resets. While the limited attempts observed to maintain persistence have not been successful to date due to a lack of logic in the malware's code to account for an encryption key mismatch, it further demonstrates the lengths UNC5325 will go to maintain access to priority targets and highlights the importance of ensuring network appliances have the latest updates and patches.

Ivanti customers are urged to take immediate action to ensure protection if they haven't done so already. A new version of the external Integrity Checking Tool (ICT), which helps detect these persistence attempts, is now available. See Ivanti's security advisory and refer to our updated remediation and hardening guide, which includes the latest recommendations.

The exploitation of the Ivanti zero-days has likely impacted numerous appliances. While much of the activity has been automated, there has been a smaller subset of follow-on activity providing further insights on attacker tactics, techniques, and procedures (TTPs). Mandiant assesses additional actors will likely begin to leverage these vulnerabilities to enable their operations.

To date, Ivanti has disclosed the following five vulnerabilities affecting Ivanti Connect Secure and other products. 

In our previous blog post, we described a mitigation bypass that was used to drop a newly identified BUSHWALK webshell. The mitigation bypass is now tracked as CVE-2024-21893. It is a server-side request forgery (SSRF) vulnerability in the SAML component of Ivanti Connect Secure (CS), Policy Secure (PS), and Neurons for Zero Trust Access (NZTA) appliances that was addressed in the patches and mitigations released on Jan. 31, 2024. 

Since that post, an additional vulnerability was reported on Feb. 8, 2024, by Ivanti, CVE-2024-22024, related to an XML External Entity (XXE) vulnerability in the SAML component that allows unauthenticated attackers to gain access to restricted resources on patched appliances.

Attribution

UNC5325

UNC5325 is a suspected Chinese cyber espionage operator that exploited CVE-2024-21893 to compromise Ivanti Connect Secure appliances. UNC5325 leveraged code from open-source projects, installed custom malware, and modified the appliance's settings in order to evade detection and attempt to maintain persistence. UNC5325 has been observed deploying LITTLELAMB.WOOLTEA, PITSTOP, PITDOG, PITJET, and PITHOOK. Mandiant identified TTPs and malware code overlaps in LITTLELAMB.WOOLTEA and PITHOOK with malware leveraged by UNC3886. Mandiant assesses with moderate confidence that UNC5325 is associated with UNC3886.

UNC3886

UNC3886 is a suspected Chinese espionage operator that has compromised network devices at targets where they leveraged novel techniques against virtualization technologies. They installed custom malware built for such technologies by leveraging code from open-source projects as well as exploiting zero-day vulnerabilities. UNC3886 has primarily targeted the defense industrial base, technology, and telecommunication organizations located in the US and APJ regions. We are continuing to gather evidence and identify overlaps between UNC3886 and other suspected Chinese espionage groups, including targeting and the use of distinct tactics, techniques, and procedures (TTPs). 

New TTPs and Malware

Since our last blog post on Ivanti exploitation, Mandiant has identified UNC5325 exploiting CVE-2024-21893 (SSRF) to deploy additional malware and maintain persistent access to compromised appliances. In addition, we have observed new TTPs that attempted to enable the custom backdoors to persist across factory resets, system upgrades, and patches. The limited attempts observed to maintain persistence have not been successful to date.

Exploitation of CVE-2024-21893 (SSRF)

Mandiant identified active exploitation of CVE-2024-21893 by UNC5325 as early as Jan. 19, 2024, targeting a limited number of Ivanti Connect Secure appliances.

On Jan. 31, 2024, Ivanti disclosed CVE-2024-21893, a server-side request forgery (SSRF) vulnerability in the SAML component of Ivanti Connect Secure, Ivanti Policy Secure, and Ivanti Neurons for ZTA. To date, we have only identified successful exploitation against Ivanti Connect Secure appliances.

In the same Jan. 31, 2024, announcement, Ivanti released a new XML mitigation to prevent exploitation of all four (4) disclosed CVEs at the time of the announcement. This included:

• CVE-2023-46805 (authentication bypass)
• CVE-2024-21887 (command injection)
• CVE-2024-21888 (privilege escalation)
• CVE-2024-21893 (server-side request forgery)

CVE-2024-21893 allowed for an unauthenticated attacker to exploit an appliance by chaining the previously disclosed command injection vulnerability as described in CVE-2024-21887. This includes appliances with the XML mitigation released on Jan. 10, 2024.

Chaining CVE-2024-21893 (SSRF) and CVE-2024-21887 (Command Injection)

Shortly after the disclosure of CVE-2024-21893, Mandiant observed threat actors chaining the SSRF vulnerability with the command injection vulnerabilities described in CVE-2024-21887 to exploit vulnerable devices.

In some instances, publicly available services, such as Interactsh, were used to validate whether the target was vulnerable to CVE-2024-21893.

Shortly after a vulnerable target was identified, the threat actor executed follow-on commands to perform reconnaissance and, in some cases, establish a reverse shell.

Identifying Exploitation Attempts

Exploitation of the SSRF vulnerability in the SAML component generates up to two (2) log events and some host-based artifacts on an affected appliance.

If the Ivanti Connect Secure appliance is configured to log unauthenticated requests, event ID AUT31556 is generated when an unauthenticated attacker requests the vulnerable SAML endpoint, /dana-ws/saml.ws. The event includes the source IP address of the unauthenticated request.

In addition, the server fails to gracefully handle the maliciously crafted SAML payload to exploit CVE-2024-21893. The appliance generates an error event log entry with event ID ERR31903 when the saml-server process crashes, which is potentially indicative of an exploitation attempt.

We recommend analyzing both allocated and unallocated disk space on the forensic image for the presence of the log events as we have observed the threat actor deleting the relevant log files.

Lastly, the crash of the saml-server process generates core dumps located in /data/var/cores/. If the core dumps are available, it is possible to extract the crafted SAML message, HTTP headers of the request, and the source IP address. We have observed the threat actor deleting the contents of the cores directory, but we have successfully recovered relevant fragments of the core dumps through file carving.

BUSHWALK Variant

In Cutting Edge, Part 2, we introduced a new web shell tracked as BUSHWALK associated with the exploitation of CVE-2024-21893 and CVE-2024-21887. Similar to other web shells observed in this campaign, BUSHWALK is written in Perl and embedded into a legitimate Ivanti Connect Secure component, querymanifest.cgi.

Mandiant identified a new variant of BUSHWALK through our incident response engagements. This new variant of BUSHWALK was identified on a compromised appliance less than twelve (12) hours following Ivanti's disclosure of CVE-2024-21893 on Jan. 31, 2024. The variant is similar to the BUSHWALK sample described in our previous blog post, but with a new function named checkVerison that enables arbitrary file read from the appliance. The function is executed when the decrypted payload contains the string check. Figure 5 shows the relevant checkVerison function.

Note that we have observed the same RC4 key for decrypting issued commands across the two BUSHWALK variants and all identified samples.

In addition, we have seen the threat actor demonstrate a nuanced understanding of the appliance and their ability to subvert detection throughout this campaign. We identified a technique allowing BUSHWALK to remain in an undetected dormant state by creatively modifying a Perl module and LotL technique by using built-in system utilities unique to Ivanti products.

To accomplish this, the threat actor first modifies a Perl module, DSUserAgentCap.pm, that evaluates incoming user agents. The modification enables the threat actor to either activate or deactivate BUSHWALK depending on the incoming HTTP request's user agent.

Figure 6 provides the excerpt of the modification in DSUserAgentCap.pm. Note the difference in spelling between App1eWebKit and AppIeWebKit in the two user agent strings.

An encrypted version of BUSHWALK is placed in a directory excluded by the integrity checker tool (ICT) in /data/runtime/cockpit/diskAnalysis. 

The activation routine (the if block) uses a built-in utility on the appliance located in /home/bin/configdecrypt used for decrypting the system's configuration. The routine executes the configdecrypt utility to decrypt diskAnalysis containing the BUSHWALK web shell. It then makes a backup of the original querymanifest.cgi file, adds it to the exclusion_list, moves BUSHWALK to the web server directory, and restarts the web server to load the web shell.

The deactivation routine (the elseif block) restores the original querymanifest.cgi file, timestomps it using touch to hide their activity, removes the path of BUSHWALK from exclusion_list, and restarts the web server. However, the encrypted version of BUSHWALK remains dormant in a dynamic directory and therefore is not scanned by the integrity checker tool. It continues to quietly persist in /data/runtime/cockpit/diskAnalysis until the threat actor activates it again.

The internal ICT is configured to run in two-hour intervals by default and is meant to be run in conjunction with continuous monitoring. Any malicious file system modifications made and reverted between the two-hour scan intervals would remain undetected by the ICT. When the activation and deactivation routines are performed tactfully in quick succession, it can minimize the risk of ICT detection by timing the activation routine to coincide precisely with the intended use of the BUSHWALK webshell.

SparkGateway Plugin Abuse

In a limited number of instances following exploitation of CVE-2024-21893, we identified the use of SparkGateway plugins to persistently inject shared objects and deploy backdoors. SparkGateway is a legitimate component of the Ivanti Connect Secure appliance that enables remote access protocols over a browser, such as RDP or SSH. The functionality of SparkGateway can be extended through plugins.

PITFUEL Plugin

Mandiant identified a SparkGateway plugin named plugin.jar (PITFUEL) that loads the shared object libchilkat.so (LITTLELAMB.WOOLTEA) through the Java Native Interface (JNI) by calling System.load(). The shared object persistently deploys backdoors and contains capabilities to persist across system upgrade events, patches, and factory resets.

Figure 7 shows the relevant excerpt of the PluginManager class in PITFUEL.

Upon execution, libchilkat.so (LITTLELAMB.WOOLTEA) performs a number of initialization routines to ensure that it persistently runs in the background on the compromised system. It accomplishes this by daemonizing itself, attempting to trap SIGPIPE, SIGKILL, and SIGTERM signals, and adjusting the out of memory (OOM) adjustment value (oom_adj) to -17 to keep the process running even when the system is out of memory.

Persistence Across System Upgrades and Patches

Upon first execution, LITTLELAMB.WOOLTEA executes the first_run() function. It calls the edit_current_data_backup() function that appends its malicious components to an archive, /data/pkg/data-backup.tgz. Figure 8 provides the equivalent command sequence.

During a system upgrade or when applying a patch, data-backup.tgz contains a backup of the data directory that is restored after the upgrade event. In addition, the function timestomps data-backup.tgz by calling utimensat. This modification would ensure its malicious components (plugin.jar, libchilkat.so, and gateway.conf) persist across system upgrades and patches.

In addition, the malware contains a function named upgrade_monitor() that supports persistence across system upgrade and patch events. We assess that this acts as a secondary persistence method by making a modification at the precise moment of a system upgrade or patch event.

It monitors for system upgrade events by continually checking the filesystem for the existence of /tmp/data/root/dev. This path is used to support a system upgrade process. In other words, the presence of the path indicates to the malware the existence of a system upgrade event.

If the path exists, it intervenes the system upgrade process by appending itself and its constituent components into the archive /tmp/data/root/samba_upgrade.tar. During a system upgrade process, the appliance decompresses samba_upgrade.tar for data migration purposes. Figure 10 provides the command executed by upgrade_monitor() when it detects the existence of /tmp/data/root/dev.

During the system upgrade or patch process, the post-install bash script executes the following to decompress samba_upgrade.tar, copying the malicious components (libchilkat.so, plugin.jar, and gateway.conf) to the new active partition. Figure 11 provides the relevant command sequence from post-install.

Attempted Persistence Across Factory Resets

Next, LITTLELAMB.WOOLTEA executes first_run(), which reads and checks the hardware of the appliance by reading the first four (4) bytes of the motherboard serial number at /proc/ive/mbserialnumber and adjusts its behavior to mount the root partition of the factory reset image for further modification.

If the four (4) bytes match the strings 0331, 0332, 0340, 0481, or 0482, the malware executes the following command to mount /dev/md5 (factory reset root partition) on /dev/loop5.

Each of the four-byte strings corresponds to a physical Pulse Secure Appliance (PSA) or a Ivanti Secure Appliance (ISA) product.

Otherwise, the malware executes the following command to mount /dev/xda5 (factory reset root partition) on /dev/loop5 if the four (4) bytes do not match any of the machine ID strings or if it fails to read /proc/ive/mbserialnumber.

Next, LITTLELAMB.WOOLTEA mounts the newly created loop device (/dev/loop5) to /tmp/tmpmnt to modify the factory reset root partition. Figure 14 provides the equivalent command sequence.

It's important to note that /bin/losetup uses an embedded encryption key within the running version's kernel used to decrypt the running version's partition. This encryption key is hardcoded at the time of build compilation and is unique for each appliance version.

However, the factory reset partition maintains its own independent encryption key embedded in the factory kernel. If the current running version and the factory reset deployment versions differ (i.e., the appliance or VM has been updated at least once), then /bin/losetup will fail to decrypt the factory reset partition due to the encryption key mismatch and thus the malware will not persist after factory reset.

Note that Mandiant and Ivanti conducted forensic analysis on an affected appliance after factory reset to confirm no evidence of malware persistence. Because the appliance had undergone at least one update since its initial deployment, the malware failed to persist through the factory reset as the encryption key of the factory reset kernel and the running version kernel were different.

If losetup had succeeded in decrypting the factory reset image, the malware would continue its persistence workflow. To modify the factory reset process, it calls the edit_factory_reset() function that renames the tar binary to tra in the mounted factory reset partition.

Then, the malware writes a trojanized version of the tar binary to /tmp/tmpmnt/bin/tar, makes the tar binary executable, and preemptively appends its malicious components (using the legitimate tar utility) to the archive /tmp/tmpmnt/bin/samba_upgrade.tar inside the factory reset partition.

The trojanized tar binary checks for a set of specific conditions to copy the malicious /bin/samba_upgrade.tar to /tmp/samba_upgrade.tar during the factory reset process. 

• There are four arguments provided (argc is equal to 4)
• The second argument, argv[1], is -cf 
• The fourth argument, argv[3], is no-data

If any of these conditions are not met, the trojanized tar binary executes the legitimate tar (/bin/tra) utility backed up in Figure 15.

The conditions are satisfied by a component of the factory reset procedure responsible for resetting the configuration (dsconfigreset). The utility creates an empty file in /tmp/no-data and archives it using /bin/tar -cf. Figure 17 provides the relevant command sequence.

When dsconfigreset executes /bin/tar -cf $tmp_part no-data, the trojanized tar copies the contents of /bin/samba_upgrade.tar containing its malicious components to /tmp/samba_upgrade.tar in the factory reset root partition (mounted on /tmp/tmpmnt).

Next, similar to the previously described system upgrade persistence flow, the appliance executes the post-install bash script during the installation process of the new system. This script decompresses the samba_upgrade.tar archive in the factory reset partition, copying the malicious components (libchilkat.so, plugin.jar, and gateway.conf) to the new active partition created after the factory reset.

Hooking the Web Server Process

The httpd_monitor() function ensures the persistent injection of another shared object, libaprhelper.so (PITSOCK), into the web process using a built-in injection function named inject_loop(). 

PITSOCK hooks the functions accept and setsockopt of the web process by modifying its procedure linkage table (PLT). This enables backdoor communication via the Unix socket /tmp/clientsDownload.sock when it receives a specific 48-byte magic byte sequence in the incoming buffer.

Creating the Malicious SparkGateway Plugin

Lastly, libchilkat.so calls persist(), which modifies the SparkGateway configuration file. Figure 18 shows an excerpt from the modified SparkGateway configuration file to support and load the plugin.

Backdoor Features

libchilkat.so also serves as a stand-alone backdoor that supports expected features such as command execution, file management, shell creation, SOCKS proxy, and network traffic tunneling. It communicates over SSL using the private key located on the Ivanti Connect Secure web server (/home/webserver/conf/ssl.key/secure.key) and communicates using the socket /tmp/clientsDownload.sock.

PITDOG Plugin

Mandiant identified a second malicious SparkGateway plugin named security.jar (PITDOG) that uses Kubo Injector (memorysCounter) to inject a shared object, mem.rd (PITHOOK), into the web process memory, and persistently executes a backdoor, dsAgent (PITSTOP). Figure 19 shows the relevant excerpts from security.jar.

The SparkGateway configuration is modified to load the plugin. Figure 20 shows the relevant excerpt from gateway.conf.

The security.jar plugin is executed during the negotiation of an RDP connection when the system invokes the Handshake plugin. The getHandshakePlugin() method creates a new thread from a Runnable interface that repeatedly calls SparkPlugin.watchdog() every ten (10) seconds. This acts as a persistence method to ensure the continuous execution of the malicious watchdog method without interfering with the primary operation of the SparkGateway application.

The watchdog method first checks if the shared object mem.rd (PITHOOK) is mapped within the web process memory. If not, it injects mem.rd into the web process. PITHOOK hooks the accept and accept4 functions within the web process by modifying the PLT. When PITHOOK receives a buffer matching the predefined magic byte sequence, it will duplicate the socket and begin communication with the Unix domain socket /data/runtime/cockpit/wd.fd.

Figure 21 shows the command executed to inject PITHOOK (mem.rd) into the web process, where %d represents the process ID (PID) of the web process.

We determined that /data/runtime/cockpit/memorysCounter is a direct instance of Kubo Injector without any additional modifications or changes. Kubo Injector is based on the popular linux-inject project, a utility that can inject a shared object into an arbitrary process given a process name or process ID.

Lastly, the watchdog method will execute the PITSTOP backdoor (/data/runtime/cockpit/dsAgent) if it is not already running.

PITSTOP listens on the Unix domain socket located at /data/runtime/cockpit/wd.fd created by PITHOOK when it receives the predefined magic byte sequence. Then it duplicates the socket for further communication over TLS. When the socket is established, PITSTOP uses Base64 and a hard-coded AES key to evaluate the incoming command. It supports shell command execution, file write, and file read on the compromised appliance.

Outlook and Implications

UNC5325’s TTPs and malware deployment showcase the capabilities that suspected China-nexus espionage actors have continued to leverage against edge infrastructure in conjunction with zero days. Similar to UNC4841’s familiarity with Barracuda ESGs, UNC5325 demonstrates significant knowledge of the Ivanti Connect Secure appliance as seen in both the malware they used and the attempts to persist across factory resets. Mandiant expects UNC5325 as well as other China-nexus espionage actors to continue to leverage zero day vulnerabilities on network edge devices as well as appliance-specific malware to gain and maintain access to target environments.

The material in this blog post is being shared as cyber threat indicators and defensive measures solely for cybersecurity purposes in accordance with the Cybersecurity Information Sharing Act of 2015 (“CISA/2015”).  This information is subject to the provisions of CISA/2015, including 6 U.S. Code § 1504(d)(1).

Indicators of Compromise (IOCs)

Host-Based Indicators (HBIs)

YARA Rules

Mandiant Security Validation Actions

Organizations can validate their security controls using the following actions with Mandiant Security Validation.