Note: This is a developing campaign under active analysis. We will continue to add more indicators, hunting tips, and information to this blog post as needed.

On Oct. 10, 2023, Citrix released a security bulletin for a sensitive information disclosure vulnerability (CVE-2023-4966) impacting NetScaler ADC and NetScaler Gateway appliances.

Mandiant has identified zero-day exploitation of this vulnerability in the wild beginning in late August 2023 as well as n-day exploitation after Citrix’s publication. Mandiant is investigating multiple instances of successful exploitation of CVE-2023-4966 that resulted in the takeover of legitimate user sessions on NetScaler ADC and Gateway appliances. The session takeovers bypassed password and multi-factor authentication.

In this blog post, we will discuss artifacts that can be used to identify exploitation activity and highlight some of the post exploitation techniques we observed during the incident response investigations.

Vulnerable Endpoints

When Citrix released firmware updates to address CVE-2023-4966, Mandiant used the same approach outlined by Assetnote on its blog to determine the vulnerable functions and develop a proof of concept (PoC). Prior to Oct. 10, Mandiant was conducting investigations where a threat actor was taking over user’s NetScaler sessions through unknown means. Prior to Citrix’s publication and our development of a PoC, we believed the session takeovers were the result of zero-day exploitation of an unknown vulnerability. Using differential firmware analysis, we identified the vulnerable endpoint and developed a PoC to validate the vulnerability.

By crafting an HTTP GET request with an HTTP Host header greater than a certain length, a vulnerable appliance will return contents of system memory. The memory contents can include a valid Netscaler AAA session cookie. This cookie is issued post-authentication, which can include multi-factor authentication checks. An attacker with access to a valid cookie can establish an authenticated session to the NetScaler appliance without knowledge of the username, password, or access to a multi-factor authentication token or device.

Investigation

The challenge of investigating a vulnerable appliance for the exploitation CVE-2023-4966 is that the webserver running on the appliance does not record requests (or errors) to the vulnerable endpoint. Mandiant is not aware of any configuration change that can be made to force request logging for these endpoints. To identify attempted exploitation requests, organizations will need to rely on web application firewalls (WAF) or other network appliances that record HTTP/S requests directed toward the NetScaler ADC or Gateway appliances.

In light of this, Mandiant has used the following approaches to identify potential exploitation of CVE-2023-4966 and subsequent session hijacking. These include the following techniques:

• Investigating requests to the vulnerable HTTP/S endpoint from WAF
• Identifying suspicious login patterns based on NetScaler logs
• Identifying suspicious virtual desktop agent Windows Registry keys
• Analysis of memory core dump files

Note: Due to the limited forensic evidence left by the exploitation of this vulnerability, identifying historic exploitation is challenging. Patterns of suspicious activity related to session hijacking might differ from organization to organization, and the techniques outlined as follows might not be applicable or feasible in all scenarios.

Suspicious Login Patterns on NetScaler

There are two types of logs identified that can be useful in identifying historical evidence of session hijacking after the successful exploitation of CVE-2023-4966. While these logs can be used to hunt for evidence of suspicious login patterns, determining if a session was hijacked by an attacker requires contextual knowledge of the organization and further analysis. For example, analysts need to consider during analysis whether the appliance is recording the true client IP addresses (opposed to network address translation [NAT] addresses), and what normal user behavior looks like (do users shift IP addresses frequently).

Ns.log Analysis

Citrix NetScalers record syslog information in the ns.log files, stored under /var/log/ns.log. The log messages recorded in the nslog can include connection statistics for SSLVPN and ICA proxy sessions.

Note: The default log rotation configuration on NetScaler allows 25 files per log type (e.g., ns.log) and 100 Kilobytes per log, therefore recording 2.5 megabytes in total. In most cases, this log size will not have enough space to cover the full period of an investigation. Additionally, log forwarding via syslog may be configured to forward logs outside the NetScaler appliance.

1. Sessions accessed from a different source IP address
   Suspicious SSLVPN sessions can be identified by comparing the IP addresses recorded in the Client_ip and the Source fields in TCPCONNSTAT events. TCPCONNSTAT events record TCP connection-related information for a connection belonging to an SSLVPN session. See Figure 2 for a sample log event (Mandiant has replaced field values with placeholders).

   Figure 2: Example TCPCONNSTAT message with mismatched Client and Source IP addresses

   <log timestamp> GMT hostname 0-PPE-1 : default SSLVPN TCPCONNSTAT 87607893 0 : Context userA@1.2.3.4 - SessionId: 1234 - User userA - Client_ip 1.2.3.4 - Nat_ip 192.168.1.1 - Vserver 192.168.2.1 - Source 5.6.7.8 - Destination 192.168.0.1:443 - Start_time " 2023/10/10:01:00:00 GMT" - End_time "2023/10/10:01:00:00 GMT" - Duration 00:00:00  - Total_bytes_send 0 - Total_bytes_recv 585 - Total_compressedbytes_send 0 - Total_compressedbytes_recv 0 - Compression_ratio_send 0.00% - Compression_ratio_recv 0.00% - Access Allowed - Group(s) "Group A"

   From Mandiant’s analysis, the Client_ip field is based on the client IP address from which the SSLNVPN session was originally created, and the Source field is based on the client IP address of the connection being logged. For hijacked sessions, Mandiant has observed TCPNCONNSTATS events where there was a mismatch between the Client_ip and Source fields, and the source IP address was suspicious upon analysis. Bear in mind that a change of IP addresses during a session can also happen due to legitimate circumstances like changing networks (e.g., enabling/disabling Wi-Fi, switching to VPNs, etc.). Investigating ASN and geo-location of both Client_ip and Source values can give some insight if this behavior is potentially malicious or a false-positive. NetScaler appliances may record similar connection statistics in other log messages as well.

2. Multiple sessions from the same IP address
   Another detection opportunity identified by Mandiant was source IP addresses that access multiple user accounts in a short period of time recorded in the ns.log files or forwarded logs via syslog. While a threat actor can choose only to access a single account from a single source IP address, Mandiant has observed that multiple accounts were accessed within hours from the same source IP address by a threat actor.
   
   Note that multiple user sessions originating from the same IP address can also occur legitimately, depending on the networks used by the organization (e.g., employees from a remote location using the same Internet Point of Presence to access Citrix sessions, NAT IP addresses, etc.).

Windows Registry Keys

When a user session is successfully brokered by the NetScaler appliance, the destination Citrix Virtual Delivery Agent (VDA) system records the source host, source host local IP, and associated UserSid in the Windows Registry of the Citrix endpoint.

By correlating these Registry values with the timestamps and user accounts recorded in the ns.log and potentially associated with hijacked sessions, the attacker’s originating hostname (i.e., DESKTOP-1ABC2DE3) and local host IP (i.e., 10.0.2.15) can be determined. Once identified, the attacker hostname and local IP address can be used to hunt for other attacker sessions across the Citrix VDA environment in both Registry and Windows event logs (specifically Event IDs 4624, 21, 23, 24, and 25). This data can also be used to add confidence to suspicious sessions identified in the log files, for example, if the recorded client hostname is a default Windows hostname (i.e., DESKTOP-########).

Memory Analysis of NetScaler Memory Core Dump Files

In some cases, evidence of exploitation can be found in the memory space of the NSPPE process of the appliance. Exploitation of CVE-2023-4966 will not crash the NSPPE process and generate memory core dump files. To perform analysis of NetScaler memory core dump files, they need to be collected. It is important not to reboot the appliance prior to generating a core dump for analysis.

To collect these files, Mandiant recommends following the instructions published by Citrix and summarized here.

1. Check that there is more than 5 GB of disk space on the NetScaler device available (instructions from vendor). 
2. Create NSPPE core dump files on NetScaler (instructions from vendor).
3. Collect NSPPE core dump files from NetScaler. Copy the core dump files from /var/core/<N>, where N is the highest numbered folder. The dump files will start 'NSPPE-'.

Within core files of appliances that were exploited with CVE-2023-4966, there were human-readable strings with more than 120,000 characters, filled with more than 99% “0” characters. This string was part of the response object that disclosed memory regions. Searching for strings contained in the legitimate response to the vulnerable endpoint can also point to exploitation if the FQDN of the appliance in the response string is a long sequence of characters rather than the legitimate FQDN. Figure 3 shows a one-liner to print all the readable strings in the core files with their length and additional processing.

Figure 4 shows the expected server response to the OAuth configuration endpoint. The placeholder value will be replaced server-side by the value of the HTTP Host header.

Figure 5 shows a truncated sample of evidence of exploitation attempt taken from a core dump file. In this case, the threat actor sent a request with the HTTP Host header set to 24,799 “0” characters.

Important note: The aforementioned evidence does not guarantee the exploitation was successful. Evidence of exploitation attempts may be found in core dump files generated on the latest NetScaler versions that are not vulnerable to CVE-2023-4966. Unfortunately, it's not possible to distinguish between a failed and successful exploitation attempt with this data alone.

Post-Exploitation Activity

Following the successful exploitation of CVE-2023-4966, Mandiant has observed a variety of post-exploitation tactics, techniques, and procedures (TTPs). Once an actor was able to successfully achieve session hijacking, the threat actor performed actions including host and network reconnaissance of the victim environment, credential harvesting, and lateral movement via RDP. Mandiant identified evidence of Active Directory reconnaissance using living-off-the-land binaries such as net.exe. Additionally, Mandiant has observed the use of the SoftPerfect network scanner (netscan.exe) to perform internal network enumeration.

In several cases, the threat actor used 7-zip to create an encrypted segmented archive to compress the reconnaissance results. The threat actor then used the built-in certutil utility to Base64 encode the segments.

In one case, certutil was used to decode multiple files related to credential theft. Mandiant observed the threat actor use e.exe to load d.dll into lsass process memory. When run, the utility creates a memory dump file located at %temp%\1.png and prints success to the console when done. The memory dump file can be processed offline by the threat actor to extract credentials. Mandiant identified sh3.exe as a utility suspected to run the Mimikatz LSADUMP command.

In another instance, a threat actor used certutil to decode a file that Mandiant identified as a newly tracked backdoor that uses Slack as its command and control. Tracked by Mandiant as FREEFIRE, it is a lightweight backdoor written for .NET. FREEFIRE communicates to a hard-coded channel to retrieve commands and upload responses. It supports loading arbitrary .NET assemblies encoded as Base64 sent to it via chat comments. Mandiant observed FREEFIRE being deployed by a threat actor through the following certutil command.

Mandiant has also observed the deployment of various remote monitoring and management (RMM) tools following the successful exploitation of CVE-2023-4966. Currently, Mandiant has observed the deployment of Atera, AnyDesk, and SplashTop to establish and maintain a foothold following exploitation of CVE-2023-4966.

Victimology

Mandiant is investigating intrusions across multiple verticals, including legal and professional services, technology, and government organizations. Given the widespread adoption of Citrix in enterprises globally, we suspect the number of impacted organizations is far greater and in several sectors. The victims have been in the Americas, EMEA, and APJ as of writing.

Attribution

Mandiant is currently tracking four distinct uncategorized (UNC) groups involved in exploiting this vulnerability. We have observed some lower degrees of confidence overlaps in post-exploitation stages among these UNC groups, like using the same recon commands and utilities available on Windows. The common tools observed across multiple intrusions were:

• csvde.exe
• certutil.exe
• local.exe
• nbtscan.exe

Two threat clusters used Mimikatz for dumping process memory. Notably, there were no overlaps in infrastructure between these clusters of activity. The exploits were sourced from different VPN provider IP addresses and previously compromised third-party devices.

Remediation

Mandiant published a blog post with remediation recommendations for this vulnerability on Oct. 17, 2023. The post includes a link to our remediation guide document.

Acknowledgements

Mandiant would like to thank Dhanesh Kizhakkinan of the FLARE OTF team for their help analyzing the CVE and the many incident responders and analysts who helped develop analysis methodologies and identify IOCs.

YARA Rules

MITRE ATT&CK Techniques