Security Bulletin: Inject Splunk to Detect HTML Injection in Citrix NetScaler

Cece Kintner
November 8, 2023
12:05 pm

By Bryan Bollou, Team Lead, Cybersecurity Engineering

Purpose of TekStream Security Bulletins

With the TekStream Security Bulletin, we are presenting some specific detection use cases using everyone’s favorite SIEM, Splunk. We’ve cherry-picked vulnerabilities that are not only intriguing but also directly impactful for our valued clients. These vulnerabilities were chosen based on a multitude of factors, ranging from the technology in the crosshairs to the specific sectors being targeted. This is not a blog post to fully explain or give recommendations on remediating the vulnerability – this has been discussed at length by various resources. The goal here is to aggregate the detections to maximize your chances of detecting an attempt to exploit these vulnerabilities. A part of that is gathering the list of IOCs scattered in multiple locations on the internet and looking at activity that could point to the vulnerability being exploited. Here at TekStream, we have an amazing team of cybersecurity engineers armed with a deep knowledge of logs and the secrets they hold, ready to fortify your cyber resiliency.

Introduction

Confirmed to have been in use since September of 2023, the CVE-2023-3519 is an unpatched vulnerability exploited to capture credentials by inserting a malicious script into the HTML content of the authentication web page. This is vulnerability that could be exploited by doing the following;
• A vulnerable NetScaler device is identified.
• The script is appended to the index.html using the vulnerability.
• This script then loads a remote JS file that creates a “Log On” button on the login page.
• Once an authentication event occurs the username and password information are sent to a malicious remote server.

Threat Overview

The vulnerability has been acknowledged by Citrix and they have suggested various mitigation steps to prevent its exploitation. These steps include:

“Cloud Software Group strongly urges affected customers of NetScaler ADC and NetScaler Gateway to install the relevant updated versions as soon as possible. Note: NetScaler ADC and NetScaler Gateway version 12.1 is now End Of Life (EOL). Customers are recommended to upgrade their appliances to one of the supported versions that address the vulnerabilities.”

So far, we have seen many good options and would like to present a few more specific cases using everyone’s favorite SIEM Splunk. This is not a blog post to fully explain or give recommendations on remediating the vulnerability, this has been discussed at length by various resources such as: Citrix ADC and Citrix Gateway Security Bulletin.

The goal here is to aggregate the detections to maximize your chances of detecting this critical attack. A part of that is gathering the list of IOCs scattered in multiple locations on the internet and looking at activity that could point to the CVE-2023-3519 vulnerability being exploited. Here at TekStream, we have several security engineers that are intimate with cybersecurity knowledge and especially the logs of our clients.

In this blog post, we go through the various steps in CVE-2023-3519 vulnerability exploitation and detection. With this format, we are providing a more generic data model “tstats” command. This is taking advantage of the data model to quickly find data that may match our IOC list. We then provide examples of a more specific search that will add context to the first find. There will be a wide variety of specific searches as each client has their own specific technologies. To get help building detection for your specific client, fill in the form below and get access to our Splunk/security expertise.

Note: Something specific. Detecting this part is not covered in this blog post. Reach out to TekStream by filling out the form below if you’d like specific guidance on that aspect.

The following detection steps mirror the stages and TTP-related indicators used in the attack:

Detect the initial web request attackers use to append custom HTML code to the index.html hosted on a vulnerable NetScaler device
Detect the hashes and file names that point to malicious files and scripts injected into the vulnerable NetScaler device
Detect Ip Addresses associated with known ATP group that used this exploit as well as addresses seen associated with the attack
Detect connections to domains that hosted the malicious JS script file created by threat actors who exploited this vulnerability
Detect a possible spike in DNS traffic that could point to attempts to hit a command-and-control destination from a victim’s network

Threat Implication

A threat actor discovering an unpatched NetScaler Device on a victim machine can attempt this exploit. It involves injecting a malicious script that creates an element on a login page that directs user credentials upon login activity.
The affected Citrix products are as follows:

Reference: Citrix ADC and Citrix Gateway Security Bulletin

Threat Detection

Step 1

Attacked would first send a web request to trigger a memory corruption a write a PHP web shell on a NetScaler endpoint. This PHP web shell would then be used to append custom HTML code to the index.html that points to a malicious a JavaScript code on infrastructure controller by the attacker. This code would add a custom function with a “Log On” element that would harvest credentials entered by a victim.

Actions to Detect

A web request to your NetScaler servers with any of the following strings;

/gwtest/formssso? event=start&target=

/items/accounts/

/api/now/sp/widget/widget-simple-list

/api/now/sp/widget/5b255672cb03020000f8d856634c9c28

SPL

| tstats `summariesonly` sum("Web.bytes") as bytes,values("Web.http_content_type") as http_content_type,values("Web.http_method") as http_method,values("Web.http_user_agent") as http_user_agent,values("Web.status") as status from datamodel="Web"."Web" where (http_method = GET OR http_method = POST) AND Web.url IN ("*/gwtest/formssso? event=start&target=*","*/items/accounts/*","*/api/now/sp/widget/widget-simple-list*","*/api/now/sp/widget/5b255672cb03020000f8d856634c9c28*") by "Web.src", "Web.dest", "Web.user", "Web.url", _time | fields _time src, dest, user, url, http_content_type, http_method, http_user_agent, status, bytes | fieldformat Time = strftime(_time,"%m/%d/%Y %T")

Note: The destination IPs for the NetScaler devices can be added to filter down this search.

Step 2

As part of Step 1, new files would be introduced on the NetScaler server. This step would be used to detect the hashes and file names that point to malicious files and scripts injected into the vulnerable NetScaler device. These hashes and files were recorded from victims of this attack.

This is thoroughly explained here: CISA.GOV Cybersecurity Advisories

Actions to Detect

Appearance of the following file names in the victim’s Netscaler devices:

Logouttm[.]php
pykeygen
info[.]php
NPS
The

Appearance of the following hashes in the victim’s environment:

SHA-1 Hash

3345ad8b43d6771532ca55acf7e95fe98aeadb27 – Logouttm[.]php

b8f9258e9c7c882944e2b773557b49e5821517fe – pykeygen

55c83cb25be5c521e4874e22b1422c360abf0a29 – info[.]php

ec17f7b3acb2334a062c37e0271c31c3c8b6879c – NPS

dec36a2aa4ecb86eafd01015e27c06f51d104317 – the

MD5 Hash

47b8d29319a9e85aaf829f4f2f9c6c00 – Logouttm[.]php

ee51f599d266be5fd89b423ca24521f1 – pykeygen

c0b32901f9c6ce3f965b8552f7d058d4 – info[.]php

9b6f12aed2369a99a1132edd2ede779d – NPS

b383c4ed6712a0a06fb85b1d8405e0f1 – the

SPL 1: File Names

| tstats `summariesonly` latest(_time) as _time, latest(Filesystem.file_create_time) as file_create_time, latest(Filesystem.file_modify_time) as file_modify_time, latest(Filesystem.file_access_time) as file_access_time from datamodel="Endpoint"."Filesystem", values(Filesystem.dest) as dest, values(Filesystem.action) as action, values(Filesystem.file_name) as file_name, values(Filesystem.file_hash) as file_hash, values(Filesystem.file_path) as file_path, values(Filesystem.file_size) as file_size where Filesystem.file_name IN ("*Logouttm.php*","*pykeygen*","*info.php*",”*NPS*”,”*the*”)  | fields _time, dest, action, file_name, file_hash, file_path, file_size, file_create_time, file_modify_time, file_access_time
| fieldformat Time = strftime(_time,"%m/%d/%Y %T")

SPL 2: Hashes

| tstats `summariesonly` latest(_time) as _time, latest(Filesystem.file_create_time) as file_create_time, latest(Filesystem.file_modify_time) as file_modify_time, latest(Filesystem.file_access_time) as file_access_time from datamodel="Endpoint"."Filesystem", values(Filesystem.dest) as dest, values(Filesystem.action) as action, values(Filesystem.file_name) as file_name, values(Filesystem.file_hash) as file_hash, values(Filesystem.file_path) as file_path, values(Filesystem.file_size) as file_size where Filesystem.file_hash IN ("3345ad8b43d6771532ca55acf7e95fe98aeadb27","b8f9258e9c7c882944e2b773557b49e5821517fe","55c83cb25be5c521e4874e22b1422c360abf0a29","ec17f7b3acb2334a062c37e0271c31c3c8b6879c","dec36a2aa4ecb86eafd01015e27c06f51d104317","47b8d29319a9e85aaf829f4f2f9c6c00","ee51f599d266be5fd89b423ca24521f1","c0b32901f9c6ce3f965b8552f7d058d4","9b6f12aed2369a99a1132edd2ede779d","b383c4ed6712a0a06fb85b1d8405e0f1")  | fields _time, dest, action, file_name, file_hash, file_path, file_size, file_create_time, file_modify_time, file_access_time
| fieldformat Time = strftime(_time,"%m/%d/%Y %T")

Step 3

IP addresses have been associated with exploiting this NetScaler vulnerability. This includes source IP addresses of ATP groups that have been recorded with attempted exploitations and destination IP addresses of the malicious infrastructure. Any occurrence of these known bad IP addresses should be investigated if they appear in an environment.

Actions to Detect

Appearance of the following hashes in a client’s environment:

5.2.64[.]155
5.255.101[.]76
172.94.124[.]18
154.6.91[.]149
216[.]41[.]162[.]172
216[.]51[.]171[.]17

SPL

| tstats `summariesonly` latest(_time) as _time, sum("All_Traffic.bytes") as bytes, values("All_Traffic.src_port") as src_port, values("All_Traffic.transport") as transport, values("All_Traffic.dest_port") as dest_port from datamodel="Network_Traffic"."All_Traffic" where All_Traffic.src IN ("5.2.64.155","5.255.101.76","172.94.124.18","154.6.91.149","216.41.162.172","216.51.171.17") OR All_Traffic.dest IN ("5.2.64.155","5.255.101.76","172.94.124.18","154.6.91.149","216.41.162.172","216.51.171.17") by "All_Traffic.action", "All_Traffic.src", "All_Traffic.dest", "All_Traffic.user" | head 10000 | `drop_dm_object_name("All_Traffic")`| eval earliest_time = IF(isnum("$earliest_time$"),"$earliest_time$",strftime(relative_time(now(),"$earliest_time$"),"%s")) | eval latest_time = CASE(isnum("$latest_time$"),"$latest_time$","$latest_time$"="now",now(),1=1,strftime(relative_time(now(),"$latest_time$"),"%s")) | where _time >= earliest_time AND _time <= latest_time | head 10000 | table action, src, src_port, dest, transport, dest_port, user, bytes

Step 4

The smoking gun is connections out to the malicious domains for credential harvesting. Multiple domains have been identified as being used in this exploit. These domains were stood up by threat actors with their origins masked. The malicious JS script that would inject the fake login element on NetScaler index.html would be hosted here. The JS files were almost all identical. Credentials taken from victim users would also be redirected to these domains. The credential endpoint was a similar path for most of the malicious domains.

Actions to Detect

Any connections to the following domains from a client network:

jscloud[.]ink

jscloud[.]live

jscloud[.]biz

jscdn[.]biz

cloudjs[.]live

cloud-js[.]cloud

Any connections with the following URL string:

/items/accounts/

SPL#1 – Web: Malicious Domains + URL String

| tstats `summariesonly` sum("Web.bytes") as bytes,values("Web.http_content_type") as http_content_type,values("Web.http_method") as http_method,values("Web.http_user_agent") as http_user_agent,values("Web.status") as status from datamodel="Web"."Web" by "Web.src", "Web.dest", "Web.user", "Web.http_referrer", "Web.url", _time | `drop_dm_object_name("Web")` | where match(url, "(?i)(jscloud.ink)|(jscloud.live)|(jscloud.biz)|(jscdn.biz)|(cloudjs.live)|(cloud-js.cloud) | (\/items\/accounts\/)") | fields _time src, dest, user, http_referrer, url, http_content_type, http_method, http_user_agent, status, bytes | eval ip_pair = src . " - " . dest  | stats values(_time) as Time values(ip_pair) AS src_dest, values(user) as user, values(http_referrer) as http_referrer, values(http_user_agent) as http_user_agent, values(http_method) as http_method, values(status) as status, values(bytes) as bytes count by url | fieldformat Time = strftime(Time,"%m/%d/%Y %T")

SPL#2 – DNS: Malicious Domains

| tstats `summariesonly` latest(_time) as _time  values("DNS.dest") as dest, values("DNS.query") as query, values("DNS.query_count") as query_count, values("DNS.message_type") as message_type, values("DNS.answer") as answer, values("DNS.reply_code") as reply_code from datamodel="Network_Resolution"."DNS" where DNS.query IN ("jscloud.ink","jscloud.live","jscloud.biz","jscdn.biz","cloudjs.live","cloud-js.cloud") | table _time, dest, query, query_count, message_type, answer, reply_code | fieldformat Time = strftime(_time,"%m/%d/%Y %T")

Step 5

In a compromised environment, it would be expected to see an increase in the amount of traffic to control and control destinations. Thus, detecting a possible spike in DNS traffic that could point to an attempt or successful attack on this NetScaler vulnerability.

Actions to Detect

A DNS Spike in an environment that may indicate attempts to connect to a C2C destination.

SPL

| tstats `summariesonly` count from datamodel="Network_Resolution"."DNS" by "DNS.query", _time | rename "DNS.query" as query | fieldformat Time = strftime(_time,"%m/%d/%Y %T")

Note: This is very simple SPL to look at the DNS resolutions by query destination and time. It needs to be tuned for each environment. It would take a Splunk professional to present the data in a more digestible form and cybersecurity professional to use this visualization to spot an anomaly. Reach out to TekStream using the form below to get in contact with both.

IOC List

Below is an aggregate of IOCs found related to the CVE-2023-3519 vulnerability. It has been presented here for you to have it consolidated in one spot and made easier to export to a lookup for notable detections.

Auto Download: TekStream CitrixNetScaler3519 IOC List

Conclusion

Ideally, you could make scheduled searches like what is shown above with lookup tables for matching more specific and new detections. These can be set to run on a continuous schedule to ensure the monitoring of your assets from new vulnerabilities. Old vulnerabilities that have been patched (and applied) can also be aged out of this lookup table to ensure it is timely and efficient. For questions on how to build such a process that is dynamic and customizable for your environment, ask one of our consultants by filling in the form below. Also, subscribe to the TekStream blog to catch the next monthly security bulletin and apply the latest detections to protect your systems. Happy Splunking!

References

Bleeping Computer: Hackers Hijack Citrix NetScaler Login Pages to Steal Credentials

CISA: Cybersecurity Advisories

Security Intelligence: X-Force Uncovers Global NetScaler Gateway Credential Harvesting Campaign

SOC Radar: Critical and High Vulnerabilities in Citrix ADC and Citrix Gateway

Citrix: Citrix ADC and Citrix Gateway Security Bulletin

Disclaimer

The approaches recommended herein have not been tested broadly across the TekStream customer base. They are preliminary in nature and come without any certification of efficacy.