Detecting and responding to ESXi compromise with Splunk
Threat actors are finding novel techniques to attack and compromise networks, and in line with this, we have been seeing a spike in attacks on virtualization platforms like VMware vSphere. These include — the ESXiArgs ransomware campaign, Royal Ransomware targeting Linux ESXi servers, RTM Locker targeting ESXi servers, Black Basta ransomware now targeting ESXi hosts, Python backdoor detected on ESXi servers, A three-fold increase in ransomware attacks targeting ESXi, VIRTUALPITA & VIRTUALPIE backdoors detected on ESXi by Mandiant, to list a few.
We can see the number of external-facing ESXi hosts looking at Shodan.
A lack of AV/EDR on platforms such as these shouldn’t limit a defender’s ability to detect & respond to threats on virtualization platforms. However, due to the lack of specific detection analytics available online, I’m making a few available here. Depending on your environment, you might need to finetune these to limit false positives. Also, note that I’ll use wildcarded search strings to cast a wide net and consider the possible variations.
There are a ton of resources online detailing how to secure your ESXi hosts. I particularly enjoyed reviewing the guides from VMware & Mandiant, and they both recommend centralized logging of vSphere components — vCenter server & ESXi hosts. In addition, I recommend following the official VMware documentation to identify the log files that should be shipped to your SIEM.
Assuming you have the necessary logs on Splunk, let’s focus on detecting anomalous events that malware/ransomware or a threat actor might perform.
Firstly, after going through the previously referenced intelligence reports on ransomware targeting ESXi hosts, I’ve noticed the usage of the following command to list & then terminate all VM processes using the ESXCLI, respectively.
esxcli vm process list
esxcli vm process kill*Next, let’s take a look at the authentication logs available in auth.log & sshd.log. These logs contain authentication events, as noted below. Ideally, these logs should be mapped to your Authentication data model and generate alerts based on anomalous access detection rules.
2023-03-27T11:54:50Z esxi3.cloud.net sshd[249482]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.2.2 user=root
2023-03-24T03:28:44Z esxi1.cloud.net sshd[210119]: Failed keyboard-interactive/pam for invalid user agent.smith from 192.168.2.2 port 57477 ssh2A malicious actor might attempt to enumerate the available access, permissions, accounts, etc. Keeping this in mind, let’s look at the commands a malicious actor might execute while performing enumeration.
List role & permissions:
esxcli system permission listListing the accounts, enabling shell for an account, the addition of accounts as persistence, removal of accounts to prevent troubleshooting etc., via the following commands:
esxcli system account list
esxcli system account set -i * -s t*
esxcli system account add*
esxcli system account remove*VMware states that with secure boot enabled, a machine refuses to load any UEFI driver or app unless the operating system bootloader is cryptographically signed.
esxcli system settings encryption set - require-secure-boot=F*A malicious actor might allow custom code execution inside ESXi by setting the execInstalledOnly setting to false via the following commands.
esxcli system settings encryption set - require-exec-installed-only=F*
esxcli system settings kernel set -s execInstalledOnly -v F*Ransomware operators might disable or change the firewall setting to relax the network access restrictions. They could also alter the audit records settings via the following commands.
esxcli network firewall set –enabled f*
esxcli system auditrecords local disable*Let’s not forget about reverse shells and the need to detect them. The following wildcarded strings give pretty good coverage.
*bash -i >& /dev/tcp/*
*/dev/tcp/*
*/dev/udp/*
*/socat exec:'bash -li'*
*socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"))*
*new IO::Socket::INET(PeerAddr*
*python -c' import socket*Once the threat actor has disabled execInstalledOnly, they can perform a — force installation of custom malicious VIBs (vSphere Installation Bundles). Sample logs are available in the Mandiant blog post referenced above.
Putting it all together and creating a Splunk query to detect them.
1. Detecting suspicious command execution
2. Detecting malicious VIB installation using — force flag
3. Detecting anomalous access events
If you’d like to see more awesome Threat Hunting queries, check out my Threat Hunting with Splunk GitHub page.
