So this is based upon a real-life case where one of my customers had a Linux virtual machine in Microsoft Azure that got compromised and I wanted to use this blogpost to showcase how it was detected and the information that came from Microsoft and what was the reason how it got hijacked. (Really been some interesting work for me as well)
So there was a virtual machine running a default Ubuntu 18.04 setup with very poor set of NSG rules configured (Meaning available to the world). The server was available externally on port 22 and some other ports for Nexus repository and Jenkins UI was open as well on the default port.
The server was up and running for about 2 months now until it got compromised. “Luckily” the customer tenant was running Azure Security Center Standard, so they got notified about it trough the alerts feature in Azure Security Center.
The machine was having multiple outgoing connections on port 22 against multiple hosts. So clearly the machine was compromised, Microsoft has also included threat intelligence about the VM which came from the reputation service from Team Cymru.
I also got an follow-up email from Microsoft with more information about what kind of targets that have been attacked from the compromised source.
So I started by locking down access to the virtual machine, since I wanted to do some further investigation but I still had access to the virtual machine using the serial access solution in Azure. Since SSH was open I started with checking the auth.log, and then I noticed this where it had successfully authenticated against the machine using the Jenkins user from some unknown addresses (in France and in the US)
When looking into the virtual machine I noticed that the Jenkins user was running a series of cron jobs that was consuming all of CPU on the machine. First guess was bitcoin mining. (You can read more about how Azure detects Bitcoin mining here –> https://azure.microsoft.com/nb-no/blog/how-azure-security-center-detects-a-bitcoin-mining-attack/)
I also noticed from the cron job that is was schedule to run after reboot as well when looking at the crontabs of the Jenkins user
@reboot /var/lib/jenkins/.firefoxcatche/a/upd>/dev/null 2>&1
Then I checked for outgoing connections which was connected other user jenikins, seems like it has been compromised using the default web UI or some sorts based upon the Jenkins logs. This is also where I noticed the outgoing connections to different IP ranges on ports 22, 222 and 22222 using a service called tsm.
Now the tsm service itself consisted of multiple files where one of them, (pretty similiar to this case that was setup here –> https://www.reddit.com/r/linuxquestions/comments/9fnggy/my_ubuntu_server_has_a_virus_ive_located_it_but_i/
The Service itself had replicated itself across the OS, pretty simple setup though. But to be honest the culprits also had another script running to clean up after other bitcoin mining malware
So to sum it up and a rule of thumb for virtual machines in Microsoft Azure.
-
Have proper NSG rules in place to lock down access or have Jumphosts in place.
-
Lock down SSH to know addresses and accounts
-
Having monitoring in place for authentication and service usage
-
Don’t setup stuff with default settings, such as Jenkins