Getting started with Azure Defender and Azure Monitor for Kubernetes using Azure Arc

As part of Azure Arc, Microsoft provides integration with Kubernetes to provide a couple of features.

• Connect Kubernetes running outside of Azure for inventory, grouping, and tagging.
• Deploy applications and apply configuration using GitOps-based configuration management.
• View and monitor your clusters using Azure Monitor for containers.
• Enforce threat protection using Azure Defender for Kubernetes.
• Apply policies using Azure Policy for Kubernetes.

In this blog post, I want to highlight how to set up an Azure Arc integrated Kubernetes Cluster and configure Azure Defender and Azure Monitor (which is a requirement if you want full threat detection for Azure Arc)

1: Connect an existing Kubernetes deployment to Azure Arc

Azure Arc for Kubernetes requires that you have an existing Kubernetes environment running (can be Rancher, Openshift or vanilla) deployment. It does technically work with another setups such as GKE and EKS but not officially supported. To do the integration you need to have a couple of things in place.

• A kubeconfig file and context pointing to your cluster.
• ‘Read’ and ‘Write’ permissions on the Azure Arc enabled Kubernetes resource type (Microsoft.Kubernetes/connectedClusters).
• Install the latest release of Helm 3.
• Install or upgrade Azure CLI to version >= 2.16.0
• Install the connectedk8s Azure CLI extension of version >= 1.0.0:

Then run the commands

az extension add --name connectedk8s
az extension add --name k8s-extension

You need both extensions to be able to setup Azure Arc and to configure Azure Defender one step later. 

Then you need to configure the resource provider for Azure Arc enabled Kubernetes

az provider register --namespace Microsoft.Kubernetes 
az provider register --namespace Microsoft.KubernetesConfiguration 
az provider register --namespace Microsoft.ExtendedLocation

Next create a resource group which will contain the Azure Arc enabled resource.

az group create --name AzureArcTest -l NorwayEast -o table

Lastly connect the Kubernetes cluster to the Azure Arc Resource

az connectedk8s connect --name AzureArcTest1 --resource-group AzureArcTest

NOTE: There are some flags that you can use together with the az connectedk8s command that you can use to define what kind of infrastructure the Cluster is running on using flags such as
–infrastructure “azure, gcp, vsphere” and also –distribution “aks, eks, gke” which are used as metadata tags in Azure, but is not required.

You can verify the connection by using the command

az connectedk8s list -g AzureArcTest -o table

or viewing in the portal under Kubernetes Services in Azure.

Now as part of the connection, Azure Arc will create its own namespace where it will place all the pods containing the services it uses to communicate with the cluster. Which you can view by using the command

kubectl -n azure-arc get deployments,pods

2: Configure Azure Defender for Kubernetes Cluster

Once the Kubernetes cluster is Azure Arc enabled, we can enable Azure Defender. I have previously written about Security Center/Defender which is called now (Azure Security Center Overview and Security Ecosystem | Marius Sandbu (msandbu.org)) but it provides threat detection against the Kubernetes environment. It should be noted that it does this on two layers.

1: For the Kubernetes Cluster (Provides detection mechanisms at the cluster level)
2: For the Kubernetes Worker Nodes (Provider detection mechanisms at the worker level)

As some examples, the first layer can detect, for instance, Exposed Kubeflow dashboard detected while at the worker node level it can detect Privileged command run in a container. 

To onboarding Azure Defender for Kubernetes you need to do a couple of things first. First you need to enable Azure Defender for Kubernetes, which is not available using the Azure CLI, you can either enable it using the Portal or for instance Terraform

resource "azurerm_security_center_subscription_pricing" "example" {
  tier          = "Standard"
  resource_type = "Kubernetes"
}

Next you need to install the extension to the Kubernetes Cluster (this requires that you have kubectl access to the existing cluster when running the command)

az k8s-extension create --name microsoft.azuredefender.kubernetes --cluster-type connectedClusters --cluster-name <cluster-name> --resource-group <resource-group> --extension-type microsoft.azuredefender.kubernetes

This will create a new namespace within the Kubernetes cluster which will contain the extension using an API extension. You can trigger a test alert by running the command

kubectl get pods --namespace=asc-alerttest-662jfi039n

This command will fail with “No resource found” but after a while, you should get an alert in Security Center, which you can view using the command

az security alert list -g "nameofresourcegroup"

Or you can view it in the portal.

3: Configure Azure Monitor for Kubernetes Cluster

Now as it is now, it will only highlight Security alerts that are related to the Kubernetes Cluster, so to provide insight to the worker nodes we also need to install the Azure monitor agent as well. This can be done using the commands

az k8s-extension create --name azuremonitor-containers --cluster-name <cluster-name> --resource-group <resource-group> --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers

With this, it will deploy a pod per host using a daemon set to collect data from the local worker node.

Now by default enabling Azure Monitor for an existing cluster, it will use a default log analytics workspace to store data. The type of information the agent will be collected is listed here –> Container Monitoring solution in Azure Monitor – Azure Monitor | Microsoft Docs

But in summary, it is.

• ContainerInventory
• ContainerImageInventory
• ContainerLog
• ContainerServiceLog
• ContainerNodeInventory_CL
• KubePodInventory_CL
• ContainerProcess_CL
• KubeEvents_CL

It will also provide Container Insight which collects a lot of telemetry data from the different worker nodes.

4: Verify and reference for security alerts

Once all these components are up and running, you can also configure custom alerts and you can also view the reference table for all security alerts in Azure defender in the URL below –> Reference table for all security alerts in Azure Security Center | Microsoft Docs

But here is an example of a Security Center alert