Upgrade Azure Kubernetes Service using Terraform

resource "azurerm_kubernetes_cluster" "aksresource" {
  name                = "name"
  location            = azurerm_resource_group.aksrg.location
  resource_group_name = azurerm_resource_group.aksrg.name
  dns_prefix          = "q-aks"
  node_resource_group = "nonprod-aks-np01-rg"
  sku_tier            = "Paid"
  kubernetes_version = "1.21.2"

By changing the Kubernetes version from one to another, Terraform will automatically change the version SKU which will trigger the update change from Microsoft.

In the latest AKS Provider for Terraform you also have the option to define automatic Kubernetes upgrade with the following command

automatic_channel_upgrade = patch

Possible values can be patch, rapid, node-image and stable. Omitting this field sets this value to none. It should be noted that Cluster Auto-Upgrade only updates to GA versions of Kubernetes and will not update to Preview versions.

• Patch – automatically upgrade the cluster to the latest supported patch version when it becomes available while keeping the minor version the same.
• Rapid – automatically upgrade the cluster to the latest supported patch release on the latest supported minor version.
• Stable – automatically upgrade the cluster to the latest supported patch release on minor version N-1, where N is the latest supported minor version.
• Node-Image – automatically upgrade the node image to the latest version available.

Maintenance Window

Planned Maintenace can also be defined on AKS clusters using Terraform

maintance_window {
 allowed {
 day = "friday"
 hours = [1,6]
 }
}

It is important however that when the upgrade is running that you have enough node capacity to handle the upgrade and ensure that your applications will be running. By default, AKS configures upgrades to surge with one additional node. This enables AKS to minimize workload disruption by creating an additional node before the cordon/drain of existing applications to replace an older versioned node.

So, for instance, having a max surge value of 100% will provide the fastest upgrade process but means that all nodes will be drained simultaneously. The best approach would be to have a controlled upgrade process. Microsoft recommends having a surge value of 33%. 

Surge value can also be configured using Terraform

    upgrade_settings {
      max_surge = "30"
      } 
  }

Setting up Kured for AKS

While Microsoft will automatically patch Kubernetes and the worker nodes will automatically get security patches from Ubuntu, it will not automatically restart nodes that require start after patches. Therefore, we also recommend that setting up Kured (Kubernetes Reboot Daemon)

That will deploy a Deamonset that will monitor for the presence of a reboot sentinel file e.g. /var/run/reboot-required

Kured can be deployed using a Helm Chart, where the pods should be deployed into a separate namespace and define nodeselector since Kured does not work on Windows.

helm repo add kured https://weaveworks.github.io/kured
kubectl create namespace kured
helm install kured kured/kured --namespace kured --set nodeSelector."kubernetes\.io/os"=linux

Now you can test this by adding a reboot-required file to one of the AKS nodes, in order to do this you would need to login into one of the AKS worker nodes using SSH (following these instructions –> SSH into Azure Kubernetes Service (AKS) cluster nodes – Azure Kubernetes Service | Microsoft Docs

Then create the file using

sudo touch /var/run/reboot-required

Then exit the pod and wait for the kured daemon to trigger a reboot.

kubectl get nodes -w

You can also define a custom schedule since it by default can restart on any given day. This can be done by changing the default download manifest and once you are done with defining the schedule (or using Helm args)

  --reboot-days=sat,sun
  --start-time=9am
  --end-time=5pm
  --time-zone=UTC

That way you have more control up the restarting window.