Consider the following, you are building a new set of services that is using the Public Cloud and the service consists of different resources such as virtual machines, PaaS services, and container-based workloads.
You want to leverage PaaS services to minimize operational overhead and be able to gain from the evolution of the PaaS services of the provider. Secondly, you are using some services to provide front-end features, backend storage, and SQL database.
You are also running some microservices on Kubernetes that handles a smaller portion of the services and virtual machines that are also handling some batch processing of the data.
While the front-end is stateless and just handles traffic flow all the backend data is stored on storage accounts, SQL database, persistent workload on Kubernetes and the virtual machines.
If you were building this on an on-premises setup using only virtual machines. The backup process to ensure that you have data protection would be pretty straight forward. Where we could define a group of machines that should be taking a consistent backup of at the same time. This ensures that we can restore the services to any point in time given that we have backup.
Now my point about this article, how would this then be in public cloud and using these different services?
For Microsoft Azure, we would need to use the following services.
- Azure Backup for Virtual Machines
- Azure Backup for Storage Accounts
- Built-in backup service for Azure SQL
- No-native backup service for Kubernetes (would need to use product like Velero)
In addition there are some differences in how the backup service works for Azure SQL compared to Storage Accounts.
The built-in backup service Azure SQL creates a full backup every week, you cannot define which day that occurs. It does differential every day and transaction log backups every 5 – 10 minutes. The backup service for Storage accounts is using built-in services in the storage account and the data is not moved so the backup data is stored within the storage account. The backup uses soft delete, change feed and blob versioning to ensure that it has backup of each change. So that you can do point in time restore.
The last part is for virtual machines in Azure, and if using Azure Backup (with a Recovery Services Vault) we have a new option now called enhanced backup policy which allows us to do backup multiple times today. (This feature is currently in preview).
Now for the last part which is the Kubernetes workloads, which also can run with persistent data. Microsoft doesn’t have a built-in service for those workloads yet do to backup. However they recommend using tools like Velero or Kasten.
These tools can then be used to do backup of the data stored on the persistent volumes which are typically stored as Azure Files or Azure Managed Disks. I’ve written about set up of Kasten here –> Get started with Kasten for data protection on Azure Kubernetes Service | Marius Sandbu (msandbu.org)
Now the point I want to make is data protection is often overlooked when it comes to the PaaS services. However our responsibility with these PaaS services is still much the data and in most cases the cloud providers have a backup service built-in or we need to build some logic ourselves.
But not all cloud services have a backup service available which means that you will need to mix different tools/services to provide the same backup mechanism across your services unlike what you did on-prem where you most likely only had one backup vendor that could take care of everything.
So ensure that you do testing of your backup not just the isolated service but ensure that you can restore all of your data across your services to an earlier state, so you don’t end up like this At last, Atlassian sees an end to its outage – in two weeks • The Register