The last 6 months I have been involved in numerous discussions related to virtualization and specifically related to “how do I move from A to B? what do I loose of functionality? what do I gain?”. Reason behind these discussions are often related to license cost and that their footprint within their own datacenter is become smaller and smaller. Therefore in some scenarios it makes sense to move to another platform. Therefore I decided to write this blog post to discuss 1: What are my options? and 2: What considerations to I need to take when moving from platform A to B.
Alternatives
Before diving into the various alternatives, it’s important to first define your overall goal. Are you managing 1000 virtual machines that will remain virtualized for the next 10 years, or are you transitioning to containerized workloads with an increasing reliance on Kubernetes? Your current and future workloads will heavily influence the type of virtualization platform or alternative you choose.
If your business will continue to use virtual machines for a long time, you will most likely continue using traditional virtualization platforms like VMware, Azure Stack HCI or Nutanix. These options offer robust VM management, along with integrated solutions for storage, networking, and security. However, if your organization is leaning towards containerization, you might want to explore platforms that are optimized for container workloads, such as Red Hat OpenShift, VMware Tanzu, or even cloud-native solutions like Google Kubernetes Engine (GKE), Amazon EKS Anywhere, or Azure AKS. Since many of these services are now bundled together, it would also make sense to look at a platform that can offer both at the same time.
If our goal is to minimize the footprint and run workloads primarily at the edge, that’s a different conversation altogether, but I will come back to that at the end here with some opinions on which direction one should take. Also rather than listing all the pros and cons of each option, I’ve summarized the key points in a table to highlight the differences. I’ve also categorized them into several groups: one for container-based workloads, another for traditional virtualization platforms, a third for cloud-based deployments, and finally one for edge scenarios.
Container platforms on-premises – While there are other options as well, such as Rancher, native or other managed offerings, these ones can be used in combination with either a cloud provider or part of an existing virtualization platform.
| Feature | VMware Tanzu | Red Hat OpenShift | Azure Kubernetes Service | GKE on-premises | AWS: EKS Anywhere |
|---|---|---|---|---|---|
| Airgap Support | Yes | Yes | No (Only if on Azure Stack Hub) or Essentials. | Yes | Yes |
| Required Hypervisor | vSphere, AWS and Azure | KVM, vSphere, Azure Stack HCI | Azure Stack HCI (Edge supports on VMware as well) | vSphere | vSphere, Bare Metal, Nutanix |
| Multi-cluster Management | Yes | Yes | Yes (via Arc) | Yes | No |
| Integrated Service Mesh | Yes (VMware Tanzu Service Mesh) | Yes (Istio) | No (Add-on) | Yes (Istio) | No (Add-on) |
| Application Catalog | Yes | Yes | Limited | Limited | Limited |
| CI/CD Integration | Yes | Yes | Yes | Yes | Yes |
| Windows Container Support | Yes | Yes | Yes | Yes | Yes |
| Bare Metal Support | No | Yes | No | Yes | Yes |
| Edge Support | Yes, on top of VMware Edge Compute Stack | Yes | Yes, AKS Edge Essentials | Yes | Yes |
| Integrated Monitoring | Yes via Aria | Yes | Azure Monitor | Yes, Prometheus | Yes (Optional) |
| Configuration Management | Yes | Yes (Ansible) | Azure Policy | Yes | Yes |
| VM Management | No (runs on top of vSphere) | Yes (Red Hat Openshift virtualization | No (runs on top of Azure Stack/Hyper-V) | Limited | No |
| Pricing | VMware Tanzu Standard 108$ per core per year | $1500 for 4vCPU per year (375$ per vCPU) | 292$ per virtual core per year | 288$ per virtual vCPU per year | $2,000 per cluster per month |
Virtualization
While I do not cover all features of a virtualization platform, but there are some considerations. Think if you today are running VMware with VSAN Nodes, you cannot directly switch to Nutanix and AHV since it requires you to have Nutanix supported nodes. Then could I do the switch to Citrix Hypervisor? well that does not support software-defined storage which means that you would need to get a SAN to store the virtual machines. You could possibly have Azure Stack HCI installed on the nodes as long as the components are on the supported list from Microsoft. This is just to highlight that you need also to verify your hardware if you plan to switch to another platform.
| Feature | Proxmox | Azure Stack HCI | VMware | Nutanix AHV | Citrix Hypervisor |
|---|---|---|---|---|---|
| Type | Open-source | Proprietary | Proprietary | Proprietary and requires Nutanix nodes | Open-source |
| Virtualization | KVM & LXC | Hyper-V | ESXi | KVM-based | Xen-based |
| Air Gap Support | Yes | No (Unless you run Hyper-V natively) where there are some feature differences Compare Azure Stack HCI to Windows Server – Azure Stack HCI | Microsoft Learn | Yes | Yes | Yes |
| SDN Capabilities | Yes | Yes | Yes | Yes | No |
| SDS Capabilities | Yes (via Ceph) | Yes | Yes | Yes | No |
| Live Migration | Yes | Yes | Yes | Yes | Yes |
| High Availability | Yes | Yes | Yes | Yes | Yes |
| Storage Types | Local, SAN, NAS. SDS | Local, SAN, NAS, SDS | Local, SAN, NAS, VSAN (SDS) | Hyperconverged | Local, SAN, NAS |
| Backup Solutions | Built-in & 3rd party | 3rd party | 3rd party | 3rd party | 3rd party |
| Clustering | Yes | Yes | Yes | Yes | Yes |
| Container Support | Yes (LXC) | Yes (AKS) | Yes (Tanzu) | Yes (NKE) | No |
| GUI Management | Yes | Yes | Yes | Yes | Yes |
| vGPU Support | No | Yes (GPU-P) NVIDIA | Yes | Yes | Yes |
| API Support | Yes | Yes | Yes | Yes | Yes |
| Price | $558 Yearly / CPU Socket (with support) | $10/physical core/month (100$ month/support) | Not publicly available | Not publicly available | Only sold with Citrix licenses |
Cloud platforms – Then we also have some of the different cloud platforms from the big three. These options differ a bit since they provide a ful platform that also includes other PaaS services and are also more of a black box solution that offers much of the public cloud services as well. They offer tight integration with the cloud provider and are often deployed as an extension to the cloud provider. Meaning that if you are using Google Cloud mostly, but you want to run some applications locally for low latency processing then Google Distributed Cloud might be an option. However it should be noted that these platform tend to have a limited set of features, not the same consistent APIs (while they advertise as much..)
| Feature | Azure Stack Hub | AWS Outposts | Google Distributed Cloud |
|---|---|---|---|
| Deployment Model | On-premises (rack or servers) | On-premises (rack or servers) | On-premises (rack or servers) |
| Hardware Requirements | Specific OEM hardware (Dell, Lenovo, HPE) | AWS-designed hardware | Google-designed hardware |
| Air-gapped | Both connected or offline supported | Only connected | Both connected or offline supported |
| Core Services | Compute, Storage, Networking | Compute, Storage, Networking | Compute, Storage, Networking |
| Scaling | Scale-out by adding nodes (4-16 servers per scale unit) | Scale by adding racks or servers | 6-12 server machines |
| Features | Virtual Machines, Blob Storage, App Service, AKS, Azure container Registry | Elastic Compute Cloud (EC2) Elastic Container Service (ECS) Elastic Kubernetes Service (EKS) Elastic Block Store (EBS) EBS Snapshots Simple Storage Service (S3) Relational Database Service (RDS) Elasticache EMR Application Load Balancer (ALB) Route 53 Resolver IoT Greengrass Elastic Disaster Recovery VMware Cloud | GKE, Virtual Machines, Block and object storage, Vertex AI. Database engine (PostgreSQL, Oracle and AlloyDB) |
Considerations
1: Knowledge and Expertise – Transitioning from one virtualization layer to another can be technically straightforward, but it also requires acquiring new knowledge and skills specific to the new technology. This learning process is not something that happens overnight and requires careful planning. Additionally, depending on the technology you choose, the availability of skilled professionals may be much more limited compared to the new platform you are moving to. One often overlooked thing is the community, moving to a new platform while it might offer significant price reduction but has a pretty small community means that it can be difficult finding people that can manage it, and the pool of people to recruit from will be significant smaller.
2: Integrations and Dependencies with the Virtualization Layer – While this may not necessarily drive the decision to switch platforms, it’s an important factor to consider. Many technologies integrate directly with hypervisor APIs, which may not be supported by other platforms. Examples include:
- Backup services like Veeam, which uses either the APIs or has an agent installed that can handle snapshots and copy of the data.
- VDI services such as Omnissa and Citrix
- Storage services, such as CSI drivers for Kubernetes
- Networking appliances like F5, NetScaler, Cisco, and Palo Alto
- Security tools like Trend Micro, which integrate into network APIs
- Monitoring tools and monitoring solutions.
- Or in general just VM appliance support. Make sure that if you have some 3.party virtual appliance running on your platform today that it supports any future virtualization platform that you plan to move too.
- Many are also using file-services from the hypervisor/virtualization layer, so make sure you plan how to replace this as well.
You might also rely on certain technologies, such as Kubernetes platforms that may not be fully compatible with other hypervisors. This means that switching virtualization platforms could require replacing or modifying third-party tools that aren’t supported by the new environment.
3: Technology Stack Considerations – If your current setup includes components like software-defined storage or software-defined networking, moving to a different virtualization layer means losing these native features. You would need to rely on third-party alternatives, which may not offer the same capabilities. There might also be security requirements such as encryption or vTPM or even having the ability to run completely offline (air-gapped) platforms where some of the alternatives do not actually support it.
4: Hardware usage – Another important factor to consider is the compatibility of your existing hardware with the new virtualization platform. While most modern servers support a wide range of virtualization platforms, you’ll still need to assess whether specific hardware components—such as network drivers, RAID controllers, underlying storage systems, and specialized hardware like GPU cards—are compatible.
For example, if your current virtualization stack relies on GPUs for VDI environments using vGPU, you’ll need to ensure that the new platform offers similar functionality. Microsoft, for instance, provides GPU-P, which allows partitioning and sharing of GPUs with Azure Stack HCI virtual machines. Nutanix also supports NVIDIA vGPU, as does Citrix XenServer. Ensuring that your new platform can accommodate these hardware capabilities is essential for a smooth transition.
5: Support for workloads. In many cases you might be running virtual workloads such as Oracle RAC on Linux, MSSQL Clusters, Exchange and if you want to run this virtually it also need to be supported by the vendors.
6: Cost The final aspect to consider is the cost of moving to a new platform. While VMware can be expensive, transitioning to a different platform involves more than just migrating virtual machines. There are several cost factors to account for, including:
- Product and Licensing – The cost of the new platform’s licenses and any additional hardware required.
- Ecosystem Tools – You may need to invest in new tools, such as backup solutions or other components, that are compatible with the new platform.
- Support – Ongoing costs for support services.
- Training – The expense of training your staff to manage and operate the new platform effectively.
- Migration Project – The costs associated with planning and executing the migration from the existing virtualization platform to the new one.
Each of these elements can significantly impact the overall budget and should be carefully considered during the decision-making process.
My thoughts
We have a lot of alternatives to choose from when it comes to the placement of our workloads. As I mentioned initially we have to assess what our current workload is (and what they are) then also what out future state might look it. If we plan to go from virtual machines to having 90% running in Kubernetes within a few years then it needs to be added into consideration on which platform we might look at.
As one example, if we have today a set of virtual machines running mostly Citrix on VMware with perhaps 3D GPU workloads, if our future state is to run these services for a long time then either moving to Citrix Hypervisor (since the license cost is covered in Citrix) or move to Azure Stack HCI with AVD running on-prem with GPU-P could also be a viable option. However if we want to also have a platform that supports native Kubernetes then Azure Stack HCI or Nutanix (with NKE) could be the most viable platform.
However when doing these considerations, we also have to factor in if there are features that we are using today that we cannot easily replicate on the new platforms. For instance if we are using VMware NSX today, replicating this over to Azure Stack HCI is difficult since their SDN stack is quite limited, while Nutanix has some of the capabilities with Flow.
While there is not single right answer to the question “how do I move from A to B? what do I loose of functionality? what do I gain?” all of the platforms in the market today has their own strengths and weaknesses (either it is license cost or features).