HANA on VMware – Maintenance
last updated: 2023-09-27
A lot of customers are using VMware as Hypervisor to run the HANA workload. Since the initial certification of this platform there are some limitations regarding features and sizing. Some of this limitations depending on the vSphere version. With every maintenance you have to take attention to compatibility and parametrization. The most VMware administrators don’t know any of the SAP HANA dependencies. May be this blog is the chance to bring them and your landscape up-to-date before you get support issues.
So, we have two possibilities as maintenance scenarios: the update of the vSphere version and a hardware change (security is another topics which I don’t want to discuss in the HANA context of VMware, because there is no SAP dependency). Both can take place due to support, limitation or feature dependencies.
1. vSphere version
At first check if your vSphere version is supported with your current hardware. I heard from some customers that the matrix on the help.sap.com is pretty hard to interpret because it is too big and not handsome.
Long story short: Haswell and Broadwell are currently not supported for vSphere 7.0+. As you can see it is planned for Broadwell. I will update the blog if anything changes.
=> Broadwell is supported for vSphere 7.0+ (limitation 8-socket wide VM)
This means with these old processors ( older Haswell) you can max. go for vSphere 6.7 U3. Be aware that vSphere 6.5 and 6.7 general support end was Oct 15, 2022!
Currently (as of 2023-09-27) this versions are supported :
Source: ©2022 VMware, Inc
If you want to go for vSphere 7.0 you have to buy new hardware. If you have such new processor architecture and want to migrate your VMs via vMotion you have to check the EVC compatibility (see below).
|vCLS: Starting with vSphere 7.0 the VMware Clustering Services vCLS needed to be considered for the sizing. For details check Erik’s blog.|
|vMotion: VM moves over CPU families limits (e.g. Broadwell => Cascade Lake) should be handled carefully due to some limits and performance brakes. Check the EVC modes and compatibilities, but I recommend a general VM restart if you migrate VMs between CPU families although the EVC compatibility is guaranteed.|
|EVC: EVC is short for Enhanced vMotion Compatibility. EVC allows you to migrate virtual machines between different generations of CPUs.|
Source: ©2023 VMware, Inc
Why you should go for vSphere 7.0? Normally the virtual performance overhead is about 10-15%. This is the factor the sizings are calculated in the past. The bare metal benchmark of the Fujitsu PRIMEQUEST 3800B2 vs. virtualized benchmark vSphere 7.0 U1 BW benchmark 448 vCPU VM on Fujitsu PRIMEQUEST 3800B2 shows us an overhead of ~3% in phase 1 and an overhead of ~9% in phase 2. Both times below 10%! This means besides the features the performance is another argument for vSphere 7.0.
Just to name some other arguments besides the benchmark performance (more details):
- vMotion performance is 2-3 times faster compared to vSphere 6.7U3
- general support till April 2025
From the perspective of support there is now (2023) the need to upgrade from ESXi 6.5 / 6.7 to ESXi 7.0+ or 8+!
2. Hardware change
- 6TB can be only achieved with Skylake, Cascade Lake and Cooper Lake (only with 7.0 U2+) within HANA on VMware (6.7 U2+ / 7.0 U2+)
- 12TB can achieved with Cascade Lake and Cooper Lake (7.0 U2+)
- vSphere 6.7U2/U3 : Skylake and Cascade Lake up to 224 vCPU
- vSphere 7.0 incl. U1/U2: Skylake and Cascade Lake up to 224 vCPU (4-sockets) or 448 vCPU (8-sockets / Cascade Lake only)
- vSphere 7.0 incl.U2/U3: 6 TB Skylake, Cascade Lake and Cooper Lake; 12 TB Cascade and Cooper Lake
- vSphere 8.0 incl. U1/U2: 4TB Sapphire Rapids (half socket not supported!)
- Ice Lake: currently 4 TB and 160 vCPU (40cores per socket)
- Sapphire Rapids: 120 vCPUs / 2 TB per socket
- OLTP max: 240 vCPUs and 4 TB
- OLAP max:
- Class L: 3 TB (1.5 TB per socket)
- Class M: 4 TB (2 TB per socket
If you don’t want or even can run a scale-out setup because it is not supported for your current environment you have to go for bare metal or IBM Power architecture.
Every time you upgrade the hardware, you also have to check the hardware version (now known as Virtual Machine Compatibility) of each VM running on the ESXi host. If your VM is e.g. still on version 10, the limits regarding number of cores and the former memory limits are still valid for the VM which means the former ESXi 5.5 limits. I recommend to upgrade the hardware version least on every major release upgrade. If the minor versions have no limitation which hit you and downtime is a driving factor in your business, skip the hardware version update till you really need them. It is just a simple VM restart – not really time intensive. In the end it depends on how many VMs you have to handle and how big is your downtime window.
Source: ©2023 VMware, Inc
“Prior to vSphere 7.0 Update 2, this was actually not possible but now a new vSphere API property has been introduced called maximumHardwareVersionKey which will allow a customer to set the maximum supported VM Compatibility for a specific vSphere Cluster and prevent users from creating a VM that is greater than what has been configured. To change this property, you will need to use the ReconfigureComputeResource_Task() API method.
In my example, I have vCenter Server running 7.0 Update 3e but as you can see from the screenshot, I can not create a VM that has VM Compatibility that is greater than ESXi 6.7 Update 2 or later.”
- Virtual machine hardware versions (1003746)
- Upgrading a virtual machine to the latest hardware version (multiple versions) (1010675)
Another topic is NUMA and the vCPU parametrization. So, check the NUMA settings and vCPU settings at least once a year. If you change the hardware which normally means also that the number of cores per socket are increasing, you have to adjust your settings. Check e.g. parameters like numa.autosize.vcpu.maxPerVirtualNode and numa.vcpu.preferHT regarding your sizing, because this parameters won’t be adjusted automatically. This means also have an eye on your NUMA settings! Therefor another blog exists.
|Pmem is available since vSphere 6.7 EP14 or vSphere 7.0 P01. Please check the help.sap.com for details.
Check the help.sap.com for more details about vSAN
Check the table from VMware
The SAP HANA installation must done by an SAP certified engineer, qualified as SAP Certified Technology Specialist – Required exam: C_HANATEC_16+
If there is no one with valid certification in your company you know there are some outstanding consultants out there – ready to support you 😉
VMXNET3 SAP workload specific network latency considerations
As mentioned in SAP note 3102813 and documented in VMware KB 83957, virtualized network based on VMXNET3 NICs, adds typically 60 µs (no load) and up to 220 µs latency (high CPU load((~80%)) to every network package sent, when compared to a bare metal installed SAP HANA system. This can impact SAP OLTP and OLAP type workloads issued by remote application servers / users.
Your check list for VMware maintenance in a HANA context:
- check compatibility of hardware and vSphere versions
- check feature compatibility
- check EVC
- check sizing (overcommitment and noisy neighbours)
- check hardware version
- check VMware parameters
In the end – like every maintenance with SAP HANA – you have to do some pre-checks, because not every scenario is supported.
help.sap.com: SAP HANA on VMware vSphere
Virtual Machine Compatibility
VMware best practice and reference architecture guide
SAP Note: 2393917 – SAP HANA on VMware vSphere 6.5 and 6.7 in production
SAP Note: 2937606 – SAP HANA on VMware vSphere 7.0 in production
SAP Note: 3372365 – SAP HANA on VMware vSphere 8
help.sap.com: SAP HANA based Applications on VMware vSAN (SAP HANA HCI)
help.sap.com: SAP HANA with Persistent Memory on VMware vSphere
SAP Note: 2954515- SAP HANA Persistent Memory – memory mode
SAP Note: 2913410 – SAP HANA on VMware vSphere with Persistent Memory
SAP Note: 2786237 – Sizing SAP HANA with Persistent Memory
SAP Note: 2700084 – FAQ: SAP HANA Persistent Memory
SAP Note: 2779240 – Workload-based sizing for virtualized environments
SAP Note: 3102813 – SAP HANA on VMware vSphere 7.0 U2/U3 with up to 12 TB 448 vCPUs VM sizes