SAP HANA on VMware vSphere
SCN Content has been Migrated to 1DX NEW !
All content of the SCN space will be migrated to the Virtualization and Cloud Infrastructure Community and to the Virtualization and Cloud Infrastructure Wiki.
Therefore, the content of this page has been migrated to SAP HANA on VMware vSphere Wiki.
SCN Content Only up-to-date until October 4th 2016
By running the SAP HANA platform virtualized on VMware vSphere, SAP customers can leverage a industry standard data center platform, optimized for agility, high availability, cost savings, and easy provisioning. SAP customers will not only gain the ability to provision instances of SAP HANA in virtual machines much faster, but also benefit from unique capabilities like:
- Increased security and SLAs, e.g. through NSX or DRS.
- Live migration of running SAP HANA instances, with VMware vSphere® vMotion®
- Standardized High Availability, based on VMware vSphere® High Availability (HA)
- Built-in multi-tenancy support, through system encapsulation in a virtual machine (VM)
- Abstraction of the hardware layer
- Higher hardware utilization rates
News
- Since May 2016, SAP supports also SAP HANA on vSphere 6 deployments for production workloads. See SAP Note 2315348 and blogs.vmware.com for more details on what’s supported, which deployment options can get used and summary of the best practices.
- The support for vSphere 6 allows customers to increase the RAM to up to 4 TB (4080 GB) of existing virtual SAP HANA systems when migrated to vSphere 6. Beside increased RAM sizes, vSphere 6 supports also more vCPUs: Up to 128 vCPUs may now be configured and used by a single SAP HANA VM.
- Supporting more physical compute resources inside a VM ultimately provides more “power” to a virtualized SAP HANA system – this alone is worth considering an upgrade from a vSphere 5.5 to a vSphere 6.0 based SAP HANA environment.
Support Information
- Customers running SAP HANA on VMware shall follow the Architecture Guidelines and Best Practices for Deployments of SAP HANA on VMware vSphere.
- We have lately seen some issues with VMs configured with large virtual hardware resources and memory-intensive workloads becoming temporally unresponsive after vMotion has completed. We recommend to read VMware KB 144984 and configure hosts correspondingly.
- Noticing a certain interest in our CPU hot-add feature, we would like to highlight that the use of this capability also requires the OS and the SAP HANA database itself to be capable of leveraging the additional resources instantly. Please be aware that the use of CPU hot-add feature disables vNUMA and hence may lead to performance degradation.
The below table states capabilities, supported deployment options and best practices, like minimal vCPU count or maximal vRAM sizes for SAP HANA VMs on VMware vSphere as of May 2016. For additional reference, see SAP Note 1788665 et. al.
|
Capability / Option |
vSphere 5.5 |
vSphere 6.0 |
||
|
|
Supported in Production |
Supported in non-Production |
Supported in Production |
Supported in non-Production |
|
SAP HANA Scale-Up VM ≤ 1024 GB |
Yes |
|||
|
SAP HANA Scale-Up VM ≤ 4080 GB |
No |
Yes |
||
|
SAP HANA Scale-Out VM ≤ 1024 GB [1] |
Yes |
No |
Yes |
|
|
SAP HANA Scale-Out VM ≤ 4080 GB [1] |
No |
Yes |
||
|
SAP HANA Multi-VM [2] |
Yes |
Limited [3] |
Yes |
|
|
SAP HANA Version |
SPS07 and above |
SPS11 and above |
SPS09 and above |
|
|
VMware vSphere and SAP HANA HA and Operation Features [4] |
||||
|
VMware HA |
Yes |
|||
|
SAP HANA Host Auto-Failover [5] |
Yes |
|||
|
SAP HANA System Replication |
Yes |
|||
|
VMware FT |
No |
|||
|
VMware SRM |
Yes |
|||
|
vSphere vMotion |
Yes |
|||
|
VMware DRS [6] |
Yes |
|||
|
Supported HW configurations [7] |
||||
|
Supported SAP HANA Systems for VMware virtualization |
Only SAP HANA and VMware certified two-, four- and eight socket Intel E7 v2 (Ivy Bridge) and later Intel processor-based server systems, as well as Intel Xeon E5 v3 and v4 based two-socket single node SAP HANA entry level systems, with a minimum of eight cores per CPU are supported. |
|||
|
Intel Xeon E7 CPU Support |
Ivy Bridge, Haswell |
Ivy Bridge, Haswell, Brodwell[8] |
||
|
NUMA Nodes per server |
4 |
8 |
||
|
NUMA Node Sharing |
No |
Yes |
No |
Yes |
|
Enable Hyperthreading |
Yes |
|||
|
Maximal RAM installed in server[9] |
4/6 TiB |
6/12 TiB |
||
|
Supported Storage Configuration |
||||
|
Supported SAP HANA Storage Systems for virtualisation [10] |
All SAP HANA TDI and VMware certified / supported storage solutions can get used. |
|||
|
TDI Storage KPI requested |
Yes, per HANA VM |
No |
Yes, per HANA VM |
No |
|
SAP HANA Virtual Machine Configuration [11] |
||||
|
Max. VM size #vCPU |
64 |
128 |
||
|
Max. VM size #vRAM |
1024 GB |
4080 GB |
||
|
Minimal VM size #vCPU |
All threads of a single CPU socket |
10 |
All threads of a single CPU socket |
10 |
|
Minimal VM size #vRAM |
RAM of CPU socket |
As sized |
RAM of CPU socket |
As sized |
|
CPU reservations [12] |
No |
|||
|
RAM reservation |
Yes |
|||
|
Supported OS for virtualized SAP HANA |
||||
|
SLES 11 and 12 [13] |
Yes |
|||
|
RHEL 6 and 7 [12] |
Yes |
|||
Table 1. SAP HANA on vSphere supported capabilities and options
- SAP support for SAP HANA scale-out deployments of SAP Business Warehouse (BW), powered by SAP HANA, on VMware vSphere as outlined in the table in context of SAP BW, powered by SAP HANA. Further supported workloads of OLAP-type (HANA Data Mart, HANA Data Warehouse) generated by SAP and non-SAP tools:
- Documentation and / or tool-support for table distribution, sizing, partitioning, scale-out and how to configure SAP and 3rd party products on SAP HANA has to be provided by SAP, partners or customers resp.
- 3rd party scenarios will not be tested by SAP. In specific cases, SAP HANA support for specific products or scenarios might be excluded by SAP in the future. If tested, then these will be listed in the SAP HANA on vSphere support note.
- 3rd party tool support can be checked at http://global.sap.com/community/ebook/2013_09_adpd/enEN/search.html (select a partner –> details –> SAP Certified Solutions)
- The defined KPIs for Data Throughput and Latency for production SAP HANA systems has to be fulfilled for each VM. SAP has released a special tool, the SAP HANA HW Configuration Check Tool, to measure if the used storage is able to deliver the required IO capacity, see SAP Note 1943937.
- Just like with the vSphere 5.5 SAP HANA support release in the beginning 2014, vSphere 6 supports currently only one production level VM that may get co-deployed with non-production level SAP HANA VMs. No resource sharing with other VMs is supported for production level SAP HANA VMs.
- Generally all VMware vSphere features, like distributed switches are supported, no support for VMware FT for SAP HANA, FT may get used with an (A)SCS instance. vMotion and DRS are explicitly listed since running SAP HANA VMs can get moved from one host to another host while the application is online and users are connected and any negative impact needs to get excluded.
- Only available with specific SAP HANA TDI Storage Vendors, see SCN page DOC-60470.
- VMware DRS should get configured in “manual mode”
- Both SAP HANA appliance and SAP HANA Tailored Datacenter Integration (TDI) delivery methods are supported for SAP HANA on VMware vSphere. Where the SAP HANA system has either been delivered pre-configured on certified SAP HANA appliances, as listed in SAP HANA Product Availability Matrix (PAM), with VMware vSphere hypervisor installed by SAP HANA hardware partner or the SAP HANA installation was done by an SAP certified engineer, qualified as “SAP Certified Technology Specialist – SAP HANA Installation” on SAP HANA certified hardware. It is recommended to verify the configuration with the SAP HANA hardware configuration check tool prior the installation.
- Brodwell CPU vSphere 6 support in validation, yet not supported. Please check SAP support release support note 2315348 for an updated status.
- Maximal RAM installed in physical server as specified by SAP, in the Certified SAP HANA Hardware Directory. The listed memory figures are the maximal memory configurations supported by VMware vSphere. Up to 6TB is supported for ESXi 5.5 Update 2 and later and up to 12 TB is supported with vSphere 6.0 on specific OEM certified platforms.
- For details please refer to the Certified SAP HANA Hardware Directory and go to “Certified Enterprise Storage” configurations.
- The maximum size of a virtual SAP HANA instance is limited by the maximum size of a virtual machine of the VMware vSphere release. Each SAP HANA instance / virtual machine is sized according to the existing SAP HANA sizing guidelines and VMware recommendations. CPU and Memory overcommitting must not be used.
- Since production level VMs have to run on dedicated NUMA nodes, there is no need in configuring CPU reservations. Configuring CPU reservations is only required when production level SAP HANA VMs would share a NUMA node, which is currently not supported. Non-prod. level VMs do not require CPU reservations.
- Both, SUSE Linux Enterprise Server (SLES) 11 and 12 and Red Hat Enterprise Linux (RHEL) 6 are supported operating systems when virtualized. Due to different scope and test coverage during corresponding validation activities, the underlying server must be certified for corresponding operating system, however.
SAP HANA on vSphere Best Practices
In a nutshell: SAP HANA follows general published vSphere Best Practices for databases:
- Set Memory Reservations for SAP HANA Virtual Machines
- Configuring Paravirtual SCSI Controllers and Network Adapters
- Right sizing of SAP HANA VMs to ensure local NUMA node memory access and high 2nd level cache hit ratios
- No NUMA node sharing for production SAP HANA VMs
- Enable Hyper Threading on the ESXi host
- Use dedicated networks for vMotion, management and client network
- Use vMotion and VMware snapshots during non peak times
Untill the new “Architecture Guidelines and Best Practices for Deployments of SAP HANA on VMware vSphere guide will be available from www.vmware.com, see the best practices and especially the configuration settings as described in the Best Practices and Recommendations for Scale-Out Deployments of SAP HANA on VMware vSphere document (page 71-73) for more detailed guidance. In addition, please ensure that following configuration parameters are used in addition to achieve optimal performance for SAP HANA:
- halt_in_monitor = “TRUE”
- idleLoopSpinBeforeHalt = “TRUE”
- Lat. Sensitivity – normal
Comparison: Physical vs. virtual SAP HANA deployment (as state of February 2016)
|
Capability |
Physical SAP HANA |
Virtual SAP HANA |
|
SAP HANA Single Node Memory per node |
128 GB – 4 TB RAM with 8 sockets |
≤ 1 TB RAM per VM (vSphere 6.0 ≤ 4** TB per VM possible) |
|
SAP HANA Scale UP SoH Memory per node |
768 GB – 12 TB RAM with 16 sockets |
≤ 1 TB RAM per VM (vSphere 6.0 ≤ 4** TB per VM possible) |
|
SAP HANA Scale Out BWoH Memory per node |
256 GB – 4 TB RAM with 8 sockets |
≤ 1 TB RAM per VM (vSphere 6.0* ≤ 4** TB per VM possible) |
|
Appliance Model |
Yes |
possible***, normally a TDI deployment |
|
Server Hardware |
Certified Appliances |
Certified Appliances or TDI |
|
Storage Hardware |
Certified Appliances or TDI |
Certified Appliances or TDI |
|
Scale Up Configuration |
Supported |
Supported |
|
Scale Out Configuration |
Supported |
Supported |
|
CPU Sockets per server |
2 to 16 CPU sockets |
2 to 4 CPU sockets (8 with vSphere 6) |
|
Flexible HANA sizing’s |
No, T-Shirt sizing’s |
Yes |
|
HANA DB Live Migration |
No, not possible |
Yes |
|
Standby-Node / System Replicaktion / VMware HA |
Yes / Yes / No |
Yes / Yes / Yes |
|
Multi-Tenancy / full tenant isolation |
Yes, since SPS09 / No |
Yes / Yes |
* vSphere 6 for non-prod. use cases **vRAM configuration is not allowed to exceed installed memory of a physical server system and must follow SAP HANA sizing guidelines. ***Some SAP HANA HW vendors offer VMware vSphere as part of their SAP HANA appliance system. This is not required and optional. The customer can choose between a pre-installed appliance or the certified server configuration and may perform the installation of vSphere on its own.
VMware SAP HANA System Health Check
SAP and VMware are offering joint services to help you design, deploy, and implement your virtual SAP HANA platform.
Examples of services which may be offered are:
- Pre-Call to discuss PoC, goals, timeline and success factors (SAP, HW Partner, Customer and VMware)
- Architecture review and introduction to the SAP HANA on vSphere Configuration Guidelines and Requirements
- Introduction to SAP HANA Hardware Check Configuration Tool (HWCCT)
- Online review session to check prepared environment like:
- Host including storage and network configuration
- VM configuration
- Linux OS configuration
For more information on SAP Consulting and VMware Professional Services, visit vmware.com/consulting or get in contact with your local VMware contact.
I am curious to know if anyone has researched or tested using NUMA affinity settings with the guests and Numa.PreferHT=1 setting for the ESX 5.5 host?
For example:
HOST has 512 GB RAM and 4 Sockets (10 core each)
Guest could have 124 GB of RAM and 1 Socket with 20 core configured. It is also set to only use NUMA node 0.
If we enable the Numa.PreferHT=1 on the HOST, it seams to me that the guest would be able to have 20 v-cores plus use only the memory assigned to NUMA Node 0. Curious to know if that would help the SAP HANA guest given that it could have more cores without the penalty of crossing NUMA nodes.
Hi Jonathan,
that is correct, but you would use the hyper threads, so it would be the roughly 20% more performance compared to the 10 pure cores under full load.
So if you want to keep the NUMA locality and take also advantage from the hyper threads, you can use the PreferHT=1 for a HANA VM.
Cheers
Sebastian
Great to see that customers wanting application aware HA when deploying SAP HANA on VMware can now achieve more demanding Recovery Time Objectives with the validated storage connectors.
Great job EMC on being the first one on the list! I have quite some requests for this stuff!
Great document! Could anyone provide a table which mentiones the sizing guidlines for Haswell CPUs (CPU/Memory ratio for SoH/BWoH)? In the best practice guides are only tables for Westmere and Ivy Bridge CPUs included. Would be great to have an update here.
Cheers
Robert
Hi Paul,
I am working on a new SAP HANA on vSphere sizing guide, which will hopefully be ready in May. Here I will cover in more depth SAP HANA sizing. I will publish the guide here once it is available.
The CPU/memory ratios are relatively easy to calculate, just divide the maximum RAM of a certified SAP HANA appliance by the amount of CPU sockets, lime for BWoH (SPS11) 2 TB / 4 CPU sockets = 512 GB, or for SoH 3 TB / 4 CPU sockets = 768 GB.
That's the maximum RAM available per CPU socket.
Please remember to subtract the RAM needed for ESXi from these figures. As a generic guideline we calculate with 2-3% for ESXi.
Also remember that these are the maximum figures and that a HANA server system may have less memory installed, like only 1.5 TB instead of 2 TB for a BWoH system.
Hello Erik,
If you have sizing document,could you please share.
Thanks and Regards,
Dileep
Hi Erik ,
Great document, based on SAP HANA SOH Haswell EX reference sheet 1.5TB is supported for 2x Processors, If I want to virtualize a two Processor scaleup appliance with VMware 5.5 for Production what is the maximum number of virtual machines which I can run in virtual environment for Production usage, Do I need to limit 1 virtual machine per processor, or i can have for example 10VMs of 128GB in 1.5TB System, Leaving approx. 15% spare memory for ESX. Do you have any guidelines for Virtual core to memory ratio?
Hi Vikas,
you should limit the virtual machines to 1 per NUMA node in a productive environment. There are 2 NUMA nodes in a 2 socket server . If you are running a non-productive environment you can also have 10 active virtual machines depending on the needed computing and memory resources.
BR
Robert
Hi Vikas,
some additional comments on Roberts statement. It is advisable to set memory reservations
for your 10 non-production HANA VMs. Please subtract 2-3% from the overall RAM reservations for the hypervisor, otherwise the last VM want start. This is not a must for non-prod systems, but you may want to avoid in any case memory swapping for SAP HANA due to memory over-commit situations.
It is also advisable to set the parameter numa.vcpu.preferHT=TRUE per VM to keep the processes and memory access of the VMs local to the NUMA node the VM got started. This implicates that HT is enabled on the host.
Best,
Erik
Hi, Erik:
I would like to check if Productive and Non-Productive HANA VMs are allowed to share the same physical host.
BR
Jorge
Erik,
May we have an answer?
Many thanks
Jorge
Sorry for the delay and the answer is yes!
vSphere provides full resource isolation and as long the production SAP HANA VM runs with full resource commitments on an exclusively used NUMA node(s) this is possible.
Beside this it also has to get ensured that the host is able to provide enough storage and network bandwidth to support the load of all planned systems.
Example: 4 CPU socket server (4 NUMA nodes) with 3 SAP HANA VMs of following sizes:
This would be a completely supported environment.
Advice: Verify inside the Prod HANA VM the HANA TDI Storage KPIs with the HWCCT test tool. This would allow you to rate the storage and to ensure that the Prod HANA VM meets the SAP recommendations.
Erik, many thanks.
Is it right to always assume "CPU socket = NUMA node"?
Many thanks
Jorge
Hi Jorge,
to be sure you can check the NUMA distribution via esxtop.
There you can see how many numa nodes and which size ESXi has recognized.
BR,
Andreas
Andreas:
My request is generic, for sizing purposes, so I don't have a particular server to run esxtop. Is it safe to assume that any certified HANA appliance/server with X sockets maps to X NUMA nodes?
BR
Jorge.
May we have an answer to the above question: Is it safe to assume that any certified HANA appliance/server with X sockets maps to X NUMA nodes? My request is generic, for sizing purposes, so I don't have a particular server to run esxtop.
Many thanks
Jorge
NUMA node = CPU socket. A VM can get configured as small as a NUMA node or span more then one NUMA node (wide VM)
now, can i do multi-vm with standby node ? does't it support ? how to do this.
Eric,
In previous generation of HANA servers there was 4x10core CPUs with 512GB of RAM, translating into 12.8 GB of RAM to Core ratio. We at our non-production lab (COIL) were provisioning VMs with 10 virtual cores and 60GB of RAM, giving us 8 such VMs on above mentioned server. The ESX host also had numa.vcpu.preferHT=TRUE enabled and as a result, each socket was running 2 VMs at any time, or one VM if that VM had 120GB of RAM.
Newer servers are coming with 1.5TB of RAM and 2x18cores of CPU. That translates into Core to GB of ram ratio of 42. So does this mean that for these new servers/CPU for the same memory size (60GB) of VM we need to provision 3 virtual cores instead of 10?
Thanks.
Hi Irakli,
this ratio is only viable for i5, but the math is correct.
Going further with this, we see that the SAP HANA is spawning a bunch of threads which could make a VM with 3 vCPU suffer from poor performance under load. To prevent this we would like to see even in this scenarios at least 10 vCPUs.
But for test system you can try to use less vCPUs, but please use always a number which can be divided by two.
I hope this information will help you.
Cheers
Sebastian
Will the section "Supported OS for virtualized SAP HANA" and note #13 include support for RHEL 7 in the near future or is there another document/note that specifies VMware support for HANA on RHEL 7?
Thanks...Alisa
Hi Alisa,
Thanks for making me aware, this was a typo in the table. RHEL 7 is already supported. I updated the table accordingly.
Erik
Terrific thanks for making the update!
Alisa
Hi Erik,
Is there an update on vSphere 6 scale-out and multi-VM for production HANA?
-Mark
Hi Mark,
support status hasn't change yet. As soon there's an updated, we will post it here (and the corresponding SAP Note). Feel free to send me a PN to discuss more detail your particular scenario and needs, so I may provide you with further guidance within current scope of support.
Arne
Thanks Arne!
-Mark
We are in the same situation. We started with VMware 5.5 with scale out for BW on HANA with a master and two salve nodes of each 1 TB.
As per the sap notes, it is supported in TEST and PRODUCTION
We are in the testing phase, recently we have upgraded BW on HANA host to VMware 6.0. Currently this is not supported in production, that worries us because our go-live is end of March 2017, about 6 months to go. I have three questions related to this topic.
1) When is it anticipated to support VMware 6.0 HANA scale out for BW GA?
2) Does the controlled availability available now for production BW on HANA on VMware 6.0 scale out?. or When will it be available?
3) What memory/cpu ratio supported for BW on HANA scale out on VMware 6.0 ( < 1 TB, < 4 TB or both)
Kumar
Erik: Can you please provide a list of which storage vendors have been validated with SAP Host Auto-failover and VMware for a scale-out? It used to be shown on this page but now it's not. Thanks
Hi Erik,
this is a great and helpful document. Unfortunately the following link to VMware seems to be broken: http://www.vmware.com/files/pdf/sap-hana-scale-out-deploymentson-vsphere.pdf
Regards,
Carsten
Hi Carsten,
thanks for making me aware of the broken link it seems that the link got changed. Please try this one:
http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/sap-hana-scale-out-deployments-on-vsphere.pdf
Thanks and best!
Erik
Hi Erik,
is there anything known about Broadwell Support for vSphere 6 in production-environments?
Regards,
Stephan