Skip to Content

SCN Content has been Migrated to 1DX NEW !

All content of the SCN space will be migrated to the Virtualization and Cloud Infrastructure Community and to the Virtualization and Cloud Infrastructure Wiki.

Therefore, the content of this page has been migrated to SAP HANA on VMware vSphere Wiki.


SCN Content Only up-to-date until October 4th 2016

By running the SAP HANA platform virtualized on VMware vSphere, SAP customers can leverage a industry standard data center platform, optimized for agility, high availability, cost savings, and easy provisioning. SAP customers will not only gain the ability to provision instances of SAP HANA in virtual machines much faster, but also benefit from unique capabilities like:

  • Increased security and SLAs, e.g. through NSX or DRS.
  • Live migration of running SAP HANA instances, with VMware vSphere® vMotion®
  • Standardized High Availability, based on VMware vSphere® High Availability (HA)
  • Built-in multi-tenancy support, through system encapsulation in a virtual machine (VM)
  • Abstraction of the hardware layer
  • Higher hardware utilization rates


These and other advanced features – to an large extend found exclusively in virtualization – lower the total cost of ownership and ensures the best operational performance and availability.  

News

  • Since May 2016, SAP supports also SAP HANA on vSphere 6 deployments for production workloads. See SAP Note 2315348 and blogs.vmware.com for more details on what’s supported, which deployment options can get used and summary of the best practices.
  • The support for vSphere 6 allows customers to increase the RAM to up to 4 TB (4080 GB) of existing virtual SAP HANA systems when migrated to vSphere 6. Beside increased RAM sizes, vSphere 6 supports also more vCPUs: Up to 128 vCPUs may now be configured and used by a single SAP HANA VM.
  • Supporting more physical compute resources inside a VM ultimately provides more “power” to a virtualized SAP HANA system – this alone is worth considering an upgrade from a vSphere 5.5 to a vSphere 6.0 based SAP HANA environment.

Support Information

  • Customers running SAP HANA on VMware shall follow the Architecture Guidelines and Best Practices for Deployments of SAP HANA on VMware vSphere.
  • We have lately seen some issues with VMs configured with large virtual hardware resources and memory-intensive workloads becoming temporally unresponsive after vMotion has completed. We recommend to read VMware KB 144984 and configure hosts correspondingly.
  • Noticing a certain interest in our CPU hot-add feature, we would like to highlight that the use of this capability also requires the OS and the SAP HANA database itself to be capable of leveraging the additional resources instantly. Please be aware that the use of CPU hot-add feature disables vNUMA and hence may lead to performance degradation.

The below table states capabilities, supported deployment options and best practices, like minimal vCPU count or maximal vRAM sizes for SAP HANA VMs on VMware vSphere as of May 2016. For additional reference, see SAP Note 1788665 et. al.

Capability / Option

vSphere 5.5

vSphere 6.0

Supported in Production

Supported in non-Production

Supported in Production

Supported in non-Production

SAP HANA Scale-Up VM 1024 GB

Yes

SAP HANA Scale-Up VM 4080 GB

No

Yes

SAP HANA Scale-Out VM 1024 GB [1]

Yes

No

Yes

SAP HANA Scale-Out VM 4080 GB [1]

No

Yes

SAP HANA Multi-VM [2]

Yes

Limited [3]

Yes

SAP HANA Version

SPS07 and above

SPS11 and above

SPS09 and above


VMware vSphere and SAP HANA HA and Operation Features [4]

VMware HA

Yes

SAP HANA Host Auto-Failover [5]

Yes

SAP HANA System Replication

Yes

VMware FT

No

VMware SRM

Yes

vSphere vMotion

Yes

VMware DRS [6]

Yes


Supported HW configurations [7]

Supported SAP HANA Systems for VMware virtualization

Only SAP HANA and VMware certified two-, four- and eight socket Intel E7 v2 (Ivy Bridge) and later Intel processor-based server systems, as well as Intel Xeon E5 v3 and v4 based two-socket single node SAP HANA entry level systems, with a minimum of eight cores per CPU are supported.

Intel Xeon E7 CPU Support

Ivy Bridge, Haswell

Ivy Bridge, Haswell, Brodwell[8]

NUMA Nodes per server

4

8

NUMA Node Sharing

No

Yes

No

Yes

Enable Hyperthreading

Yes

Maximal RAM installed in server[9]

4/6 TiB

6/12 TiB

Supported Storage Configuration

Supported SAP HANA Storage Systems for virtualisation [10]

All SAP HANA TDI and VMware certified / supported storage solutions can get used.

TDI Storage KPI requested

Yes, per HANA VM

No

Yes, per HANA VM

No


SAP HANA Virtual Machine Configuration [11]

Max. VM size #vCPU

64

128

Max. VM size #vRAM

1024 GB

4080 GB

Minimal VM size #vCPU

All threads of a single CPU socket

10

All threads of a single CPU socket

10

Minimal VM size #vRAM

RAM of CPU socket

As sized

RAM of CPU socket

As sized

CPU reservations [12]

No

RAM reservation

Yes


Supported OS for virtualized SAP HANA

SLES 11 and 12 [13]

Yes

RHEL 6 and 7 [12]

Yes

Table 1. SAP HANA on vSphere supported capabilities and options


    1. SAP support for SAP HANA scale-out deployments of SAP Business Warehouse (BW), powered by SAP HANA, on VMware vSphere as outlined in the table in context of SAP BW, powered by SAP HANA. Further supported workloads of OLAP-type (HANA Data Mart, HANA Data Warehouse) generated by SAP and non-SAP tools:
      • Documentation and / or tool-support for table distribution, sizing, partitioning, scale-out and how to configure SAP and 3rd party products on SAP HANA has to be provided by SAP, partners or customers resp.
      • 3rd party scenarios will not be tested by SAP. In specific cases, SAP HANA support for specific products or scenarios might be excluded by SAP in the future. If tested, then these will be listed in the SAP HANA on vSphere support note.
      • 3rd party tool support can be checked at http://global.sap.com/community/ebook/2013_09_adpd/enEN/search.html (select a partner –> details –> SAP Certified Solutions)
    2. The defined KPIs for Data Throughput and Latency for production SAP HANA systems has to be fulfilled for each VM. SAP has released a special tool, the SAP HANA HW Configuration Check Tool, to measure if the used storage is able to deliver the required IO capacity, see SAP Note 1943937
    3. Just like with the vSphere 5.5 SAP HANA support release in the beginning 2014, vSphere 6 supports currently only one production level VM that may get co-deployed with non-production level SAP HANA VMs. No resource sharing with other VMs is supported for production level SAP HANA VMs.
    4. Generally all VMware vSphere features, like distributed switches are supported, no support for VMware FT for SAP HANA, FT may get used with an (A)SCS instance. vMotion and DRS are explicitly listed since running SAP HANA VMs can get moved from one host to another host while the application is online and users are connected and any negative impact needs to get excluded.
    5. Only available with specific SAP HANA TDI Storage Vendors, see SCN page DOC-60470.
    6. VMware DRS should get configured in “manual mode”
    7. Both SAP HANA appliance and SAP HANA Tailored Datacenter Integration (TDI) delivery methods are supported for SAP HANA on VMware vSphere. Where the SAP HANA system has either been delivered pre-configured on certified SAP HANA appliances, as listed in SAP HANA Product Availability Matrix (PAM), with VMware vSphere hypervisor installed by SAP HANA hardware partner or the SAP HANA installation was done by an SAP certified engineer, qualified as “SAP Certified Technology Specialist – SAP HANA Installation” on SAP HANA certified hardware. It is recommended to verify the configuration with the SAP HANA hardware configuration check tool prior the installation.
    8. Brodwell CPU vSphere 6 support in validation, yet not supported. Please check SAP support release support note 2315348 for an updated status.
    9. Maximal RAM installed in physical server as specified by SAP, in the Certified SAP HANA Hardware Directory. The listed memory figures are the maximal memory configurations supported by VMware vSphere. Up to 6TB is supported for ESXi 5.5 Update 2 and later and up to 12 TB is supported with vSphere 6.0 on specific OEM certified platforms.
    10. For details please refer to the Certified SAP HANA Hardware Directory and go to “Certified Enterprise Storage” configurations.
    11. The maximum size of a virtual SAP HANA instance is limited by the maximum size of a virtual machine of the VMware vSphere release. Each SAP HANA instance / virtual machine is sized according to the existing SAP HANA sizing guidelines and VMware recommendations. CPU and Memory overcommitting must not be used.
    12. Since production level VMs have to run on dedicated NUMA nodes, there is no need in configuring CPU reservations. Configuring CPU reservations is only required when production level SAP HANA VMs would share a NUMA node, which is currently not supported. Non-prod. level VMs do not require CPU reservations.
    13. Both, SUSE Linux Enterprise Server (SLES) 11 and 12 and Red Hat Enterprise Linux (RHEL) 6 are supported operating systems when virtualized. Due to different scope and test coverage during corresponding validation activities, the underlying server must be certified for corresponding operating system, however.

SAP HANA on vSphere Best Practices

In a nutshell: SAP HANA follows general published vSphere Best Practices for databases:

  • Set Memory Reservations for SAP HANA Virtual Machines
  • Configuring Paravirtual SCSI Controllers and Network Adapters
  • Right sizing of SAP HANA VMs to ensure local NUMA node memory access and high 2nd level cache hit ratios
  • No NUMA node sharing for production SAP HANA VMs
  • Enable Hyper Threading on the ESXi host
  • Use dedicated networks for vMotion, management and client network
  • Use vMotion and VMware snapshots during non peak times


Untill the new “Architecture Guidelines and Best Practices for Deployments of SAP HANA on VMware vSphere guide will be available from www.vmware.com, see the best practices and especially the configuration settings as described in the Best Practices and Recommendations for Scale-Out Deployments of SAP HANA on VMware vSphere document (page 71-73) for more detailed guidance. In addition, please ensure that following configuration parameters are used in addition to achieve optimal performance for SAP HANA:

  • halt_in_monitor = “TRUE”
  • idleLoopSpinBeforeHalt = “TRUE”
  • Lat. Sensitivity – normal



Comparison: Physical vs. virtual SAP HANA deployment (as state of February 2016)

Capability

Physical SAP HANA

Virtual SAP HANA

SAP HANA Single Node Memory per node

128 GB – 4 TB RAM with 8 sockets

≤ 1 TB RAM per VM

(vSphere 6.0 4** TB per VM possible)

SAP HANA Scale UP SoH Memory per node

768 GB – 12 TB RAM with 16 sockets

≤ 1 TB RAM per VM

(vSphere 6.0 4** TB per VM possible)

SAP HANA Scale Out BWoH Memory per node

256 GB – 4 TB RAM with 8 sockets

≤ 1 TB RAM per VM

(vSphere 6.0* 4** TB per VM possible)

Appliance Model

Yes

possible***, normally a TDI deployment

Server Hardware

Certified Appliances

Certified Appliances or TDI

Storage Hardware

Certified Appliances or TDI

Certified Appliances or TDI

Scale Up  Configuration

Supported

Supported

Scale Out Configuration

Supported

Supported

CPU Sockets per server

2 to 16 CPU sockets

2 to 4 CPU sockets (8 with vSphere 6)

Flexible HANA sizing’s

No, T-Shirt sizing’s

Yes

HANA DB Live Migration

No, not possible

Yes

Standby-Node / System Replicaktion / VMware HA

Yes / Yes / No

Yes / Yes / Yes

Multi-Tenancy / full tenant isolation

Yes, since SPS09 / No

Yes / Yes

* vSphere 6 for non-prod. use cases **vRAM configuration is not allowed to exceed installed memory of a physical server system and must follow SAP HANA sizing guidelines. ***Some SAP HANA HW vendors offer VMware vSphere as part of their SAP HANA appliance system. This is not required and optional. The customer can choose between a pre-installed appliance or the certified server configuration and may perform the installation of vSphere on its own.

VMware SAP HANA  System Health Check

SAP and VMware are offering joint services to help you design, deploy, and implement your virtual SAP HANA platform.

Examples of services which may be offered are:

  • Pre-Call to discuss PoC, goals, timeline and success factors (SAP, HW Partner, Customer and VMware)
  • Architecture review and introduction to the SAP HANA on vSphere Configuration Guidelines and Requirements
  • Introduction to SAP HANA Hardware Check Configuration Tool (HWCCT)
  • Online review session to check prepared environment like:
    • Host including storage and network configuration
    • VM configuration
    • Linux OS configuration

For more information on SAP Consulting and VMware Professional Services, visit vmware.com/consulting or get in contact with your local VMware contact.

To report this post you need to login first.

31 Comments

You must be Logged on to comment or reply to a post.

  1. Jonathan Haun

    I am curious to know if anyone has researched or tested using NUMA affinity settings with the guests and Numa.PreferHT=1 setting for the ESX 5.5 host?

    For example:

    HOST has 512 GB RAM and 4 Sockets (10 core each)

    Guest could have 124 GB of RAM and 1 Socket with 20 core configured. It is also set to only use NUMA node 0.

    If we enable the Numa.PreferHT=1 on the HOST, it seams to me that the guest would be able to have 20 v-cores plus use only the memory assigned to NUMA Node 0. Curious to know if that would help the SAP HANA guest given that it could have more cores without the penalty of crossing NUMA nodes.

    (0) 
    1. Sebastian Lenz

      Hi Jonathan,

      that is correct, but you would use the hyper threads, so it would be the roughly 20% more performance compared to the 10 pure cores under full load.

      So if you want to keep the NUMA locality and take also advantage from the hyper threads, you can use the PreferHT=1 for a HANA VM.

      Cheers

      Sebastian

      (0) 
  2. Antonio Freitas

    Great to see that customers wanting application aware HA when deploying SAP HANA on VMware can now achieve more demanding Recovery Time Objectives with the validated storage connectors.

    Great job EMC on being the first one on the list! I have quite some requests for this stuff!

    (0) 
  3. Robert Paul

    Great document! Could anyone provide a table which mentiones the sizing guidlines for Haswell CPUs (CPU/Memory ratio for SoH/BWoH)? In the best practice guides are only tables for Westmere and Ivy Bridge CPUs included. Would be great to have an update here.

    Cheers

    Robert

    (0) 
  4. Erik Rieger Post author

    Hi Paul,

    I am working on a new SAP HANA on vSphere sizing guide, which will hopefully be ready in May. Here I will cover in more depth SAP HANA sizing. I will publish the guide here once it is available.

    The CPU/memory ratios are relatively easy to calculate, just divide the maximum RAM of a certified SAP HANA appliance by the amount of CPU sockets, lime for BWoH (SPS11) 2 TB / 4 CPU sockets = 512 GB, or for SoH 3 TB / 4 CPU sockets = 768 GB.

    That’s the maximum RAM available per CPU socket.

    Please remember to subtract the RAM needed for ESXi from these figures. As a generic guideline we calculate with 2-3% for ESXi.

    Also remember that these are the maximum figures and that a HANA server system may have less memory installed, like only 1.5 TB instead of 2 TB for a BWoH system.

    (0) 
  5. Vikas Agrawal

    Hi Erik ,

    Great document, based on SAP HANA SOH Haswell EX reference sheet 1.5TB is supported for 2x Processors, If I want to virtualize a two Processor scaleup appliance with VMware 5.5 for Production what is the maximum number of virtual machines which I can run in virtual environment for Production usage, Do I need to limit 1 virtual machine per processor, or i can have for example 10VMs of 128GB in 1.5TB System, Leaving approx. 15% spare memory for ESX. Do you have any guidelines for Virtual core to memory ratio?

    (0) 
    1. Robert Paul

      Hi Vikas,

      you should limit the virtual machines to 1 per NUMA node in a productive environment. There are 2 NUMA nodes in a 2 socket server . If you are running a non-productive environment you can also have 10 active virtual machines depending on the needed computing and memory resources.

      BR

      Robert

      (0) 
      1. Erik Rieger Post author

        Hi Vikas,

        some additional comments on Roberts statement. It is advisable to set memory reservations

        for your 10 non-production HANA VMs. Please subtract 2-3% from the overall RAM reservations for the hypervisor, otherwise the last VM want start. This is not a must for non-prod systems, but you may want to avoid in any case memory swapping for SAP HANA due to memory over-commit situations.

        It is also advisable to set the parameter numa.vcpu.preferHT=TRUE per VM to keep the processes and memory access of the VMs local to the NUMA node the VM got started. This implicates that HT is enabled on the host.

        Best,

        Erik

        (0) 
  6. Erik Rieger Post author

    Sorry for the delay and the answer is yes!

    vSphere provides full resource isolation and as long the production SAP HANA VM runs with full resource commitments on an exclusively used NUMA node(s) this is possible.

    Beside this it also has to get ensured that the host is able to provide enough storage and network bandwidth to support the load of all planned systems.

    Example: 4 CPU socket server (4 NUMA nodes) with 3 SAP HANA VMs of following sizes:

    1. Prod HANA VM on NUMA node 0 + 1
    2. Dev HANA VM on NUMA node 2
    3. est HANA VM on NUMA node 3

    This would be a completely supported environment.

    Advice: Verify inside the Prod HANA VM the HANA TDI Storage KPIs with the HWCCT test tool. This would allow you to rate the storage and to ensure that the Prod HANA VM meets the SAP recommendations.

    (0) 
        1. Jorge Navarro

          Andreas:

          My request is generic, for sizing purposes, so I don’t have a particular server to run esxtop. Is it safe to assume that any certified HANA appliance/server with X sockets maps to X NUMA nodes?

          BR

          Jorge.

          (0) 
  7. Jorge Navarro

    May we have an answer to the above question: Is it safe to assume that any certified HANA appliance/server with X sockets maps to X NUMA nodes?  My request is generic, for sizing purposes, so I don’t have a particular server to run esxtop.

    Many thanks

    Jorge

    (0) 
  8. Irakli Natsvlishvili

    Eric,

    In previous generation of HANA servers there was 4x10core CPUs with 512GB of RAM, translating into 12.8 GB of RAM to Core ratio. We at our non-production lab (COIL) were provisioning VMs with 10 virtual cores and 60GB of RAM, giving us 8 such VMs on above mentioned server. The ESX host also had numa.vcpu.preferHT=TRUE enabled and as a result, each socket was running 2 VMs at any time, or one VM if that VM had 120GB of RAM.

    Newer servers are coming with 1.5TB of RAM and 2x18cores of CPU. That translates into Core to GB of ram ratio of 42. So does this mean that for these new servers/CPU for the same memory size (60GB) of VM we need to provision 3 virtual cores instead of 10?

    Thanks.

    (0) 
    1. Sebastian Lenz

      Hi Irakli,

      this ratio is only viable for i5, but the math is correct.

      Going further with this, we see that the SAP HANA is spawning a bunch of threads which could make a VM with 3 vCPU suffer from poor performance under load. To prevent this we would like to see even in this scenarios at least 10 vCPUs.

      But for test system you can try to use less vCPUs, but please use always a number which can be divided by two.

      I hope this information will help you.

      Cheers

      Sebastian

      (0) 
  9. Alisa Martinelli

    Will the section “Supported OS for virtualized SAP HANA” and note #13 include support for RHEL 7 in the near future or is there another document/note that specifies VMware support for HANA on RHEL 7?

    Thanks…Alisa

    (0) 
    1. Erik Rieger Post author

      Hi Alisa,

      Thanks for making me aware, this was a typo in the table. RHEL 7 is already supported. I updated the table accordingly.

      Erik

      (0) 
    1. Arne Arnold

      Hi Mark,

      support status hasn’t change yet. As soon there’s an updated, we will post it here (and the corresponding SAP Note). Feel free to send me a PN to discuss more detail your particular scenario and needs, so I may provide you with further guidance within current scope of support.

      Arne

      (0) 
      1. Anbukumar Sivasubramanian

        We are in the same situation. We started with VMware 5.5 with scale out for BW on HANA with a master and two salve nodes of each 1 TB.
        As per the sap notes, it is supported in TEST and PRODUCTION

        We are in the testing phase, recently we have upgraded BW on HANA host to  VMware 6.0. Currently this is not supported in production, that worries us because our go-live is end of March 2017, about 6 months to go. I have three questions related to this topic.

        1) When is it anticipated to support VMware 6.0 HANA scale out for BW GA?

        2) Does the controlled availability available now for production BW  on HANA on VMware 6.0 scale out?. or When will it be available?

        3) What memory/cpu ratio supported for BW on HANA scale out on VMware 6.0 ( < 1 TB, < 4 TB or both)

        Kumar

        (0) 
  10. Fanny Osorio

    Erik: Can you please provide a list of which storage vendors have been validated  with SAP Host Auto-failover and VMware for a scale-out? It used to be shown on this page but now it’s not. Thanks

    (0) 

Comments are closed.