SAP recently completed an official HANA performance test jointly with IBM (See press release for additional details). The highlight of the test was SAP HANA achieving over 10,000 queries per hour of throughput while performing analytical queries against 1.3 TB of data, producing response in seconds. The performance test was verified by independent 3rd party WinterCorp.
Note that additional information about the SAP HANA performance test will be forthcoming. In addition to providing the audit report, WinterCorp is in the process of preparing a whitepaper to review and explain their findings in complete details.
Rather than repeating the results, or stealing the thunder from the upcoming whitepaper, I thought it may be more interesting if I share some of the thinking behind SAP HANA’s performance testing methodology and offer some of my personal viewpoints.
In the SAP HANA performance test, we ran a series of analytical and operational reporting queries on an enriched test database to more closely approximate the demands of operational BI in a typical SAP customer scenario.
Some of the concepts found in the industry performance testing practices were employed in the test:
- The test database is a Sales Delivery database, with 8 different tables containing 6 years of sales data.
- The test database is populated using a data generator parameterized to generate sizes according to a variable scaling factor (SF).
- The test queries include multiple joins and aggregates
- The performance metrics include standalone query response time, as well as throughput in a concurrent user environment. SAP HANA’s performance test follows the industry guidelines in generating a number of query streams that depends on the database generator scaling factor.
However, SAP HANA’s performance test has added the following:
- Two of the tables in SAP HANA’s performance test database are much larger. Modeled after the VBAK (Sales Document) and VBAP table (Sales Document Item) in SD subsystem of SAP ERP, the Lineitem and Orders tables in SAP HANA performance testing database have been enriched with hundreds of fields, with data types and distributions cover the ranges typically found in SD enterprise data.
The following table provides more information on the test database table-size
| Table Name | # of Columns | # of Rows (with scalability factor 100) |
| Region | 3 | 5 |
| Nation | 4 | 25 |
| Customer | 8 | 15,000,000 |
| Supplier | 7 | 1,000,000 |
| Part | 9 | 20,000,000 |
| Partsupp | 5 | 80,0000,000 |
| Orders | 149 | 150,000,000 |
| Lineitem | 250 | 600,000,000 |
- SAP HANA’s performance test queries were designed to emulate the business conditions of performing ad-hoc queries to analyze in-depth business data. What this translates into is that all analytical queries were tested without extensive database tuning (e.g. multiple indexes, materialized views…etc) to better emulate the situation where a business user needs to perform ad-hoc analysis on datasets that are not finely tuned already. Typically in the traditional disk-based database scenario, one would need the help of highly skilled DBAs to improve query speed and throughput via database tuning.
What do all these mean?
- The announced performance test results, in both throughput number and response time, are impressive. For example, HANA outperformed traditional disk-based systems by over 100 times on both throughput and response time in these tests.
- One should avoid comparing the SAP HANA performance results with the results published by TPC-H, as the performance results are not comparable. Heed the caution statements provided in the audit report.
- Will this be the end of SAP HANA performance testing? I think this is only the first step. Many of you are also familiar with the SAP Benchmark Council, which has its own set of standardized benchmarking conditions, reporting a hardware independent unit of performance measurement called SAPS unit (SAP Application Performance Standard). The benefit of measuring SAP’s in-memory database in SAPS unit is its ability to measure in-memory computing performance in application usage context, independent of hardware. Additionally, one can learn many good things from the industry benchmarking groups such as TPC and it would not surprise me if there are future collaboration opportunities down the road.
