Skip to Content

SAP recently completed an official HANA performance test jointly with IBM (See press release for additional details). The highlight of the test was SAP HANA achieving over 10,000 queries per hour of throughput while performing analytical queries against 1.3 TB of data, producing response in seconds. The performance test was verified by independent 3rd party WinterCorp.

 

Note that additional information about the SAP HANA performance test will be forthcoming. In addition to providing the audit report, WinterCorp is in the process of preparing a whitepaper to review and explain their findings in complete details.

 

Rather than repeating the results, or stealing the thunder from the upcoming whitepaper, I thought it may be more interesting if I share some of the thinking behind SAP HANA’s performance testing methodology and offer some of my personal viewpoints.

 

In the SAP HANA performance test, we ran a series of analytical and operational reporting queries on an enriched test database to more closely approximate the demands of operational BI in a typical SAP customer scenario.

 

Some of the concepts found in the industry performance testing practices were employed in the test:

  1. The test database is a Sales Delivery database, with 8 different tables containing 6 years of sales data.
  2. The test database is populated using a data generator parameterized to generate sizes according to a variable scaling factor (SF).
  3. The test queries include multiple joins and aggregates
  4. The performance metrics include standalone query response time, as well as throughput in a concurrent user environment. SAP HANA’s performance test follows the industry guidelines in generating a number of query streams that depends on the database generator scaling factor.

     

 

However, SAP HANA’s performance test has added the following:

  1. Two of the tables in SAP HANA’s performance test database are much larger. Modeled after the VBAK (Sales Document) and VBAP table (Sales Document Item) in SD subsystem of SAP ERP, the Lineitem and Orders tables in SAP HANA performance testing database have been enriched with hundreds of fields, with data types and distributions cover the ranges typically found in SD enterprise data.

The following table provides more information on the test database table-size

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table Name

 

 

# of Columns

 

 

# of Rows (with scalability factor 100)

 

 

Region

 

 

3

 

 

5

 

 

Nation

 

 

4

 

 

25

 

 

Customer

 

 

8

 

 

15,000,000

 

 

Supplier

 

 

7

 

 

1,000,000

 

 

Part

 

 

9

 

 

20,000,000

 

 

Partsupp

 

 

5

 

 

80,0000,000

 

 

Orders

 

 

149

 

 

150,000,000

 

 

Lineitem

 

 

250

 

 

600,000,000

 

 

  1. SAP HANA’s performance test queries were designed to emulate the business conditions of performing ad-hoc queries to analyze in-depth business data. What this translates into is that all analytical queries were tested without extensive database tuning (e.g. multiple indexes, materialized views…etc) to better emulate the situation where a business user needs to perform ad-hoc analysis on datasets that are not finely tuned already. Typically in the traditional disk-based database scenario, one would need the help of highly skilled DBAs to improve query speed and throughput via database tuning.

 

What do all these mean?

  1. The announced performance test results, in both throughput number and response time, are impressive.  For example, HANA outperformed traditional disk-based systems by over 100 times on both throughput and response time in these tests.
  2. One should avoid comparing the SAP HANA performance results with the results published by TPC-H, as the performance results are not comparable. Heed the caution statements provided in the audit report.
  3. Will this be the end of SAP HANA performance testing? I think this is only the first step. Many of you are also familiar with the SAP Benchmark Council, which has its own set of standardized benchmarking conditions, reporting a hardware independent unit of performance measurement called SAPS unit (SAP Application Performance Standard). The benefit of measuring SAP’s in-memory database in SAPS unit is its ability to measure in-memory computing performance in application usage context, independent of hardware. Additionally, one can learn many good things from the industry benchmarking groups such as TPC and it would not surprise me if there are future collaboration opportunities down the road.
To report this post you need to login first.

4 Comments

You must be Logged on to comment or reply to a post.

  1. Bala Prabahar
    Audit report doesn’t talk about disk-based system(or did I miss it?). Information on disk-based database product(I assume DB2), version etc would be helpful.  What advanced features(of disk based DB) were used?

    How does HANA compare with BWA? Also TCO comparison between BWA, HANA and Disk based systems would be helpful. BWA probably can’t handle 460b records; it can definitely handle 600m records/4TB. Would additional information that you mentioned in the blog provide these details?

    Thanks for writing this blog.

    Bala

    (0) 
    1. Ken Tsai
      Hello Bala,

      DB2 was not part of the HANA performance testing project. The entire project was focused on measuring the throughput and speed of SAP’s in-memory DB performance with realistic datasets that one would typically find in SAP DB tables.

      With that said, the performance testing team did some additional works and was able to mesaure the peformance of disk-based RDBMS with SSD running the same query, same dataset, with larger hardware, and found that SAP’s IMDB outperformed disk-based RBDMS by one to two orders of magnitude, as stated in the blog.

      More detail about the performance testing conditions and results will be covered in the upcoming whitepaper.

      Cheers, Ken

      (0) 
  2. Alistair McEwan
    Hi Dina,
    we have been reading about the great performance characteristics of HANA for months. However and what strikes me, it seems that SAP shys away from engaging into a 1:1 comparison with traditional RDBMS products. Why don’t you run some TPC benchmarks that would allow price and cost comparisons? Or why not even the SAP SD benchmark? If HANA is superior then it should show up there, even if those benchmarks are not tailored towards HANA as the one you are describing here -> see the emphasis on “analytic queries” which – by the way – has been known for 5+ years from BWA and is not so novel at all by now.
    Alistair
    (0) 

Leave a Reply