Fast Is Not A Number

Let me preface this post with a few disclaimers.  I’m an SAP Mentor, an SCN forum moderator, an ASUG volunteer, and I work for a company that runs SAP.  What follows is my personal opinion, or rant if you will, and represents none of the preceding responsibilities.  I’m no SAP HANA expert; I do have experience in capacity planing at the enterprise level.  When the Business Suite on HANA was recently announced, my question was: what are the numbers?  Before I evolve that question further, some background on benchmarking, and on the hype that surrounds new technology.

Measurements and comparisons occur constantly. When I was growing up, it was the era of “muscle cars” and the benchmark number for automobiles was “Zero to 60”. How fast could a vehicle go from a dead stop to what was then the general interstate highway speed limit? The implication was, the faster your car could accelerate, the easier and safer you could maneuver in traffic. But it was really a child’s game (“My car is faster than your car. Neener neener.“). Then the first gas crisis hit, and the benchmark number was MPG – how far does your vehicle travel with a finite amount of fuel? To get back to HANA for just a second, if MPG is the metric, would it makes sense to ask what the MPG of an electric vehicle is?  Probably not, though distance traveled per money spent, or something like that, would certainly allow comparisons of varied components.

The first computer benchmarking tests I recall learning about was back in the day of the Radio Shack TRS-80, if not the first computer anyone could purchase, then close to it. I had one (before 1980) and learned to program in BASIC on it (though I had learned FORTRAN way before that). I was at a computer club meeting, and yes, there were so few of us we needed to form clubs. One of the members asked how long the BASIC interpreter took to dimension variables.  I recall being a bit perplexed by the question, and had thought this operation was, well, so fast as to be instantaneous.  But, as it turned out, it wasn’t instant, it was quite time-consuming to allocate a lot of memory.  The classic statement (including required line numbers) would look like:

(Page 37: http://bitsavers.trailing-edge.com/pdf/dartmouth/BASIC_4th_Edition_Jan68.pdf )

Adding more memory to a program than was needed would add time to the program execution.  A simple benchmark program of one dimension (DIM) statement would indicate how quickly the processor, using the instruction set provided, would do a certain amount of work. As I recall, the scalability was linear:  if it took 10 seconds to allocate 100 array entries, it would take 100 seconds for 1,000 entries.

I won’t bore the reader with more “tales from the old days”.  I’ll just say I’ve collected, run, and recorded, many benchmark programs, from the Sieve of Eratosthenes to ABAP counters (see:SLEEPing on the JOB ).  Sometimes what I find is just what the literature claims (yup, it goes from 0 to 60 in 7 seconds, or it gets 40 miles per gallon), and sometimes I find things that the account rep left out of their pitch.

What I can’t run is an SD benchmark. Presumably, nearly everyone in the SAP world knows what this is, and what SAPs ratings are.  If you don’t, take a gander at – http://www.sap.com/solutions/benchmark . The earliest results on the two-tier benchmark page, from 1996, show 12,000 to 76,000 fully processed line items per hour.  For nearly 20 year now, these tests have shown what improving hardware and software can do.  Results are still being certified, and as I write this, the most recent values are from a week ago, and the champion at order processing clocked in at nearly 14 million line items per hour. By my watch, that is an increase of 3 orders of magnitude. 

The reason I can’t run this test is that a benchmark test result, I’ve been told, could budget out at close to $1,000,000, counting hardware put on the shelf instead of shipping to a customer, and, primarily, the expert(s) time to run iterations in the prescribed manner to obtain official SAP sign-off. What you can’t see on the SD benchmark certifications are the tricks used to increase the numbers, allowed under the rules, but not something a typical customer could or should do.

I have asked for SD benchmark numbers for “Suite on HANA” since it was announced, in private, and in public:

I’ve gotten pushback, or no answer.  I am a little surprised, though I guess I shouldn’t be.

Here’s what I think may be the reasons this classic benchmark hasn’t been made public:

• The test results are worse than most existing hardware and database combinations.
• The thing just doesn’t scale the way it should.
• The mantra is “it’s really really fast” and you don’t need to put a number on “fast”.

I can wait for the hardware vendors to get their performance tuning teams all over this, like the pit crews at car races. In the meantime, here is why I think the numbers should be available:

• As it says in the blog title, “fast is not a number”.  However the platform performs, slap a (repeatable) number on it.
• People who do capacity planning know one number doesn’t tell the story, any more than a miles per gallon rating on a car makes sense if you don’t fit behind the wheel.
• Though I’ve been asked “what do I expect the numbers to be?”, any scientist worth their pocket protector would tell you the data are the data.  I don’t expect the results to be high, low, or exactly the same as an existing “legacy” R/3 system. Test it and share the results.
• SMART – measurable, repeatable, etc. If the number today is 2 million line items per hour, that gives the hardware and software mechanics a plateau from which to climb upwards.

Hopefully, some numbers (other than the BW-like ones that don’t matter to a transaction system) will be forthcoming.

References:

• The Transaction Processing Council – http://www.tpc.org/
• The “Byte” benchmark – http://code.google.com/p/byte-unixbench/
• Stream benchmark – http://www.cs.virginia.edu/stream/
• HBench-OS – http://www.eecs.harvard.edu/~vino/perf/hbench/
• Whetstone – http://www.roylongbottom.org.uk/whetstone.htm

And:

http://www.sap.com/campaigns/benchmark/appbm_bweml.epx

(Note this benchmark has one result only)

(image below linked from the sap.com/benchmark site – I liked the juxtaposition of charts and roadside billboard)