Skip to Content

Benchmarking…my lessons learned…

Yesterday, I wrote a blog called MySQL, PostreSQL and SAP HANA – A friendly comparison which I thought it was fun and cool…and maybe it was…but…thing can’t be taken that lightly…no even by myself…

My blog might be entertainment…but…there’s a few points that must considered when doing something like this…

  • Benchmarking must be done using the same Hardware. There’s no point on comparing a DB running on a single core to another one running on 4 or more cores.
  • Benchmarking must be done on the same environment. In this case I used local copies of MySQL and PostgreSQL while SAP HANA was running on AWS.
  • Benchmarking must be done with real knowledge of the technologies being used…I have almost no experience using MySQL and PostgreSQL…and that might lead to poor performance.
  • Benchmarking must be done on DB’s that were designed to fulfil the same goal. SAP HANA was created for ERP/Enterprise applications. MySQL and PostgreSQL might be as well…but the difference is that they were not designed thinking on Big Data or the special kind of information managed by SAP.
  • Benchmarking results must be published making completely sure that the code is well done. I made a huge mistake on measuring the speed taking the Python processing time into account…Should have been focused on the DB’s time only…but again…without the same hardware…it’s pointless…

This said…that was my first and last blog on the Benchmarking topic…I made a huge mistake trying to put myself on a territory that is clearly not my main area of expertise…sorry about that…sometimes, you let your passion for technologies to drive you in the wrong direction…lesson learned and lesson taken…

Rest assure, that I will keep blogging and doing my crazy experiments…but the same mistake will be not repeated twice…



You must be Logged on to comment or reply to a post.
  • NO, NO, NO and  1000 times NO.

    I am not letting you get off that lightly. I totally applaud you for getting on a wild brumby that has given you a few jolts as it tried to buck you off but get back on the horse and give it another go.

    It was a great idea to make comparisons like this after all why should anyone pay top dollar for HANA unless it can really deliver. No one is expecting exceptional performance from Mysql but with the correct architecture it can run a site like Wikipedia.

    Let’s keep doing these comparisons and see what we learn from them.



    • Nigel:

      I would keep making comparison and tests, but only for myself (and anyone who wants to join) but I will for sure not blog about it…I don’t want to lead anyone to confusion or make public benchmarks using scenarios that are not the same…if I had MySQL or PostgreSQL using the same hardware as SAP HANA…then I could consider it…but as I don’t…I prefer to remain quiet about this topic 🙂



    • I agree with Nigel on the importance of doing such comparisons in order to obtain measurements and base decisions on facts. In my mind, having facts available for comparison, even if the facts might be only 50% reliable, the methodology wasn’t bulletproof, etc. is still infinitely better than having no facts at all going into a decision. Also, someone will always find a flaw in any methodology so aiming for a binary choice of “perfection or nothing” is rarely helpful.

      Maybe the term “benchmarking” should not be applied to such activities though because it is highly overloaded with marketing-sensitive meaning, and 200-page documents used by vendors, analysts and customers to make large commerical decisions… So agree there is danger here.

      Maybe I would encourage people to share their findings with some up-front disclaimers to clearly state the un-official nature of the results and other factors which might bias the results. But if you found the activity useful, then there’s a good chance others will also find the results interesting.

      However, I would never let all of this discourage anyone from embarking on at least some measurement-based testing or comparison. The alternative is far worse, and doesn’t help anyone in the long run.


      • Sascha:

        Thanks for your comments, but I don’t think I’m discouraging anyone…not even myself…it’s just that I believe that I should have wrote my blog in a different way and present the facts in a more clear way…last thing I wanna do is to misguide or confuse people…

        It was my choice to write the blog and it was my choice to write this “mea culpa” blog as well…not as an SAP employee, but as a Developer…

        I would love to read blogs from other people doing some “comparisons”, but I just don’t feel like doing one myself again…



        • Hi Blag,

          sorry, I didn’t mean to say you were discouraging. I just didn’t want someone else reading this and thinking “oops, better not start something like this then”! 🙂

          All good, but I do hope you’ll change your mind and keep sharing 🙂


          • No harm done 🙂 I always take things on the good side 😉

            I will keep sharing for sure…just not this kind of “comparisons”…my strongest suit is the integration of SAP and external technologies…make them work together and not compete against each other 🙂

            Like I said…I would love to see people (more prepared than I am in the DB field) doing what I tried…like they say “Shoemaker to your shoes” (I think it sounds better in Spanish) 😛