Skip to Content
Author's profile photo Former Member

How Real-Time is SAP’s Real-Time Data Platform ?

This blog post is 3 months in the making. I started writing it on the way back from Sapphire in May, and due to the manically hectic schedule that is called “my life”, never got to finish it and am now starting fresh.

In hindsight, I’m glad this happened, as between now and then the “To BW or not to BW” debate has been a hot topic. This blog post is not going to go down that road as I feel my peers have covered those points very well, and as Ethan Jewett pointed out so eloquently in a tweet two days ago People (especially experts) tend to prefer tools that they understand.

As I sit here on flight QF63 from Sydney to Johannesburg, I am pondering the sentiments and comments from all the attendees (customers, partners and SAP employees) of the Mastering BI with SAP Conference in Melbourne the past few days. The general message, from a Database & Technology (D&T) point of view, is still very much HANA for everything which is slightly disappointing. I was hoping that SAP would have started to embrace all the products in the Real-Time data platform with one integrated view for the customer instead of pushing the HANA message which I feel customers are getting a bit tired of.


Those that know me well often ask me if I go to all these conferences to surf or to attend the event. A surfboard is always part of my luggage, and the odd detour from the convention center to the ocean is always on the cards for me. The picture above represents my Nirvana. Warm tropical water and beautiful uncrowded surf. Throw in the wife and kids and this nerd is one happy man.

For my customers, I want them to have this Nirvana when it comes to accessing their data – one entry point, and not by using a BI4 universe, which kills your performance with large data sets. To put things bluntly, I feel that my customers should not really care where the data comes from. The words “hot and cold” data get thrown around loosely but this always leads to a large amount of ETL (Extract-Transform-Load) and users needing to be educated on which “entry point” they need to access their data based on the date range they are trying to access. One of my customers referred to this solution as “clunky” and I could not agree more!

I guess by now you are wondering what I am after… what is the “silver bullet”? I want my customers to have one entry point (the real time data platform) and by using slick and intelligent archiving mechanisms “under the hood”, they must be oblivious to where the data is stored without moving and duplicating data backwards and forwards between database technologies. In the “old days” we were lucky enough to work in batch driven systems, so reconciliation was pretty simple as there was always an end of day point. Those days are now long gone as many customers load data into their BI solution 24/7, with no end of day, which always makes reconciliation a challenge (and that’s putting it nicely). 

Fun Fact: We were fortunate enough to have 90 minutes with Hasso Platner and Vishal Sikka in May in Orlando and it was fascinating for me to hear (from Hasso Plattner) that the hot/cold concept of storing data was in the original design of R1, but due to the low volumes of data back then, everything was put in the “hot storage”.

So where are we today and how close are we to that Nirvana? As is the case in the current SAP eco-system, there are two distinct scenarios: BW or non-BW customers, so I am going to break down my opinions that way.

Many thanks to fellow SAP Mentor Ethan Jewett for checking my BW facts, as my hands-on experience is limited.

BW World

Things are actually looking very good for BW customers. Right now, there are two third party companies ,that I know of, that offer the Near Line Storage (NLS), namely:

  1. PBS NLS: I have been exposed to this solution through my long-term love affair with Sybase IQ. The concept is excellent, where your read only data gets archived into a Sybase IQ data store. The benefits to the company are immediate and are as follows:
    1. Shrink the size of your native BW database, which ultimately leads to query performance enhancements
    2. Get the benefit of compression in Sybase IQ, dropping your data foot print
    3. The benefit of column based storage in Sybase IQ for fast query response times

          It almost sounds too good to be true typing this,  and by no means is the PBS solution cheap.

2.     SAND – I am less familiar with this solution, but thanks to fellow SAP Mentor, Sascha Wenninger, for his input, as they implemented this strategy at one of          his customers.

Ethan also let me know about this solution by IBM using DB2

The Future

By all accounts, native NLS is meant to be coming in one of the service packs of BW 7.3. This will allow BW customers, without paying the additional licenses to third party vendors, to have their truly hot data in HANA with the cold archived data in Sybase IQ. For me, this is an awesome solution, and once the price of HANA slowly drops over time, customers can increase the amount of the hot data in HANA.

One thing that does concern me, from a purely theoretical point of view, is that the BW application layer does most of the processing.  I understand why SAP did this, to ensure that they were database agnostic, but now that they are in the database game, there is a huge need to push the processing down to the database engines to really reap the benefits of the technology of HANA and SAP Sybase IQ. 

Enterprise Data Warehouse (EDW) / Non BW World

Many thanks to fellow SAP Mentor Josh Fletcher for checking my sanity on this section.

This is the area that is close to my heart and the one that I mainly focus on.  In this space, there are currently no options whatsoever (yet again I am not looking at a BI4 multi source universe). The EDW world is filled with ETL jobs and moving data backwards and forwards. The challenges brought in by the constant movement of data are vast, with reconciliation efforts often being quite tedious and time consuming and the “single version of the truth” is seldom met.

There is SAP Sybase Replication Server for replicating data from SAP Sybase ASE to SAP Sybase IQ. Replication Server has got a new “Real time data load” function that polls data up in replication server, and then bulk loads this into Sybase IQ. It is solid technology and works well, but all you are doing here is creating an operational data store (ODS) in your EDW by replicating the OLTP system sitting in SAP Sybase ASE.

I have such a clear vision for this space, and to me, it is clean, simple and has the potential for SAP to entrench themselves in many customers, and more importantly set themselves up for future opportunities.

During my conversations with the APJ SAP team over these last few days, the number of SAP BusinessObjects customers (2500) in the region got mentioned. Now, let’s assume that the majority of those are classic BOBJ customers and not BW ones.

If I was running D&T in APJ (doubt they would have me 😉 ) I would do all I could to make contact with those +- 2000 existing BOBJ customers and talk about Sybase IQ. Now I know a few of you might be sighing and say: Clint’s on about Sybase IQ again! But, wait, there’s more…………….

Similar to the BW space, I feel SAP need to offer the same NLS type solution for EDW customers. This way SAP can go out with confidence and start selling Sybase IQ into the EDW/classic BOBJ space immediately, and know that the HANA sales will come. As soon as SAP get this integration/archiving sorted out then, in my opinion,  SAP will be hard to beat. 

In my experiences, HANA is suffering in the EDW space (note I am based in EMEA) and when it comes down to the TCO argument with no special content all falls completely flat.

But, here is the Nirvana for both the customer and SAP – let’s also remember that SAP sell software! SAP need to approach all their EDW customers and get them to look at Sybase IQ. MANY customers out there are dying with performance of the traditional row based databases that are often seen under those classic BOBJ systems.

Customer Win

  • Smaller data footprint due to compression
  • Drastic query performance improvements
  • Faster load times of data with lower latency 
  • Not having to archive old data off due to performance constraints
  • Lower TCO

I could go on and on but that will be another blog.


  • They sell software and increase the D&T footprint. More importantly they sell their customers a tried and tested technology that will add huge and instant value to their customers. 

Let’s look 12-18 months down the line, and the HANA/Sybase IQ integration in the EDW space is out of beta and is now a slick solution. SAP can now approach the same customers with large data volumes, who I can pretty much assure you, are going to be happy with the SAP database technology (Sybase IQ), and look to start loading the hot data into HANA with seamless archiving into the already existing Sybase IQ instance

Customer Win

  • Benefit of speed of in memory technology
  • The ability to buy a small instance of HANA as an entry point and not worry about their users having multiple entry points into the data
  • Continue to leverage their previous investment in Sybase IQ


  • Sell more software
  • A non disruptive way to offer their customers a cheap entry point to in memory technology
  • A very scalable solution for customers being able to add on more and more HANA where needs be

I know the HANA message is strong right now at SAP; However, I honestly feel that the above message is one that will be more palatable to customers, and seeing that it is based on solid, scalable technology, will make SAP look good.

SAP is working hard at trying to change their image of long, expensive projects that often break the bank at customers. In my experience, within a few short weeks, you could replace a classic DBMS with SAP Sybase IQ, and amaze your customers with the benefits.

The key word for me in both my personal and business world is trust.  Customers are tired of the “HANA message” and in most cases it falls out of budget. If SAP look into their tool kit and see that SAP Sybase IQ is a great 2-4 year plan that will give customers huge value at a fraction of the price, it will buy them credibility and most importantly trust from their customers.

The investment won’t be a throw away, as their customers can use Sybase IQ as their archiving strategy to assist with long term (10+ years) trend analysis.

Don’t get me wrong; I am not a HANA hater. I think the technology is amazing and truly do feel that in memory is where everything is going to be in the next 5 years. The key is in how we get there!

I have just noticed that this blog post is over 5 pages – the joys of a  15-hour flight and post conference thoughts…

So to summarize:

BW folks: This is looking OK with a roadmap in place and third party tools for those less patient.

EDW folks: There is a real-time-data platform filled with amazing technology but right now the integration is either replication or ETL

SAP have been quite bullish in their statement of “being the 2nd largest database vendor in the world by 2015”, and, perhaps walking the road with their customers, from disk based to in memory technology over the next few years, will work better than going straight to the HANA message upfront.

Assigned Tags

      You must be Logged on to comment or reply to a post.
      Author's profile photo Former Member
      Former Member

      HI Clint.

      Thanks for blog.

      I think it's very good idea to use IQ as EDW for Non-SAP BO-customers and as NLS for BW on HANA customers. But IMHO Sybase IQ license cost now is to much higher.

      IQ license policy must more accessible. Elsewhere SAP HANA wll be more advantageous for customers and IQ will die.

      Advanced Options in IQ like security, Multiplex Grid Option and etc. must not to be sale for additional money, they options must be include in enterprise package by default.

      And also Sybase and SAP must work hurd to integrate Sybase poduct to SAP Empire, and Sybase must be more open for infromation (roadmaps, plans, blogs, community grows and etc.). They must popularize Sybase products.

      Author's profile photo Former Member
      Former Member

      Hi Mikhail,

      Thanks for the comments and I agree completely with your sentiments. For me it makes sense for SAP to really make it known to the world that they are a big player in the D&T space.

      If they can come in at a lower cost point on IQ and rapidly increase their footprint in the industry it is, well I feel, setting them up for future success down the road.


      Author's profile photo Former Member
      Former Member

      APJ SAP team ?

      Can you provide more details about this team ?

      Author's profile photo Former Member
      Former Member

      Asia-Pacific and Japan SAP Team. There were a few members of that team in Melbourne last week at the conference we were at.

      Author's profile photo Former Member
      Former Member

      Clint, completely agree with you on this front.  What's interesting is your initial statement about hot/cold data, and the problem that the BI4 universe kills performance on big data if attempting multi-source connections. Couple this with the fact that BW currently can use HANA and IQ for NLS seamlessly to the end user.

      So, being controversial here, again we have this difference between what BW offers as a semantic layer, and what BOBJ offers.  Perhaps SAP need to do more work in ensuring performance at the semantic layer in BOBJ so that performance isn't an issue, and that hot/cold data is given one entry point for the user.

      As for the NLS option, there is not only the querying part - what about the actual NLS process, which I think Data Services is making progress towards, with their new Workbench Eclipse tool, which will replicate to HANA or IQ automatically and very simply - all it needs is some controls on hot/cold data, and it can do it.  Replication Server doesn't get much airtime at the moment either, which is a little concerning - will it replace SLT as the HANA replication option, or not?

      End of the day - still really keen to see the overall DB&T roadmap... 😆


      Author's profile photo Former Member
      Former Member

      Hay Josh,

      Thanks for the comments. Offline Ethan and I are having the discussion of how best to do this NLS/EDW scenario without hammering performance.

      If memory servers me correctly then BusinessObjects do have a Data Federator tool but I am not to sure how it handles large volumes of data ?

      Thanks also for pointing out the DataServices option.

      As for the road map... can't wait ! 😉


      Author's profile photo Ethan Jewett
      Ethan Jewett

      The short version of my offline comments is that transparently accessing solutions like NLS is a *really hard problem*. Making it worse, SQL wasn't designed for this problem (which is one reason the BW OLAP engine isn't just SQL). If you want to have a full-featured and reasonably performant SQL interface to a table that looks like a normal database table but is actually half in one database and half in another, it's going to be brain-meltingly hard 🙂

      Author's profile photo Former Member
      Former Member

      Agreed - as much as I was trying to avoid the data federation option it seems once thinking about it a bit deeper that it is probably going to be the best way to solve the problem OR an application layer sitting on top on the RTDP ?

      Either way the performance contraints worry me 😕

      Author's profile photo Sascha Wenninger
      Sascha Wenninger

      So we need NoSQL then, right? 😉

      Author's profile photo Former Member
      Former Member

      Exactly right 😆 😆

      Author's profile photo Ethan Jewett
      Ethan Jewett

      That should solve the problem. No sweat 🙂

      Author's profile photo Former Member
      Former Member

      A very good summary of the state of things as it pertains to EDW in the SAP space.  I share your views all round and also agree with Mikhail that SAP needs to be more creative in making Sybase IQ attractive as an NLS option from a pricing perspective.

      One suggestion might be to introduce an NLS-only license for IQ at a reduced price that limits the use of IQ to an NLS store for BW.  This would be similar to what SAP did by introducing a lower cost HANA license when using it as a platform for BW.

      In my view IQ NLS is a critical enabler to allow BW customers to migrate to HANA and SAP should be doing more to get customers moving in this direction.

      Author's profile photo Former Member
      Former Member

      Hi Herman,

      Thanks for the comments.

      A reduced price of IQ for NLS is a great suggestion and one I hope SAP will listen to. I guess the "problem" with that is you can access IQ via an ODBC connnection and the licensing around controlling that may be tough.

      Looking forward to catching up next week.



      Author's profile photo Former Member
      Former Member

      Do we need ASE in SAP Real-Time Platform ?

      What for ? Is ASE is better then HANA?

      ASE - OLTP (disk), HANA  as OLTP (in-mem)

      Author's profile photo Former Member
      Former Member

      You could say the same thing about Sybase IQ then I guess ?

      Both ASE and IQ are disk based solution and right now a better fit for those with budgets.

      In theory, and with no budgetary contraints, disk based technology is going to fall away.

      Author's profile photo Former Member
      Former Member

      Different is that

      HAHA TCO is high. HANA is multimillion EUR solution (licence, hardware)

      IQ - thousands EUR solution (licence, hardware). 


      BW on HANA avg cost. (little error - not usd, eur).

      Author's profile photo Former Member
      Former Member

      Dear Clint.

      >  The Future

      > By all accounts, native NLS is meant to be coming in one of the service packs of BW 7.3. > This will allow BW customers, without paying the additional licenses to third party vendors, to have their truly hot data in HANA with the cold archived data in Sybase IQ.

      Are you shore about this ?

      Author's profile photo Former Member
      Former Member

      Great blog, Clint, but what I think you are missing is that SAP can't sell IQ to customers because it will put their need for HANA out to far. If I've been muddling through with an RDBMS for 10 years, and I accelerate it today with IQ, it will likely be another 10 years before that isn't enough for me and I need in-memory. If I'm looking to accelerate now for the first time, SAP had better sell me HANA or they aren't goign to have another chance for a while.

      Author's profile photo Former Member
      Former Member

      Be truly.

      If SAP will do SAP BW on Sybase IQ - for mostly BW-customers  SAP BW on HANA will not neeeded for a long time. 🙂

      SAP BW on Sybase IQ will be ideal from speed/cost perspective.

      Author's profile photo Former Member
      Former Member

      Hi Clint,

      there is a third NLS solution called OutBoard (from company DataVard) SAP certified since 2010. The solution is unique, because it does not need any additional database. NLS data is stored in separate tables, but in the same SAP BW database (any supported) and highly compressed (up to 95%). This way you have all NLS benefits without the necessity of having an additional system with Sybase IQ.