BW on HANA … Now What? (Part 3 of 3)
Prologue
Previous parts on SDN:
- http://scn.sap.com/community/bw-hana/blog/2016/09/04/bw-on-hana-now-what-part-1-of-3
- http://scn.sap.com/community/bw-hana/blog/2016/09/20/bw-on-hana-now-what-part-2-of-3
Apologies for the delay in posting part 3. Reason: bock beer season!! (http://www.beeradvocate.com/beer/style/32)
Miscellaneous Topics
I like miscellaneous topics. They are so … miscellaneous.
Right. Enough introduction. Let’s get on with it.
Apologies to Support Consultants & HANA to the Rescue
In Part 1, I described the formation of “data holes” in BW cubes and more or less suggested that support consultants are to blame for these holes. My apologies. I am an implementation consultant. Blaming support consultants is like second nature to me. But sometimes, implementation consultants screw up as well.
Below an example in which an anonymous implementation consultant (ok, I confess … it was me) built a wrong solution for data lookup from another DSO. A certain set of fields in the left dataflow is read from the right data flow. The solution assumes that a value change for any field within this set triggers not only the right data flow but also the left data flow.
An existing solution that worked properly – assumption was apparently valid – was enhanced with an additional field for which the assumption was not always valid. Subtle issues like this are usually not found during testing, but are noticed after go-live. This is a typical example of the type of data integrity issues found all through a traditional data warehouse. Fixing the data issue involves carrying out some full loads instead of delta loads.
And this is where HANA technology can come to the rescue. In a traditional database, one would think twice before carrying out a data load involving millions of records in full mode on a daily basis. With the speed of HANA technology this becomes an option seriously to be considered. Below a solution quite successful in practice, i.e. a “self-update in full mode”. All lookup actions are carried out in a transformation from a DSO to itself in full mode.
No delta-trigger is required to pick up new values. Preferably, the self-update transformation is kept “ABAP-free” for optimal performance. Application of the “self-update in full mode” solution for all lookup actions can and does strongly increase data integrity in a traditional data warehouse.
Convergence of BW objects
Below, SAP’s standard graph displaying the convergence of BW objects moving towards BW on HANA versions. Only one object for data persistency remains, i.e. the “Advanced Data Store Object” or ADSO. The Open ODS View and Composite Provider are virtual objects, which do not even require the use of InfoObjects. InfoObjects remain, but will in future probably only be used in case of complex master (hierarchies, authorizations, multi-language descriptions, …).
This convergence is a good thing. At first sight, it may look like functionality is disappearing, but all options existing in the separate objects on the left side appear as settings on the limited number of objects on the right side. These new objects are thereby much more flexible than the old ones. A type of object cannot be changed after creation, but a setting on an object can.
The same tendency is noticed in Native HANA objects. Attribute Views and Analytical Views are replaced by more flexibility in Calculation Views. SAP itself is using only Calculation Views in its Hana Live offering, and most customers are currently doing the same.
Hurray for SAP!
Convergence of Frontend Tools
An even better idea was the convergence of frontend tools. The overall idea is sketched in the graph below (not the latest version, but this version most clearly demonstrates the convergence).
After takeover of Business Objects by SAP, a huge number of reporting tools was scattered all over SAP’s product portfolio confusing not only BI veterans like me, but also most customers, who decided to postpone investments in reporting tools or look outside of the SAP portfolio. It was a necessity that SAP would make a statement like “These are the frontend tools for the future. We will continue to invest in these tools.” Luckily they did, although a bit late. The strongest and most future-proof tools – Analysis Ofice, Lumira and Design Studio – were all new developments by a joint team of SAP and Business Objects developers after the takeover, and not attempts to integrate mature Business Objects products with SAP technology, which for some reason does not work well. For more details, please read the blog on this topic, which is also the source of the graph shown above: http://blogs.sap.com/analytics/2014/06/25/run-simple-convergence-of-the-sap-businessobjects-bi-product-portfolio/.
More convergence: Lumira and Design Studio are joined in next releases. On the downside: the cloud version of Lumira is recently replaced by Cloud for Analytics aka Business Objects Cloud, a new product. Are we diverging again? If not, hurray for SAP!!!
Everything in one HANA box?
In Part 1 I stated at some point that HANA technology makes it possible to put transactions and reporting back together in one box. Somewhere else I also stated that data storage is cheap. The second statement clearly is not true in case of HANA in-memory storage. Putting everything in one box may be feasible for smaller companies, but larger companies will rarely choose to do this. System architectures arising will include multiple something (servers, clients, tenants …), often to facilitate “Multi-Temperature Data Management”. See the SAP-graph below for an explanation.
The client I currently work for has a Native HANA analytics system on a database which hardly contains any data, only virtual models. The data is kept in separate Operational Data Stores, in most cases also HANA databases, connected to the analytics system via Smart Data Access. SAP source systems also run on HANA databases. SAP source system data in the Operational Data Stores is synchronized with minimal delay using System Landscape Transformation. This type of setup will be seen often in practice for larger companies. But if you want … you can put everything in one box.
There is a lot more to say regarding system architectures for Native HANA, BW on HANA and hybrid solutions. One of my colleagues is currently working on a blog on this specific topic.
Skills: from ABAP to SQL
In the (near) future, SAP BI consultants will need new skills: less ABAP and more SQL. In Native HANA, enhancements can be built with “ABAP Managed Procedures”. Don’t let the name fool you: there is no ABAP involved in this type of enhancement, only SQL. HANA operation of BW transformation is already disabled after pressing the Start/End routine button alone. Any coding involving something like “loop at datapackage” is by definition row-based and not compatible with high-speed HANA-operation. Also, frontend tools like Design Studio require more SQL expertise than before.
SQL brains are needed instead of ABAP brains. Advice: In a team of 4 SAP BI consultants, fire 1 and replace him/her by someone with traditional database skills, someone not spoiled by the fancy development shells SAP is providing. There are plenty of them around in the non-SAP world. They are usually quite cheap.
How about Historical Data?
The openSAP training series introducing S/4HANA were quite shocking for BW consultants, as BW was not mentioned once during the entire series. With disbelief we watched the lesson on “Data Footprint Reduction” by Wieland Schreiner, who for some reason wanted to shrink the database of an ERP system in such a way that it would fit in an iPhone. Why? Don’t ask me. I am not Wieland Schreiner. The first steps I could relate to: compression, column store, remove redundant tables, all fine by me. But in a “grande finale” Wieland Schreiner said, and I quote: “If we now take out of that system all that is not needed, so all historical informations …”.
There he lost me. Maybe there are organizations living so much in the moment that they are not interested in data older than, say, a year, but personally I have not encountered such organizations. This is where some sort of data persistency is required in the data integration platform: for historical data that is not kept in the source systems anymore. But also for “snapshots” or frozen states of data that are either not stored in the source systems, or only in obscure logfiles. For this purpose, a traditional data warehouse system like BW is an obvious choice, although one can think of alternatives like physical tables in the HANA database created with HANA Studio and loaded with data using Data Services. In recent presentations on BW/4HANA, SAP is positioning BW for historical data (and historical data only).
Which is an excellent bridge to the final miscellaneous topic …
What is BW/4HANA?
BW/4HANA is the “new-objects-only” version of BW on HANA.
What is BW/4HANA? – Revisited
Well, maybe that was a bit short, although it sort of covers it. What does one do if one wants to get rid of old technology hampering future developments, like cubes and stuff? A two-step approach is required. First, one releases a version with old and new technology side-by-side. Second, one releases a version with new technology only. The side-by-side version is supported till the end of time, but all innovations are focusing on the new technology. In time, all clients will need to migrate to the new-objects-only version.
How easy will it be to migrate from BW on HANA to BW/4HANA? Let’s see what SAP itself says about it in its BW/4HANA FAQ document: “Although, SAP BW/4HANA is not a legal successor of SAP BW (based on SAP NetWeaver), a conversion path will be provided.”. Oh boy, SAP is going into “legal-mode”. Be afraid, be very afraid! Here is another quote: “Classic objects in BW-on-HANA can be semi-automatically migrated into SAP BW/4HANA compliant objects.”. Ouch. Translation: “It will hurt”. However, nothing a good implementation partner won’t be able to help you through.
Finally: Say goodbye to the good old “Business Explorer” frontend. But let’s be honest: wasn’t that about time?
Epilogue
Feedback from one of my colleagues on the draft version of this blog: “In parts 1 and 2 you gave your opinion a lot. In part 3 you are more ruminating SAP slides.” He may be right. Therefore, in a nutshell, my opinion on today’s topics:
- Frontend tools: Stop using the old SAP stuff. Focus on Lumira, Design Studio and Analysis for Excel. Do not underestimate how much report users like Excel and the amount of self-service it provides them. In a situation with predominantly SAP source systems, stop using QlikView, Tableau and other non-SAP tools. These tools may currently have some benefits in functionality, but integration with the backend is worse and in the long run more important.
- System architecture: To me it seems that the customer I currently work for is on the right track with its approach. Separate servers for analytics and for data storage combined with Smart Data Access technology provide a lot of flexibility. Still some child diseases, but it makes sense for the future.
- BW versions: Convergence of BW objects is an improvement, but is by no means revolutionary. You can stick to your older BW version for a couple of years more. BW/4HANA only for customers that are new to BW.
- Positioning of BW: As a BW-veteran it hurts to admit that the golden years of BW – if there ever were any – appear to be over. For the future, I agree with SAP that BW should only be considered for historical data. But even there Data Services with persistency in a HANA database is a strong alternative.
This blog and the two previous ones more or less wrap up what I have to say about the topic in the title. Please contact me if you would like to cheer, disagree, complement or chat à freek.keijzer@ciber.nl.
Hi,
I got some questions or maybe doubts about all these new things that SAP has been spouting recently.
1. SAP has been pushing for Calculation View instead of Attribute and Analytical Views. I think we got a pretty heated old thread where some SAP guys just say something like, “Don’t ask, just do it” when asked why only use Calculation View instead of utilizing Attribute and Analytical View. But as far as I know, HANA got OLAP engine which is only utilized by Analytical View. Now, is it wise to calculate aggregation without the OLAP engine? How far can we abuse the brute force of HANA, depending on simplicity without taking design and performance into thoughts?
2. We still need to use InfoObjects for Navigation Attributes, Time Dependant, and Variable functionality. Navigation attributes is a comfort zone to me that I will ask questions as to why someone would do multiple joins in Composite Provider for master data lookup which will make it confusing to look at. Time Dependency is too complex to code in ABAP or SQL. Field-based modeling is also subject to abuse, especially for the BEx variables. I’ve seen multiple times when people create BEx variables based on the Field, then they need to make multiple variables with the same logic for multiple fields which basically the same thing.
3. When I heard Lumira and Design Studio become a single product, I was kind of happy. But then it’s revealed that actually it’s only sharing Lumira name (Lumira Discovery and Lumira Designer) while it’s still two different product with two different license model, then I’m speechless.
4. BW only for Historical data gives me some laughs. Maybe they can say it again when the new tools are already mature enough that it can handle time-dependent data, lookup on million of rows of non-persistent data in memory. Maybe by the introduction of S4/HANA it’s possible, because it will transform how the transaction data stored to be easier for reporting. But until when the customers need to wait for all modules to complete the transformation? Also let’s not go into the Industry Specific solutions, cause the situation is more severe over there.
But yeah, maybe it’s time to polish our SQL skills because SAP sure starting to push BW to the edge while they actually still depends on them. But I think visual tool is still better to use because of less efforts for maintenance and reading other people’s code hasn’t gotten easier even now. ALV reports based on CDS view getting data from Analytical/Calculation Views should be ideal for operational reporting and lift some weights off our shoulder.
Donnie,
I am honored by your comment, which is more like a blog in response to my blog. I think I can let most of your statements speak for themselves, and I can agree to the bigger part of them.
Some points I would like to clarify to you and other readers:
Regards, Freek
Hi Donnie,
just a few comments that might help understand:
With the introduction of SAP BW/4HANA is should be very clear that we re not "pushing BW to the edge". To the contrary, data warehousing is now back on everyone's agenda again.
Best,
Marc
Product Management SAP HANA DW
Hi Marc,
Thanks for the positive response.
I hope that SAP will make BW great again.
Hi Freek,
first of all congratulations and thanks for writing an excellent 3-part blog. We appreciate your opinions and feedback a lot and it's important to hear how customers and consultants are adopting the innovations we deliver.
A few comments from my side:
"BW/4HANA only for customers that are new to BW." That is certainly not how we position it. SAP BW/4HANA provides great value for all existing BW customers whether they are still on other database platforms or already on SAP HANA. The simplification that customers can achieve by transitioning to SAP BW/4HANA is tremendous. Much of this you describe yourself but how many customers have done the switch to Advanced DSOs, Open ODS Views, and CompositeProviders for their complete BW system? How many have reduced the layers? Who many are struggling with data life-cycle management and dream of automating this "no business value" task? And who has really done integration with Big Data or tapped into logical data warehousing?
When customers are relatively "far away from SAP BW/4HANA" and would benefit a lot from it, justifying the investment. When customers are relatively "close to SAP BW/4HANA", for example if they started on SAP BW powered by SAP HANA and are using SAP BusinessObjects exclusively, then the investment to get to SAP BW/4HANA is relatively low and the transition is pretty easy putting these customers quickly back on the innovation track.
Anyway, lot's of words and we need to prove that this plays out in projects as well. I have been a BW consultant for much of my SAP career and I'm very confident that customers will love SAP BW/4HANA. Hopefully, you will find a project where you can give it a go soon!
Thanks again and I'm looking forward to more blogs from you,
Best,
Marc
Product Management SAP HANA DW
PS: Modern Data Warehousing is certainly not just for historical data. That's an over simplification (and a flaw on marketing slides). For example, one of the reasons data warehousing even exists is to be independent of the source systems since they are not available, reliable, and fast enough especially if you aggregate across all your sources. Until customers have their complete landscape for example on SAP HANA and full master data management implemented across all their systems, I'm sure data warehouses will be loaded continuously with all data - not just the history.
Marc, first of all my apologies for responding only after 2 years. Mitigating circumstances: like many SCN-members I was in “Former-Member-Limbo” for quite some time until I managed to solve it together with SAP’s Community people some weeks ago.
I greatly appreciate your remarks. I also agree to most of them. Going through my own blog 2.5 years later, some statements do indeed seem oversimplified or even outdated. Let me reconsider some of my arguments.
As a consultant working in the field, I tend to go wherever our customers lead me. At the time of first publication of the blog this was away from BW. Customers tend to follow SAP’s promotional activities - like marketing material, conferences and OpenSAP courses – with a couple of years delay. During the promotional campaign for S/4HANA – e.g. the OpenSAP courses from “s4h1” in 2015 to “s4h13” in 2018 – BW was to my knowledge never mentioned as being part of the SAP system landscape the customer needed to go to. Customers drew conclusions like “a data warehouse is not needed anymore” or “BW is outdated, we need to move to Native HANA”. Also, at the time I wrote the blog I was working for two customers simultaneously: one with Native HANA using state-of-the-art tooling, the other one with old version BW “on any db”, BEx and old version BO. Not a fair comparison of course, but this experience made me quite pessimistic about the future of BW. Also, request for BW expertise seems to have dropped sharply in the NL-market.
BW for historical data only? Did I really say that? Re-reading my blog, I see that it was actually SAP’s marketing slide stating this. My statement was historical data + snapshots. I tend to still stick to this opinion. For all other data I agree with SAP’s direction to “leave the data where it is”, e.g. by using the Data Hub tool. Data persistency should me minimized, exceptions only for performance reasons.
“BW/4HANA only for customers that are new to BW.”? My full quote was this: “BW versions: Convergence of BW objects is an improvement, but is by no means revolutionary. You can stick to your older BW version for a couple of years more. BW/4HANA only for customers that are new to BW.” As I am not working for SAP ?, I can afford to advise customers to “wait it out” whenever new SAP products enter the market. Let the companies that are desperate for the new features deal with the child diseases. Meanwhile, the product has matured, we had an OpenSAP training in Nov/Dec 2018, but there is no proper migration from previous BW versions to BW/4HANA; only tools that can assist. For complex BW data warehouses, moving to BW/4HANA is not a straightforward choice. Alternatives like Native HANA and non-SAP are also considered. What I see in the NL-market is that the number of BW/4HANA implementations is still small, and all of them seem to be new implementations.
I recently wrote a new overview article on modern data warehousing (https://www.linkedin.com/pulse/sap-hana-data-warehousing-non-experts-freek-keijzer/) briefly dealing with topics like "Leave the data where it is" and "Mixed data warehousing BW + SQL".
I still go to wherever customers lead me, and at the moment this regretfully still is not BW/4HANA. However, the marketing campaign for S/4HANA Cloud does seem to have caught on in my immediate vicinity. As a result, I am now concentrating on SAC and Cloud Extensibility scenarios (https://blogs.sap.com/2019/05/15/sap-end-to-end-scenario-from-s4hana-cloud-via-hana-modeling-in-scp-to-sac/ , https://blogs.sap.com/2019/05/15/data-visualization-app-on-sap-cloud-platform-versus-graphical-report-in-sap-analytics-cloud/). I would be interested in your opinion on these BI topics.