Breaking free from BI (or BW)
Sooo, have you thought about buying HANA? Ha-ha, just kidding! No, folks, this is not another sales pitch for HANA or some expensive “solution”, but a simple customer experience story about how we at Undisclosed Company were able to break free from the BI (*) system with no external cost while keeping our users report-happy. Mind you, this adventure is certainly not for every SAP customer (more on that below), so YMMV. The blame, err… credit for this blog goes to Julien Delvat who carelessly suggested that there might be some interest in the SAP community for this kind of information.
It might be time to part with your BI system if…
… every time you innocently suggest to the users “have you checked if this information is available in BI?” their eyes roll, faces turn red and/or they mumble incoherently what sounds like an ancient curse.
… you suspect very few users actually use the BI system.
… your have a huge pile of tickets demanding an explanation why report so-and-so in BI doesn’t match report so-and-so in SAP.
… your whole BI team quit.
… the bill for BI maintenance from your hosting provider is due and you can think of at least 10 better things to do with that money.
What went into our strategic decision
- Tangible cost to run BI. Considering number of active users and value, we were not getting our money’s worth.
- Relatively small database size. The Undisclosed Company is by no means a small mom-and-pop shop but due to the nature of our business we are fortunate to have not as many records as, say, a big retail company might have.
- Reports already available in SAP. For example, it just happened that few months before the “BI talk” even started our financial team already made a plea for just one revenue report in SAP that they could actually rely on. Fortunately, we were able to give them all the information in (gasp!) one SQ01 query.
- No emotional attachment to BI. As far as change management goes, we had the work cut out for us (see the eye rolling and curse-mumbling observation above). The users already hated BI and SAP team didn’t want anything to do with it either.
We’re doing it!
Personally I have suggested that we simply shut down BI and see who screams, but for some reason this wasn’t taken by the management with as much excitement as I was expecting.
Instead we took a list of the users who logged into BI in the past few months (turned out to be a rather small group) and our heroic Service Delivery manager approached all of them to find out what reports they’re actually using in BI and how did they feel about it. Very soon we had an interesting matrix of the users and reporting requirements, which our SAP team began to analyze. Surprisingly, out of the vast BI universe the users actually cared about less than 15 reports.
For every item we identified a potential replacement option: an existing report in SAP (either custom or standard), a new query (little time to develop), a new custom ABAP report (more time to develop). With this we were able to come up with a projected date for when we could have those replacements ready in SAP and therefore could begin the BI shutdown. It was an important step because having a specific cut-off date puts fear into the users’ minds. Otherwise if you come asking them for the input or testing and no specific due date we all know it’s going to drag forever (there seems to be always “end of month” somewhere!).
Drum roll, please
So what did 15 BI reports come down to in ECC? We actually ended up with just 2 custom ABAP reports and 2 new queries, everything else was covered by standard SAP reports and just a couple of existing custom reports. Interestingly, we discovered that there were sometimes 3 different versions of essentially the same report delivered as 3 different reports in BI. In those cases we combined all the data into one report/query and trained the users on how to use the ALV layouts.
The affected functional areas were Sales, Finances and QM (some manufacturing reports were and are provided by our external MES system). There was very little moaning and groaning from the user side – it was definitely the easiest migration project I’ve ever worked on. Breaking free from BI felt like a breeze of fresh air.
Are you thinking what I’m thinking?
If you’ve already had doubts in the BI value for your organization or this blog just got you thinking “hmm”, here are some of our “lessons learned” and just random related observations and suggestions. (Note – HANA would likely make many of these points obsolete but we have yet to get there.)
- If you feel you don’t get adequate value from your BI system it is likely because you didn’t really need it in the first place.
- If you are already experiencing performance issues in the “core” SAP system, you might want to hold on to your BI for a bit longer (unless it’s BI extraction that is causing the issues). Adding more reporting workload to the already strained system is not a good idea.
- Find the right words. If we just told our business folks that we’re shutting down BI the hell would break lose (“evil IT is taking away our reports!!!”). But when you start conversation with “how would you like to get a better report directly from SAP?” it’s a different story. And don’t forget to mention that they will still be able to download reports into Excel. Everybody loves Excel!
- Always think ahead about your reporting needs. I can’t stress this point enough. For example, in our scenario one of the reporting key figures is originally located in the sales order variant configuration. If you’ve never dealt with VC, let me tell you – good luck pulling this data into a custom report. (The same problem with the texts, by the way – makes me shiver when some “expert” suggests on SCN to store any important data there). So our key VC value was simply copied to a custom sales order (VBAP table) field in a user exit. Just a few lines of code, but now we can easily do any kinds of sales reports with it. It only took a couple of hours of effort but if you don’t do it in the beginning, down the line you’ll end up with tons of data that you cannot report on easily.
- Know your SAP tables. Many times custom SAP reports get a bad rep because they are simply not using the best data sources. E.g. going after the accounting documents in BSEG is unnecessary when you can use index tables like BSAD/BSID and in SD you can cut down on the amount of data significantly if you use status tables (VBUK/VBUP) and index tables like VAKMA/VAKPA. I’m sure there are many examples like that in every module – search for them on SCN and ask around!
- Queries (SQ01) are awesome! (And we have heaps of material on SCN for them – see below.) If you have not been using them much, I’d strongly encourage you to check out this functionality. You can do authorization checks in them and even some custom code. And building the query itself takes just a few button clicks with no nail-biting decisions whether to use procedural or OO development. SAP does everything for you – finally! 🙂
- Logistics Info System (LIS) – not so much. Even though I wouldn’t completely discount it as an option for reporting (yet), it is usually plagued by the same problems as BI – inconsistent updates and “why this report different from that report” wild goose chase.
- When it comes to reports – “think medium”. You’ve probably noticed that in our case number of reports reduced greatly in SAP compared to BI. Why was that? Turned out that we had many reports that essentially used the same data but were presenting it slightly differently. There is no need to break up the reports when display customization can be easily achieved by using the ALV layouts, for example. And on the other side of the spectrum are the “jumbo reports” that include data from 4 different modules because someone requested the report 10 years ago and thought it was good, so he/she told other users about it and other users liked it too BUT they needed to add “just these two fields” to make it perfect, then more and more users joined this “circle” and everyone kept asking for “just these two fields” but nothing was getting removed because the first guy left the company years ago and now no one even remembers what the original requirement was. So you end up with the ugly Leviathan of a report that has to be sent to the farm upstate eventually. Try to avoid those.
- Be creative. If a “jumbo report” cannot be avoided (ugh!), you might want to consider creating a “micro data warehouse” in a custom table that can be populated in a background job daily (or more frequently, if needed). Such reports usually do not require up to the second information and we can get the best of both worlds – minimize impact on performance by pre-processing the data and allow the users to run the reports on their own. Another tip – if a report is used by different groups of users and includes certain fields that are more time-consuming than others, you can add an option to selection screen to exclude those fields when they’re not needed. Also simple training the users on ALV functionality can be very helpful. For example, we noticed that some users ran a report for one customer, then went back to the selection screen and ran it for another. But running the report for two customers and then using ALV filter would actually be more efficient.
- Don’t let the big picture intimidate you. The big goal of taking down a large (as we thought!) productive system seemed pretty scary in the beginning, but, as you could see, we broke it down into pieces and just got it down one by one. And this was done by the team of just 5 people in 3.5 months while supporting two productive SAP systems and handling other small projects as well. If we did it, so can you!
Next Generation ABAP Runtime Analysis (SAT) – How to analyze performance – great blog series on the SAT tool. My weapon of choice is still good old ST05, but in some cases it might be not enough.
There are many SCN posts regarding ABAP performance tuning, although quality of the posts varies greatly. This wiki page could be a good start, use Google to find more. Look for the newer/updated posts from the reputable SCN members. (Hint – I follow them! 🙂 )
Some tips on ABAP query – this is Query 101 with step by step guide, great for the beginners.
10 Useful Tips for Infoset Queries – good collection of miscellaneous tips and tricks
Query Report Tips Part 2 – Mandatory Selection Field And Authorization Check – great tip on adding a simple authority check to a query.
(*) or BW? – I’m utterly confused at this point but had the picture already drawn so let’s just stick with BI
Congrats for posting such an innovative idea.
But still the questioning of fetching information from different source systems and reporting them remains unanswered, in your case you had a single source system but what about integrating data from different sources?
Thanks for the comment, Prathish (or Antony?). You're correct and there could be legitimate cases when a BW system would have an advantage. Thanks for pointing this out. As I mentioned, this is not for every customer. But I hope it will help the customers that may have just been bullied by the consultants into implementing something that is causing more trouble than it's worth.
I work mainly with IS Retail, and in retail it's impossible to use SAP ECC for sales/stock reporting without HANA. I hate BW with a passion, but it's unavoidable.
Gasp! It should be ok in your case considering the user base and the value addition by the BW installation.
But the BW problems can be solved by sound architecture, good datasource reconciliation techniques ensuring BW data is one with ERP data and good BW consultants. In my humble opinion, the values that a good BW installation can create far outweighs the installation and operating cost. for e.g. quick reporting, OLTP/OLAP split with reduced stress on OLTP system, predictive analysis, data mining, easy report realization, uniform report template, huge volume transactional reporting, planning, complex data transformation leading to decision making reports etc.
Thanks for sharing your decomissioning story, it made me think (scared). 😛
Yes, but then what actually happens is that the architecture runs into compromisses, the extraction jobs fail, and users complain. Even if the BW consultants are good, someone less qualified ends up maintaining the system on a daily basis.
Not to mention that Bex Explorer is one horrible piece of badly designed software, and many installations rely on it. No one explained to the developer rule nº 1 of UI developement, don't lock the UI thread since it leads to "Application not responding" and users quiting mid extraction.
Lets start this way if you have the best car in the world,wouldn't you be fueling it with best fuel available in the market ? In first place we need to ensure that we have well qualified team who monitors the daily loads and thereby making sure that load happens on time.
I understand that once in a blue moon load fails,extraction gets delayed and mail box starts flooding with the mails of all stake holders this is something inevitable.
In most of the cases i feel its not the issue with the system but probably with the person who handles it.
If I had infinity money yes, but it's a fact the maintenance budgets are controlled by bean counters, and it's not very likely that you'll have top of the line consultants maintaining your BW. That's the reality I have faced for many years.
Sure, but people are still part of the process, and if the process depends too much on people then it will fail. ERP reports never fail to update, if I go to MB52 the stock will be correct even if the IT MM person falls asleep, goes into labor, etc. With BW, "something" always happens.....I don't even know what, BW people just say "The job failed" or some other thing, all I know is that it happens too much to be acceptable.
Good BW consultants can stabilize the system in quick time, you will not face load failures or report-data inconsistency every now and then after stabilization. I have worked in development and operations of BEx catering to huge user base with complex queries (they were pretty happy with what we offered). In my H opinion, I think that the product BEx is pretty cool and can do loads. The BEx could fail to respond in huge data extraction and heavy calculations, we have to optimize it with appliances or optimization techniques. I feel that good BW consultants can turn around any BW system in quick time.
Although I'm not an expert I feel exactly the opposite. Separating reports from transactions just doesn't seem like a good idea in general. I believe even Hasso Plattner himself points this out in his introduction to the In-memory DB course at openHPI (it's available online).
Keeping with the car analogy, I'm sure many customers would just prefer a Toyota Camry that gets them from point A to point B reliably and doesn't require a personal mechanic on site. 🙂
Interesting perspective, Jelena! We don't have BW/BI at my organization, and there is much talk about a need for it, mainly for offloading reporting and making large dataset reports (like year-end, etc) more manageable. Everyone complains about the performance of our reports (which are mostly custom, because why use something out of the box when you can build your own version instead, or better yet half a dozen versions, each presenting the same data slightly differently?), and the general thinking seems to be "If we had BW, these reports would be so much faster!" I admit I've been a subscriber to this view, too, but your article has got me rethinking my position. I do strongly suspect that we're making many of the classic mistakes in our custom reports (using poorly indexed custom tables, not using index tables instead of BSEG, etc, using FOR ALL ENTRIES instead of effective joins, and other well-known performance no-nos), but more than this we have such a confusing jumble of reports, all very similar to each other, that replacing them feels like unwinding the proverbial Gordian knot.
So, a part of me wants BW for the standardized reporting platform, but a part of me worries about the extra basis management overhead involved (we aren't going to gain any extra staff to manage it, so I know who it will fall to), and whether the result will really justify the cost. If it could help us move our standard reporting into the Portal, that would justify it right there. Also, SRM these days comes with very little built-in reporting, but lots of hooks to BW, and our SRM/MM lead has been wanting access to that for a while.
Thank you for this perspective. I'll pass this along to my boss (who is chief among the BW supporter camp).
Matt, thanks for the comment. Obviously BW/BI is still a relatively large revenue source for SAP that they might not want to let go of easily. That's why I feel it's especially important for the customers to have an honest information exchange about the reality. Really glad to see all the different comments here.
Hi, very good perspective. Congratulations.
Instead shut down BI and see who screams (or ask users) You can use BI statistics to check who/how often/what reports use.
Interesting point of view... I know of some IT managers who will love this! Personally, I'm still in favor of BW, however I don't think every report should be made in BW/BI (as SAP used to promote). I have setup "monsters" in BW just because it had to be done in BW (where in ABAP it would have taken a fraction of the time to create, plus it would be more performant). The problem is: I seem to be talking to walls in those situations. It's always funny after the facts when they ask why it took so long to set up such an "easy" report? I tell them that in ABAP it would have taken me a day or 2, but they requested me to do it in BW 😉
It would be easy to extract a report in ECC with real time data from few tables for a period of one/two months.
But the concept of BW is altogether different. The reports analysis would be multidimensional with huge data sets. User cannot forecast his business without Business Intelligence tool.
For storing small sets of data with simple reports there are many OLTP systems no need of SAP ECC as well.
I strongly believe that there is no replacement for SAP BW Intelligence !!!
Passion for the standard!
That is a motto that have very deep meaning and should be the heart of every SAP Implementation.
If ECC has reports and tools for creating them exploit them. If you are sure you've reach those tools limits (no the consultant limits 🙂 ; that often happens) then you can start thinking of other tools, such as BI.
The idea sounds interesting, but I think this case was very specific,very unlikely to happen again, and really small datasets. (Example: I did not hear anything about reporting archived data).
I agree with passion for the standard, but BI is a development, usually a huge one since the standard cubes and extractors don't fit the requirements.
Folks, I appreciate all your inputs and it's important to the readers / SAP customers to hear different opinions as well. Thank you!
But surely you can't truly believe in your heart that separation of OLAP and OLTP is in general a great idea? Even Hasso Plattner himself thought it wasn't good (you might want to watch his introduction to the In-memory DB course at openHPI if you don't believe me). C'mon now...
The only legitimate case when you do need some kind of data warehouse is when data from multiple systems is involved (as was pointed out in the first comment). That seems quite logical. The companies with enormous datasets would probably do very well with HANA or another in-memory database. And the rest should think twice about implementing BI (and read my blog 🙂 ).
By the way, our custom reports run perfectly fine for the whole year and possibly longer periods. But in reality our customers almost never look for data that is over a year old. We were also concerned that the users may need to run the reports for multiple years, but turned out it was not needed. Nowadays in many industries the products become obsolete, the customers taste changes and the whole companies get bought and sold faster than, say, 20 years ago. So would the data from 3 years back even be relevant in such case?
"By the way, our custom reports run perfectly fine for the whole year and possibly longer periods."
Do you have jobs that calculate aggregates, or is it all online?
All of our reports are online, there are no separate background jobs to support them.
But in one of my old jobs we used the approach I've mentioned - pre-populating a custom table in background. In that case the report included information from many purchasing and material documents (e.g. stock information, last receipt date, etc.) and was too "heavy" to run on demand.
I don't exactly what is your line of business, but there are reports that can't be run online, it's way too much information. I hate BW as much as the new guy, but there are things that need to be preprocessed.
I had reports from Global Trade Management that needed to display the whole flow from PR, to Invoice, and they wanted the information for whole months in a matter of seconds. You can't do this online.