Skip to Content

 

 

 

As we debate on who’s to blame for financial crisis, I am puzzled no one is faulting BI for not providing an ‘competitive advantage’.  Why has BI not been a resounding success for many customers?  The problem is not that BI is not a stable product, but because SAP BI is not an application product, but a toolkit.  It depends on how the tool is used in building this Data warehouse from ground up.  A very different approach to implementing SAP R/3 that’s like a mobile home that needs to be connected to the business environment and lights can be turned on, so to speak.  And companies spent millions of dollars doing that, is it realistic to expect a ‘plug and play’ BI?

Most companies use SAP R/3 system to integrate their business operations. This has lead to one central accumulation of data about their customers, suppliers and processes.  Businesses needed more than ABAP Reports to analyze their data.  Several BI vendors tried to fill this void by providing an ETL and Reporting tools. But, most of these software needed direct access to R/3 tables, which meant SAP Basis administrator would lose control of data access from within the SAP Layer.  Instead of fighting this losing battle, some software vendors like ACTA (acquired by Business Objects) integrated their ETL process with SAP R/3 by creating ad-hoc ABAP programs to pull data from R/3using file transfers, i.e., they generated ABAP Code from Transformations that were created as Y or Z Programs in the R/3 systems. 

Only a few bleeding edge customers went with independent BI Vendors and most were waiting for SAP to solve this problem.  And SAP did with its BW 1.2 back in 1998.  Since then, SAP BW has not changed much.  Yes, we now have Multi-providers, data stores, transformations, web-based client, mobile delivery etc, but InfoCubes, Extraction process and OLAP has remained stable.  This is good news for customers as the pieces they developed ten years back are still valid today, with incremental upgrade changes.  Even with SAP getting the BI technology right for the past ten years, I don’t think many SAP BI customers would say the same about their overall BI experience.  Here is a modest attempt to identify why this happened and what can be done to fix it.

a.     Wrong Assumption – SAP BW is a Packaged Toolbox, not packaged Application.

Unlike SAP R/3, SAP BW is not an out of box product. Transaction processing can be standardized, but Reporting requirements vary significantly across companies. SAP had rich pre-delivered content for extractors and master data objects, but left much of cube / report building to individual businesses.  This flexibility made sense and is logical as each customers’ reporting needs are different.  But without a formal structure that SAP customers were used to, bad designs started propping up.

b.      Skill gap and Cost of Bad Design.

SAP BW is a Data Warehouse product, but most BW technical staff and consultants have R/3 or ABAP background and have not worked with large databases.  Consultants created cubes and built elaborate levels of data stores for the first time at several large SAP BW installations.  A badly written ABAP program can be analyzed, re-written or plainly left unused.  But, a badly designed extractor, data store design or transformation is very unforgiving.  Bad design shows its rear–end either in nightly wakeup calls or long query times.

c.       Data Explodes – Careful what you wish for.

A large portion of BW cubes were built to handle production data volumes at the time of go-live.  Data has grown since then and has provided the rich set of data to report and analyze.  Unlike hardware, design does not scale up easily.  In the intent of promising the world, consultants and BI Managers built cubes with 70 base fields, granular level detail that can provide individual document numbers for R/3 processes that accumulates Millions of records per month.  This gave customers the flexibility and rich dataset at go-live, but soon started sinking in its own weight. Good design that scales is deliberate and needs to be put in place and traded off with Flexibility.

 d.      Cost of Dormant Data

Customers access recent data in greater numbers that decreases in frequency as the data ages.  As data grows, these sparingly used historical data that’s still stored at detailed level slows down analysis of the relevant recent data.  Eventually, businesses resort to purging historical data from cubes and holding them at the summary level.  A lot of frustration could have been avoided if BI teams constantly looked for dormant data and collateral damage they cause.

 e.      More Hardware to solve Design issues

SAP BIA solved issues relating to query performance.  In the name of Stability, IT Managers preferred to buy hardware than to fix design problems.  But, BIA cannot logically partition the data and index only recent years. It will index all data in the cube.  Therefore, if a cube has ten years of detailed data, expensive BIA space is taken and the queries can still result in blade errors due to too much data pulled into memory.

All of these issues are not unique to SAP BI, but are much more expensive in an SAP environment.  For the benefits of integration, SAP BI demands a thoughtful process.   For example – Info Objects provide a building block for sharing master attributes across various info providers, but is not very flexible if a mistake is made with regards to compounding, field length, type etc.  Most of these restrictions were put in place by SAP to ensure data integrity, a very good reason, but is difficult to change without tearing down the walls.

 

What can companies do to increase Odds of BI Success?

Data Warehousing is an Iterative process and does not lend itself to one grand big-bang SAP implementation.  Therefore, BI Project Managers need to invest time in understanding business problems before just turning on pre-delivered BI cube content and lettingbusiness explore data. Based on the pain points, BI team can build appropriate lean cubes /reports that are built on pre-delivered content and extractors.  Although this may sound common sense, most customers have already implemented BI.  BI Team needs to build strategy documents for the following three main challenges before building the data warehouse.

a.       Data Integrity / Accuracy – How trustworthy is the data

b.      Data Timeliness – How often.

c.       Data Granularity – What level of detail in reports

d.      What does and what doesn’t to report in BI.

In a subsequent blog, I will explore each of these options in more detail.

 

 

 

To report this post you need to login first.

7 Comments

You must be Logged on to comment or reply to a post.

  1. Jose Garcia
    Its good to see that we’re not the only ones with these growing pains.  I’m in absolute agreement that a standard structure/process from SAP would greatly reduce these headaches; and would provide customers a leg up on Business Warehousing.

    Look forward to subsequent blogs.

    Jose

    (0) 
  2. anil kamsala
    Ramesh,

    Its really good to see the blog on the on going pains of BI and their detailed analysis.

    Waiting for the next series.
    – anil kamsala

    (0) 
  3. jan ter avest
    if I’m not mistaken, it is SAP policy to discourage reporting in ECC and have BW take on this task. BI end users who always got lists at a very granular level, mostly extracted through 3rd partly tools like impromptu, expect the same from BW. We frequently get requests for plain lists, rather than reports with well-thought -thru key figures and navigation  So basically if your organization doesn’t have a clear view on BW and how to use it, you’re on your own.
    (0) 
  4. Gregory Misiorek
    …no one is faulting BI for not providing an ‘competitive advantage’…

    Because it’s suspended between r/3 and a spreadsheet, both having a strong and vocal user base.

    …Why has BI not been a resounding success for many customers?…

    Competition, SAP BI (OLAP?) is only one of many BI tools. The most obvious competition are (now Oracle) Essbase and the MS Excel pivot tables. I also think that the end users never really understood the difference between OLAP and OLTP and have been asking to have both in one.

    …is it realistic to expect a ‘plug and play’ BI?…

    No, it’s not realistic, especially for exponentially increasing data volumes.

    …This has lead to one central accumulation of data about their customers, suppliers and processes…

    In many environments, SAP is not the only ERP system being used, but is shared with mainframe, Oracle, small vendor, and the host of internally written applications.

    …they generated ABAP Code from Transformations that were created as Y or Z Programs in the R/3 systems…

    Just another bolt-on application, one of many that live off r/3.

    …most were waiting for SAP to solve this problem…

    I think the opposite is true, customers never waited and implemented alternative OLAPs, their financial analysts ignored it altogether and have been maintaining reporting data manually in their spreadsheets.

    …Since then, SAP BW has not changed much…

    It changed a lot, we got better performance, the user interface has improved and most of all it’s all connected through the web.

    …Unlike SAP R/3, SAP BW is not an out of box product…

    Yes, it is standardized very much by having business content.

    …Transaction processing can be standardized, but Reporting requirements vary significantly across companies…

    Agreed. The end users will ask for the world if IT lets them.

    …But without a formal structure that SAP customers were used to, bad designs started propping up…

    Agreed, flexibility comes at a price. It’s hard to correct mistakes made early in the process. Many customers were used to their legacy reports and they just wanted them the same way without adjusting their work around it. Accountants don’t appreciate reengineering as that’s only a burden on the work they have been doing for years and some of them wouldn’t survive due to new skills being required.

    …a badly designed extractor, data store design or transformation is very unforgiving. Bad design shows its rear–end either in nightly wakeup calls or long query times…

    Agreed, after millions spent on training on r/3 there wasn’t any fuel left for BW. My training was only for 2 days out 20 total and the instructor didn’t have a clue.

    c. Data Explodes – Careful what you wish for.

    …Good design that scales is deliberate and needs to be put in place and traded off with Flexibility…

    Do customers really need all that aggregation and than the drill-down or do they simply ask for what is free to them (but not their companies)?

    …A lot of frustration could have been avoided if BI teams constantly looked for dormant data and collateral damage they cause…

    This is true for any database. Backup, storage, and archiving are needed for all systems, not just BI.

    e. More Hardware to solve Design issues

    …if a cube has ten years of detailed data, expensive BIA space is taken and the queries can still result in blade errors due to too much data pulled into memory…

    Agreed, the lowest level of data is the hardest to change, but again it’s not BI specific issue.

    What can companies do to increase Odds of BI Success?

    …BI Project Managers need to invest time in understanding business…

    I have always said that accountants make bad programmers and now I can also say they make bad database administrators, so keep them out of IT or have them trained in data formats, entity relationship model, star schemas, database administration, sparcity, indexing, you name it.

    Consult the industry standards like formerly OLAP report. Raad Erik Thomsen’s book about OLAP.

    learning how to use foramtting and those pesky html tags is not a bad idea, either.

    (0) 
  5. Ramesh Sampath Post author
    Thank you everyone for your feedback.  This was my first blog and didn’t know what to expect.  This is great.  Thanks Greg for your detailed Response.  We are in full agreement.

    Appreciate your responses.  The world does feel small and connected.

    Thanks
    Ramesh Sampath

    (0) 

Leave a Reply