As we debate on who's to blame for financial crisis, I am puzzled no one is faulting BI for not providing an 'competitive advantage'. Why has BI not been a resounding success for many customers? The problem is not that BI is not a stable product, but because SAP BI is not an application product, but a toolkit. It depends on how the tool is used in building this Data warehouse from ground up. A very different approach to implementing SAP R/3 that's like a mobile home that needs to be connected to the business environment and lights can be turned on, so to speak. And companies spent millions of dollars doing that, is it realistic to expect a 'plug and play' BI?
Most companies use SAP R/3 system to integrate their business operations. This has lead to one central accumulation of data about their customers, suppliers and processes. Businesses needed more than ABAP Reports to analyze their data. Several BI vendors tried to fill this void by providing an ETL and Reporting tools. But, most of these software needed direct access to R/3 tables, which meant SAP Basis administrator would lose control of data access from within the SAP Layer. Instead of fighting this losing battle, some software vendors like ACTA (acquired by Business Objects) integrated their ETL process with SAP R/3 by creating ad-hoc ABAP programs to pull data from R/3using file transfers, i.e., they generated ABAP Code from Transformations that were created as Y or Z Programs in the R/3 systems.
Only a few bleeding edge customers went with independent BI Vendors and most were waiting for SAP to solve this problem. And SAP did with its BW 1.2 back in 1998. Since then, SAP BW has not changed much. Yes, we now have Multi-providers, data stores, transformations, web-based client, mobile delivery etc, but InfoCubes, Extraction process and OLAP has remained stable. This is good news for customers as the pieces they developed ten years back are still valid today, with incremental upgrade changes. Even with SAP getting the BI technology right for the past ten years, I don’t think many SAP BI customers would say the same about their overall BI experience. Here is a modest attempt to identify why this happened and what can be done to fix it.
a. Wrong Assumption - SAP BW is a Packaged Toolbox, not packaged Application.
Unlike SAP R/3, SAP BW is not an out of box product. Transaction processing can be standardized, but Reporting requirements vary significantly across companies. SAP had rich pre-delivered content for extractors and master data objects, but left much of cube / report building to individual businesses. This flexibility made sense and is logical as each customers’ reporting needs are different. But without a formal structure that SAP customers were used to, bad designs started propping up.
b. Skill gap and Cost of Bad Design.
SAP BW is a Data Warehouse product, but most BW technical staff and consultants have R/3 or ABAP background and have not worked with large databases. Consultants created cubes and built elaborate levels of data stores for the first time at several large SAP BW installations. A badly written ABAP program can be analyzed, re-written or plainly left unused. But, a badly designed extractor, data store design or transformation is very unforgiving. Bad design shows its rear–end either in nightly wakeup calls or long query times.
c. Data Explodes – Careful what you wish for.
A large portion of BW cubes were built to handle production data volumes at the time of go-live. Data has grown since then and has provided the rich set of data to report and analyze. Unlike hardware, design does not scale up easily. In the intent of promising the world, consultants and BI Managers built cubes with 70 base fields, granular level detail that can provide individual document numbers for R/3 processes that accumulates Millions of records per month. This gave customers the flexibility and rich dataset at go-live, but soon started sinking in its own weight. Good design that scales is deliberate and needs to be put in place and traded off with Flexibility.
d. Cost of Dormant Data
Customers access recent data in greater numbers that decreases in frequency as the data ages. As data grows, these sparingly used historical data that’s still stored at detailed level slows down analysis of the relevant recent data. Eventually, businesses resort to purging historical data from cubes and holding them at the summary level. A lot of frustration could have been avoided if BI teams constantly looked for dormant data and collateral damage they cause.
e. More Hardware to solve Design issues
SAP BIA solved issues relating to query performance. In the name of Stability, IT Managers preferred to buy hardware than to fix design problems. But, BIA cannot logically partition the data and index only recent years. It will index all data in the cube. Therefore, if a cube has ten years of detailed data, expensive BIA space is taken and the queries can still result in blade errors due to too much data pulled into memory.
All of these issues are not unique to SAP BI, but are much more expensive in an SAP environment. For the benefits of integration, SAP BI demands a thoughtful process. For example – Info Objects provide a building block for sharing master attributes across various info providers, but is not very flexible if a mistake is made with regards to compounding, field length, type etc. Most of these restrictions were put in place by SAP to ensure data integrity, a very good reason, but is difficult to change without tearing down the walls.
What can companies do to increase Odds of BI Success?
Data Warehousing is an Iterative process and does not lend itself to one grand big-bang SAP implementation. Therefore, BI Project Managers need to invest time in understanding business problems before just turning on pre-delivered BI cube content and lettingbusiness explore data. Based on the pain points, BI team can build appropriate lean cubes /reports that are built on pre-delivered content and extractors. Although this may sound common sense, most customers have already implemented BI. BI Team needs to build strategy documents for the following three main challenges before building the data warehouse.
a. Data Integrity / Accuracy – How trustworthy is the data
b. Data Timeliness – How often.
c. Data Granularity – What level of detail in reports
d. What does and what doesn’t to report in BI.
In a subsequent blog, I will explore each of these options in more detail.
