Skip to Content

The latest hot tech vs. fundamental datawarehousing tradeoffs – Remember BW?

You know, in all the excitement in the SAP datawarehouse and business intelligence areas about HANA, analytical applications, BPC, and other new applications, I have the feeling that we sometimes forget the basics. Basics like the fundamental challenges we struggle with in the datawarehousing and business intelligence profession, or just sitting down and taking some time to think about whether it is the tool or the design that is to blame for all our problems.

Some time ago, I wrote a couple of posts on my personal blog that were a brain dump on the fundamental challenges that we struggle with in datawarehousing and business intelligence. I’m sure I missed some things, but the list amounted to the following:

  • Data volume
  • Data quality
  • Data consistency
  • Semantic integration
  • Historical data
  • Unstructured data
  • Aggregating silo-ed data sources
  • Representing data in a meaningful way
  • Representing reporting structures
  • Performance
  • Security
  • Data loading

If you want definitions, take a look at the posts for a bit more detail.

What is interesting to note here is that each of these new and exciting products or innovations that we occasionally see heralded as The Next Big Thing are really only addressing one or two of these challenges, often at the expense of more difficulty in addressing others of the challenges!

HANA? Data volume (sort of) and query performance at the expense of tools for semantic integration, challenges around data silos, and the operational complexity of limited deployment options.

BPC? Operational simplification and user empowerment at the expense of ETL tools, data siloed in the BPC application, and semantic integration and master data management challenges.

Business Objects Enterprise 4.0? A virtual semantic integration layer (formerly Universes) addressing semantic integration and siloing issues at the expense of performance.

The same goes for all non-SAP wunderkind software products I’ve had exposure to. Hadoop, Qlikview, and Tableau, to name but a few.

Don’t get me wrong – the versions of these products that I’ve worked with are really powerful. The improvements and sacrifices that these products make are perfect for certain use cases, but we shouldn’t kid ourselves into thinking that they are complete general purpose tools. Sometimes they will be the wrong tool for the job. One of the important functions of a technical governance group (or solution architecture group, or whatever you want to call it) is to work to match workloads to tools in a clear-headed and analytical way.

BW is pretty unique in that it is a truly general purpose datawarehousing tool. This means that it is probably not the best tool on the market for solving any of the challenges above, but it may well be the best tool on the market for solving all of them. I’m not saying it’s simple or that it is a joy to work with, but it is good at what it does, it is flexible, it provides very good performance as long as you are nice to it, and it is very open to non-SAP data and to using external tools when you want that best-of-breed experience or a specialized tool for a specialized workload.

I’m at least as guilty of this as anyone else, but sometimes in our excitement about new and innovative products, we forget that the hardest and most important decisions are the ones in which we acknowledge tradeoffs and decide between two or more not-ideal options.

The BW product does an admirable job of helping us work through and design around the tradeoffs inherent in the fundamental datawarehousing and BI challenges listed above. In my view, this is an important niche that I don’t see addressed by other tools. Perhaps it is a niche that is destined for extinction, but I don’t think so. These tradeoffs don’t go away just because of a new tool or a new technical innovation. Sometimes an innovation will change the relationship or the tradeoff a little bit, but the tradeoff is still going to be hiding there somewhere.

If we take one thing from this discussion, I hope it is the realization that most of the challenges listed above are more human in nature than they are technical. They tend to derive from the difficulty in making tradeoff decisions, standardizing interfaces and architectures, identifying and focusing on the problem space, and understanding how people may actually use these systems to greatest effect. Because of this, these challenges are often at least as susceptible to design solutions as they are to technical solutions. There is a tendency in the industry to focus on technical answers to these challenges over design answers, perhaps because technical solutions are often more impressive, in some sense more physical, and in many senses more sellable. I think this tendency is unfortunate.

15 Comments
You must be Logged on to comment or reply to a post.
  • Hello,

    I thought I am alone on this island :). 200 % agree. Marketing has overtook some IT minds. Some people forget why and what for they have to use data warehouse.

    Regards,
    Dominik

    • Accepted! +1

      Business Tranasaction Events aka BTEs had a fundamental transactional / relative issues back then, viz Normalization etc;

      We can’t just forget this fact, the reason why was DW introduced.

      My Question **** Is an DW still Required for Business Warehousing/Intelligence? ****

      BI tools are increasing steadily in the market
      Developers must focus to understand proper usage of BI for DW;

      It was indeed a crux a few years back. But guess it’s not now…

      DW is a crucial factor for many BIW projects, but its still no longer the foundation for all BI projects to be built.

      Classify your business problem… Identify if its one of the 3 types….

      1. Operational BI
      2. Classic/Tradiational BI
      3. Plan Vs Actual Data / Actual Vs History Data

      Good post…

      • Hi,

        “Classify your business problem” that is the clue. But i don’t see that software providers put this point as first one. Usually it is announced as product for everything in any conditions. Maybe IT guy will be aware of points mentioned in this blog and posts, but non-IT customers will say: Great, we  don’t need this complicated DW models, we just need new HANA solution. That leads to discouraging to BI/DW solutions at all for end customers.

        I will appreciate that software providers start kind of education program for non-it clients, explaining what is really important in BI/DW. How they should prepare themselves to implementations, adoptions or at the end choosing solutions.

        Sales Dept. are the guys as for now. Shame.

        Regards,
        Dominik

  • Overtaken by Marketing…
    For example, Solution Manager Business Content for Service Desk is *outstanding* – I may write a blog about this later.

    BW is still a workhorse.

    Probably the biggest challenge is lack of attention.

    Great blog.
    Tammy

    • Thought provoking. Like something that becomes old and taken for granted 🙂 I read this on a blog here (not the exact words but something to the effect of) – It is a sad situation when users start dictating what tool they think needs to be used…

      Regards,
      Bhanu

    • Hi Tammy,

      I agree 100%. BW is a great product. Unfortunately it is not being used for what it was designed. The foundation on which BW was built is very strong. In my opinion, RSA1 made BW look like a very simple tool. RSA1 is like airplane’s “auto-pilot” mode. Yes, auto-pilot mode would simplify flying airplanes; however it wouldn’t replace the experience and skills of a pilot. Auto-pilot mode wouldn’t help landing on Hudson when there is a crisis. The customers rely on RSA1 too much without understanding “what/why/how/when/where”.
      Ten years ago, BW needed a different mindset. Unfortunately it was not communicated. As a result, several mistakes were made. Fixing those mistakes today is not easy nor cheap.  So sales/marketing team is busy selling a new set of products. I don’t believe there is much I(we) can do about it.

      Regards,
      Bala

  • You say BI doesn’t get enough attention but the folks in ERP have been beaten over the head with BI for several years now.

    If I had to rank the SAP apps over the past 5 years in terms of SAP AG focus, marketing, speed of development, new functionality, interest level (SCN forums, Expert articles et al), BI would be #1 on my list.

    • That BI is quite different from BW.

      BI in SAP’s message now is mainly BOBJ suite; BW being a footnote if not an afterthought in it.

      To me it is all much ado about nothing.

      • I wasn’t trying to make a technical differentiation.  For this post, BI=BW=BOBJ.  Either way, it’s a mature product just like ERP is so the market at large has adapted to that change.  Companies still use these apps and rely on them, but they’ll move on to wherever SAP leads them next if they want innovation or the next big-dollar differentiator.

        The real pain that I feel is that as SAP moves on, their R&D dollars go with them…  meaning that future enhancements and fixes to existing problems slow dramatically.  There are still plenty of issues in ERP but they aren’t being dealt with at a late-’90s speed.  Ex: I’m sure SAP could come up with a slick GR/IR Reconciliation & Analysis Workbench (maybe it’s RSA1_GRIR!) but there’s no way they’ll spend the EUR to develop something like that.  They might have done it at some point but that window was open sometime back circa 1997…  about the time their attention turned to the new dimension products.

        Welcome to old age BW.

  • at many things; flip side being it likely subtracts from making it great at ‘any’ thing.

    As data volumes increase, traditional RDBMS barriers combined with complexities in design and maintenance in BW (a trait of all SAP products) make people reconsider the value they get out of BW.

    Once HANA is in place we may well find it(HANA) to not bring the Nirvana it is supposed to bring. In practical usage for most customers it may be nothing more that BWA+. A good progress or a big letdown, depending on how much hype you have bought into.

  • but i hope not our lunches.

    i also subscribe to the “BW (BI) is great camp” as i have seen tremendous progress made since i saw it for the first time in 1999. similar to workflow and a number of other traditional SAP technologies, BW is there and it will continue to hum without big promotions and budgets dedicated to “newer”, but i’m not sure if necessarily “better” technologies. my personal favorite is BCS. i know Cartesis and Crystal Reports in SAP are the newer kids, but they need that “semantic layer” further developed before becoming really useable by our non-IT customers.

    we don’t have to blame our marketing brethren too much either. they have a job to do and need to create interest and excitement and it’s not their fault that sometimes (uhm), the offering is not fully developed because of deadlines, competition and the rest of the business world waiting for the next epiphany (oh, yes… there’s was a company by that name, too).

    tweet by @greg_not_so

  • Ok – we probably do, but is it such a big deal anymore?

    With the advances in hardware and database/software design, I think pretty soon we will reach a stage where we don’t need a separate physical DW.

    But till such time, I agree – we should manage the basics of BW well.

  • Hello Ethan,

    I couldn’t agree more. One thing I believe is that Enterprise Data Warehousing won’t go away any time soon. It’s been there fore decades and I’m sure will be a necesity by the time I retire. That doesn’t mean that EDW changes over time. To the contrary, it continues to evolve at ever increasing speed, and I don’t mean query runtimes but EDW theory (i.e. Layered Scalable Architectures), technologies (in-memory, real-time), combining structured and non-structured data (i.e. Enterprise Search + OLAP), etc.

    BW? Even if our BW Solution isn’t being pushed by sales and marketing, it IS the second most successful solution at SAP (behind ERP) with over 15,000 productive systems. The biggest companies in the world run on it. BW is not great? Well check out BW 7.3!

    HANA? This is certainly one of the most exciting things I have worked on. The good news is that we are in the process of porting BW on top of this platform. Imagine a complete EDW running in-memory (and unlike BW Accelerator there are no jobs required to index data).

    BPC (and I throw in BW Integrated Planning, too)? We need both worlds: IT designed planning applications and business driven planning apps. Great news: We are making the impossible happen with the coming EPM/BPC release (if you just need the IT lead apps, you can of course use BW-IP only). Even better news, the next generation planning apps will be able to take advantage of the in-memory technology because they run on BW which runs on HANA.

    BusinessObjects? All I say is 4.0.

    And the best news: Much is already available in ramp-up so the complete package should become available in the near future. I love the vision but will leave it to our customers to judge if the next “BW” will be great altogether.

    Stick with BW. Talk about it. Help customers become a Best Run EDW company.

    Best,
    Marc

  • … for the moment I would like here to ask one question: So what? Like many others, I have been ranting here for a while including “What should be the Next Big Thing for SAP in BW/BI?” (What should be the Next Big Thing for SAP in BW/BI?) or “BW path to the Data Warehouse plus the role of New Technologies” (BW path to the Data Warehouse plus the role of New Technologies). Do Ethan’s, Bala’s, Vijay’s posts reach the target audience? I doubt, although they are causing positive resonance in our technical community, which SDN is.
    “Companies do not live from what they produce, but from what they sell”. As techies we may hate this fact, but it is true. If you know some techie who grew to CEO in enerprise software to prove the opposite – I am ready to pay for her or his book to read it.
    Public traded companies exist for one reason – to return money to their shareholders. In order to do this these companies have to constantly increase revenue on existing products or generate the new revenue streams. And in order to generate new stream vendors sometimes invent the problems (“create the markets” to put nicely in marketing jargon) that customers have not even thought about.
    Is battle completely lost for existing products as BW? It does not have to be! Customers have yearly support fee for implemented products, and have right to expect the value in return in form of better and quality new releases, including BW. Customers’ groups, like ASUG (www.asug.com) are in a primary position to keep SAP rolling on constant and even breakthrough improvements in existing products. And this is what they are doing through better or worse Influence Councils and alike.
    Coming back to my initial question “So what?”. Does it mean there is nothing the SDN community can do other than complain? No! There is something called SAP Idea Place (https://ideas.sap.com/index.jspa), where community members can post and vote recommendations for SAP products. BW is not there today, but no one stops anyone from requesting BW to be added to list of products on Idea Place. If the relevant Product Team agrees to accept ideas from the community, BW will pop up there and then we can provide ideas on how to improve the product directly to the Product Team.
    Why haven’t I done this if it’s so easy? I know – shame on me 🙁 Hopefuly someone will have time and energy to do this right.