The latest hot tech vs. fundamental datawarehousing tradeoffs – Remember BW?
You know, in all the excitement in the SAP datawarehouse and business intelligence areas about HANA, analytical applications, BPC, and other new applications, I have the feeling that we sometimes forget the basics. Basics like the fundamental challenges we struggle with in the datawarehousing and business intelligence profession, or just sitting down and taking some time to think about whether it is the tool or the design that is to blame for all our problems.
Some time ago, I wrote a couple of posts on my personal blog that were a brain dump on the fundamental challenges that we struggle with in datawarehousing and business intelligence. I’m sure I missed some things, but the list amounted to the following:
- Data volume
- Data quality
- Data consistency
- Semantic integration
- Historical data
- Unstructured data
- Aggregating silo-ed data sources
- Representing data in a meaningful way
- Representing reporting structures
- Data loading
If you want definitions, take a look at the posts for a bit more detail.
What is interesting to note here is that each of these new and exciting products or innovations that we occasionally see heralded as The Next Big Thing are really only addressing one or two of these challenges, often at the expense of more difficulty in addressing others of the challenges!
HANA? Data volume (sort of) and query performance at the expense of tools for semantic integration, challenges around data silos, and the operational complexity of limited deployment options.
BPC? Operational simplification and user empowerment at the expense of ETL tools, data siloed in the BPC application, and semantic integration and master data management challenges.
Business Objects Enterprise 4.0? A virtual semantic integration layer (formerly Universes) addressing semantic integration and siloing issues at the expense of performance.
The same goes for all non-SAP wunderkind software products I’ve had exposure to. Hadoop, Qlikview, and Tableau, to name but a few.
Don’t get me wrong – the versions of these products that I’ve worked with are really powerful. The improvements and sacrifices that these products make are perfect for certain use cases, but we shouldn’t kid ourselves into thinking that they are complete general purpose tools. Sometimes they will be the wrong tool for the job. One of the important functions of a technical governance group (or solution architecture group, or whatever you want to call it) is to work to match workloads to tools in a clear-headed and analytical way.
BW is pretty unique in that it is a truly general purpose datawarehousing tool. This means that it is probably not the best tool on the market for solving any of the challenges above, but it may well be the best tool on the market for solving all of them. I’m not saying it’s simple or that it is a joy to work with, but it is good at what it does, it is flexible, it provides very good performance as long as you are nice to it, and it is very open to non-SAP data and to using external tools when you want that best-of-breed experience or a specialized tool for a specialized workload.
I’m at least as guilty of this as anyone else, but sometimes in our excitement about new and innovative products, we forget that the hardest and most important decisions are the ones in which we acknowledge tradeoffs and decide between two or more not-ideal options.
The BW product does an admirable job of helping us work through and design around the tradeoffs inherent in the fundamental datawarehousing and BI challenges listed above. In my view, this is an important niche that I don’t see addressed by other tools. Perhaps it is a niche that is destined for extinction, but I don’t think so. These tradeoffs don’t go away just because of a new tool or a new technical innovation. Sometimes an innovation will change the relationship or the tradeoff a little bit, but the tradeoff is still going to be hiding there somewhere.
If we take one thing from this discussion, I hope it is the realization that most of the challenges listed above are more human in nature than they are technical. They tend to derive from the difficulty in making tradeoff decisions, standardizing interfaces and architectures, identifying and focusing on the problem space, and understanding how people may actually use these systems to greatest effect. Because of this, these challenges are often at least as susceptible to design solutions as they are to technical solutions. There is a tendency in the industry to focus on technical answers to these challenges over design answers, perhaps because technical solutions are often more impressive, in some sense more physical, and in many senses more sellable. I think this tendency is unfortunate.