The Past, Present, and Future of Data Warehouses

[former_member766315]

Warehouses aren’t sexy. A warehouse is just a big, multi-purpose building that’s primary role is accessible storage. There’s not a lot of hot debate about warehousing in its traditional sense. So why are data warehouses something enterprise companies can’t stop talking about?

There’s arguably no better topic to discuss as I kick off my new LinkedIn Live series: Data Champion Roundtable, powered by SAP. In this series, I’ll be sitting down with enterprise data experts from across the industry. Each episode will focus on deep dives into the complexities, challenges, and opportunities of using big data to drive success.

In our inaugural episode, data warehouses take center stage. What are they and how do they work? How can enterprise companies leverage them to de-silo data? Are data warehouses worth exploring in the modern age of data sharing and cross-platform collaboration?

It’s these questions I pose to my guests, Tony Baer, Principal at dbInsight LLC, Independent Researcher, and Judith Hurwitz, President & CEO at Hurwitz & Associates LLC. Their insights show how crucial data warehousing is and why enterprise companies need it to power actionable insights.

Here’s what the experts have to say.

Data Warehouses Aren’t a New Concept

To understand data warehouses and why there’s so much chatter surrounding them, I chose to ask my guests the obvious question. What is data warehousing, exactly?

“Data warehouse definitions have changed a lot over the years,” explains Judith. “What we have today is basically a platform that allows an organization to consolidate its data from across the business—whether it’s traditional relational databases, NoSQL databases, a variety of other data sources—so it becomes available across the business. It’s used to understand trends and what customers want, so you can really apply deep analytics across business silos.”

Already, Judith’s definition conjures up a much more interesting picture than the traditional concept of a dusty old warehouse. Tony rebuts with a different perspective.

“I would say the definition of data warehouses hasn’t changed—rather, the sources of data have broadened over the years,” says Tony. “Back, 20-25 years ago, when data warehouses first emerged, it was a platform to consolidate that data from a mainframe or a relational database. Today, as Judith said, we have many more data sources.”

Judith goes on to explain that in the past, data relevancy was also a major obstacle for warehousing. Bits and pieces of data compiled in a data warehouse required manual updating—a cumbersome process without the real-time accessibility we experience today.

This process—called data virtualization—is what makes today’s data warehouses so useful for enterprise organizations that rely on data to drive crucial decision-making insights.

Are Data Warehouses Dead?

In preparing for the show, I came across no small faction of organizations and service providers declaring the death of data warehousing. Of course, I found an equal number of warehousing evangelists. I decided to ask the experts. Are data warehouses dead?

“A data warehouse isn’t dead because the problem hasn’t gone away,” says Judith. “When people initially wanted to do a warehouse, they’d say ‘I have all this data and I need to understand it, but it’s all over the place and I can’t get a handle on it. Let’s do a warehouse.’ What happens today? You hear the same exact conversation. ‘I have 27 business units all holding onto their data, but I have to have all this data so I can make decisions.’”

The problem isn’t any different than it has ever been, according to Judith—which means warehousing continues to be the answer for data aggregation. Tony agrees, but emphasizes with the challenges data warehousing has faced over the last two-to-three decades—from fragmented data to lack of logistical solutions to aggregate that data.

Tony doesn’t see the death of warehousing. On the contrary, he sees opportunity in the cloud. “The challenges are still with us, but I think that with the cloud, it can start to provide that bridge to solutions. Hopefully, we can start to think about solutions that make everyone happy.”

Data as the Key to Something More Important: Analytics

The underlying theme in my conversation with Judith and Tony ultimately comes down to analytics. How can companies use warehousing to drive better decision-making? How does all that raw data become actionable insights?

“It took companies 20 years to master basic query and reporting. Now, we have this whole palette of capabilities and it’s getting to the point where we’re working with AI and machine learning. The fact is, a lot of organizations are looking at their need for data scientists to start taking advantage of some of these predictive and prescriptive analytics.”

“What it comes down to is that companies have always been challenged. They mastered the last war; now they need to fight a new one,” says Tony.

Judith brings the conversation back around to warehousing. “When you start talking about AI and machine learning models, and you have data from all these sources, how do you know you’re selecting is the right data? Is it clean? Does it have errors in it? Can you make decisions from it?”

So many moving parts make it difficult for enterprise companies to maintain the integrity of their data. The role of good warehousing quickly becomes evident, harkening back to the traditional definition of “a multi-purposed, accessible storage space.”

Warehousing and the Rise of Self-Service Insights

The cloud is an imperative cog in the future of successful data warehousing and is key in enabling self-service insights via data that’s housed in (you guessed it) a data warehouse.

“We are moving into a period I’m calling ‘The Industrialization of the Cloud.’ Over the next few years, the cloud will come to be at the very center of what we do with data. The idea of self-service is a premier way that the cloud can start to change things. If you set things up right with the right model and the right tools, you can then allow your business’ users to get access to the data they need to make well-informed decisions. That really changes a lot,” says Judith.

“The cloud allows several things. First, it allows solution providers to realize the potential for an end-to-end experience. A data warehouse used to be, essentially, a specialized database. Now, thanks to self-service and the cloud, there’s no reason why that ETL tool needs to be separate. There’s no reason why the visualization tool has to be separate. There’s no reason why the data federation tool needs to be separate. They may be separate tools, but they can be packaged and delivered as a single service.”

The concept of self-service insights is one that’s appealing to any enterprise organization. Business analysts, data scientists, and even members of the C-suite can delve into a complete data warehouse concept to get their questions answered. It’s the be-all and end-all of what warehousing has promised to deliver for the last 20-25 years.

The Next Iteration of Data Warehousing

The only thing left for you to wonder is what does the future of warehousing look like?

“I think it’s the idea of a distributed warehouse,” says Judith. “One of the things the cloud gives us is cloud-native capabilities—the ability to have this virtualized, highly distributed environment that can manage huge volumes of data when you have to scale up and down. It’s very elastic and changes the nature of warehousing into something that’s very dynamic, very fluid, and very effective.”

Judith’s vision of data warehousing covers the infrastructure of the future. Tony sees its potential for application.

“There’s a top layer to this, and that’s a guided experience. As a service provider, companies like SAP have lots of customers asking very similar types of questions,” says Tony. “On the consumer services side, we’re used to technologies asking us ‘is this the question you want to ask?’ There’s all this machine learning going on in the background that can provide us with a guided experience. It doesn’t make the decisions for us, but it helps us decide how to make those decisions and what questions to ask.”

Join us for an Even Deeper Dive into Data Warehousing

In just 15 minutes, Judith and Tony provided wonderful insight into data warehousing, to show exactly why it continues to be an exciting prospect for enterprise companies. We know the concept, purpose, and challenges behind warehousing data, as well as how the cloud is driving new warehousing strategies.

For an even deeper dive and expert insights on this topic, catch the first Data Defined episode on Wednesday, August 19th! Register for the webcast here.