At SAP's North America conference "Conversations on the Future of Business" at MetLife Stadium on Wednesday, there was a lot of interest around big data and how can that help businesses. That makes a lot of sense. Big data is a buzzword for huge volumes of data that is generated in part by millions of customers who interact daily with online and mobile services. Private corporations now have access to an unprecedented amount of data that can be analyzed to provide quicker solutions to customers.
However, it is important to differentiate between data that is just 'noise' and the part that is relevant. That is not an easy task, given the huge volumes we are talking about. In the year 2000, only a quarter of the world’s stored information was digital. The rest was preserved on paper, and other analog media like film. Today, 98% of all stored data is in digital form.
This issue was addressed by Nate Silver (pictured), who won national acclaim after correctly predicting the winner of all states in the 2008 presidential elections. He is the author of the acclaimed book, The Signal and the Noise: Why So Many Predictions Fail — but Some Don't, and blogs at fivethirtyeight.com. Over the last five years, he has used his model to predict everything from winners of Major League Baseball to politics and even, dating.
Silver asked the audience to have a bit of skepticism when dealing with big data. I found all of that to be very relevant. Imagine organizations relying on big data analysis to make changes to the customer experience only to realize that their analysis was off the mark.
First, it is important to incorporate the margin of error inherent in any study. Often big data is seen as a means to eliminate all error because there is no sampling anymore - you are analyzing the whole dataset through powerful programs. However, Silver emphasized that errors still happen, and that organizations must be aware of them before making decisions based on the analysis of that data, be it trends in retail or predicting consumer behavior.
Secondly, people who analyze data must be aware of their own biases. To illustrate this, Silver talked about his experience of predicting wins in the 2012 presidential election. He ignored what the pundits and politicians were saying and instead stuck to what the polls said. Relying on polls, which had more credible data, led to accurate predictions.
Thirdly, it is important to have access to the right data. He pointed out the flawed analysis of mortgage-backed securities leading up to the 2008 economic crash. "You need to survey the data landscape carefully," he said. This factor has been important also at the United Nations Global Pulse, where I have worked with colleagues who analyze call data records for development purposes. One of the most important steps is finding partner organizations who can share data sets that are authentic and reliable.
Silver's final advise was to "try and err." There might be errors in initial analysis or in the type of data used, but eventually organizations would get it right. So while the media often writes about the big wins (such as Silver's election predictions) "in the real world, it's usually lots of little wins instead," he said.
Some have done a better job of big data analysis than others. For example, big retail organizations have worked out models that are now helping them analyze huge troves of customer data to design optimal inventory and pricing strategies. Their data is high quality, and reliable and they can make better decisions as a result. Not so for the financial firms that never saw the economic crisis coming.
Later, on the sidelines of a roundtable on "Insight Driven Innovation", I talked with SAP's Ken Koffler, and he told me that more clients are using HANA to combine their own data with third party data sets to better understand customer habits. For them, and for the thousands of organizations looking to tap big data for business, Silver's words could not have come at a better time.
Continue the conversation on the Future of Business here.
Related:
Millennial Engagement Dominates SAP’s “Conversations on the Future of Business”
- SAP Managed Tags:
