If not the term ‘Big Data’ then what?
When asked, ‘What is Big Data?’ Responses almost invariably boil down to three V words at the core – volume, velocity, variety – with another one or two sometimes thrown in for good measure. Frankly, I think there is a bigger trend afoot; a trend that the term ‘Big Data’ is ill equipped to define. And I have heard many others share the same thought.
And that of course is the million dollar question: What is the right term to define this trend? Or maybe it’s the billion dollar question since everyone points to Google, Facebook, and eBay when they think of Big Data. This blog describes where my thinking is at today. Be sure to comment and share your thoughts! Maybe together we can find a better term.
What is truly powerful about Google, Facebook, and eBay is how they transform data into insights, driving massive valuations in only a few years. Of course it’s not just about the ‘cha-ching’ of shareholder returns. Stop and think for a moment what truly impresses you about these companies. Here’s how Google and Facebook inspire me:
Google. I personally love how Google finds the information I’m looking for across the Internet and at lightning speed! Of course there is a really impressive architecture to take the network of information (that’s all the content on the Web), turn it into sophisticated intelligence (using nifty algorithms), and drive the resulting insights to a network of users (that’s us).
Facebook. No denying the fun in stalking long forgotten classmates and old neighborhood chums. But what’s truly interesting about Facebook is how they turn your profile and your network into insights about what matters to you, and then match you to the right advertisers. Over $5 billion a year in ad revenue is not too shabby for only 9 years of work!
No, Big Data is not what is impressive about Google and Facebook; Big Data is only a part of what makes them successful. What’s impressive is how they apply intelligence in their networks. In the case of Google, their network is the Internet. In the case of Facebook, their network is the 800 million users and the companies that want to advertise to them.
For enterprises wanting to harness data to become the leaders of their industry, a Big Data strategy is not enough. No, their strategy needs to insert intelligence into their networks of employees, customers, processes and resources. And that’s what some of SAP’s most innovative customers are doing. They are building Networked Intelligence into their organizations and changing how we work, play and live.
So what is Networked Intelligence? It is the second phase of Insight-based Computing, and builds upon Big Data and data warehousing. Watch for my next post describing my thesis on Insight-based Computing, blog style.
Hi David, thanks for sharing your thoughts on Big Data. If "big" is relative to the size of the organization dealing with the data, then we can say that organizations of all sizes need to manage Big Data. It also sounds like the term Big Data only highlights challenges and problems. What if we looked at it from a solution or goal-oriented perspective - could we call it "Big Insight?"
BigInsights is actually a product of IBM based on Hadoop for Big Data! I have written a post on my personal blog yesterday about the Hype on Bigdata, maybe you find it inteesting as well
Wow, good for IBM to capture this name! 🙂 Thanks also for sharing the link to your blog post, Carsten.
I've been hosting SAP Inside Track events in Vancouver for the last few years, and have considered the Big Data theme for this year's event. I will be following the Big Data discussion closely.
Thanks for sharing the blog post, Carsten. I agree with you and Jason that insights are central to the trend.
Some of your comments near the end of your blog post really resonated with me. You mentioned:
"The paradox is that we have more and more information available, this makes it even more difficult to determine what is relevant and what not." and also the importance of the 'human decision factor'.
I think this next phase in the trend, that I'm labeling Networked Intelligence, is how to leverage insights to improve the 'human decision factor' by building a network that can turn available information into insights.
I wonder if we can't truly harness the potential of data until we get past the paradox you mention to impact the 'human decision factor'?!
Good question you are raising. It will be interesting to see how things evolve. There are always fashions and hypes. I remember in 2000 when I was working at Ariba, people were desperately trying to buy our solution (we were able to choose the customers). The solution at that time was a marketplace. If you did not talk in 2000 about Marketplace and B2B you were just so 1990's. How many talk today about creating a Marketplace ???
That said the Human decision factor will be probably hard to eliminate but that is something that is beyond my knowledge. Like very much the idea of Networked Intelligence. Sounds very promissing!
For years IT has been engaged in crunching numbers and spitting out reports based on old data. Then came predictive analytics that allows them to look into the future based on past trends. Now we have Twitter, Facebook and other social networking engines that have data that needs to be tracked. Ths is the touchy feely or left brain data. The convergence of these two worlds is what is being referred to as 'Big Data'. How does a company take hard numbers, combine it with social sentiment (tweets, likes, etc) and determine where they should go next? A great hockey player once said "You don't skate to where the puck is, you skate to where its going to be" WG. I think business are struggling with figuring out where the puck "customers" will be. Ask us how we are doing that today/
you are making very good point but on the other side the use of realtime and even more social media bears its risks especially if you consider social media in Real time. I have refered to it in my blog with the incident on Wallstreet just last week. A Tweet made the Dow loose $200bn in a few seconds.
I second the thought that companies need to have everytime more their ear on the rail, and listen very closely what their customers are thinking yet it is not all they need to do. Using "old" data as you are referring to remains very important. More than that I would challenge that most companies today are not even using in most of the cases the data they have available (hard numbers) since the tools they are using are not user friendly and even worse EXCEL continues to be probably one of the most used tools to analyze and present data.
In short yes we should use "soft data" for our decision making as we are doing in "normal" life as well. But just like we do most of the time in normal life do it in balanced way. To most companies I would recommend today to focus much more on their real internal data and make sure it is being used.
Hey Rich - I agree. Big Data seems to be this next phase of 'data warehousing' which is taking new data (i.e. social media) and more data to uncover insights.
BUT here's what I wonder. Isn't 'data warehousing' analogous to the old mainframe days where it was handled be experts. I'd argue that the PC revolutionized information processing by pushing it beyond the experts to non-technical folks in the organization.
In my mind the next phase, Networked Intelligence, is analogous to the PC revolution. Networked Intelligence is about pushing insights beyond the experts - data scientists as well as the MapReduce and SQL geeks - and into the broader organization.
Looking back in time and/or looking at human behavior - Greed, need and to a certain extent 'herd instinct' are considered to be 3 key emotional drivers for business behavior (including stock markets). So I'm not surprised when one thought about a relevant word, they came up with a term that make you notice and instills a degree of fear ( if you think its a challenge) or drive (if you think its an opportunity).
I agree that the word 'Big data' denotes to a state of 'challenge/opportunity' instead of a solution. Then again, to gain mass attention, people resort to a tried and tested method of defining a big problem, making you agree that the 'elephant is in the room', then coming up with a solution ' Analytics' to give you an 'aha' moment. In 2001, Doug Laney from Meta( now Gartner) mentioned about Big data, but the momentum picked up only recently. Just to add bit more to the question - now we have also words such as 'right data', 'lean data' etc floating around 🙂
David, nice post. It didn't take long for the term 'Big Data' to turn into a buzz word, did it? 😛
That's the problem with buzz words. Everyone has a different interpretation. I think as long as companies that are serious about big data adhere to some of the principles you describe in this IT Business Edge article, it's all good. Naming conventions shouldn't matter so much.
Thanks for the comments, Tim and Ramesh.
Some of my thoughts shared in the IT Business Edge interview highlight the importance of terms, in my mind. The problem with Big Data is that the word 'Big' connotes large. So people so often equate Big Data primarily with volume. Yet when I talk with CIOs, their bigger problems seem to be data velocity and associating together data from different sources.
I wonder if in the larger trend, there are terms like you mention Ramesh - big data, right data, lean data, machine data, fast data, etc. BUT none of them capture the essence of the trend, simply technical objectives that need to be achieved to grab hold of the broader trend.
Agreed, the terms such as big data, right data etc don't capture the essence of the trend - its just that each sub-sector of the overall market is carving out such words to either define their 'problem statement' or 'value proposition'. For eg. within the whole Big data story , there might be only a portion that's relevant to the company in a phased manner. So now with companies in an execution mode, we can expect to see more of this. Just like the term Cloud, it triggered so many sub-segments during the execution phase. Interesting times ahead.
One more term - Little data 🙂
Little Data Makes Big Data More Powerful - @HBR
I think it is besides the velocity the variety and to see the needle in the haystack. There is actually this cool video of SAP & IBM to show a nice combination of what could make sense for a "BigData" scenario. I have seen a lot of organisations struggle with implementing Hadoop based solutions. Hadoop is not a "database" in the sense that we used to know it is much more a "development platform". This makes it great but also quite complex --> good services business (very good one I would dare to say). But this in turn leads back to the question of VALUE. Is not much easier and with much higher ROI to buy something standard, rapid and easy to deploy, even though it does not allow you to customise 100% but only 95% ?
The reason why CIO focus on velocity more might be linked to the fact that it gives quite a strong "show effect".
Nice post David - I know Big Data is going to be covered in a big way at SAPPHIRE Orlando (excuse the pun) in a few weeks in the Databse & Technology Campus. What customers do you have speaking and what sessions would you recommend people attend?
Thanks, Margaret. If people aren't planning to be at Sapphire they should probably fix that quick 🙂
On the Big Data front we have some really smart customers and project collaborators speaking on the topic, including McLaren, Home Shopping Network 24, Pirelli, Stanford University and Burberry.
We also have Rick Smolan, an internationally renowned photo journalist, who recently published a visually stunning book on the topic called, The Human Face of Big Data.
Speaking of McLaren...wait...what's this? Is that ME on the SAPPHIRE NOW homepage?
Catch SapphireNOW even if you are not there: http://www.sap.com/events/sapphirenow/index.epx?url_id=banner-us-scn-view0515-sapcom-live-SANOW2013