I understand some people consider the rise of big data as little more than hyperbole. Indeed, for the first time, it made Gartner’s 2012 hype cycle, which purports to “highlight the common pattern of over enthusiasm, disillusionment and eventual realism that accompanies each new technology and innovation.” The research firm placed big data approaching its “peak of inflated expectations,” meaning we’re all about to be disappointed.
Sorry, Gartner, with all due respect, you’re dead wrong. Actually, the expectations from big data are underinflated as far as business, science, healthcare, government, and education are concerned. That’s because the benefits these market segments get from big data are not theoretical, they’re real.
In a post here some months ago I highlighted a few big data success stories for the SAP HANA in-memory database. Others have also underscored big data-driven breakthroughs for a variety of industries. A conference to be held later this month at Temple University that will highlight big data success stories at Wal-Mart, Merck, NASA, and many other organizations. The fact is, if you’re not motivated by the “hype” around big data, your company will be outflanked by competitors who are.
The arrival of big data today is not unlike the appearance in businesses of the personal computer, circa 1981. Like the PC, big data existed long before it became an environment well-understood enough to be exploited. That is, PCs existed in the 1970s, but only a few forward-looking businesses used them before the 1980s because they were considered mere computational toys for hobbyists.
What changed? Combining the PC with nascent tools, such as VisiCalc, Lotus 1-2-3, WordStar, dBase II and other packages made the difference. Suddenly, company executives everywhere could run sophisticated software without needing IT (MIS departments back then) developers to spend months creating programs.
Similarly, we’ve known about big data or “information overload” inside organizations since the term was popularized by Alvin Toffler in 1970 in his best seller Future Shock. But, as with the early years of the PC, we lacked the technologies to exploit the vast amounts of data inside companies.
As they say, that was then. Now we have the technology to exploit the data with relatively low-cost flash memory, multicore processors, hyper-fast in-memory and columnar databases, real-time complex-event processing, Hadoop/MapReduce, rapid deployment of complex analytical models through standards like Predictive Model Markup Language (PMML), and much more. These tools will make big data as pervasive and as useful to business as the PC has been.
Gartner got its hype cycle wrong this time. Big data is already well along on the so-called Plateau of Productivity as its countless success stories already prove. Skeptics who doubt this are like people who once derided the PC as not being a serious computer. Today, it is those big data skeptics that we should not take too seriously.
