Managing Big Data
Big Data is currently the main disruptive force driving many companies to rethink their data strategies. In their new book, The Human Face of Big Data, Rick Smolan and Jennifer Erwitt explain that soon every object on earth will generate data – our homes, our cars, even our own bodies. By gathering and analyzing data on a real-time basis, this new knowledge may lead us to address some of humanity’s biggest challenges, including pollution, world hunger, and illness. They reveal that we are exposed to more data in a single day than our 15th century ancestors did in a lifetime. Smart devices are turning us in to human sensors, as we leave a continuous digital trail of texts, calls, location data and more.*
This may feel overwhelming, even threatening. But as our planet begins to develop its own nervous system – of which we are all a part – it’s worth considering how we can benefit from Big Data.
First, we should examine what it is. In essence, Big Data comes in three flavours:
- Structured data generated from business processes.
- Machine-generated data, such as clickstream or sensor data.
- Human-sourced data, such as social media, images, audio and unstructured documents.
This presents problems for traditional data warehouses, which cannot keep up with new sources of information, new types of data, more complex analysis and greater speed needed. In fact, despite the explosion in data volume, variety and velocity, most data is never actually collected.
Fortunately, we have now developed new ways of capturing and using Big Data with in-memory platforms such as SAP HANA® that delivers near-instantaneous access to and analysis of real-time data.
SAP HANA allows us to develop new processes and eradicate limitations of the past. This means we can not only use Big Data to see and understand information in more detail, at the moment when we need to see it, but we can also use advanced analytics to discover entirely new ways of doing things.
To meet the evolving needs of Big Data projects, many businesses are embracing a hybrid data ecosystem – a flexible data management environment that can comprise of a changing set of technologies, such as operational platforms, cloud-based platforms, Hadoop, analytical platforms, discovery platforms, data marts and data warehouses.
*SAP is a proud sponsor of “The Human Face of Big Data”, a fascinating look at the affect that Big Data has on humanity. To learn more about this and other stories in the book: http://amzn.to/UhubvR