Abundant Big Data Opportunities for Organizations both Large and the Small

The more relaxed pace in the workplace during the week with the Canadian and American national holidays has given me a chance to catch up on the on some things. In addition to tying up some loose ends I have also had opportunity to read and digest in more depth some of the plethora of material that floods my way. On piece in particular, Unisphere Research’s 2013 Big Data Opportunity Survey (link), struck a cord and provided me with some interesting insights I would like to share here:  

• Big Data is not just volume

• Even small and mid-size companies have massive amounts of data

• Organizational size is not a significant factor in undertaking Big Data projects

• Predictive Analytics and Understanding Customers are the top two Big Data opportunities

• Big Data is not just about new data types

• Over half of Big Data initiatives use production and transactional data and ERP data is used in over 40% of Big Data initiatives

• SAP is in an excellent position to help companies large and small with Big Data

• SAP can currently help companies overcome many of the barriers they see around Big Data

Released in May 2013 – told you I needed to catch up on some reading – the report covers companies both large and small. Over 25% of the respondents were from companies with 100 or fewer employees and over 25% of respondents were from companies with 10,000 or more employees, giving an exceptionally good world view. DBA’s, data architects, IT consultants, IT managers and executives were well represented with each of these groups individually making up from 11% to 17% of the over 300 data managers and professionals who responded so it is pretty real world. 

Not surprisingly the survey reveals Big Data is a big deal to a lot of companies with over 75% of respondents indicating they had users who were demanding more data in order to do their jobs and 58% indicating that Big Data was very or extremely important to their organizations. And it is not just more data it is faster data access with one third of respondents indicating that over the next 3 years they anticipate an over 50% increase in user demand for access to real time data (within 1 hour of creation) and over half of respondents anticipating an over 26% increase in user demand for real time data.  At present only 16% of organizations deliver 50% or more of their data in real time and just under a third deliver between 10% and 25% of their data in real time. Real time data is
something SAP, the holder of the Guinness World record for data loading and its flagship in-memory real time SAP HANA platform excel. As companies roll out projects to meet the substantive anticipated increase in user demand for more real time data SAP is well positioned to help. There is a need for speed and in-memory and SAP is leading the way in the paradigm shift. 

So Big Data is more than just volume, but speaking of volume which many people equate with Big Data, the survey reveals that some companies are dealing with massive amounts of data indeed. Half of the organizations have data stores of 50TB or greater, 39% have data stores 100TB and larger and a taggering and 17% noting stores of 1PB or greater. That is a lot of data. And over a quarter of respondents feel that data volumes will increase by over 100% in the
next few years.

Not too surprisingly company size is seems somewhat related size of data managed with 68% of companies with over 10,000 having data stores in excess of 100TB. But interestingly some 8% of small companies with 100 or fewer employees and well over a quarter of mid-sized companies of between 101 and 1000 employees manage more than 100 TB of data. Again this speak well to SAP positioning and strategy with flexible affordable cloud and on premise
solutions that can meet the needs of companies large and small. Not to mention the blazing performance of the SAP HANA platform which has been proven to be able to handle a petabyte of raw data and SAP Sybase IQ a multi-petabyte scale columnar database.  

The two top opportunities organizations saw for Big Data to improve and increase business were predictive analytics (62%) and gaining a better understanding of customers (53%) and of the Big Data projects under way 55% are focused on customer analysis / segmentation.  As a result it is not surprising that Big Data efforts were strongest in retail / services at 61% of respondents in that industry group having Big Data projects followed closely by financial services / insurance close behind with 58% of the organizations in that industry running Big Data projects.  Again this speaks well for the offerings SAP provides from the built in predictive analytics and the extended natural language processing text mining capabilities in SAP HANA.; including Simplified Chinese for multi-lingual “voice of the customer” analysis. To the simplified predictive application development application function modeler in SAP HANA Studio that enables model-driven access to native functions and predictive algorithms and natural language processing capabilities in SAP HANA. Big Data initiatives around historical / archived data analysis that make up 46% of Big Data initiatives can also be well served by SAP though SAP HANA’s smart data access capability. This allows non-disruptive evolution and integration of SAP HANA into existing architecture letting SAP HANA access data where it exists and integrates with systems holding data from Teradata to Hadoop to SAP Sybase IQ and SAP Sybase ASE. Indeed it positions SAP Sybase IQ, a Petabyte scale massive parallel process capable system to ask as a near-line store and accessible archive to SAP HANA. 

I find it very interesting that organizational size does not seem to be a factor in undertaking Big Data projects. Some 37% of companies having 100 or fewer employees having Big Data initiatives under way compared to 53% of companies with over 10,000 employees.  The graph below shows only a moderate slope increase in the number of organizations that have Big Data projects as company size increases.  So big and small company size does not seem to
be a major factor but it is odd to note that where 50% of companies with 100 or fewer employees and 65% organizations with between 101 and 1000 employees saw Big Data as very or extremely important only 35% of companies with over 10,000 employees did. But this may be more of a perspective thing and what you think of as big. Regardless it shows that company size is not a key factor and that solutions must be affordable enough and flexible enough to fit companies from small to large. This bodes well for the path and strategy that SAP has set forth. SAP HANA platform and SAP HANA One provide flexible, affordable solutions to in the cloud and on premises helping companies to get going quickly and that meet the needs of organizations from large to small.

Not surprisingly with the focus on predictive and understanding customers the survey showed Big Data efforts were strongest in retail / services at 61% of respondents in that industry group having Big Data projects followed closely by financial services / insurance close behind with 58% of the organizations in that industry running Big Data projects.  When it comes to the drives noted as motivating Big Data projects the need to improve existing processes was noted
as a driver by 59% of respondents while 39% noted a need to reduce time to access data. This ties in well with what I noted about users wanting more data
and an anticipated need of users demanding faster access to data and how SAP is well positioned to provide solutions for this.

Big Data projects do not necessarily mean new data types. Over half of the Big Data initiatives are using production or transactional data and about 40% were using ERP and CRM data with about the same number that were using textual (documents) data. Only about a third of projects are using weblogs, social media and multimedia and only about one fifth of use sensor or machine to machine data. To me that a substantive number of Big Data initiatives are using existing internal data is good shows there was considerable untapped value that can now be extracted with new techniques and that combining it with data from new sources is providing invaluable insights.  When it comes to Big Data strategies one third of organizations are looking at extending current nfrastructure and one quarter are looking at adding Hadoop and in-memory.  Again SAP is leading the way and well positioned to help organizations lever BiG Data. In-memory – SAP HANA the innovative first mover paradigm shifter. ERP data – think SAP. Levering existing production or transactional data, extend and optimize current infrastructure interact with and lever Hadoop – SAP HANA’s smart data access provides it all allowing for non-disruptive evolution.  SAP HANA smart data access provides non-disruptive intelligent architecture to provision enterprise-wide data from heterogeneous systems such as Hadoop, Teradata, SAP Sybase ASE and SAP Sybase IQ. As well it provides the capability for SAP HANA to SAP HANA queries which can facilitate SAP NetWeaver Business Warehouse on SAP HANA to SAP Business Suite on SAP HANA query use cases among others. SAP brings in-memory computing to data federation and it will optimize where data is stored and acted upon. Now customers can easily build unified transactional-analytical applications with secure access to data across their business networks without large data transfers. In-memory computing lets then get results from the core process in real-time, providing a single version of truth on which business can act.

SAP can help organizations realize the value of all the data they have across their enterprise and to gain real-time insight from all the information they have at their disposal. 

There seems to still be a bit of inertia around getting Big Data projects underway with 43% of organizations indicating they had specific Big Data projects underway and 42% indicating that it was just business usual in their data environment. There were several barriers, technical and business that organizations noted in the survey. Luckily SAP can easily help address the majority of these – if only these companies knew. Lack of budget (43%), lack of skills, (35%) and data governance (31%) were the top three business barriers to Big Data initiatives noted by organizations. As I noted previously SAP with SAP HANA platform and SAP HANA One provides flexible, affordable solutions to in the cloud and on premises to help companies to get going quickly and that meet the needs of organizations from large to small in a very cost effective manner.  SAP has several offerings around governance that can be levered for Big Data and SAP HANA smart data access allows leverage of Big Data information where it sits without highly specialized skills. And organizations can lever Data Science services from SAP to uncover new signals hidden in their data. No need to wait months attempting to hire a data scientist they can hire data scientists who also know your industry. For organizations that are unsure of the value of Big Data (32%) and the 29% that were unsure of the technology requirements, organizations can engage Design Thinking experts to uncover critical business needs and opportunities. This can help focus Big Data projects on critical business problems, align business and IT stakeholders to maximize adoption, and bridge the gap between business vision and IT requirements.

The top three technology barriers to Big Data projects noted were faster querying of the data (51%), faster access to large data volumes / sets (43%) and faster acquisition and storage of data. Again I see SAP as very well positioned to address these three barriers. Faster querying of data ties into the increased need for real time data access that began this piece and as I noted there SAP is leading the paradigm shift.  Real time data is something SAP, the holder of the Guinness World record for data loading and its flagship in-memory real time SAP HANA platform can really help customers address. Just as the members of the 10,000 club who SAP helps running applications 10,000 times faster on the SAP HANA Platform.  http://www.computerweekly.com/news/2240184309/Sapphire-2013-Plattner-Sikka-hail-HANA-growth-beyond-SAP   As to faster acquisition and storage of data – did I mention SAP is currently holds the for data load Guinness world record – there is SAP Sybase ESP for streaming data feeds, SAP
Sybase SQL Anywhere to synchronize machines and mobile devices, SAP Sybase Replication Server to replicate data from transactional systems in real-time can and SAP Data Services to Load and transform any type of data. Quite the comprehensive portfolio to provide real world Big Data solutions.

To me, the study indicates that there is a long way to go as companies gird themselves to work with Big Data and the opportunities it offers and sadly that many Big Data initiatives are doomed to struggle and fail especially if they try to lever inadequate existing technology and fail to make the paradigm shift to and integrate the in-memory real time processing that the SAP HANA platform can offer.

The Unisphere Research’s 2013 Big Data Opportunity Survey can be found here