Big Data Facts and its importance

[vivekbhoj]

Hi Everyone,

Lately, I have heard a lot of buzz around Big Data and Hadoop.

There is a very good blog series by mahesh.kumar8on Big Data, Hadoop and HANA.

You can check it here :

After reading this blog series, I got basic overview of Big Data and Hadoop and decided to explore more on this topic.

In this blog, I would like to share with you what I have learned while exploring Big Data.

Just to give everyone a brief Overview,

Big Data is huge amount of data that exceeds the processing capacity of our traditional database systems.

Big Data is difficult to capture, store, manage, and analyze via traditional database systems. It generally deals with data in Petabyte and Exabyte

Now a days, Big data is one of the most talked topic in IT industry.

Every organization wants to harness this data in order to make better decisions.

Many companies like Splunk and Cloudera are developing solutions and services that would give businesses the power to harness big data.

Now lets see what is the situation today:

As per the Digital Universe Ticker, around 3.2 Zettabyte of information has been created since 1st January, 2013.

This ticker also shows that Terabytes of data is being generated every second.

Check the Ticker here:

http://www.emc.com/leadership/digital-universe/index.htm (The Digital Universe is an EMC-sponsored study by IDC measuring and characterizing all the digital data created, replicated, and consumed worldwide in a single year).

As per 2012 Digital Universe Study:

• The amount of information in the Digital Universe is doubling every two years, currently growing at a rate of more than 7,600 Petabytes per day.
• Less than 1% of the world’s data is analyzed today, presenting an enormous opportunity for Big Data analytics.

The main reasons for growth in digital data are:

• Continued growth of Internet usage, social networks and smartphone adoption:
  • Now a days, everyone of us is using Social network sites especially Facebook and twitter.
    • Now a days more than 9000 tweets are being generated every second.
    • Check this link to know about more twitter statistics: http://www.statisticbrain.com/twitter-statistics/
    • Now a days, around 3 lac statuses are updated and more than 1 lac photos are uploaded to Facebook every minute.
    • Check this link to know about more Facebook statistics: http://zephoria.com/social-media/top-15-valuable-facebook-statistics/
  • For the very first time in history, more Smartphones were sold more than feature phones in first quarter of this year.
    • Check this press release by IDC:http://www.idc.com/getdoc.jsp?containerId=prUS24085413
• Falling costs of the technology devices that create, capture, manage, protect, and store information:
  • Now a days, digital products are getting cheaper and cheaper - Now we can buy smartphones at less than 10,000 Rupees - this was one of the main reasons as to why Nokia Lumia 520 is selling so well all around the world - Smartphones like these can take images, create documents and do all sorts of things thus increasing digital data
  • We upload lots of videos on Youtube as its free and there are lots of Reviews of different products available in Youtube and thus lot of data is being generated
• Migration from analog TV to digital TV:
  • Now we all use digital TVs - most of them have Full HD resolution - and all the content that is being transmitted to TV is also now being shot in Full HD or even 4K which takes a lot of space and it is increasing digital data
• Growth of machine generated data:
  • The data generated from Satellites, sensors readings from factories, data generated by equipment(tractors, vehicles) like fuel consumption, temperature etc. This data is growing at a great speed with more industrialization and growth of developing countries like India.
  • The machine generated data will account for 40% of the Digital Universe by 2020, up from just 11% in 2005 as per Digital Universe estimation.

Now let me quote some facts about Big Data:

According to Foreignaffairs:

As recently as the year 2000, only one-quarter of all the world’s stored information was digital. The rest was preserved on paper, film, and other analog media. But because the amount of digital data expands so quickly — doubling around every three years — that situation was swiftly inverted. Today, less than two percent of all stored information is non-digital.

According to Informationweek:

A recent survey by database vendor RainStor of mid-senior level executives shows the majority of respondents understand the value of big data to their businesses. Three-quarters of respondents said that better management of big data helps their organizations make smarter business decisions. And yet more than a third (37.5%) also said that analyzing big data is their biggest challenge.

According to MGI Survey:

A retailer using big data to the full could increase its operating margin by more than 60 percent.

If US healthcare were to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in value every year. Two-thirds of that would be in the form of reducing US healthcare expenditure by about 8 percent. In the developed economies of Europe, government administrators could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data, not including using big data to reduce fraud and errors and boost the collection of tax revenues. And users of services enabled by personal-location data could capture $600 billion in consumer surplus.

As per the article from Washingtonpost:

A Big Data Factsheet was put out in the White House in March 2012 — upon the launch of its $200 million Big Data Research and Development Initiative

that listed more than 85 examples of such efforts across a number of agencies. They include the Cyber Infrastructure for Billions of Electronic Records (CI-BER), led in part by the National Archives and the National Science Foundation, and NASA’s Global Earth Observation System of Systems (GEOSS), which the fact sheet described as a “collaborative, international effort to share and integrate Earth observation data.” And the Defense Department is putting about $250 million a year into the research and development of such projects — “a big bet on big data,” as the White House called it.

According to Gartner:

By 2015, 4.4 million IT jobs globally will be created to support big data, generating 1.9 million IT jobs in the United States. In addition, every big data-related role in the U.S. will create employment for three people outside of IT, so over the next four years a total of 6 million jobs in the U.S. will be generated by the information economy.

As seen from above Big Data Facts, it is clear that Big Data is going to play very important role in the future.

Now a days, Big Data is being used in many Industry Sectors, be it Healthcare, Banking, Retail etc.

Also read the Business Insider article on How Mobile Marketing Is Being Shaped By The Explosive Growth In Big Data and it tells us how big data can be used to enhance and personalize mobile experience and also how it can affect mobile advertising.

Also read Bigdataweek article on The Value in Unstructured Data and know more on how Businesses can benefit from Unstructured data(such as data from Facebook)

Now a days whenever a new Product is launched, people share their views on twitter or Facebook.

As we all know last year, Iphone5 was launched and lots of people tweeted their views

Derek Ruths, an assistant professor at McGill University analyzed this data and created an Infograph that showed popular phone features,countries that tweeted most about the Iphone, most frequently used words in tweets about Iphone announcement and few more analyses.

As this analysis shows, we can gain a lot of insight about any product, such as we can know if our new product was well received by the consumers, which features they liked, which features they didn't like etc.

There is need of Big Data because traditional databases can't analyse data from Social media, data from Videos, data from sensors as this type of data grows at very fast speed.

If you are interested in knowing Big Data History, you can read the article  A Very Short History of Big Data by Forbes

If you are interested to know more about Big Data and its uses, you can download the free ebook Big Data Now from O'Reilly or  Understanding Big Data from IBM.

Also Check my second blog Hadoop,Its Importance and Use Cases

Also read about Hadoop & HANA Integration

Thank You for reading my blog.