Skip to Content

Hi Everyone,

Lately, I have heard a lot of buzz around Big Data and Hadoop.

There is a very good blog series by Mahesh Kumar CV on Big Data, Hadoop and HANA.

You can check it here :

After reading this blog series, I got basic overview of Big Data and Hadoop and decided to explore more on this topic.

In this blog, I would like to share with you what I have learned while exploring Big Data.

Just to give everyone a brief Overview,

Big Data is huge amount of data that exceeds the processing capacity of our traditional database systems.

Big Data is difficult to capture, store, manage, and analyze via traditional database systems. It generally deals with data in Petabyte and Exabyte

Now a days, Big data is one of the most talked topic in IT industry.

Every organization wants to harness this data in order to make better decisions.

Many companies like Splunk and Cloudera are developing solutions and services that would give businesses the power to harness big data.

Now lets see what is the situation today:


As per the Digital Universe Ticker, around 3.2 Zettabyte of information has been created since 1st January, 2013.

This ticker also shows that Terabytes of data is being generated every second.

Check the Ticker here:

http://www.emc.com/leadership/digital-universe/index.htm (The Digital Universe is an EMC-sponsored study by IDC measuring and characterizing all the digital data created, replicated, and consumed worldwide in a single year).

As per 2012 Digital Universe Study:

  • The amount of information in the Digital Universe is doubling every two years, currently growing at a rate of more than 7,600 Petabytes per day.
  • Less than 1% of the world’s data is analyzed today, presenting an enormous opportunity for Big Data analytics.

The main reasons for growth in digital data are:

  • Continued growth of Internet usage, social networks and smartphone adoption:
  • Falling costs of the technology devices that create, capture, manage, protect, and store information:
    • Now a days, digital products are getting cheaper and cheaper – Now we can buy smartphones at less than 10,000 Rupees – this was one of the main reasons as to why Nokia Lumia 520 is selling so well all around the world – Smartphones like these can take images, create documents and do all sorts of things thus increasing digital data
    • We upload lots of videos on Youtube as its free and there are lots of Reviews of different products available in Youtube and thus lot of data is being generated
  • Migration from analog TV to digital TV:
    • Now we all use digital TVs – most of them have Full HD resolution – and all the content that is being transmitted to TV is also now being shot in Full HD or even 4K which takes a lot of space and it is increasing digital data
  • Growth of machine generated data:
    • The data generated from Satellites, sensors readings from factories, data generated by equipment(tractors, vehicles) like fuel consumption, temperature etc. This data is growing at a great speed with more industrialization and growth of developing countries like India.
    • The machine generated data will account for 40% of the Digital Universe by 2020, up from just 11% in 2005 as per Digital Universe estimation.

Now let me quote some facts about Big Data:

According to Foreignaffairs:

As recently as the year 2000, only one-quarter of all the world’s stored information was digital. The rest was preserved on paper, film, and other analog media. But because the amount of digital data expands so quickly — doubling around every three years — that situation was swiftly inverted. Today, less than two percent of all stored information is non-digital.

According to Informationweek:

A recent survey by database vendor RainStor of mid-senior level executives shows the majority of respondents understand the value of big data to their businesses. Three-quarters of respondents said that better management of big data helps their organizations make smarter business decisions. And yet more than a third (37.5%) also said that analyzing big data is their biggest challenge.

According to MGI Survey:

A retailer using big data to the full could increase its operating margin by more than 60 percent.

If US healthcare were to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in value every year. Two-thirds of that would be in the form of reducing US healthcare expenditure by about 8 percent. In the developed economies of Europe, government administrators could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data, not including using big data to reduce fraud and errors and boost the collection of tax revenues. And users of services enabled by personal-location data could capture $600 billion in consumer surplus.

As per the article from Washingtonpost:

A Big Data Factsheet was put out in the White House in March 2012 — upon the launch of its $200 million Big Data Research and Development Initiative

that listed more than 85 examples of such efforts across a number of agencies. They include the Cyber Infrastructure for Billions of Electronic Records (CI-BER), led in part by the National Archives and the National Science Foundation, and NASA’s Global Earth Observation System of Systems (GEOSS), which the fact sheet described as a “collaborative, international effort to share and integrate Earth observation data.” And the Defense Department is putting about $250 million a year into the research and development of such projects — “a big bet on big data,” as the White House called it.

According to Gartner:

By 2015, 4.4 million IT jobs globally will be created to support big data, generating 1.9 million IT jobs in the United States. In addition, every big data-related role in the U.S. will create employment for three people outside of IT, so over the next four years a total of 6 million jobs in the U.S. will be generated by the information economy.

As seen from above Big Data Facts, it is clear that Big Data is going to play very important role in the future.

Now a days, Big Data is being used in many Industry Sectors, be it Healthcare, Banking, Retail etc.

Also read the Business Insider article on How Mobile Marketing Is Being Shaped By The Explosive Growth In Big Data and it tells us how big data can be used to enhance and personalize mobile experience and also how it can affect mobile advertising.


Also read Bigdataweek article on The Value in Unstructured Data and know more on how Businesses can benefit from Unstructured data(such as data from Facebook)

Now a days whenever a new Product is launched, people share their views on twitter or Facebook.

As we all know last year, Iphone5 was launched and lots of people tweeted their views

Derek Ruths, an assistant professor at McGill University analyzed this data and created an Infograph that showed popular phone features,countries that tweeted most about the Iphone, most frequently used words in tweets about Iphone announcement and few more analyses.

As this analysis shows, we can gain a lot of insight about any product, such as we can know if our new product was well received by the consumers, which features they liked, which features they didn’t like etc.

There is need of Big Data because traditional databases can’t analyse data from Social media, data from Videos, data from sensors as this type of data grows at very fast speed.

If you are interested in knowing Big Data History, you can read the article  A Very Short History of Big Data by Forbes

If you are interested to know more about Big Data and its uses, you can download the free ebook Big Data Now from O’Reilly or  Understanding Big Data from IBM.

Also Check my second blog Hadoop,Its Importance and Use Cases

Also read about Hadoop & HANA Integration


Thank You for reading my blog.

To report this post you need to login first.

15 Comments

You must be Logged on to comment or reply to a post.

  1. Raman Korrapati

    Hi,

    For me also its new.Just recently hearing more about Big data and Hadoop terminology on SCN. By reading i got an idea about big data. Thanks for your time and efforts Vivek.

    Thanks

    (0) 
    1. Vivek Singh Bhoj Post author

      Hi Raman,

      Big data is going to be very crucial for businesses in the near future and there is a prediction that by 2015 there will be  a lot of big data and hadoop jobs in the market.

      Hadoop is really very interesting, I have read a lot about it and about HANA and Hadoop integration too, will post some detail about it sometimes later

      Regards,

      Vivek

      (0) 
      1. Raman Korrapati

        Thanks for more information Vivek. Mean while i will try learn some basic knowledge on this two concepts. it may add more values to our profiles.

        Thank You

        (0) 
          1. Kotha Kris

            Hi Vivek hoping to get answers/views on below questions:

            a) Is HANA BIG data or does HANA deal with BIG data?

            b) Why HADOOP when we have HANA or vice versa?

            c) How HANA will make use of ETL such as BODS? I am a BODS developer and would want to start learn BIG data

            d) Can I call HANA and HADOOP as BIG data from now on? Does HANA makes BODS be out of the game i.e. No more ETL with BODS

            Thanks in advance, Krishna 

            (0) 
            1. Vivek Singh Bhoj Post author

              Hi Krishna,

              a) HANA is not Big Data but it can help in dealing with Big Data

              For rest of the questions, please read below blogs also and then ask me any queries you have:

              Hadoop, Its Ecosysytem and Use Cases

              Hadoop and HANA Integration

              d) HANA doesn’t make BODS out of the game as BODS is a ETL tool that is also used to load data into HANA and can also be used with Hadoop also.

              Getting started with Data Services & Hadoop

              Regards,

              Vivek

              (0) 
  2. Vikash Kumar

    Really informative blog.

    With growing nature and size of data, it will be really interesting how technologies can adopt themselves to this massive growth.

    (0) 
  3. Praveen Kumar

    Dear Vivek,

    Thanks for such Great full work of Knowledge sharing .I wold like to Learn SAP HANA kindly guide me with steps .

    Thanks for your kind support

    (0) 

Leave a Reply