Skip to Content
Author's profile photo Ravi Verma

MDM and Big Data – Discovering the obscure link

In the following blog I have tried to detail my understanding of the subject.For a clearer understanding I have kept it in a question answer format.Please feel free to add to it.Here we start.

Big data and MDM seem to have a significant connection, but for now the connection is still very unclear. At first thought they seem like an odd pair with great contrast between them.

MDM can be defined as a set of tools, procedures, and policies to govern, create and maintain a trusted data. This data serves as the very base of business transactions and hence many times referred to as “DNA of a business”.

Big Data on the other hand comprises of environments which is composed of huge volume of data coming from variety of sources. It is the overall umbrella of social media data, unstructured documents, streaming data from instrumented devices, and more. Unlike MDM its main character is not Trust.

In Gartner’s words “Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.”

So how are the two related?

Big data provides a high volume data, this high volume and high velocity data needs to be processed and refined to come at valuable information. And here lies the link – The MDM hub can keep a traditional ‘golden record’ of trusted information along with a less-trusted view of the same person or product based on the findings from big data. When you combine the traditional ‘golden record’ with new information found from  your big data, the superset of information can power even better business insights and business decisions that were not possible before. The combined view can provide a more insightful complete view, but they can still be presented separately in cases where business can’t afford to base decisions on the less-trusted view. So Big data is supported by MDM, it tells a business what does a predictive analysis or classification mean for its Customers, Vendors and more.

How to arrive at valuable information?

To achieve this business would have to mine the Big data. In other words, it would involve exploring and analyzing large amounts of data to find patterns for big data. The techniques came out of the fields of statistics and artificial intelligence (AI), with a bit of database management thrown into the mix. Generally, the goal of the data mining is either classification or prediction. In classification, the idea is to sort data into groups. For example, a marketer might be interested in the characteristics of those who responded versus who didn’t respond to a promotion. Based on such information he can change his strategy to target a group of responders. To add to this there are softwares enabling a process called Opinion mining. It is a type of natural language processing for tracking the mood of the public about a particular product. It is also called sentiment analysis, involves building a system to collect and examine opinions about the product made in blog posts, comments, reviews or tweets. Automated opinion mining often uses machine learning, a component of artificial intelligence (AI).

Together big data and MDM can help extract insight from the increasing volume, velocity, variety, and mapping that to the truthful data, in context, beyond what was previously possible. This could lead to creating master data entities, loading new profile information into the MDM system, sharing master data records or entities with the big data platform as the basis for big data analysis, and much more.

Example –Let us take example of Health care system. Here master data could be Patients, Providers, Payers, Households, Employees, Reference data etc. To have complete view of a Patient we can have data from Hospital, Doctor, Web, Health insurance, Pharmacy etc. One can leverage predictive analysis to come at patients who are at risk for readmission, leverage classification to measure and identify causes for adverse events, assist in drug discovery ,this can help insurance companies devise personalized programs etc.

How do innovations in existing MDM portfolio enable this connect?

SAP is employing a multipronged approach.  Firstly, the current Master data solutions would continue to evolve in line with business, addressing their needs. Secondly, SAP is leveraging SAP HANA as the common data platform for SAP enterprise MDM .It would allow businesses to handle huge volume of data in large scale master data hub.  Additionally, the Master Data Services package from SAP and Business Objects enables customers to improve business process efficiency, effectiveness, and responsiveness, and make better informed decisions based on high-quality, accurate master data.

I would like to conclude by saying that to have MDM in place is to already have a strong foundation for Big data,together they can provide a Vantage point to business.

Assigned Tags

      You must be Logged on to comment or reply to a post.
      Author's profile photo Former Member
      Former Member

      Hello Ravi,

      Thanks for sharing the wonderful insight into BIG Data and MDM solution.

      But i have few questions on the same.

      1. How is an MDM solution going to handle the huge velocity of data from a BIG data system.

      2. Since trust is not a factor for BIG Data Solution, how is an MDM solution to trust the correctness of the data.

      I do understand these are a bit on the technical side, just need your inputs on the concerns.

      Thanks & Regards,


      Author's profile photo Ravi Verma
      Ravi Verma
      Blog Post Author

      Hi Abhishek,

      Thanks for going through. Please find my answers below -

      1.Since every insight based on data has a shelf life,it should come fast.The pace with which data is gathered and analysed is important for success.In my view this is not something which would be handled directly by MDM. For example, MDM would tell which Customers can stop doing business with a organization based on one insight (obtained using Big data analysis).The result would again have a shelf life and hence result may change at some other time depending on various parameters.Thus based on the insight,the High Risk Customers may change at some other time.So huge data Velocity does play a role in a way providing fast changing insights to MDM but it does not have a direct bearing on MDM. Innovations as listed above should help MDM to handle the indirect consequence of high velocity data.

      2.MDM holds the data which is based on TRUTH.Data correctness is an attribute of Master Data.If data is not correct then it could lead to wrong Customer/Vendor contacts,wrong names etc. The results of having such data are obvious. SAP MDM would provide access to real world entities to the Predictions/Hypothesis/Insight generated using BIG Data analysis.

      Hope I could answer your questions.  

      Best Regards,


      Author's profile photo Deepak Kr Gupta
      Deepak Kr Gupta

      This is very good insite on MDM & BIG data integration.

      Best | Deep.

      Author's profile photo mohan kumar
      mohan kumar

      Hi Ravi,

      Thanks for the beautiful insight on MDM and Big Data connection. Yes, I do agree that there is a significant connection between analysis of data collected over the Big Data and data in MDM. May be MDM data will act as a “search index” for Big Data giving scope for more targeted analysis.

      When you say “Enterprise MDM”, I assume you are talking about SAP NW MDM, MDG and MDS as a bundle because SAP terms it this way (pls correct me if I am wrong). However I have a big concern here, if I want to deep dive into Enterprise MDM and see who is going to handle the Big Data- NW MDM or MDG? What do I find?

      Considering that SAP is aggressively adding support for HANA to most of its products (if not all), which also means replacing the traditional disk based storage as it is doing for “BW on HANA” for high speed analytics irrespective of the volume of data. Also SAP’s solution for Big Data in place with HADOOP and HANA integration, now it is planning for “MDG on HANA” considering that Data Governance plays a vital role in Big Data too. But nothing about HANA or Big Data is mentioned in the SAP NW MDM road map, which clearly indicates that SAP is gearing MDG to handle the “Big Data” in coming future and it also indicates that NW MDM has very little or nothing to do with the Big Data. So will the SAP’s analytical solution for MDM-Big Data will be “MDG on HANA”?

      This is just my understanding kindly give your thoughts on this.


      Mohan Kumar

      Author's profile photo Former Member
      Former Member

      Thanks for sharing!

      Author's profile photo Gaurav Nayar
      Gaurav Nayar

      Good blog Ravi.

      Thanks for sharing.

      Gaurav Nayar