What is mass enrichment?
Mass enrichment is a function in SAP Cloud Platform Data Enrichment that enables you to enrich high volumes of database records. With this feature, you can enrich your existing business partner data with data from third-party data providers such as Dun & Bradstreet.
You can select the existing business partner data records you want to enrich. These records will then be used to search for similar data records in the data from third-party data providers. Based on the results, you can carry out a mass enrichment of your business partner data records after deciding the entities that you want to enrich.
What are the challenges faced by customers in this area?
Currently master data is widely available. However, the challenge is to keep the huge scale of master data (sometimes in millions) up to date. It is not practical to update the individual master data records one-by-one. Some challenges that users face include the following:
- Data formats: Varying data formats (for example, json, csv) might have to be followed for data from different sources
- Authentication methods: Different authentication methods would have to be followed for different data sources (for example, OAuth 2.0, SAML, basic authentication, and so on)
- Maintenance effort: Field names would be different for data from different data providers (for example, country might be called country name) and there would be an effort that would be required for maintenance to adapt to the data provider’s interface. Even for extensibility, in the case when the data provider adds more fields, a maintenance effort would be required.
- Coordinating with data providers: Reaching out to different data providers, understanding their various pricing models that requires expertise in the field, and negotiating with them individually would also be difficult
- Different landscapes: Enriching mass data in different landscapes such as SAP S/4HANA and SAP MDG is tough for customers
- Multiple licenses: Different departments of an organization might have different licenses and agreements with the same data providers instead of having one license for the entire organization. Even in the case of mergers, acquisitions, and subsidiaries the issue of multiple licenses might arise.
What is the solution that mass enrichment offers?
Data enrichment supports mass enrichment of general data and address information of business partners consisting of the name of the business partner, business partner ID given by the data provider, unique identifier generated by SAP for the business partner, street information, telephone number, name of the city and country, country code, postal code, and so on.
The mass enrichment solution provided by SAP Cloud Platform Data Enrichment provides a simplified way of enriching huge scales of master data records in batches. It tackles the various challenges listed in the following ways:
- Data formats: Data enrichment defines a single data format per domain such as business partner for all different data providers simplifying the process for users
- Authentication methods: Users can follow a single authentication method defined by data enrichment
- Maintenance effort: Users can follow the standardized fields that data enrichment offers for data from different data providers thereby cutting down the efforts required for maintenance from the user’s side. Extensibility or addition of different fields by different data providers would also be taken care by data enrichment.
- Coordinating with data providers: SAP would be the go between to reach out to the different data providers for pricing and negotiations. Also, the offerings from the different data providers can be easily accessed from SAP App Center.
- Different landscapes: Out-of-the-box integration is available with different landscapes such as SAP S/4HANA, SAP MDG and SAP Data Hub
- Multiple licenses: The data enrichment solution helps users to have a unified license for an organization that cuts cost and reduces effort to handle different licenses.
What are the prerequisites to using mass enrichment from data enrichment?
- You must purchase the data enrichment solution from SAP Store.
- You should purchase the required enrichment data to third-party data providers from SAP App Center (in case you do not already have license to a particular data provider already).
How can you consume mass enrichment using data enrichment?
1. SAP S/4HANA Cloud and SAP MDG
Mass enrichment is integrated with the Consolidation and Mass Processing for Business Partner Data application from SAP Master Data Governance, which can be consumed using SAP S/4HANA Cloud (starting from the 1902 release) and SAP MDG on S/4HANA (starting from the 1909 release).
This diagram shows the SAP MDG consolidation flow and where mass enrichment from data enrichment fits in.
This screenshot shows the Start Mass Enrichment and Manage Mass Enrichments applications from data enrichment that is integrated with SAP S/4HANA Cloud.
2. SAP Data Hub
Mass enrichment is integrated with SAP Data Hub where only configuration must be maintained. The minimum SAP Data Hub version required is Data Hub 2.4.56 which includes vrelease 2.4.50.
This is a diagram of a sample pipeline configuration.
- Send match criteria to data enrichment: To support massive scale of processing master data to be enriched should be uploaded in batches
- Message queue: As part of buffering, messaging queues are used that are specially designed to support faster read/write
- Submit match criteria to data provider: Messages are read from the queue and processed (mapping of fields) and passed on to the data provider, i.e., D&B.
- Polling: Since the scale of data can be huge, the data provider can take some time to enrich the master data. Hence, this is a mechanism to poll periodically for the status.
- Status check: Customers can always check on the status of their mass enrichment processing.
- Read from data provider: Once the master data is enriched from the provider, data enrichment downloads the results and processes the response in mapping of SAP fields. The results are saved in an object store.
- Download results: Enriched master data can be downloaded by the customer as needed from the object store.
Benefits of Using Mass Enrichment from Data Enrichment
- SAP is the single point of contact between the various data providers and you.
- It is also easy for you to switch between different data providers as SAP standardizes the fields and authentication methods.
- You do not have to face the hassle of getting into pricing and other negotiations with the data providers as the data enrichment solution would take care of this.
- Performance is ensured since mass enrichment is handled in batches and auto scaling helps in handling huge loads of data.
- Reprocessing and error handling is simplified as the user’s interaction is minimal as it is an automated solution.
- Mass Enrichment through SAP Data Hub
- Mass Enrichment through SAP S/4HANA Cloud
- SAP Cloud Platform Data Enrichment Documentation
- SAP Store
- SAP App Center