Skip to Content
Product Information
Author's profile photo Oliver Huth

Bi-directional Data Exchange for Individualized Data Products in the Data Marketplace

In this blog, you will learn how data providers and data consumers can exchange data on an individualized basis due to the Data Marketplace technology and processes. If you want to learn more about the Data Marketplace for SAP Data Warehouse Cloud in general, check out the following overview blog post.

The entire process – from setup to operations phase for seamless and governed update – is demonstrated in the demo video below in 10 simple steps. The use case demonstrates a retail customer that runs SAP Data Warehouse Cloud wants to retrieve the amount of e-charging stations around his outlets from a Data Provider that has data about all charging stations in Germany. Instead of simply “throwing over” the entire data set, the data is enriched and provided as a private data product.

<20′ youtube video to be embedded once uploaded>.

The 10 steps – 7 for setup and 3 for operation – are shown on the below marchitecture where you can see how the data from the ultimate data consumer is shared as reference data to produce the individualized data on the provider side and share back the individualized product for the final data enrichment.

The%20full%20set%20of%20data%20sharing%20functionality%20and%20processes%20of%20the%20Data%20Marketplace%20for%20SAP%20Data%20Warehouse%20Cloud%20is%20leveraged%20by%20both%20the%20data%20consumer%20and%20data%20provider.

The full set of data sharing functionality and processes of the Data Marketplace for SAP Data Warehouse Cloud is leveraged by both the data consumer and data provider.

The 10 steps with some additional explanations are the following:

DESIGN PHASE

  1. Check View To Be Enriched – Consumer in Data Builder

    As a first step, the data consumer needs to make his internal data available to the data provider as reference data. If he wishes to do that in an integrated fashion via the Data Marketplace, he needs to have the data in a deployed view that he can use in the subsequent step in the Data Builder.


    In this demo scenario, it is a master data view that contains store locations with initially 542 stores that have IDs below 300.000. For the demo, we increase the number of stores by actively filtering a larger set. In reality, obviously, this data would be updated based on changes in the source systems or the ETL processes.

  2. Make Data available to Provider – Consumer in Data Sharing Cockpit

    As a second step, the data consumer creates a data product that makes the previously shown view available in the Data Marketplace. To not unveil unnecessary information to other users, the listing is created in a minimal fashion while a contract ID is used in the header for reference by the data provider (This will be enhanced with the visibility management with comes in Q2/22 on the SAP Data Warehouse Cloud roadmap). As not the entire view shall be shared as reference data, the data provider makes use of the column selection which allows sharing only the relevant subset of the actual columns for the individualization on the provider side. To manage the actual access, it is set as a license key product and a license key is generated and passed on to the Data Provider. Lastly, the current set of data is published by creating a data release in the publishing management.

    The%20Listing%20can%20also%20be%20done%20in%20a%20minimalistic%20way%20to%20share%20data%20securely%20and%20fast%20between%20companies%20via%20the%20Data%20Marketplace.

    The Listing can also be done in a minimalistic way to share data securely and fast between companies via the Data Marketplace.

  3. Load Reference Data – Provider in Data Marketplace

    Now the Provider can act as a data consumer and can load the reference data into his Data Factory Space to create the private data product by using the license key that the retail customer has provided. He can either deal with all data products in one space or have one space per customer – this has no cost effect and depends on the subsequent steps. He can use the Delivery Tracking to ensure that the data transfer job runs successfully. He also gets information on the number of records that have been transferred and that he now needs to enrich.

    Sharing%20is%20for%20everyone.%20Consumption%20is%20for%20everyone.%20To%20retrieve%20the%20reference%20data%2C%20the%20data%20provider%20uses%20the%20Data%20Marketplace%20the%20same%20way%20a%20consumer%20does%20and%20benefits%20from%20all%20supporting%20processes%20such%20as%20Delivery%20Tracking.

    Sharing is for everyone. Consumption is for everyone. To retrieve the reference data, the data provider uses the Data Marketplace the same way a consumer does and benefits from all supporting processes such as Delivery Tracking.

  4. Prepare & Harmonize Data – Provider with Intelligent Lookup

    Now the individualization takes place. The Data Provider brings together the reference data – stores with geolocation data in form of longitude and latitude columns – with his database of e-charging stations in Germany that also have such geolocation data. For the demo setup we have chosen a straightforward approach of reducing the number of decimals of the geolocations to create 111x555m grids that we try to match. There are obviously more sophisticated options, e.g. leveraging the SAP HANA Spatial engine to calculate distances of the geo tuples. 


    The%20Intelligent%20Lookup%20is%20used%20to%20match%20the%20reference%20data%20with%20the%20entire%20database%20and%20retrieve%20full%20transparency%20of%20the%20matching%20result.
    The Intelligent Lookup is used to match the reference data with the entire database and retrieve the full transparency of the matching results.

    To derive maximum transparency in the process and reserve the freedom to add matches manually, the data provider uses the Intelligent Lookup feature of SAP Data Warehouse Cloud. Learn more about it in the overview post of Jan Fetzer. As a result of the Intelligent Lookup, we get a table based on which we create a view. This is required to be able to add it into a Listing in the Data Sharing Cockpit in the next step. In this demo scenario, out of 538 stores, 54 have matched with an e-charing station in the same grid.

  5. Create Private Data Product – Provider in Data Sharing Cockpit

    After the actual creation of the data, the data provider can proceed with the listing. He is free to maintain the details based on the agreement with the consumer and context details the ultimate user needs. Most likely, the Data Provider will have a separate public data product that gives all information of the private product with sample data to try it out will this listing is only executing the actual one. The same way the consumer has done before, the Data Provider creates the License and a Release for the governed and transparent update process later on.

    For%20Private%20Data%20Products%2C%20the%20License%20Management%20is%20essential%20to%20only%20provide%20access%20to%20consumers%20that%20have%20a%20commercial%20entitlement.

    For Private Data Products, License Management is essential to only provide access to consumers that have a commercial entitlement.

  6. Load Individualized Data Product – Consumer in Data Marketplace

    Everything is set for the most important step now. The consumer can load the individualized data product into his space of choice – in this scenario into his initial POS 360° space. The Delivery Tracking shows transparently that all 538 records have been shared back as not only the enriched records have been shared but also the ones where no charging stations have been identified. If part of the commercial agreement, the Data Provider could have also shared a second view with the same data product with the list of charging stations that have been matched.

    Governance%20and%20Transparencies%20are%20core%20values%20of%20the%20Data%20Marketplace%20for%20enterprise-grade%20cross-company%20data%20sharing%20as%20here%20in%20the%20Delivery%20Tracking.

    Governance and Transparencies are core values of the Data Marketplace for enterprise-grade cross-company data sharing as here in the Delivery Tracking.

  7. Use Acquired External Data – Consumer in Data Builder

    Last but not least, the Data Consumer can use the entire data management functionality to process the data further in SAP Data Warehouse Cloud or use it with SAP Analytics Cloud or 3rd Party Tools. In this demonstration, we end with joining the enriched data back with the initial store data set to end with the additional KPI: Number of Charging Points.

    Harmonization%20of%20Internal%20and%20External%20Data%20is%20the%20key%20use%20case%20of%20the%20Data%20Marketplace.%20With%20such%20private%20data%20products%2C%20the%20main%20harmonization%20effort%20is%20outsourced%20to%20the%20Data%20Provider.

    Harmonization of Internal and External Data is the key use case of the Data Marketplace. With such private data products, the main harmonization effort is outsourced to the Data Provider.

    OPERATIONS PHASE —————————————–

  8. Publish New Data for Enrichment – Consumer in Publishing Management

    The data consumer and data providers are now wired together and if new data shall be enriched, the process can be easily be triggered in a transparent way. To simulate new available data, in this demonstration, we adjust the filter to all records from IDs below 300.000 to IDs below 500.000 which changes from 538 stores to 1.058. To request data enrichment for the new data, the consumer simply uses the Publishing Management to hand over the new records.

    With%20Publishing%20Management%2C%20the%20Data%20Consumer%20can%20actively%20communicate%20the%20request%20for%20a%20new%20update.%20This%20could%20also%20be%20done%20with%20a%20live%20query.

    With Publishing Management, the Data Consumer can actively communicate the request for a new update. This could also be done with a live query.

  9. Load, Harmonize & Publish New Reference Data – Provider

    In the design time, the Data Provider has chosen the update mode “manual”. As a consequence, in the Delivery Tracking, he can see that the status of his activated Data Product is “Outdated”. He can now pull the new update to start his update process. Alternatively, with Update Mode “Immediate”, the new update could have been ingested directly after the release creation on the consumer side.

    In%20the%20Update%20Tab%20of%20My%20Data%20Products%2C%20it%20becomes%20transparent%20that%20a%20data%20update%20is%20available%20and%20its%20ingestion%20can%20be%20triggered.

    In the Update Tab of My Data Products, it becomes transparent that a data update is available and its ingestion can be triggered.

    Now it also becomes apparent why for the Data Provider we have chosen the Intelligent Lookup as a harmonization environment. As the Data Marketplace has updated the records directly in the Input Table, the matching status changes and we now have 49% records for whom no matching has been conducted yet. After triggering a run of the Intelligent Lookup, we can see that we know have E-Charging Stations for 134 stores based on the matching rule.

  10. Load & Use Updated Data – Consumer

    Finally, with just one click – that could also have been made obsolete by choosing the Update Mode Immediately – the new data is ingested into the table that has been created with the initial activation of the Data Product. The join in the view now showed the 134 stores for whom the e-charging station data has been enriched.

    Access%20to%20data%20in%20clicks%2C%20not%20projects.%20It%20works%20for%20both%20public%20data%20products%20as%20well%20as%20private%20and%20individualized%20data%20products%20with%20the%20Data%20Marketplace%20functionality%20and%20processes.

    Access to data in clicks, not projects. It works for both public data products as well as private and individualized data products with the Data Marketplace functionality and processes.

In this demo setup, we have consciously chosen a highly governed process where both parties use publishing management to request new data or push new data respectively. In case you want to automate such an exchange with a live query setup of the reference data or also the data enrichment, reach out to us via mail to datamarketplace@sap.com.

While this was a demo scenario to explain the use case and supporting functionality and processes, you can actually check out the free “E-Charging Stations in Germany” Data Product by the “Open Data connected by SAP” Data Provider in the Data Marketplace. In case you are not an SAP Data Warehouse Cloud customer yet, just sign up for a free trial for up to 90 days right here.

Assigned Tags

      Be the first to leave a comment
      You must be Logged on to comment or reply to a post.