SAP Data Intelligence Metadata Explorer – Part 1

Facing Data governance issues while implementing data driven innovations across your enterprise?

Here are the latest innovations in data governance that SAP Data Intelligence has in-store for us.

This blog series will be focusing on the detailed features of Data intelligence w.r.t Metadata explorer and will have a broader comparison of different tools of Metadata management, Data Quality and Data Profiling tools that are available in technology space. This blog is intended to the people who are trying to explore more on Data Intelligence features and how it can be related to different Data Governance features.

It also dives into the intricacies of “how-to” automate the process of data profiling and Data quality.

*Firstly, let us know why any organization needs to adapt Data governance solution and what data issues does it exactly addresses

In an organization the data is distributed across multiple systems and what if your data is inconsistent between systems and doesn’t roll up well, how would you address this issue? Take a minute and try to think of such scenarios…

Now in most of the scenarios where you came up with solutions the data might be simple, usually the data problem doesn’t come in dribs and drab, it is high in number and more complicated and in large quantity. Generally, when such issues are reported, the IT doesn’t have enough context to fix the data and business users believe this is an IT Issue. This is where the organization ends up with no mans-land.

This leads to a lot of problems in variety of ways, like your reports don’t roll up well and this will further lead to the failure of data reconciliation.

Organizations use a framework called Data Governance to tackle this no-man’s land. A framework which facilitates the organization with the overall management of the availability, integrity, usability, ensuring quality and security of the data.

So all the benefits of Data Governance can be effectively achieved through the SAP Data Intelligence Metadata explorer. The Metadata explorer helps you to track the data between sources, catalog the data sets, publish the data sets, Profile the data, Data Quality and check for sensitivity of the data.

These are a brief set of features of the Metadata explorer that I have come across during my research on Metadata explorer.

1. 1. Get the Meta data of the data set
   2. Preview the data set
   3. Profile the data to see the distribution of the data
   4. Publish the data sets to allow the users to access and search for the data
   5. Can check the distribution of data using wizards like charts for the Profiled data set.
   6. Tagging and creation of tag hierarchy on the data sets that has registered and published.
   7. Creation of Validation rules, rule books and binding of the data sets.
   8. Data lineage on registered data and also on data transformed file.
   9. Creation of score cards to visualize the Quality score of the data sets, categories and rule-books.
   10. Visualizing the rule set results by creating dashboards on the data sets
   11. Monitoring the runs of Publishing, Profiling, Data Preparation and rule books
   12. Business Glossary for defining business terms and relationships.

Metadata explorer can be useful to different users in Data Governance frame work in difference ways. Below are the list of roles that are part of Data Governance Programme.

*Now let’s get familiar with the usage and the basic features of SAP Data intelligence Meta data explorer.

SAP Data Intelligence Metadata Explorer Features

The landing page of Data intelligence Metadata explorer contains different tiles that help users to navigate to the desired area.

In Browse Connections tab you can see all the connections that are created in Connection Management tile. You can view the source data, Metadata of the data set, profiling report, run the lineage analysis on the supported connections and also run the preparation tasks.

You can also view the connection capabilities that are supported with the connections available as part of Data Intelligence. In this blog we will work with AWS S3 bucket (connection created for demo) and below are the functionalities it supports.

With the current version of Data Intelligence, Lineage extraction is not supported for AWS S3 bucket.

Fact sheet gives you the detailed information of the Metadata of the data set, it provides the sheet with the columns, Data types, tags, Unique key and description. It also provides the detailed information of the data set, Connection ID, Type of Data set, Data set Size, Last Modified, Last Published and many more.

It provides you with the trends on the row count and size, it also provides charts to get a better view on the data spread and metadata of the data.

Data preview tab provides you with the sample data of the data source, by default no. of records is set to 100 and it can be extended up to 1000 rows.

Publishing the data – Once the data set is registered it can be published, publishing the data set provides easy accessibility of the data set to the team and also allows the team to search for data set in the search bar, to add comments and Tag the data set. Tagging the data set provides you with most sophisticated search on datasets.

All the published datasets can be seen from “Browse the Catalog” screen and the progress can be monitored from Monitoring tile.

Data Profiling is the process of examining and providing detailed statistical report of the data. Based on this report different users can act on the data accordingly, it helps the users to gain better insights of the data, helps in taking some cleansing actions such as running diagnosis and also helps in keen examination of the data that you have.

SAP Data intelligence has in-built feature of data profiling which provides you with additional metadata of the data set, some of the meta data attributes are Minimum-Maximum, Average Length, Null Values, Blank Values, Distinct values and many more will get appended to the fact sheet of the data set.

All the Profiled data sets can be seen from “View Profiled Datasets” tile and monitoring run can be seen from monitoring tab of Data intelligence.

Data Preparation  used to manipulate the raw data prior to data processing and data analysis. It helps in collecting the data, data cleansing, data enrichment, data standardization, Data imputation and consolidating the data into one accurate.

Data preparation in SAP Data intelligence provides you the ground to perform all the kinds of transformations that to be performed on the raw data prior to the data analysis. Some of the functionalities of SAP Data Intelligence Data preparation includes in-column transformation, custom logic on existing column, data enrichment, duplication of data, editing the recipe and run the preparation task.

All the preparation runs can be seen in “View Preparation” tile and can be monitored from Monitoring tile.

While executing the preparation task it asks to store the preparation file in any of the supported target container. Data Intelligence has tight integration with Agile Data Preparation. it also has natively available operators that helps to automate the process in more sophisticated way.

Recipe records all the transformations and manipulations that were done to the data at the time of data preparation. You can also revert the changes by deleting the recipe items.

Data Quality and Validation rules are the special form of Business rules that are defined on the data, it provides the flexibility to check the quality of the data by defining data quality rules, to check whether data meets the required standards, to check for sensitivity of the data and also to check the completeness of the data.

SAP Data intelligence can import the rules which are already defined and consumed in SAP Information steward. It is just a single step process to import all the approved rules from SAP Information steward to SAP Data Intelligence

We can also define the rules and test the rule in prior to the data set binding.

Rule book allows you to import the rules that were already defined and bind the rules to the particular data set. You can run the rules and it provides you with percentage of passed records and also the sample of records which doesn’t satisfy the rule.

Rule book runs can be monitored from the monitoring tile to check the progress of the rule book.

Rule Dashboard enables you to create interactive dashboards on the rule categories, on rule books and on datasets. It allows you to create dashboard on single data set/rule category/rule-book, comparison of multiple datasets/rule categories/rule-books and also to capture data set/rule category/rule-book trends.

Business Glossary is a business metadata that provides the semantic context to data. It helps the organization to define the terms and relationships between the objects. SAP Data Intelligence Business Glossary is a library where you can define the business terms and associate them with the business. Additionally, it also allows to create relationship between terms, datasets and columns to bring in a meaningful relevance of the data in the organizational context.

One of the Example, what does “Customer” mean to the organization? It really depends on the organization type, it is altogether a different context when compared with Finance and Sales domain. Each organization has its own terms, definitions, and relationships. Business glossary helps you to create the Business terms and relationships that are very much aligned with the organization

Categories in Business Glossary allows you to group the logical terms into a single unit. You can also add Keywords, synonyms to the Business term which helps you define the metadata of the Term.

Conclusion

Metadata explorer has a rich set of features where organization can adapt for the Data Governance program. With Metadata explorer organizations can successfully execute 5W’s and 1H called who – what – how – when – where and why of data. It not only ensures the security and compliance of the data but also delivers valuable insights of the data with 5W and 1H principle.

I hope this blog provided you with the intrinsic features of Metadata explorer in Data intelligence, starting from end to end view on data management and orchestration for the big data analytics.

You can post any queries and concerns you face in the comment section below and also feel free to share any findings you come across that I missed in this blog.

Happy Learning 🙂