SAP Data Intelligence and Data Flow Functionality in SAP Data Warehouse Cloud- When to use what
SAP Data Intelligence is our unified data management solution for data orchestration, data catalog and integration. Since some of these functionalities come in handy also in other SAP tools, we started to offer these needed functionalities as an embedded version of Data Intelligence. The best example for this is the Data Flow Capability which is now available in the Data Builder in SAP Data Warehouse Cloud. Even though this embedding strategy is great it sometimes leads to misunderstandings and confusion for some customers. The most common rumors I hear are “Data Flow can do everything that Data Intelligence can do” or “Now I don’t need Data Intelligence anymore”. Let me quickly put this right: both statements are wrong; data flow is not the new full blown Data Intelligence. But this confusion was the main reason for me to write this blog post. I hope it will help you to understand better what you can do with the Data Flow Functionality in SAP Data Warehouse Cloud and when you should use Data Intelligence.
The main purpose of the Data Flow is to ingest data into SAP Data Warehouse Cloud. It comes at no additional cost for SAP Data Warehouse Cloud Users and you don’t need any additional external tools since it is directly available in the Data Builder in SAP Data Warehouse Cloud. The basis for it is Data intelligence but it doesn’t offer all the functionalities. There are three main differentiators:
- There are no operators available to put machine learning into production. The available Python operators offer only data transformation options.
- You can integrate data from various systems into SAP Data Warehouse Cloud but not ingest it into other systems or a distributed IT landscape.
- Furthermore, SAP Data Warehouse Cloud has a different user group than SAP Data Intelligence that is why Data Flow has a simplified UI. The typical SAP Data Warehouse Cloud User is most of the time a Business User, analyst and not a developer that is why it focuses on simplicity and ease of use.
Wait a moment Sarah. You mentioned that I can’t do Machine Learning with Data Flow, but you can’t tell me there is no option to do ML with SAP Data Warehouse Cloud. Sure, you are wright there is an option you can use the Hana Cloud in SAP Data Warehouse Cloud for this. The details are described in this Blog Post.
Now let’s have a look at the full-blown SAP Data Intelligence. With this solution you are not limited to bringing data to SAP Data Warehouse Cloud, you can streamline all data integration needs to cover all scenarios also beyond SAP Data Warehouse Cloud. You can integrate any kind of data including streaming and unstructured data. Based on this you can catalog your data to get an overview of all the available data and access it from SAP Data Warehouse Cloud and all other data sources in your organization. You also have robust metadata management to ensure your data quality, check the lineage of your data and with this trust the accuracy of your data. Furthermore, you can use Data Intelligence to orchestrate any kind of processing including Machine Learning and external engines into your SAP Data Warehouse Cloud or other environment. In the Data Intelligence you have R and Python operators available that you can use to put your own R and Python code into production.
I hope this helps you to deal a bit better with such rumors and to choose the right tool for your use case. Of course, you can always contact me when you need more support, I am more than happy to discuss your Machine Learning use cases further with you.