This article is part of our series “Give Data Purpose Weekly” . In this article I share my thoughts why traditional on-prem data management is not sufficient for the management of cloud data sources.
INTRODUCTION: DATA MOVES TO THE CLOUD
Many organizations have moved core applications to the cloud. With every application moving to the cloud, there is a significant amount of critical company data moved alongside.
According to Gartner, 75% of all databases will be deployed or migrated to a cloud platform by 2022 (Gartner: “Gartner Says the Future of the Database Market Is the Cloud”). Companies and the heads of data management in particular, are being confronted with complex hybrid data landscapes comprising both on-prem & cloud data stores. In this document we outline the challenges and solutions for a successful cloud data management transformation.
Cloud data is different
Access and processing of these new cloud data stores, however, is quite different from fully controlled, operated, and managed on-prem RDBMS systems. It requires new methods.
Furthermore, additional cloud datatypes and significant amount of data coming from social media, web sites, IoT and streaming devices are available. They exceed the traditional amount of data processed by companies today and need new concepts of data processing and data storage.
Managing cloud data leads to two key questions:
- Are my current data management solutions sufficient for new data complexity?
- What is the right data strategy: (a) Extend my current landscape to the cloud or (b) recreate it from scratch in the cloud?
If you're experiencing the same challenges and like to identify your Cloud Data Management approach, please get in touch with our SAP EXPERTS:
eMail us to arrange a Cloud Data Management workshop appointment!
CLOUD DATA MANAGEMENT FOR CLOUD DATA
Follow the data gravity
Now that most data is moving to the cloud, the adjacent data management topics need to be pulled to the cloud as well to be the new center for all data and data management functions.
“The center of data gravity is shifting to Cloud” said Adam Ronthal from Gartner in a 2021 Webinar.
Why on-prem data management is no longer sufficient?
Most data management solutions are from the early 2000s with a strong focus on data in on-prem applications. They have mainly been designed to integrate, improve, and synchronize data between on-prem applications. Such traditional ETL/ELT, replication, synchronization and data quality tools run in local data centers.
Although these traditional data management solutions still have a significant relevance, their reach to cloud data is "limited" for the following reasons:
- Cloud applications provide new & different APIs
- Cloud applications (usually) do not allow database level access
- Cloud applications run in a different network
- Cloud applications require new ways of authorization & authentication
A new approach is needed to manage on-premise and cloud data in a unified way.
DATA MANAGEMENT IN THE CLOUD: CHANGE IS REQUIRED!
Cloud data scalability is tempting
PaaS offerings tempt companies to generate large data stores without a strategy to process and apply this scale of data. Today we find data lakes of petabyte size in many companies storing unused “dormant” data. Since they lack a proper cloud data management strategy and software tools, this data remains unused and does not generate any value for the company.
Realtime self-service expectation
The intuitive real-time UX of modern mobile apps, cloud applications, and internet services has developed an according user expectation for data management tools. Users expect self-service and fast availability of any data product instead of predefined, static data marts. Users want secure, business user friendly tools to access corporate data sources (on-premise & cloud). Accordingly, data management must shift from an IT-centric tooling to a business user focus.
THE GOAL:
CLOUD DATA MANAGEMENT FOR ALL DATA SOURCES
Only a Cloud Data Management platform can address all demands to data management for both cloud and on-prem data without disruption. A Cloud Data Management platform is comprised of four main capabilities:
Cloud Data Integration
Source/Target processing with multiple exchange patterns: ETL/ELT, Replication, Virtualization, Streaming
Cloud Data Quality
Data Object profiling and improvement utilizing a data quality rule framework
Cloud Data Catalog
Metadata sourcing, discovery, storage, search and data lineage
Cloud Data Store
Used to persist all relevant data as a source for analytics or new Cloud applications
All capabilities should utilize artificial intelligence wherever meaningful. Self-service is an overall supported pattern to reduce dependency from IT experts and increase agility.
GO "HYBRID" IN DATA MANAGEMENT
“Hybrid” in the context of data management means expanding an existing on-prem data management to the Cloud to leverage the best of two worlds.
A hybrid data management environment complements the existing on-prem data management solutions with a Cloud Data Management platform containing new cloud data capabilities and functionalities to orchestrate, invoke and re-use components from the on-prem data management solutions. Data virtualization plays a key role in a hybrid data management strategy. A virtual database with a unified data model and a single interface for accessing data residing in different distributed data models provides quick access to on-prem data sources and avoids complex data integration effort.
Any new data management requirement should be implemented and configured in the Cloud Data Management platform ("Cloud first" approach). The legacy data management platform will continue to serve “on-prem only” requirements.
Artefacts from the legacy data management platform will be migrated over time whenever appropriate.
Eventually the on-prem data management platform can be switched-off without any risk and the Cloud Data Management solution will be the single future-ready platform to handle on-prem and cloud data management requirements.
SAP SOLUTIONS FOR HYBRID DATA MANAGEMENT
SAP Data Intelligence Cloud
SAP Data Intelligence Cloud is SAP’s comprehensive Cloud Data Management solution. As the data orchestration layer of SAP’s Business Technology Platform, it transforms distributed data sprawls into vital data insights, delivering innovation at scale.
It consolidates data from heterogeneous on-prem and cloud sources, provides management of IoT data streams, and facilitates scalable machine learning. SAP Data Intelligence enables business applications to deliver the intelligent enterprise vision and provides a holistic, unified way to manage, integrate, and process all your enterprise data
SAP HANA Cloud
SAP HANA Cloud provides a single place to access, store, and process all enterprise data in real time.
It is a cloud-native platform that reduces the complexity of multi-cloud or hybrid system landscapes. SAP HANA Cloud provides all of the advanced SAP HANA technologies for multi-model data processing in-memory or on disk. You can benefit from cloud qualities such as automatic software updates, elasticity, and low total cost of ownership by using SAP HANA Cloud either as a stand-alone solution or as an extension to your existing on-prem environment.
Orchestrate & re-use artefacts from SAP Data Services, SAP Information Steward &
SAP Landscape Transformation Replication Server
SAP HANA Cloud and SAP Data Intelligence Cloud provide the best-in-class connectivity to existing on-prem data management solutions from SAP. SAP Data Intelligence can orchestrate any ETL job build in SAP Data Services. Business terms and data quality rules form SAP Information Steward can be imported into SAP Data Intelligence to be re-used for cloud native data sources. Data from on-premise ABAP data sources can be replicated to cloud native targets utilizing SAP Data Intelligence replication flows together with on-prem SAP Landscape Transformation Replication Server.
GET ENGAGED!
eMail us to arrange a Cloud Data Management workshop appointment!
We'd like to discuss your current data management challenges, your landscape and your roadmap towards the Cloud to identify your best Cloud Data Management journey!
SOMETHING TO READ?
General overview
What is SAP Data Intelligence Cloud?
What is SAP HANA Cloud?
Additional BLOGS
SAP Data Services, SAP Information Steward, and SAP Data Intelligence: Strategy …
SAP Data Services & SAP Data Intelligence: Optimized Data Management with Hybrid Solution
A Hybrid Use Case with SAP Data Intelligence and SAP Information Steward
SAP HANA Cloud Demo:
Data Virtualization & Relational Data Lake with AWS & Azure services
Free Software Trial
SAP Data Intelligence | SAP HANA Cloud
SAP Community
SAP Data Intelligence | SAP HANA Cloud
openSAP Micro Learning
SAP Data Intelligence | SAP HANA Cloud
Do you also want to give more purpose to your data?
Have a look in our popular paper “12 ways to give data purpose in a multi-cloud world”, where we provided a quick preview into 12 main ways to overcome typical hurdles that keep companies from using their data.
If you have any questions about SAP Business Technology Platform or data & analytics, leave a question in SAP Community Q&A or visit our SAP Community topic page.
- SAP Managed Tags:
2 Comments
