Digital Transformation – Architectures and Applications – 1 – Data Lift
“The only constant in life is change” – Heraclitus. Greek philosopher
While Heraclitus wise words is true for any time, it feels like it is more relevant now than ever. We are in times where world is changing rapidly than ever before and businesses as well as individuals and are trying to keep up with the change. If i think about some of the aspects of change, some things that comes to my mind are
- Ever changing consumer requirements and a greater need for better customer engagement and experience.
- Constant need for running business processes in an optimal way and eliminating redundancies.
- Ability to adopt continuously and efficiently to external changes and mitigating risk.
- Ability to quickly adopt to new and changing environments, policies and regulations.
- Ability to to work collaboratively with partners to envision and achieve greater results.
- Employee wellbeing and ability to meet the workplace expectations of next generation for greater employee satisfaction and engagement.
- Ability to function as a data driven organization by efficiently utilizing data and leveraging insights from data to drive operations and actions.
- Social responsibility and care for our community and our planet.
- and many more…
But if i think about it, can’t every change provide an opportunity at the same time? While we all can agree with Heraclitus that change is constant in the world, is it possible to embrace it as an opportunity and work with it in a gracious way to benefit from it. What do we need to do be able to embrace the constant changes? The answer for this is in the change itself. Most of the changes are due to external factors on which we may have little or no control. For example rapid growth in digital economy brought digitization to every aspect of our life leading to digital businesses, digital processes, digital consumers, digital markets, digital workforce, and digital partners. This digital growth also lead to a data explosion that brings a need to process, store and analyze enormous amount of data to get the insights that we need. If digitization is causing the change, it also provides an opportunity to deal with the change in the form of “digital transformation”. Individuals and organizations can find digital transformation as the solution for the constant changes that digital economy brings.
If individuals and organizations digitally transform themselves, it should empower them to embrace any change. When i say digital transformation, i was referring to transformation of business processes, not just technology transformation. One of the important aspect of digital transformation is to have a platform to run our business processes that is equipped to facilitate the digital transformation journey, a platform that is very robust to run business processes in a reliable way and very agile to accommodate rapid changes. A platform that can fulfill organization’s goal to become a data driven company by providing robust and agile data management abilities. A platform that gives the ability to accommodate new business models within less time. We also need to run business processes using applications and architectures that can accommodate changes easily. I am writing this blog series to share some of my views on some of these aspects.
In this blog i want to discuss an architecture i was working for some time. Many times, for SaaS applications the data of a SaaS application is accessible through reports and APIs. While the reports and APIs is good to analyze the data, there will be some use cases that require more in-depth data analysis or use cases where the data from SaaS application need to be analyzed together with other data. For discussion, i took the example of SAP cloud platform integration (SCPI), the cloud based integration as service that integrates any SAP on-premise and cloud solutions and any non-SAP applications. SCPI comes with rich integration features for any enterprise integration needs. It also provides out of the box plug and play type of integration packages for integrating all SAP applications and major non-SAP applications.
In this blog i will discuss a use case where i want to analyze the CPI integration data along with other data to get better visibility and deeper insights in to the integrations. Below is the architecture that helps with this need. This extracts the data from CPI using APIs and persist it to HANA cloud database on SAP cloud platform. I named this architecture as “data lift” as we are lifting the data from one place and copying to a different place. There are few technical pieces that facilitate this which will be discussed in detail below.
Data extractor is a recurring job that triggers CPI i flow and extracts data using APIs. CPI provides robust OData APIs for extracting any required data. Every OData API comes with a timestamp field which can be used for incrementally extracting the data.
SAP HANA cloud:
Data is persisted in the SAP HANA cloud database in SAP cloud platform.
An application is deployed on SAP cloud platform cloud foundry environment. Node.js backend application helps with data ingestion in to HANA database and UI5 application provides the required user interface for the application
SCPI custom adapter:
Applications running on SAP cloud platform Cloud Foundry is used by end users and secured using OAuth 2.0 standard. For accessing the application from CPI, OAuth 2.0 password grant is used. So a generic custom adapter is developed that takes care of accessing cloud foundry application from CPI. Technically the same can be done without a custom adapter using two API calls, one for token and another one for sending data to the application. I used a custom adapter for simplicity of integration.
I will conclude this blog here hoping you find it worth the time and not too boring.
This blog should leave you with a viewpoint on changes in current digital age and how digital transformation can help to embrace change. It discusses an architecture that extract data from CPI to SAP HANA cloud using SAP cloud platform. The architecture helps with use cases that need deeper analysis in to integration scenarios and combined analytics with other data.
Please do share your valuable comments if you like the blog.