First steps with Google Cloud Cortex Framework
Since 2020 I had been experimenting with different services in Google Cloud Platform, partly because I’m a very curious individual and partly because at Diff Consulting, where I joined as their ALM Practice Lead, GCP is a key component for the company’s long-term strategy. We manage our whole infrastructure and SAP landscape in GCP, and we couldn’t be happier with the ease of use of such a stable and innovate technology stack that we get to use on daily basis.
A while ago I thought in start telling you about my “adventures” in GCP and I think this a great time to do so since lately I had been getting my hands on some new SAP oriented technology that Google is putting out there. This most likely will become a series, let’s see what the future brings but for this BLOG I will like to share with you how to leverage GCP, SAP SLT Replication Server, Google BigQuery and something very special called Google Cloud Cortex framework.
Let’s first define some key concepts:
SAP LT Replication Server
With SAP Landscape Transformation Replication Server, you can synchronize your business data in real time for your system landscape while keeping data transfer volume to a minimum. You can easily replace custom-built data synchronization applications with SAP Landscape Transformation Replication Server and benefit from the tight integration with the SAP NetWeaver ABAP Application stack and data model.
Source & use cases check here: SAP LT Replication Server
BigQuery is a fully managed enterprise data warehouse that helps you manage and analyze your data with built-in features like machine learning, geospatial analysis, and business intelligence. BigQuery’s serverless architecture lets you use SQL queries to answer your organization’s biggest questions with zero infrastructure management. BigQuery’s scalable, distributed analysis engine lets you query terabytes in seconds and petabytes in minutes.
Source: Google BigQuery
BigQuery Connector for SAP (SLT Add-On)
BigQuery Connector for SAP installs into SAP Landscape Transformation Replication Server (SAP LT Replication Server) and enables near real time replication of SAP data directly into BigQuery.
With your data in BigQuery, you can perform real-time analytics with artificial intelligence or machine learning on live SAP application data. In BigQuery, you can also integrate your SAP data with data from other sources.
Source: BigQuery Connector for SAP
Google Cloud Cortex
Google Cloud Cortex Framework is a foundation with endorsed reference architectures for customers to deploy technologies more quickly. Kickstart insights and reduce the time to value with reference architectures, packaged services, deployment templates, and accelerators to guide you from planning to delivery, and getting set up quickly.
Source: Google Cloud Cortex Framework
More info: Google Cloud Cortex Framework Data Foundation for SAP
Topic of today’s blog
Since this subject is quite complex and there is a lot of material for me to share, I will probably end up creating a series of blogs. For this one, I would like to explain how to leverage all the components that I mentioned before (SAP SLT Replicaton Server, BigQuery Connector for SAP and Google Cloud Cortex Framework/SAP Data Foundation), this will be done at a high level… otherwise we will ended up with a Tutorial/Manual and that’s not my intention.
In addition, I would like to share some reasons to get into the SAP on Google world and my first set of key take aways or conclusions around the POTENTIAL for the future.
If we try to take a look at this from a “helicopter view” there are some basic components to understand how the data flows from SAP into GCP. Here is my take on it:
- Data Selection: at this stage our main objective is to define which data we want to extract from our SAP Data Source (any SAP system). This could be existing tables or new ones that we want to create for our project.An advantage that the Google Cloud Cortex Framework gives us is 50+ pre-defined BQ operational Data Marts for SAP out-of-the-box, covering functional areas like Purchasing, Sales & Distribution and Master Data. I will recommend reviewing those at this stage and try to re-use as much as we can.
- Data Replication: at this stage we will focus on how to replicate our tables into Google BigQuery by leveraging SAP SLT Replication Server and the BigQuery Connector for SAP.Another important aspect is the frequency on how the data is replicated, this will depend on what do you have to do with the data but near real-time replication is possible (which is awesome!).An great aspect of the Google Cloud Cortex Framework is that it provides a clear approach on how to replicate data from SAP into BigQuery, so by taking this as a best practice you can’t fail 😉.
- Data Enrichment: at this stage we have our data set in BigQuery and is the time to enrich it. Here there are several aspects you can onboard, this to expand in other blogs but a few ideas that come to mind:
- Leverage data from other existing BQ data sets to enrich yours.
- Replicate other SAP and Non-SAP data sources and work with that.
- Do any kind of data preparation/processing using any programming runtime or additional GCP services.The sky is the limit 😊!
- Data Usage: this is the final stage and the one with the biggest potential because here is where you decide what do with your data set. A few ideas:
- Leverage BigQuery Reporting capabilities
- Use Looker or any other tool to create Dashboards and data visualization
- Use BigQuery ML to find new insights at your companyThe sky is limit here too 😊!
Now that we have some basic concepts explained, we can look at the following architecture and understand the role of each component:
Note: Architecture for SAP data sources on Google Cloud. We’re calling “SAP data sources” to any SAP systems from which I want to extract information. (ex: S/4HANA, ECC, etc.)
Original Source: https://cloud.google.com/solutions/sap/docs/bq-connector-for-sap-planning#architecture-on-gc
As we can see in the above diagram the data is being replicated via SAP SLT Replication Server + BigQuery Connector for SAP into Google BigQuery in GCP.
What’s key to highlight here is that to make these scenarios actionable you will require an approach or methodology to work. Here is where Google Cortex Framework comes in and simplifies our lives in a wonderful way, through the SAP Data Foundation to have a baseline of analytical artifacts and everything being deployed automatically.
I had the chance to have a sneak peek into this framework and I can tell you without a doubt that this is the right way for customers to maximize the use of their SAP data and for SAP / GCP partners to join Google into delivering the next generation of applications.
Some final comments for this first blog:
- For the most part we’re seeing Hyperscaler giving us their technology and telling us to figure it out. It’s refreshing to see Google taking the lead here, giving us a framework that’s intentionally designed to exploit the best of their services/solutions and the best of SAP.
- In addition to getting the framework, we’re also already getting specific content:
- Pre-defined BigQuery Operational Data Marts for SAP: Purchasing, Sales & Distribution and Master Data.
- BigQuery ML Templates
- Analytics Scenarios
- And much more coming up
- I had/have a lot of fun working in GCP, as many of you know I have more than +20 years in the SAP space and love working in our ecosystem – now I have the opportunity to extend portfolio of service to work with all the cool technology that a company like Google brings into the Enterprise space. I see a bright and fun future ahead of all of us.
- As I mentioned before, I will try to write more blogs in the future about this fascinating topic as I’m still learning, and this is just the beginning.
- Last but not least, I recently took an openSAP training where SAP and Microsoft explore the areas for collaboration between Azure and SAP BTP. Many wonderful ideas came from the reference architectures that I found in that training, I wish and look forward to seeing a similar openSAP training between SAP and Google – I think that the potential for synergies between the platforms are tremendous.
I’m leaving you with some learning material that was very useful for me so far:
Accelerate SAP and Google Cloud innovation and value
Do we have any connector for ODP/ODQ?