Introduction
Often we come across scenarios where we need to replicate large scale Enterprise Data to Cloud Storage.
One way of doing this in the SAP Landscape is using SAP Data Intelligence (SAP DI) Cloud to facilitate the replication of ECC data.
For this blog we will be focusing on using the Replication Flow (RMS) which is the out of the box data replication solution on the SAP DI.
We will be replicating data from SLT system to an Amazon S3 bucket replicating our ECC tables.
Prerequisites
For following this blog you will need to establish connection between your SLT system and the SAP Data Intelligence Cloud .
Following SAP Note details how this connection can be achieved : https://me.sap.com/notes/2835207
Configuration
SLT
On the SLT side we will need to create a Mass Transfer Id which will be used by the SAP DI to connect to the data source which will be our ECC system.
New SLT Configuration
On the Specify Transfer settings tab make sure to select "SAP Data Intelligence(Replication Management Service)" . Selecting other options will make so that the RMS flow will not be able to connect to the ECC system via SLT.
select SAP Data Intelligence(Replication Management Service)
Detailed blog on creating this SLT configuration can be found here : https://blogs.sap.com/2019/10/29/abap-integration-replicating-tables-into-sap-data-hub-via-sap-lt-re...
Amazon S3
We will also need to establish a connection with the Target system which is an S3 bucket.
Create a new AWS S3 connection from DI
Connection
Check Connection
Create the RMS Flow on SAP DI
We go to the Modeler section on the DI tenant and select the Replication Tab
Create a new RMS Flow
We now see the properties tab on creating a new RMS flow and entering a name for it.
We can not enter the description (optional) and select the Source and target Connections along with the containers.
For target we select the S3 bucket along with the container. Here you can enter a new container or select a pre-existing one on S3.
RMS Properties
We can group the delta records in our RMS flow by date or hour.
File types currently supported are csv, parquet, json and jsonlines.
We can also select a file compression from gzip or snappy.
Selecting Tables to be replicated
We select the objects to be replicated in the Tasks tab
Some of the options given are:
Source Filter : We select if a filter is need based on which data will be considered for replication.
Mapping : We can map existing Table fields to Custom fields, Expressions on the target system.
Target : This is the target container on the S3 bucket.
Load Type : We can choose between Initial Only and Initial and Delta.
Note : If we select a container having existing data on the S3 bucket then we will need to select the Truncate option. This will delete all the existing data in the existing container before starting the replication. Not doing this will result in the RMS flow giving an error and not starting.
Activating the RMS Flow
Validate the RMS Flow
After successful validation we can go ahead and activate and run the flow.
Deploy the RMS Flow
Once the RMS flow is executed we click on the "Go to Monitoring" Button to navigate to the DI Monitoring Screen.
In that screen we gain access to the metrics of the RMS flow executed along with the current status and health of the RMS flow.
Monitoring
Files on the Amazon S3 Bucket
On replication files are created on the S3 bucket under the Initial and Delta folders in the defined Target containers in the RMS Flow
Note: Delta folder is created only in cases of Initial and Delta Load Types
S3 Initial Load Folder
S3 Delta Folder
Conclusion
This is a basic use case explaining the replication of ECC tables on the Amazon S3 bucket using RMS. We can also use complex scenarios and objects such as CDS views.
Thank you for reading. Feel Free to share and comments and feedback
- SAP Managed Tags:
