Replication of data using SLT and SAP Data Intelligence Generation 2 Operators

Introduction

SAP Landscape Transformation Replication Server (SLT) is a product that allows users to replicate data between systems. While there are a lot of blogs out there covering SLT in general, I will focus on how SLT can be used with SAP Data Intelligence (DI) for replication.

SAP Data Intelligence is a product that is used to organize a heterogenous data system landscape from SAP systems and third-party systems. It is the main tool that I work with on the Data Management Team in Ireland. Within DI you have the option to build pipelines using Generation 1 Operators or Generation 2 Operators.

If you are using Generation 1 Operators, please have a look at this excellent blog by Martin Boeckling (https://blogs.sap.com/2021/07/20/replication-and-filtering-of-data-by-using-slt-and-sap-data-intelligence/).

In this blog, I will give you a step-by-step walkthrough of how you can build a pipeline in DI to replicate data from an SLT table and store it in any target using Generation 2 Operators.

Prerequisites

To follow this blog, you will need a SLT system (either a standalone DMIS system or a SLT version on an SAP S/4HANA system) and a DI system (on-prem or cloud) with an ABAP connection to the SLT system you wish to use. This SAP Note (https://launchpad.support.sap.com/#/notes/2835207) contains details of how to connect the two systems.

SLT

Create your configuration

In order to connect a SLT system to a DI system you must create a configuration within SLT using the SLT cockpit which can be accessed with the LTRC transaction. By clicking the paper icon (in the red square below), you can create the configuration in the pop-up window that follows.

SLT Paper Icon

Follow this blog (https://blogs.sap.com/2019/10/29/abap-integration-replicating-tables-into-sap-data-hub-via-sap-lt-replication-server/) to create the connection. At the end be sure to check you have configured everything correctly in the review screen. If everything is as necessary, click create to create the configuration.

SLT Create Button

Once the configuration has been created, you can access an overview and any current replications related to it using the glasses icon.

 

SLT Replication Details

Table replication

For this blog, we will use the SFLIGHT table to demonstrate this replication scenario. The table appears as follows within SLT:

SFLIGHT Table

SAP Data Intelligence

Integrate SLT configuration within SAP Data Intelligence

Now that we have our SLT Configuration made, let’s look at building the replication pipeline in DI. For this, we are required to use the Modeler tile from the Launchpad. In order to use this tile and to be able to view the replication results, we must have the following policies assigned to the user we wish to use: app.datahub-app-data.fullAccess, sap.dh.member, sap.dh.developer. (https://launchpad.support.sap.com/#/notes/2981615)

Data Intelligence Launchpad

We will now connect to the SLT system, replicate the data to a file in DI and view the results.

Build your pipeline

For this, we will use the Read Data from SAP System Operator which can be accessed from the “Operators” tab on the left, under the ABAP category. By right clicking the operator and selecting the view documentation option, you can read the details of the operator and its parameters.

Read Data from SAP System Operator

Now to configure the operator. To do this you can either right click and choose the open configuration option or you can use the shortcut that appears when you select the operator.

Open Configuration of Operator

This will open the configuration panel where we must specify the ABAP connection to be used to connect to out SLT system and choose the version of operator we wish to use.

ABAP Connection for SLT

Specify Operator Version

Once this has been specified, we will be able to see additional fields in the configuration panel. Namely, Object Name and Transfer Mode. When specifying the Object Name, we will need to select the Mass Transfer ID we previously created and once we select that, we can search for the SFLIGHT table within it.

Selecting the Table

Next, we must specify the Transfer Mode we wish to use for our pipeline. Here we have three options: Initial Load, Delta Load and Replication. These determine what data gets replicated when the pipeline is run. Initial Load will only replicate the table and not look for changes, Delta will only replicate over the changes that occur in the table. Additionally, Delta will add a flag to show what change occurred (I – insert, U – update and D – delete). Lastly, Replication is the Transfer Mode that does both. It first does the Initial load and then any changes that are made to the table also get replicated over.

Three Transfer Modes

You may notice that an output port is automatically added by DI for the table we are going to read.

Output Port Added

Since we want to write our table data into a file, we will have to firstly convert the table. To use the Binary file producer, our output from the Read Data Operator must be compatible with the input for the Binary File Producer. If we try to connect the two operators, we will see an error due to incompatible port types.

Incompatible Port Types

To fix this, we will use the Table to Binary operator to convert the table to binary before producing the file from it. We connect the output port labelled binary to the input port of the binary file producer as follows:

Converting Table to Binary

Now that we have matching ports, we just need to configure the Binary File Producer Operator using the Open Configuration shortcut as before.

Open Configuration

Binary File Producer Configuration

Firstly, we must specify which connection we wish to use for our target system. Let’s use the S3 connection for our example.

S3 Target

Next, choose the path mode you want to give and select your path to the target. If you wish for a new file to be created, just select the folder and add the file_name.csv at the end. After this the mode must be selected, this is what happens to the file. We have three options here, Overwrite (if the file already exists, its contents will be deleted and the new contents will replace them), Append (if the file already exists, the new data will be added to the bottom and old data will be kept too) and Create Only (no changes occur if the file already exists).

Write File Modes

That completes our pipeline!

Just for convenience though, we can add the Wiretap operator to the end to be able to view the records as they come in. This will display the traffic to the browser window.

Our completed pipeline should look like this:

Generation 2 Pipeline for SLT Replication

You can now save and run the pipeline using the controls highlighted in red above.

Note that you must specify the time to capture snapshots for the Read Data from SAP Operator to work. You do this by clicking the arrow beside the run button and choosing the Run As option from the dropdown menu.

Capturing Snapshots

For errors regarding the same Mass Transfer ID being used multiple times, this SAP Note may be helpful: https://me.sap.com/notes/0003204663

Once your graph is running, you will be able to see it in the status tab:

Running Graph Status

To see the records coming in, open the UI for the wiretap. This will open in a new tab. Your results should look similar to this:

Open Wiretap UI

Wiretap Results

And if you browse the connections using the Metadata Explorer, you should be able to see the new file in the location you specified. The catalog will only contain the published files and folders.

Using the glasses icon, you can view the data that has been replicated.

View Factsheet

The data should be visible to you under the data preview tab.

Data Preview

Conclusion

Congratulations! You now have a pipeline that allows you to replicate data from SLT to DI using Generation 2 Operators.

If you have any further questions, feel free to comment down below. Feedback is also welcome.