Overview
In Part 1 of this blog series, I introduced how to use CI-DS to load full data from S/4HANA with Generic Extractor. This blog post is Part 2 to explain how to use CI-DS to load delta data from S/4HANA with Generic Extractor.
Scenario
We would like to use CI-DS to load delta Product master data from S/4HANA to IBP with custom Generic Extractor.
To make it simple, we focus on the bellowing.
- S/4HANA side: Define a simple Generic Extractor based on DB view to extract delta Product master data.
- CI-DS side: Configure the data flow and mapping to call the Extractor.
Configuration
1. S/4HANA side: Define a simple Generic Extractor based on DB view to extract delta Product master data
Generic Extractor can be defined based on DB View, SAP Query or Function module. Here we gave a simple example to define Generic Exactor based on DB View.
1.1. Define DB View (Tcd: SE11)
(Skip 1.1 if you have already defined the DB View.)
Select MARA and MAKT table for the table join.
Configure the fields of the DB View.
Set the selection condition.
1.2. Define Generic Exactor (Tcd: RSO2)
Create new Data source(Extractor) based on the above DB View.
Define Selection and Hide fields and save.
Set date field for the delta data load.
1.3. Test Extractor with ABAP Report program RODPS_REPL_TEST.
2. CI-DS side: Configure the data flow and mapping to call the Extractor
2.1. Import the definition of the Extractor into the Data Store of S/4HANA.
Subscriber
Subscriber identifier to fetch data from the delta Queue
Extraction Mode
Query: Full data load
Change Data Capture: Delta data load
2.2. Define datetime type Global Variables for delta data load which saving the timestamp of data load.
2.3. Write Preload and Post scripts to get and store the timestamp of the delta data load.
In Preload script, get the stored timestamp of last delta data load.
Preload script:
print('#####################');
$G_START_DATE = to_date(get_data('EXTRACT_PRODUCT_DELTA'), 'yyyy-mm-dd hh24:mi:ss');
If ($G_START_DATE is null)
$G_START_DATE = to_date('1900.01.01', 'YYYY.MM.DD');
print('INFO - $G_START_DATE: '|| $G_START_DATE);
$G_END_DATE = sysutcdate();
print('INFO - $G_END_DATE: '|| $G_END_DATE);
In Postload script, store the timestamp of delta data load this time.
If task failed, the timestamp of delta data load will be not stored. And the delta data load can repeated with the last stored timestamp.
Postload script:
print('#####################');
save_data('EXTRACT_PRODUCT_DELTA', to_char($G_END_DATE,'yyyy-mm-dd hh24:mi:ss'));
Configure Extractor Option in the mapping of the data flow.
Package size
Indicates the maximum number of rows the extractor reads from the source and loads into memory at one time. Once the system processes and loads these rows to the target, it reads the next set of rows. By limiting the number of rows, less memory is used. Default is 1,000.
Initial load
For changed-data capture (delta loads), indicates whether to reload all the data on a subsequent run.
Yes: Returns a full extraction
No: Returns only data that has changed since the last run
Extract from datetime
Indicates a specific date and time for when to extract changed data. Select a predefined global variable of type datetime. If the datetime value is the same as the value from the last execution, or falls before the value from the last execution, the system repeats the last changed-data extraction.
If the datetime value is later than the value from the last execution, the system returns the new data.
Parallel process threads
Specifies the number of threads used to read the data.
We recommend that you don't use this option. Setting a value can cause the software to go into recovery mode after the first iteration, resulting in sending the same rows repeatedly.
After CI-DS task executed, you can use Tcd:ODQMON in S/4HANA to check the data extracted.
Conclusion
I hope this blog post was helpful to understand how to use CPI-DS to load delta data from S/4HANA with Generic Extractor. About full data load with Generic Extractor, please refer to Part 1.
- SAP Managed Tags:
