Recently I have been working on a POC in BW/4HANA & S/4HANA for one of the biggest Consumer goods Industry, I got an opportunity to explore this new Framework of ODP and I like to share few interesting information on the same.
The operational delta queues are a new framework which enables multiple subscribers to replicate data from a set of queues of a provider.
The standard BW Extractors (SAPI) present in ECC or S/4 systems have been made ODP compatible via the mass ABAP program ‘BS_ANLY_DS_RELEASE_ODP’ (SAP Note 2232584) (This is a one-time activity). These extractors are not only used in context of BW but also for other targets (like SAP Data services, OData etc.). However, with BW/4HANA as the data target, only ODP extractors are supported.
Overview of the ODP framework.
Operational Delta Queue is a physical Queue (T-Code: ODQMON), present in the source system. It is replacement of traditional RSA7 Delta Queue.
The Source of ODQ are called Providers and the Target systems are called Consumer or Subscriber. ODQMON stores data in compressed format and is also a replacement of traditional PSA.
Target system (Subscriber) can request different types of data requests:
Below are some advantages of the ODP Framework:
The Operational Delta Queue is mainly used for the below delta types for ODP extractors:
Delta Type ‘D’ – The SAP application writes delta records directly into ODQ (PUSH) for ODP extractors with delta type ‘D’. Each data record is either stored in the ODQ individually on saving/updating the corresponding transactions in the application (for example, direct delta in the LO Cockpit) or is written in groups of delta data records (after updating the transaction) to ODQ using application-specific jobs (LO Queued delta). In each case, the delta data records are in the ODQ for the SAP source system before the delta update is requested from the target system (for example BW/4HANA). In the case of delta updates for the DataSource, the ODQ is read, and the data records that exist there are transferred to the target system (for example, BW).
Delta Type ‘E’ – The ODP data source determines the delta through the extractor on request. The extractor must be capable of providing the delta records for the DataSource on demand (PULL). The delta data records that have been determined are placed in the ODQ by the extractor and are transferred from there to the requesting target system (for example BW). If you have carried out delta initialization for more than one SAP BW system, or several delta initialization for the same BW system, the extractor has to stage delta records for all the delta initialization and store them in all the ODQs that are specific to the target systems (for example BW/4HANA.) This means there is a one-to-one relationship between each ODQ and each subscription.
Above, shows a comparison of two different SAP BI content DataSources with different delta processes and how ODQ is filled with delta records:
The ODQ uses the following three tables to store data:
DTP’s fetch data directly from the ODQ. The first time you run a DTP, the ODQ performs a delta initialization where a request for the DataSource is generated, and table ODQDATA_C is filled. In the second run, the ODQ does a delta update with those records that were created or changed (including deletions) since the last load. The delta records are stored in the ODQDATA table in a compressed format.
There are three different choices for Retention:
To recover a delta process that has been canceled:
For Data with Low Relevance:
For Data with Medium Relevance:
The ODQ Datasource comes with two additional default fields:-
These two fields are responsible for determining the RECORDMODE in BW/4HANA system.
Please check the below list of values in these two fields and how it is helping to determine the RECORDMODE in BW.
If ODQ_CHANGEMODE is 'U' it means this is a changed record and ODQ_ENTITYCNTR can be 1 or -1 which represent RECORDMODE as ‘ ’ (After Image) or ‘X’(Before Image) in BW/4HANA respectively.
These two fields are mapped to the BW Transformation to RECORDMODE field and is responsible to populate the field as discussed above.
Difference noticed compared to old 7.x Service API Datasource.
When we are activating any logistics application datasources for the very first time. It needs to be initialized first to create the subscription in ODQMON and then only we will be able to fill Set-up table.(Unlike in old 7.x API datasources).
However if you still want to fill the set up table without Initializing in BW, then the Program RODPS_REPL_TEST will help you create dummy subscription in ODQMON and then Set-up table can be filled (useful in testing environment).
This step is not necessary for every LO datasources, but necessary for the first Datasource of any Application Component (like 2LIS_11*, / 12* etc.)
The mentioned program (RODPS_REPL_TEST) is also very useful while testing ODP Datasource.
http://learningsapbw1.blogspot.com/2018/03/odqmon.html
https://wiki.scn.sap.com/wiki/display/BI/Introduction+to+Operational+Delta+Queues
The Operational Data Provisioning (ODP) Framework
The operational delta queues are a new framework which enables multiple subscribers to replicate data from a set of queues of a provider.
The standard BW Extractors (SAPI) present in ECC or S/4 systems have been made ODP compatible via the mass ABAP program ‘BS_ANLY_DS_RELEASE_ODP’ (SAP Note 2232584) (This is a one-time activity). These extractors are not only used in context of BW but also for other targets (like SAP Data services, OData etc.). However, with BW/4HANA as the data target, only ODP extractors are supported.
Overview of the ODP framework.
ODP Framework
The Operational Delta Queue (ODQ)
Operational Delta Queue is a physical Queue (T-Code: ODQMON), present in the source system. It is replacement of traditional RSA7 Delta Queue.
The Source of ODQ are called Providers and the Target systems are called Consumer or Subscriber. ODQMON stores data in compressed format and is also a replacement of traditional PSA.
Target system (Subscriber) can request different types of data requests:
- Subscribers can request a data snapshot (full or one-off request). This is supported by almost any BW DataSource and does not require a subscription. (Queue can be in ODQMON).
- Subscribers can request data changes (delta request). This requires to create a subscription first (delta initialization) and usually to transfer the initial data. This is supported by many BW DataSources, in one of the following two ways:
- The application pushes data into the delta queue (e.g. 2LIS_* Datasources).
- The delta queue pulls data into the delta queue via an extractor (e.g. 0FI_* Datasources).
Below are some advantages of the ODP Framework:
- If you use ODP, you can load the data directly into the BW InfoProviders, bypassing the Persistent Staging Area (PSA) layer by using a Data Transfer Processes (DTP)
- The ODP infrastructure (with delta queues) takes over important services such as monitoring data requests.
- The data is stored in a highly compressed (>90%) state in the delta queue. A delta request transfers data records from the queue to the subscriber (target system).
- The data changes to a queue can also be requested by more than one subscriber (target system)
- The data is retained in the delta queue for a specified time period for recovery purposes.
Delta Type for ODQ Extractors
The Operational Delta Queue is mainly used for the below delta types for ODP extractors:
Delta Type ‘D’ – The SAP application writes delta records directly into ODQ (PUSH) for ODP extractors with delta type ‘D’. Each data record is either stored in the ODQ individually on saving/updating the corresponding transactions in the application (for example, direct delta in the LO Cockpit) or is written in groups of delta data records (after updating the transaction) to ODQ using application-specific jobs (LO Queued delta). In each case, the delta data records are in the ODQ for the SAP source system before the delta update is requested from the target system (for example BW/4HANA). In the case of delta updates for the DataSource, the ODQ is read, and the data records that exist there are transferred to the target system (for example, BW).
Delta Type ‘E’ – The ODP data source determines the delta through the extractor on request. The extractor must be capable of providing the delta records for the DataSource on demand (PULL). The delta data records that have been determined are placed in the ODQ by the extractor and are transferred from there to the requesting target system (for example BW). If you have carried out delta initialization for more than one SAP BW system, or several delta initialization for the same BW system, the extractor has to stage delta records for all the delta initialization and store them in all the ODQs that are specific to the target systems (for example BW/4HANA.) This means there is a one-to-one relationship between each ODQ and each subscription.
ODQ Delta Type
Above, shows a comparison of two different SAP BI content DataSources with different delta processes and how ODQ is filled with delta records:
- Datasource 2LIS_02_ITM (Purchasing data (item level)) with delta type D (push):
- As this is delta type PUSH, the delta data records from the application are posted to the delta queue before they are extracted from ODQ as part of the delta update. If the DataSources are from the LO cockpit, this push takes place either with a direct delta (directly from the application, V1 – updating) or with a queued delta (from the application to the extraction queue (V1), and from there to ODQ with a collective run).
- Datasource 0FI_GL_4 (GL: Line items) with delta type E (PULL):
- As this is a ‘PULL’ delta-type, the delta data records are determined during the delta update by the Datasource extractor, updated to the ODQ, and passed on to the SAP BW directly from there.
ODQ Tables
The ODQ uses the following three tables to store data:
- ODQDATA_C Contains compressed Init. request data
- ODQDATA_F Contains compressed Full request data
- ODQDATA Contains compressed Delta request data
DTP’s fetch data directly from the ODQ. The first time you run a DTP, the ODQ performs a delta initialization where a request for the DataSource is generated, and table ODQDATA_C is filled. In the second run, the ODQ does a delta update with those records that were created or changed (including deletions) since the last load. The delta records are stored in the ODQDATA table in a compressed format.
ODQ Retention Period
- The data in ODQ is retained for reconciliation and recovery.
- We can have control over how long to keep the queue’s data after it has been successfully sent to all targets.
- The default is 24 hours for any data in the queue that is flagged as retrieved or as cancelled.
- The job to reorganize delta queues is created by default when delta initialization request is executed from the subscriber, e.g. BW/4HANA or SAP Data Services.
- The time and schedule of the job can be changed manually by program ODQ_CLEANUP or by selecting “Reorganize delta queues” under the "Goto" menu in transaction ODQMON.
ODQ Retention
There are three different choices for Retention:
To recover a delta process that has been canceled:
- This minimum retention period for data in the queue tables that is flagged as retrieved in the delta process or cancelled.
- The default setting is 24 Hours
For Data with Low Relevance:
- It has not yet been declared as retrieved or invalid
- All subscribers have subscribed to it with low relevance. This period is given in days. The default is 10 days.
For Data with Medium Relevance:
- It has not yet been declared as retrieved or invalid
- All subscribers have subscribed to it with at most medium relevance.
- Delta request is kept in Medium relevance as it is required for a longer time
Data Relevance
- Data that has not yet retrieved and is business critical is never automatically deleted by reorganization process until it is flagged with retrieved or invalid.
- At present system does not make any relevance related distinction of delta data and this relevance is automatically given by the system based on the type of the request.
- Because of particularly high volume expected data from delta Initialization request and standard request is classified as low relevance.
- Delta request is kept in Medium relevance as it is required for a longer period of time.
Record Mode Logic
The ODQ Datasource comes with two additional default fields:-
- ODQ_CHANGEMODE
- ODQ_ENTITYCNTR
These two fields are responsible for determining the RECORDMODE in BW/4HANA system.
Please check the below list of values in these two fields and how it is helping to determine the RECORDMODE in BW.
RECORDMODE Logic
If ODQ_CHANGEMODE is 'U' it means this is a changed record and ODQ_ENTITYCNTR can be 1 or -1 which represent RECORDMODE as ‘ ’ (After Image) or ‘X’(Before Image) in BW/4HANA respectively.
RECORDMODE mapping in BW TRFN
These two fields are mapped to the BW Transformation to RECORDMODE field and is responsible to populate the field as discussed above.
Few Takeaways:-
Difference noticed compared to old 7.x Service API Datasource.
When we are activating any logistics application datasources for the very first time. It needs to be initialized first to create the subscription in ODQMON and then only we will be able to fill Set-up table.(Unlike in old 7.x API datasources).
However if you still want to fill the set up table without Initializing in BW, then the Program RODPS_REPL_TEST will help you create dummy subscription in ODQMON and then Set-up table can be filled (useful in testing environment).
This step is not necessary for every LO datasources, but necessary for the first Datasource of any Application Component (like 2LIS_11*, / 12* etc.)
The mentioned program (RODPS_REPL_TEST) is also very useful while testing ODP Datasource.
Reference
http://learningsapbw1.blogspot.com/2018/03/odqmon.html
https://wiki.scn.sap.com/wiki/display/BI/Introduction+to+Operational+Delta+Queues
- SAP Managed Tags:
3 Comments
