Some organizations require making decision based on real time up to-date details regarding certain process or product information. For Example: Stock markets, Communications, Defense, Aeronautics, Supply chain management etc. Here each of and every transaction data matters the most in making a decision at a priority level, and it’s very crucial to transfer data to the reporting side at shorter & regular intervals.
RDA Vs Direct Access
RDA: Extracts data from BW with a high frequency from source systems & migrates it to BI system.
Direct Access: DA to source systems. Data is not physically migrated into BI, but read rights for data analysis are directly transferred to respective Data source
Basics of RDA
Real time data acquisition which supports tactical decision-making is a framework for deriving information from data as data becomes more complex.
- Time factors: Lower time scale than for scheduled/batch data acquisition
- Scalability: Stream oriented
- Frequency: Immediate availability for reporting
Frequency is data transfer is quite high. Follows regular type of data transfer, where extracted data runs through extraction & transformation process. Although not true real-time, data transferred is physical data
Example: RDA using SAP Service API
We can also set the features for SOAP Runtime & publish a Web service (as a business service in the UDDI)
Fig. A. Existing Example: RDA using SAP Service API
Existing Data Flow Concept – Example
The Fig. B illustrates the regular data flow example which we regularly follow to use.
Fig. B. Existing Data Flow Concept – Example
New Data Flow Concept – RDA using SAP Web Services API
Using new functionality of Real-time Data Acquisition (RDA) in BI 7.0 system we can now load transactional data into SAP BI system every single minute. The source system for RDA could be SAP System or it could be any non-SAP system, as most of the Standard Data Sources are real-time enabled.
Real time enabled Data Sources could only be defined since the release of BI Service API of SAP Net Weaver 2004s. For SAP 4.6C with plug-in 2004.1, this option has been available since Service pack 10.
We must start the modeling by creating an Info package & DTP for Real time data acquisition, wherein the type of adapter used for both of them must be ‘Web Service (Push)’. Refer Fig. D
Fig. D. Modeling View
RDA Example Using Service API – Administration View
The following Fig. E illustrates the administration view, also better known as Real-time data acquisition using the transaction code RSRDA. Here we must first create a daemon in the RDA monitor, which will generate an open request waiting for transactional data from the SAP system. We must further enter data through a proper channel like say Web service API to trigger the data acquisition process.
Fig. E. Administration View
Note: Info packages for real time data acquisition can be defined with SAP source systems only if the delta procedure was initiated and data transfer process was created.
Types of RDA Mechanisms
There are 2 types of Real time data acquisition which are discussed as under,
Pull: It’s a request oriented, for long tern planning, scheduled at nights & consumes less resources.
Push: For tactical decision making on a daily basis, information is processed every 1 hour or 1 min, it’s based on data availability, and resource needed is very high as it’s an active background job.
Note: Enhance established Data Flow with RDA capabilities
[+] Implement additional DSO for operational reporting
[+] Replace standard delta IP by RDA IP
[+] Regular data loads can be scheduled after closing RDA IP Request using appropriate Process Chain feature
[+] Typically data is deleted regularly from DSO supplied using RDA
[+] Standard reporting can be enhanced by operational reporting using the report-report interface
RDA – Implementation Scenario
Fig. F. RDA Implementation Scenario
Why use Delta?
In Extraction part of RDA, data sources are probably not ready to immediately supply data in small quantities, but in very large quantities instead. If several million records are transferred subsequently, latency exists. Ex: What if it takes several minutes to load data. So, in RDA, the concept of request generation or logging in BI needs to be changed. Data sources in source systems must be able to react as quickly as possible for the new requirements. Therefore data sources in RDA are delta enabled.
Situations: If the data source is small, which one will you use? RDA or direct access?
Answer: Direct Access
Processing of RDA is done by daemons. (Refer Fig. H)
Daemons originally stand for Disk And Execution MONitor in the UNIX. Each step is tracked in a control table, as and when it’s successfully executed. This also, allows restarting which can be initiated such that it starts at the next step after last successfully executed step.
The BI Daemon data load includes three steps:
Initiate BI Service-API data pull using Info Package for RDA into PSA (SAP source systems)
Track status of data transfer from source system
Initiate update of Data Store Object using DTP
Daemons are system process to initiate data loads at regular intervals: from one minute to hourly basis.
In RSRDA, we can monitor status of daemons, check if a batch job is still active, running, or not. Sometimes, it might be stopped by an error or user itself. This monitor is useful for displaying runtime information about the daemons.
Fig. H. Real Time Data Acquisitions monitoring using RSRDA
RDA in Extraction Layer
The core of RDA is the SAP Source systems.
Other source systems can be linked to the concept of RDA only if they act as a Web service and transfer data via SOAP messages to SAP BI. (Refer Fig. I)
For SAP systems, the data sources that need to be connect by RDA need to be explicitly marked as ‘Real time enabled’. For Generic Data sources, you can specify if a data source is able to handle accesses in real-time or not. The decision if a Generic Data source needs to be marked as real time enabled depends, how generic deltas are created.
Condition 1: The delta defining fields should enable a high performance access to new/modified data. I.e. delta defining field must be selective & must be max. Affected by different values
Condition 2: If there is a recent extraction, identification of the relevant data records from delta creation must contain only those recent data records.
Via the SOAP services, the SOAP messages are directly written to the PSA level of the data source. Hence, the messages don’t find a way into BI through extraction process. So a BI system does not require executing the extraction, but only the PSA data, which is further processed in the delta procedure.
RDA in BI Staging
Here in RDA, the data is extracted from the PSA in delta mode and is written to the DSOs. For Web service source systems, the data is already available in the PSA. While for SAP systems, extraction from PSA must also be considered in the RDA design.
3 Requests in RDA
The PSA Request – Contains raw data of the source system.
The DTP request – Describes transformation into DSO.
The Change Log Request – Contains activated data of the DSO
With regular staging these 3 requests are not often generated. Instead, they are created synchronously & are closed if they contain a defined amount of data records.
Control is usually transferred completely to a daemon. So, RDA is not only a technology but also a particular form of automation in standard operations.
Data source for RDA
In SAP systems, the controlling of extraction needs to be explicitly defined by the RDA. But this is one optional form of usage. Only with the selection of suitable adapter will be the execution affected in real time.
Info Package for RDA
IP for real time data acquisition can be defined with SAP Source systems only if the delta procedure was initiated & the DTP was created. For a Data source for a real time data acquisition, only one info package for RDA can be defined. However for Web services, the IP must also be provided, if data is transferred to BI via Web Services. (Even if data is written into PSA & is not extracted)
DTP for RDA
As a precondition for the creation of IP for RDA, we must choose DTP as ‘DTP for Real-time data acquisition’ with the definition of DTP.
As with the definition of IP, the qualification of DTP mostly serves the purpose of modifying request handling.
A precondition for having a DTP for RDA is that, we must have at least one DSO as data target & one data source to provide data
Controlling the RDA
The processing of the RDA, through the staging architecture involves 2 scenarios: Initialization of Delta procedure & Data transfer during RDA. The monitoring scenarios are as explained below:
Data Initialization: The initialization of delta procedure is necessary at first to create the IP for RDA. If the initialization is made with DTP, then initialization process can’t be handled by the DTP.
Switching DTP types: The handling of the initialization of the DTP cannot be done, and must be reverted back to initial state. The request needs to be switched to its default behavior, as the initialization can be only started via a daemon that controls & monitors the RDA. The DTP definition contains a key called ‘Change to Standard DTP’, which when clicked can switch DTP to initial state.
Daemon for RDA
If the delta initialization was processed, & respective request was activated in DSO, controlling IP, DTP & activation of new data in DSO can be transferred to a daemon that controls & monitors the staging for RDA.
A daemon is a continuous running back ground job that switches into sleep mode between single extraction processes. However, to improve the main memory usage, it ends the job & reschedules itself again.