Skip to Content
Author's profile photo Mark Thienpont

Streaming Process Chains

Context

To update and ADSO in near-realtime, one has to consider the Process Chain for Streaming (available since BW 7.5 SP03; information can be found at https://help.sap.com/viewer/2e90b26cf7484203a523bf0f4b1bc137/7.5.4/en-US/ac029de05e164a12ac1ce08d16180f05.html).

Using RDA (realtime data acquisition) does not support updating ADSO’s (only cubes, classic DSO’s, and alike), and has thus become yet another obsolete BW-artifact.

This blog shows the setup in a nutshell in context of both ODP-SAPI and ODP-SLT datasources.

ODP-SAPI

In this case, the basis are the well-known (sometimes rich in business-logic) extractors.

In this example, I’ve used the 2LIS_11_VAITM “Sales Document Item Data”, for which I’ve set the Update Mode to “Direct Update” in the Logistics Workbench in the source system (this aspect is covered sufficiently elsewhere).

The source-system connection is of type ODP-SAPI. This implies that after applying an initial load in BW, there is no more usage of the qRFC-mechanism (so no more RSA7), but now we have to take a look at the queues and subscriptions via at ODQMON (on the system where ODQ has been implemented) :

Remark that there is also no need to use PSA-level due to the usage of ODQ; so both qRFC as PSA level is being replaced via one ODQ-queue.

Due to the usage of Direct Update, any change in the transactions (in this case VA01 or VA02, or alike) will immediately insert the change-records into ODQMON-queue; very alike the behavior in RSA7, actually.

One can use this part for further batch-processing, with whatever target-BW-object.

The transformation is just a straightforward 1:1; the ADSO is purely field-based (just had to indicate the right key-elements and some key-figure aggregation behavior). All this can actually be performed in a couple of minutes.

Now the fun starts : we want to use this setup to update the ADSO as soon as possible.

Trying to create a DTP of type “RealTime” is not possible : it is removed from the drop-down list as soon as the target is and ADSO !!

And actually it is no longer necessary to do so. Just use a regular delta-DTP, and put it into a Process Chain. As soon as you tag the flag “streaming” in that process chain, and schedule it, the according DTP’s will behave as “realtime” :

Once (and as long as) this Streaming Process Chain is scheduled, you will find one request  in ODQMON which has status “Extraction Running”; as soon as the Streaming Process Chain is out of scheduling this request receives the status “Confirmed”.

As a consequence, the process chain is running with the indicated frequency (pull). Herewith the resulting view of updates of the ADSO.

 

ODP-SLT

In case it is necessary to replicate a table, or if a delta is not easily available via the above described extractor technique, one can use SLT. To update the SLT-queues towards BW yet another ODQ-layer can be introduced.

To activate the SLT/ODQ setup and subscription, it is sufficient to first create a new datasource in BW for the table needed, and load a first delta-DTP. The example we are showing here is VBAP :

Then I’ve created a field-based ADSO using this datasource as a template, and a 1:1 transformation between the datasource and this ADSO. This is all depending on the use-case at hand.

Once done, a normal delta-DTP can be created.

The first execution of this might take a while, as this is setting up the SLT-specific delta-capturing mechanism in the source-system; it will finally also result in an initial load data-set (and an additional initial line in SLT/ODQMON).

Again : you can use the above for batch-purposes.

To now start streaming the changes towards the ADSO, put the delta-DTP in a process-chain, flag “streaming”, and schedule. Remark : if you schedule according to ‘API’, the process chain will be triggered event-based via ODQ !! Meaning : this is a real-time push-mechanism.

As such chain will only process based on an event coming from the SLT/ODQ, the number of update-requests in the ADSO is largely reduced : only changes are resulting in an update request (in real-time)

 

Conclusion

Although SLT is purely table-based, so it doesn’t contain any business logic in itself, it it a very powerful delta-enabling technique, and it is capable of pushing changes in real-time to BW’s ADSO’s.

 

Assigned tags

      19 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Sebastian Gesiarz
      Sebastian Gesiarz

      Hi Mark,

       

      Great post!  Thank you for sharing your knowledge.

      Could you please tell whether this would mean that the only difference in time measure of ~realtime replication between SLT and SAPI is ~one minute?

       

      Best regards,

      Sebastian

      Author's profile photo Mark Thienpont
      Mark Thienpont
      Blog Post Author

      Hi Sebastian,

      There is another difference : SLT only leaves an update-package in the data-staging when there is a real update, whereas SAPI (polled way of working) is actually leaving a trace per minute. I didn't notice much more differences yet.

      KR,

      Mark.

       

      Author's profile photo Matthias Schober
      Matthias Schober

      Does streaming work on all types of Datasources ? Fi/CO ?

       

      Author's profile photo Mark Thienpont
      Mark Thienpont
      Blog Post Author

      Dear Matthias,

      Streaming is only connected (as far as I know) to SLT-triggers (or compatible). So I don't think you can use the existing extractor framework as trigger for event-based staging.

      If you would however use an SLT on 'the main' table regarding your extractor (e.g. BSEG), to know that there is a change, you could use the delta-capable regular extractors to process the delta's.

      Kind regards,

      Mark.

       

      Author's profile photo Former Member
      Former Member

       

      Printable Calendar 2018 printable 2018 calendar 2018 Calendar Printable Free Printable Calendar 2018

      Author's profile photo Sunil Paladugu
      Sunil Paladugu

      Hi - We have configured a  PC for Streaming which is extracting data from a Table EKKO using ODQ/SLT Technique. Would you suggest the idle periodic interval for the process chain?

      Author's profile photo Sunil Paladugu
      Sunil Paladugu

      Hi - It seems ODP/SLT is failing to send an event to trigger the Process Chain in Streaming Mode.Any idea on this.

      Author's profile photo Former Member
      Former Member

      Hi Sunil,

      we have noticed the same behavior in our current project. The implementation is: MSSQL -> SLT/ODP -> SAP BW 7.5 where the target of our data flows is mainly an advanced DSO.

      We have done the delta initialization and then we have put the DTP(s) in a streamng process chain (scheduled according to API ) and we can see the real time request in the ODP queue(ODQMON). The real time daemon job (ODQ_DAEMON_CLIENT_XXX) is correcty running every 15 minutes and we expect the process chain automatically triggered as soon as there ia a change done at source tables level but it does not happen (If we manually trigger the process chain or we manually execute the DTP the delta records are correctly extracted).

      Did you manage to find the root cause of the issue?

      Thanks and Regards,

      Delio

      Author's profile photo Sunil Paladugu
      Sunil Paladugu

      Hi Delio -

       

      We are doing SAP ECC--->SLT--->BW, due to Time Zone issues in SLT sever we had issues with Real Time Streaming Issues, but now after correcting the Time Zone Issue it seems Real Time streaming is working and still we are doing testing to confirm that everything looks good. will confirm you back on this.

      Author's profile photo Former Member
      Former Member

      Hi Sunil

      We are facing the same issue. Please let us know what was done to correct the time zone issue. Also let us know if the testing was successful.

      Thanks and Regards,

      Radhika

       

      Author's profile photo Former Member
      Former Member

      Hi Sunil,

      We are facing the same issue. Please let us know what was done to correct the time zone issue. Also let us know if the testing was successful.

       

      Thanks and Regards,

      Radhika

       

      Author's profile photo Former Member
      Former Member

      Hi Sunil,

      Is your testing complete? Please share the steps taken to correct the time zone issue.

       

      Thanks and Regards,

      Radhika

       

       

       

      Author's profile photo P. Verstraeten
      P. Verstraeten

      Hi all,

       

      We also have troubles with streaming process chains, I've found a very important SAP note "Important corrections for the usage of Streaming DTP-s and Process Chains". SAP also recommended me to implement the note first (with more than 30 prerequisite notes and 70 optional notes). Streaming process chains are new in BW7.50 and need some corrections first.

      Regards,

      Pieter Verstraeten
      www.pieterverstraeten.com

      Author's profile photo Former Member
      Former Member

      Hi Pieter,

      Thanks for sharing this note.

      Please let us know if you are facing a similar issue to the one below and please share any additional steps you took to resolve it once the issue is resolved.

      The implementation is: MSSQL -> SLT/ODP -> SAP BW 7.5 where the target of our data flows is mainly an advanced DSO. We have done the delta initialization and then we have put the DTP(s) in a streaming process chain (scheduled according to API ) and we can see the real time request in the ODP queue(ODQMON). The real time daemon job (ODQ_DAEMON_CLIENT_XXX) is correctly running every 15 minutes and we expect the process chain automatically triggered as soon as there is a change done at source tables level but it does not happen (If we manually trigger the process chain or we manually execute the DTP the delta records are correctly extracted).

      Thanks and Regards,

      Radhika Desai

      Author's profile photo Mihir L Kiri
      Mihir L Kiri

      Hi Mark,

      Is it possible to leverage ODP and delta enabled standard extractors for real time replication into Native HANA. We do not have BW license and hence process chain option is not available.

      Regards,

      Mihir

      Author's profile photo Hercar Carvajal
      Hercar Carvajal

      Dear Mark

      I hope you are doing well!!

      First of all, thank you for this wonderful document, secondly, I have several doubts about Stream process chain data request administration, as described as follow:

      1. imagine and Stream PC ADSO getting real-time data, what if one request fail?
      2. as per my last experience, I cannot remove that request...
      3. Must I unschedule the Stream PC first? then remove the request?
      4. will I recover all data missed during downtime, resolving ADSO request issues?

      Is there a kind of step by step to solve this situation?

      Thank you in advance

      Best Regard

      Hercar

      Author's profile photo Mamta Pathak
      Mamta Pathak

      Hello Mark,

       

      I tried this ODQ approach for 0FI_GL_4 with streaming enabled. The process chain now is running every minute. I changed the record in ECC system, the streaming process chain did not bring the delta for next 15 runs, all the time it was 0. After the chain was removed from the scheduled and streaming, ran the DTP manually and it brought those delta records.

      Could you please help to understand why streaming process is not bringing the delta?

       

       

       

      Author's profile photo Adria Triquell Cristofol
      Adria Triquell Cristofol

      Hi Mark,

      We're in SAP BW/4HANA SP1 1.0. We are scheduling a process chain from SAP ECC each 5 minutes, so I think would be better to set the chain in "streaming mode".

      After switch the flag "streaming mode", the first execution was ok, but the successive executions of the DTP are always in the same error:

      "At the moment DTP can only be executed in a streaming process"

      Do you have any idea about this issue?

      Thanks in advance

      Author's profile photo Frank Riesner
      Frank Riesner

      Hi Mark,

      maybe you will find following blog post interesting as well
      https://blogs.sap.com/2021/03/15/run-modes-of-sap-bw-streaming-process-chains/

      Regards, Frank