Run Modes of SAP BW Streaming Process Chains

[FrankRiesner]

Dear SAP BW enthusiasts!
Recently, one of my customers asked me for possible approaches to implement real-time reporting. During preparation my research also covered “Streaming Process Chains” with surprising results. For this reason I thought it would be a good idea to share “one of the best kept secrets of BW functionality” here in the SAP community. This blog post will focus on the streaming process chain run modes and provides URLs to additional sources in the end. I use the general term “SAP BW” which refers to both SAP BW 7.5 and SAP BW/4HANA (any release). 

In general, acquiring data as fast as possible should not be regarded as first choice. Instead, SAP recommends leveraging virtualization and data federation wherever possible. In combination with SAP HANA, BW provides multiple suitable modeling options, mainly based on SAP HANA integration services (SDA, SDI), SAP HANA modeling options (Calculation Views), and Open ODS Views or CompositeProviders as virtual InfoProviders on the BW application side.

Only if these approaches do not meet your business requirements, loading data in "real-time" into SAP BW should be evaluated as a serious alternative. Introduced long ago in 2006 with NetWeaver 7.0, “Real-Time Data Acquisition (RDA)” was the related concept for InfoCubes and DSOs (classic). However, with the introduction of ADSOs, SAP decided to retire RDA and to enhance the existing framework of process chains instead.

Streaming is an operation mode of BW process chains which can be set in the properties of each process chain. When a process chain is running, each step takes a certain amount of time; the interesting question is what actually happens, when the chain is started again -maybe even several times- during this period, and the execution reaches the process which is already active from a previous execution:

• A classic process chain is scheduled in the background. Then the second execution waits 10 minutes for the previous execution to complete. If the previous run is not completed within this period, the second execution is canceled, the process chain is interrupted and fails with an error.

• A process chain in streaming mode is executed using a new Queued Task Manager Framework. This enables a flexible number of “worker” jobs that get the next process from a queue. If a second, third, and further execution reaches the current active process, these requests to start this process once more are written to a queue only. No more resources are used, and the system does not wait. As soon as the current process is finished, the worker job receives all of these start requests for a given process and restarts the process just once: It takes the most recent chain execution from the queue and executes it. This means that the second and third chain executions are dropped, and only the most recent chain execution is continued.

In a nutshell, the main value-added the Streaming mode is that it facilitates the frequent data transfer from a source system to SAP BW without scheduling errors and without unreasonable waste of resources. As a result, data can be acquired in SAP BW in almost real-time. It is not 100% real-time, because the realistic latency is measured in minutes in the best case. So do not expect latency times of milliseconds to seconds which are common in SAP SLT replication scenarios for example.

In general, there are different run modes to be distinguished: A Push and two Pull run modes. The main difference between them is whether the consuming system or the source system is triggering the data transfer.

A) The PUSH Run Mode
Main Characteristics:
SAP BW receives a trigger from a so-called ODP Daemon in the source system if a new record is generated there. This event starts the immediate execution of the process chain which transfers the new data to the SAP BW InfoProvider.
This is the preferred mode for ODP sources, because the process chain runs only when new data is available in the source. The process chain is scheduled once in streaming mode, the definition of periodic repetitions is not required.
The daemon has to be scheduled as regular job ODQ_DAEMON_CLIENT_xxx in the SAP source system (tr. ODQMON, GoTo/Real-Time Daemon...). By default scheduling, It checks all 15 sec for updates and sends an event to BW if there are any.

Preconditions:

1. This mode is only available for ODP DataSources of types SAP Extractors, ABAP CDS Views, BW InfoProviders and SLT.
   
   
2. Another precondition is that DataSources are Delta-capable:
   • ODP_SAP or ODP_CDS: The delta-capability is dependent on the implementation.
   • ODP_BW: The delta-capability depends on the source InfoProvider. Supported sources are DSOs (classic), InfoCubes and ADSOs.
   • ODP_SLT: All DataSources are delta-capable in general.
     
     
3. Finally, the DataSources need to support Streaming ("Streaming can be triggered by this DataSoure"):

• ODP_CDS, ODP_BW, ODP_SLT: All delta-capable DataSources support this property by default.

• ODP_SAP: This property depends on the individual implementation of the extractor. For example, LO Cockpit DataSources (2LIS_*) are generally not streaming-enabled due to their complexity (e.g. due to the queued delta), but other SAP delivered DataSources might support this feature. However, most of the SAP delivered DataSources are not streaming-enabled in the delivered version. There are some corrections provided (see SAP notes 1692484, 1780297, 1782173), but in most of the cases so-called Realtime-flag has to be set manually in the SAP source system (e.g. by a customer report updating ROOSOURCE-REALTIME to value 'X', see also SAP note 905089, which is not released currently and so SAP internal only). However, consider that this makes only sense for DataSources which are able to provide new data in real time, which means a delta is available at every point of time.
  Customer developed DataSources do not have this property initially as well. It must be enabled explicitly in the DataSource definition (Tr. RSO2). The corresponding property is called Real-Time enabled and is available in the customizing settings for the Delta mode.Note: In some BW 7.5 systems the parameter might have a different name due to the reference to RDA. But from the a BW Streaming Process Chain perspective the same is meant by "Realtime" - "Realtime Data Acquisition Supported".
  
  

B) The Intelligent PULL Run Mode

Main Characteristics:
SAP BW checks very frequently (each 60 seconds) if a new record is generated in the source. If this check is successful, the process chain is started immediately to transfer the new data to the target InfoProvider. The process chain is scheduled once in streaming mode, the definition of periodic repetitions is not required.
This is the preferred mode for HANA sources, because the process chain runs only when new data is available in the source system.

Preconditions:

1. This mode is available for HANA DataSources only.
2. The DataSources need to be Delta-capable.
   • If the source system type is HANA_LOCAL or HANA_MDC (BW/4HANA) / HANA with connection type "Local SAP HANA Database Schema" (BW 7.5), check additional requirements in SAP note 2372534.

• If the source system type is HANA_SDA (BW/4HANA) / HANA with connection type "SAP HANA Smart Data Access" (BW 7.5), it is important which type of remote connection is in use:
  • SAP HANA Smart Data Access (SDA) does not support this run mode in general.
  • SAP HANA Smart Data Integration (SDI): The intelligent PULL mode is available only for adapters which support "Real-time Change Data Capture" as documented in the PAM functional overview of SAP HANA SDI 2.0 (pages 20-22). Currently those are: ASEAdapter, ASEECCAdapter, DB2ECCAdapter, DB2ECCLogReaderAdapter, MssqlECCAdapter, MssqlLogReaderAdapter, OracleECCAdapter, OracleLogReaderAdapter, PostgresSQLLogReaderAdapter, HanaAdapter, TeradataAdapter, AbapAdapter, CloudDataIntegrationAdapter and TwitterAdapter.

In a nutshell, the SDI adapter replicates each change to a generated BW table (/BIC/CMT*) in real-time. You can identify this table in the BW DataSource definition, there is a button called SAP HANA objects. This is the location the streaming process chain checks for new data regularly.

Note: For BW DataSources related to source system HANA_* the Streaming property is always off ("Streaming cannot be triggered by this DataSource"). Do not be confused - I was confused myself quite a long time...it seems like this property only refers to the PUSH mode and thus to ODP sources.

C) The Scheduled PULL Run Mode
Main Characteristics:
SAP BW executes a process chain which leverages the streaming framework, no matter if there are new data in the source or not.  The Streaming process chain is scheduled as a periodic job and it can be executed very frequently if required (up to each minute). If there are updates available, they will be loaded immediately to SAP BW. On the other hand, if there is no new data, then an empty request is created in the target InfoProvider.
This is the preferred mode, if the source is receiving streaming data, e.g. a record each second. In this case the PUSH or intelligent PULL would not be able to process the new data in the same speed toward SAP BW. So this simple frequent scheduling could be most efficient approach.

Preconditions:

1. This mode is available for ODP DataSources which are not streaming-enabled.
2. This mode is also available for HANA DataSources which rely on SAP HANA SDA, or they are based on SAP HANA SDI connections which do not support "Real-time Change Data Capture" (all adapters which are not listed above).
3. Another precondition is that the DataSource is Delta-capable.

Note: The fact that the preconditions the PUSH mode (A) or the Intelligent PULL mode (B) are not met, does not mean, that you cannot run process chains in streaming mode. You can still leverage the queued task manager framework to run them with high frequency for Delta-capable DataSources. In this case, the process chain needs to be scheduled as a periodic job to pull data from the source in up to 1 minute frequency, instead of the process chain being triggered only when new source data is available.

D) Appendix: Special Mode for Write Interface-enabled ADSOs 

Main Characteristics:
As described in my separate blog, there is a new SAP BW/4HANA write interface which enables SAP and non-SAP tools to provision data directly into an ADSO comparable the web service source system in BW 7.x . Technically, the data is pushed into the ADSO inbound table and the data can be processed from there by BW/4HANA standard means.
If this ADSO requires an activation process, this activity can be triggered automatically by means of a Process Chain in ‘Streaming’ mode, because this process chain can be triggered (started) as soon as a request in the inbound queue is closed. 

Preconditions:

1. Release SAP BW/4HANA 2.0 or higher
2. ADSO with modeling property Write interface-enabled
3. Process Chain in Streaming mode containing a process variant ‘Activate DSO Data’ specifying the ADSO name

SUMMARY
The Streaming property is a really cool enhancement of the process chain concept in SAP BW. It provides a flexible toolset, if you really have to load data as fast as possible into a BW InfoProvider. The preferred run modes are the PUSH mode and the Intelligent PULL mode, as they only trigger the DTP execution if new source data is generated.
I recommend following SAP Education offerings if you are eager to learn more about this topic and much much more. We have prepared a sophisticated scenario based on SDI HanaAdapter where you can explore streaming process chains hands-on. Both trainings are currently upgraded to SAP BW/4HANA 2.0 SP07 with more details about streaming process chains. Availability of this latest version expected by mid-2021.

• BW462 - SAP BW/4HANA: Delta from BW classic
• BW450 - SAP BW/4HANA Data Acquisition

REFERENCES
There are some interesting blog posts in this community which provide use cases for the run modes introduced above:

• Streaming Process Chains – Scheduled PULL via ODP_SAP and PUSH via ODP_SLT
  https://blogs.sap.com/2017/02/28/streaming-process-chains/
• Streaming Process Chains - PUSH via ODP_CDS
  https://blogs.sap.com/2018/02/04/delta-extraction-and-real-time-streaming-odp-cds-to-bw/
• Streaming Process Chains - PUSH via ODP_SLT (1)
  https://blogs.sap.com/2017/09/26/slt-odp-real-time-data-replication-from-sap-source-to-bw-adso/
• Streaming Process Chains - PUSH via ODP_SLT (2)
  https://blogs.sap.com/2017/10/13/slt-odp-real-time-data-replication-from-non-sapmssql-to-bw-adso/

For additional details refer to the SAP help portal or following SAP notes:

• SAP BW/4HANA help about Process Chains for Streaming
• SAP note 2844555: Corrections for Streaming Process Chains (BW Note Analyzer)
• SAP note 2372534: Streaming data from local SAP HANA schema to SAP BW
• SAP note 2447916: How to transfer RDA to Streaming Process Chains
• SAP note 2865995: How to check if an ODP DataSource supports PUSH Streaming or not
  
  

Thanks to Waldemar, Dirk, Nicola, Thomas, Deniz, Tobias, Jens, Alex, Andreas, Luiz, Paul and others who provided valuable input for this blog post!

[DenizOsoy]

Great blog Frank! Thank you very much. Very valuable and innovative for BW customers.

[PhilMad]

Hi Frank, thanks for this interesting blog. The pull mode was new to me and can be of interest for many scenarios. A problem I found when testing streaming process chain was to skip a record in case a record produces an error. SDI based remote subscriptions offer such a functionality. My test was to send two number from the source and then do a division as a logic in the transformation between DS and target. When I sent a 0 for the divisor I got as expected a div by 0 shortdump. But couldn't figure out how to skip this record in the the processing so that the stream returned to be operative again. Any hints on that? Take care Philipp

[Vijay4]

Hi Frank, Thanks for your information, very useful.

[Abhishek_Hazra]

Good one Frank! This blog covers a lot of unexplored areas of process chain streaming feature that BW techies were looking out for 🙂

Thanks,
Abhi

[paulvatter]

Thanks for summarizing and sharing this detailed information, Frank! Best regards Paul

[FrankRiesner]

Hi Philipp,
was machst Du nur für Sachen... 🙂
Well, I don´t know the exact answer how to skip a record with errors and fix the stream. But in general I think, there should not be any logic in the TRFN from DataSource to the ADSO for exactly the case you describe: To avoid any errors or delays on the BW target side. Here the focus is on the near-real-time transfer of data and any logic will put a danger there. At least the monitoring effort would increase.

Viele Grüsse, Frank

[Markus_Ganser]

Great stuff, very usefull! Thanks and best regards, Markus

[MarkGreen]

Frank, once again an excellent blog from you covering a topic that is often asked about, but few are able to explain in detail. Thanks for sharing the findings of your research.

[ibayraktar]

Great explanation! Thank you.

Best Regards.

[ClaudiaFiess]

Excellent blog Frank! So valuable - especially as more customers asking for "real-time" or "near real-time" options when looking at data loads into BW. Thanks for sharing 🙂

[yi]

Before this post, I thought the "push" mode is for ODP-SLT only, and the "pull" mode is for ODP-SAPI. Apparently, SLT always supports push, but ODP only sometimes does. Now it's clear that even if a data source doesn't support streaming (RSOOSOURCE-REALTIME flag is not set) and it says so, we shouldn't give up, because BW will use its own streaming, which is the "pull" mode. One way or another. Makes sense now.

Thanks,

Andrew

[alex_neary3]

Hi Frank,

I am unable to get the The PUSH Run Mode working on SAP BW 7.5 SP17.  It seems that the extraction properties are not available in the SAP GUI.  It looks like your screen-shots were taken from HANA Studio.  Can you please confirm you were able see "Streaming can be Triggered by this Datasource" option in SAP BW 7.5, you said "(any release)" at the beginning of your blog.

Best Regards,

Alex

[FrankRiesner]

Hi Alex,

yes my screenshots are from BW Modeling Tools, but the same can be found in the SAP GUI.

First of all, your ODP_SAP data source needs to be streaming enabled. Check in your source system if this is the case, e.g. in table ROOSOURCE-REALTIME = X or Tr. RSO2/RSA2.

When you replicate this object to BW, the same field REALTIME = X is in table RSDS or in the Extraction tab of Tr. RSDS (DataSource Maintenance).

Regards,

Frank

[alex_neary3]

Greetings Frank,

I naturally checked all my configuration before I asked this question.  All my configuration settings are and have been as you outlined in your reply.  There is simply no option in RSDS for "Streaming can be Triggered by this Datasource" the way it looks to me.  Also, if you have access to OSS, I have filed the a message with SAP detailing the issue with screen-shots.  Please reference Incident 279621/2021.

RSDS with no Streaming Config Line

Best Regards,

Alex

[FrankRiesner]

Hi Alex,

good point, but easy resolution. In 7.5 systems the UI might have a different naming:

Title: "Realtime" instead of "Streaming"
Value: "Realtime Data Acquisition Supported" instead of "Streaming can be triggered by this DataSource".

However, it is the very same feature.
In the backend  his setting is managed in a field called "REALTIME" for historic reasons, because RDA was the prior concept.

In BW 7.5 you still can choose between RDA and Streaming PCs, while in BW/4HANA RDA has be decommissioned. I think this is the reason of the renaming in BW/4HANA.

So your DataSource above is fine for the PUSH mode from my perspective. If you have issues, then I suggest to check your source system with the BW Note Analyzer and input from SAP note 2844555 and SAP_BW_ODPODQStreaming_<Date>.xml if all notes are up-to-date.
And check your BW system with SAP_BW_ProcessChainStreaming_<Date>.xml as well.

Regards, Frank

[alex_neary3]

Greetings Frank,

Thank you for the quick response and clarity on this.  From my experiences, and I have tested a lot of scenarios in my present release, the method you describe actually does not start a process chain for streaming.  It uses RDA, to your point, but it runs the event through the Daemon process using T-Code RSRDA, you can see this.  You also have to insert your data into a classic DSO to make it function.  I was actually hoping to use a triggered streaming chain because I could insert to an ADSO.

My conclusion is for BW 7.5, Option A is simply not available, not in the way you describe at least.  You're right, it functions the same, it triggers the event to load, but you have to understand you will not get the benefits of running any of the other Process Chain steps.  The only thing triggering the DTP via RDA does is insert the records into the Classic DSO and Activates.   Option C is our only choice for ECC extraction, we have to "pull" the data to use the process chain streaming.

Please correct me if I am misstating the facts.

Best Regards,

Alex

[FrankRiesner]

Hi Alex,

the concepts I described have nothing to do with RDA (except outdated naming of the ODP_SAP DataSource maybe). Streaming PCs and ODP are available a long time and as such option A is a valid option for BW 7.5 systems in general as well. It is not any RDA process which starts the process chain but the trigger event from the ODP daemon in the SAP source which is able to execute the BW process chain if it is in streaming mode.

Regards, Frank

[alvaro_gutierrez_villarin]

Excellent blog Frank!

 

We're setting up a Streaming process chain and we are here in scenario C - Scheduled PULL mode as we mainly need 2LIS* data sources updating a BW 7.50 SP20 on HANA

We scheduled the PC about every 5 minutes and at each PC exaction i.e ( 10:05, 10:10. 10:15...) always come two additional executions (one after the other) just after the scheduled process was executed.

Is this an expected behaviour ? -  We're not interested in such a feature as no data will be retrieved as our V3 collector jobs run also every 5 minutes

 

Regards,

Álvaro

 

[FrankRiesner]

Hi Alvaro,

no, this is not the expected behavior I think. These additional executions will not bring any new data,right? So they will only consume system resources and bring additional monitoring efforts.

I recommend to open an incident with SAP and ask for guidance.

Before make sure, you are up-to-date regarding correction provided in SAP note 2844555 (Note Analyzer and highlight this in your incident description as well).

Regards - Frank

[cyril_metge2]

Hi Franck,

 

Very interesting blog, thanks for sharing

Trying to implement a real-time data extraction with and ODP_CDS, I'm stuck at the very begining :

As far as I understood, all ODP_CDS datasources are supposed to allow realtime extraction, providing that they are delta-capable (which is my case).

But my datasource appears as non real time capable.

ODP CDS datasource

I didn't found any other pre-requisite in the different documents that I found. Do you know if there is a specific annotation to be added on the datasource ? Or any specific setup to be done ?

Regards, Cyril

 

[FrankRiesner]

Hi Cyril - can you provide the delta annotations of this CDS View?
Maybe that will help to analyze the question

Thanks - Frank

[ABner]

What version are you on, some of these capabilities were not fully supported until specific Hana versions...also - assuming this is a native CDS view not a converted extractor...

[cyril_metge2]

Hi Frank,

Thanks for your feedback.
Here is a simplifyed example that I created for testing purpose :

@EndUserText.label: 'CDS View - RC Details'

@AbapCatalog.sqlViewName: 'ZBW_RC_TEST'

@AbapCatalog.compiler.compareFilter: true

@AccessControl.authorizationCheck: #NOT_REQUIRED





@Analytics.dataCategory: #FACT

@Analytics.dataExtraction.enabled:true 

@Analytics.dataExtraction.delta.byElement.name:'LASTCHANGEDATE'

@Analytics.dataExtraction.delta.byElement.maxdelayInSeconds: 300 

@ObjectModel.usageType.dataClass: #TRANSACTIONAL



define view ZA_BW_RC_DETAILS_TEST as select from zgl_mm_scost_007 d {

    key d.mandt as MANDT                                                

    ,key d.scnum as SCNUM                                               

    ,key d.werks as WERKS                                               

    ,key d.matnr as MATNR                                               

    ,@Semantics.systemDate.lastChangedAt: true 

    case 

    when d.updat <> '00000000'

    then d.updat

    else d.crdat

    end as LASTCHANGEDATE     

    ,d.maktx as MAKTX                                                   

    ,d.matkl as MATKL                                                   

    ,d.stawn as STAWN                                                   

    ,d.ekgrp as EKGRP                                                   

    ,d.glcod as GLCOD                                                   

    ,d.mcomp as MCOMP                                                   

    ,@Semantics.amount.currencyCode: 'WAERS' d.stprs as STPRS           

    ,@Semantics.currencyCode d.waers as WAERS                           

}

Thanks,

Cyril

 

[cyril_metge2]

Hi Aaron,

Thanks for your help.

I do confirm that it is an Abap CDS view based extractor, not a convertted one (see code on my previous answer to franck).

Our HANA version (source system) is 2.00.055.00.1615413201, and the SAP APPL componet is  (SAPK-61811INSAPAPPL)

We are using a BW on HANA (hana version 2.00.057.00.1629894416) where SAP BW version is SAPK-75016INSAPBW

According to the SAP Note analyzer (2844555)  our components version are green so seems OK.

We have few notes to implement, I'm asking for implementation but not sure it will be enough regarding the description of those note.

 

Thanks,
Cyril

[FrankRiesner]

Hi Cyril,

my assumption is, that this generic delta needs to be enhanced to provide deleted items as well.
Can you please try to add Analytics.dataExtraction.delta.byElement.detectDeletedRecords Annotation? See details here https://help.sap.com/docs/SAP_NETWEAVER_750/cc0c305d2fab47bd808adcad3ca7ee9d/c2dd92fb83784c4a87e16e6...

Then create the DataSource again and see if the streaming works fine.

KR Frank

[cyril_metge2]

Hi Frank,

Thanks for your answer.

Your messages made me realise that expecting realtime on a date generic delta clealry make no sense. I was trying to reuse an existing datasource, but this scenario is not relevant.

I tryed the Addition of the detectDeletedRecords but without success (datasource still aprears as non realtime capable).

I have tryed a different example with a generic delta based on timestamp, and then it appears as a realtime datasource.

I'm also going to test with CDC based delta, but it look obvious that a timestanmp or CDC will be the best (and probably mandatory) option for realtime extraction.

Thanks again for your answer, was really helpfull!

[paulvatter]

hi Cyril

Regarding the "detectDeletedRecords"-annotation just another remark: when this is used a shadow table is created in the source and being used as reference. In one customer scenario the shadow table was consuming more space than the original table, so really to be used with caution besides the point above 😉

It would be great if you could share your results about the own-developed CDC based delta. This is very interesting option for some scenarios in my opinion!

Thanks and best regards Paul

[cyril_metge2]

Hi Paul,

Thanks for your message. I will be happy to share any experience on CDC.
If you didn't saw it already, you should read this blog from Simon Kranig about CDS based data extraction delta handling. It is very detailed, about CDC delta handling.

Best Regards,

Cyril

[former_member494405]

Question:

How do we create an alert to notify the technical team if the streaming process in the system is long running at DTP step due to huge volume of data being posted in bulk at source or the streaming process is stuck due to system issues.

 

Appreciate your inputs

 

Thanks

PN

[FrankRiesner]

Hi PN,

you surely know, that you can send notifications withing process chains for dedicated events like finished successfully or finished with error. But a long running job is not a clear event for the SAP notification framework. So, I am afraid there is not default solution available for your scenario.

KR Frank