SAP Signavio Process Intelligence: On-Premise SAP S/4HANA data
This blog is the first of two I’m publishing and has the focus on how data from an on-premise S/4HANA system is extracted (for ECC it’s analogous), transformed and loaded (ETL) into SAP Signavio Process Intelligence for business process analysis. It reflects my personal learning journey and excitement working with SAP Signavio. It’s therefore not a best-practice document or recommendation in any way.
B. Signavio Overview
SAP Signavio consists out of several modules and it’s worth looking at them briefly before moving straight to the point of this blog.
Today we will look briefly into Process Manager and in more depth into Process Intelligence. We focus on how data can be extracted in relation to a defined business process. In another, upcoming blog I’ll explain how to analyze the data against the process definition to find improvement points.
In principle you use:
- Process Explorer to organize the different artifacts you created or that were shared by others like: BPMN 2.0 diagrams, value maps, customer journey maps, UML diagrams and several more. The look-and-feel is similar to Windows Explorer or a file manager on Linux.
- Collaboration Hub to share documents with the wider community in the company (ideally every employee should be able to look at the defined company process that impacts him/her). From there feedback can be provided.
- Process Manager to define, build and manage process definitions.
- Process Intelligence to extract, transform and load data from business systems or files that contain process data for measurement and to map them to process definitions.
- Process Insights to continuously monitor what has been set up in Process Intelligence to identify bottlenecks, instabilities and compare across plants or company codes within a group.
C. On-premises extractor setup
To retrieve data from SAP S/4HANA on-premise you need the On-premises Extractor provided as part of Signavio Process Intelligence. Note that this tool can connect to a multitude of on-premises systems and S/4HANA is just one of them.
You can find it in the Process Intelligence → Manage Data menu.
Through creation you obtain secrets that you have to store at the place where you run the extractor application.
Since the extractor is a component that must be run somewhere (e.g. a local server, notebook etc.) I decided to deploy it on SAP BTP Kyma, SAP’s Kubernetes offering.
For that you need to create a Docker image and in case you would like to replicate I share the Dockerfile below. Here are the detailed system requirements.
#Build SAP Signavio On-Premises Extractor Image FROM openjdk:11.0.16-oraclelinux7 WORKDIR /usr/signavio COPY . /home RUN yum -y update && \ yum -y install nano && \ yum -y install unzip EXPOSE 443/tcp
Dockerfile for the extractor. Put the downloaded and zipped on-premises extractor file in the same folder where the dockerfile resides. It will be copied into the image.
apiVersion: v1 kind: Namespace metadata: name: dl-signavio labels: istio-injection: enabled --- apiVersion: apps/v1 kind: Deployment metadata: name: signavio-extractor-app namespace: dl-signavio spec: replicas: 1 selector: matchLabels: app: signavio-extractor-app template: metadata: labels: app: signavio-extractor-app spec: containers: - name: signavio-extractor-image image: <imagename from dockerhub>:1.0 imagePullPolicy: Always ports: - containerPort: 443 resources: requests: memory: "2Gi" cpu: "250m" limits: memory: "3Gi" cpu: "500m" volumeMounts: - mountPath: /home/config name: signavio-extractor-config-configmap command: ["/bin/sh"] args: ["-c", "if [ ! -f /usr/signavio/start.sh ]; then cd /home && unzip /home/*.zip && cd pi* && mv ./* /usr/signavio && rm /home/pi* -r && cp /home/config/config.env /usr/signavio/config.env; else echo Already installed, starting; fi; cd /usr/signavio && ./start.sh"] volumes: - name: signavio-extractor-config-configmap configMap: name: signavio-extractor-configfile --- apiVersion: v1 kind: Service metadata: labels: app: signavio-extractor-app name: signavio-extractor-service namespace: dl-signavio spec: ports: - name: "adminport" port: 443 targetPort: 443 type: ClusterIP selector: app: sapcc-app --- apiVersion: v1 kind: ConfigMap metadata: name: signavio-extractor-configfile namespace: dl-signavio data: config.env: | ############ PLEASE INCLUDE YOUR EXTRACTOR ON-PREM CONFIGURATION BELOW ##################### EXTRACTOR_ID=<my ID> EXTRACTOR_SECRET=<it's a secret! ;-)> WEBSOCKET_SERVER_ADDRESS=wss://<your signavio server>.suite-saas-prod.signav.io
You see the config.env file is added to kubernetes deployment yaml file. You’ll get the content for this file at the time you set up a new extractor in Signavio.
This yaml will extract the extractor 😀 (from its zip file) and then start it.
With that in place we can deploy it on SAP BTP Kyma. If you look for an introduction to Kyma, see the end of the blog.
We should check the log for the pod or – if we are bold go straight away to Signavio PI to check for the extractor status. After ca. 1 min of deployment it should come online.
D. Extraction method and details
The Signavio On-Premises Extractor uses an RFC connection to the S/4HANA or ECC system based on the Java Connector (JCo). In the S/4HANA itself then, the well known RFC_READ_TABLE function module serves to generically read the table data.
This module has a limitation to 512 characters per table row in the TABLES parameter. There was a recommendation to create a Z-function module and specify that in the connection settings. It seems however, that with the recent addition of the ET_DATA export parameter using strings (and therefore removing the 512 characters limitation) that was removed. Please see the relevant note 2246160 in the references at the end of the blog.
One thing I found worth to mention – make sure that in the connection settings you have all tables defined that are part of the data pipeline! If you don’t there is only a java error raised that won’t be understandable easily:
com.signavio.pi.etl.extractor.shared.exception.EtlException: ExtractionJob failed | source-system: SAP | exception: java.lang.IllegalStateException | cause: null | sql-query: [SELECT "MANDT" as "mandt", "EKGRP" as "ekgrp", "EKNAM" as "eknam" FROM T024 ... | Caused by: java.lang.IllegalStateException: Unable to obtain the schema for table: T024
See the screenshot below of the relevant settings.
E. Process data pipeline and extraction
A process data pipeline is a chain of actions with a reference to a business process defined in SAP Signavio. It is that referenced process which is measured and later analyzed using Signavio Process Intelligence or monitored through Signavio Process Insights.
Often it’s a good idea to start with a template prepared – it saves time to assemble the S/4HANA tables needed and compiling the data model.
I’ve prepared a little video explaining the steps from template to extraction.
F. Outlook and next steps
I hope this first tutorial was useful to show how on-premise data is extracted and loaded into SAP Signavio. In an upcoming blog I will show how to analyze the procure-to-pay data inside SAP Signavio PI for optimization and value realization. Happy to receive your feedback!
- Overview of connectors for SAP Signavio – link
- SAP Signavio Process Intelligence Overview – link
- SAP BTP Kyma: Build an Application in the Kyma Runtime – link
- SAP Note 2246160 – Enhancement RFC_READ_TABLE (7.40+) – link
Thank you very much for this article.
Finally someone sheds some light on the on-prem extraction.
Could you let me know from where you got the zip file of the extractor? It still eludes us so far.
Once again, thank you!
great to hear you find it useful! The zip file is obtained as indicated in this screenshot.
Hope this helps!
Thanks a lot. We are completely missing this download button.
But it is already good to know how it's supposed to look, so thanks.
will this extractor also be available via the SAP Launchpad / SAP4me?
Also, where can we find the release notes (if there are any) for the extractor?
(For example pi-etl-extractor-onprem-v1.559.0 seems to use EOL SAP JCO component version 3.0.18 from 2018)
Thanks in advance!
Good day Martin Strobel
plan is to move the extractor package in the future into the SAP Software Download Center where all other solutions are hosted for download as well. This might happen later this year.
When this happens it might also be that release notes will be a part of that package, currently it is not.
This is information from the experts at SAP for Signavio (which I am not). Hope it helps. 😌