SAP CPI and SAP SuccessFactors Monitoring EP02: The one where CPI writes logs to Execution Manager

*Note: This post is specifically targeted to my fellow SAP CPI developers who design SAP SuccessFactors integration solutions and is in continuation to my last post in the series SAP CPI and SAP SuccessFactors Monitoring.*

Firstly, a little background. Too often in the past I have seen support calls from customers about SAP SuccessFactors to/from SAP ERP integrations and a request to “find out what went wrong.” Many/most of these issues could be solved if only the SAP SuccessFactors administrators had access to any error/success messages that the integrations were producing. The aim of these blogs is to show how easy it is to incorporate logging to SAP SuccessFactors Execution Manager into all your SAP CPI integrations so that admins and developers have a simple way to monitor all SAP SuccessFactors integrations.

So, let’s begin

Now that we have an idea of how easy it is to log information into the Execution Manager Dashboard in SAP SuccessFactors, it is time to put our knowledge to use. As Dale Carnegie once said,

“Knowledge isn’t power until it’s applied.“,

It’s time for us to finally go ahead and see a bigger picture of how exactly we can harness the Execution Manager to our advantage.

In this article, I’ll be focusing on how I implemented the Execution Manager in SAP Cloud Platform Integration iFlow.

Bird’s eye view of the implementation

This implementation consists of an iFlow and a simple reusable groovy script to generate EMEvent payload groovy script. Just to make things a bit modular here, I broke the iFlow into the following integration processes:

1. Main integration process,
2. Few local integration processes for the following:
   2.1 populating demo data (in external parameters and body) to imitate the process of generating data to be posted to Execution Manager Dashboard, and
   2.2 sending data to Execution Manager Dashboard using HTTP adapter

3. A separate local integration process for exception handling, and
4. A groovy script to generate EMEvent payload to be sent to Execution Manager Dashboard in SAP SuccessFactors.

Now before I get into the details, I would like to quote an expression by Tess Flanders in 1911 that fits here perfectly,

“Use a picture. It’s worth a thousand words“.

So, here I present you how exactly the implementation looks like when it is ready to log everything we need in the Execution Manager Dashboard **drumrolls**

Fig.1: iFlow to log data in Execution Manager Dashboard

Digging into the iFlow

Now let’s see what’s really happening here in the iFlow above (I’ll try explaining things in the order I’ve mentioned above).

1. Main Integration Process

Fig.2: Main integration process

In this example, every execution of the main integration process logs 3 separate events into the execution manager dashboard, i.e. START – data/error/warning/failure (you can log as many events representing these between a START and END event) – END :

a) Initializing the external properties for EMEvent payload. Here’s what the ‘initialize properties’ step looks like. I have included a sample payload to log a single event from my previous post and how it maps to the properties initialized.

Fig.3: Properties defined for EMEvent and EMMonitoredProcess payload

*note: The EMMonitoredProcess payload needs to remain the same, as a single execution of the iFlow represents a process in Execution manager. All the events logged under a single process are clubbed together enclosed within a ‘START’ and an ‘END’ event.*

b) Once the properties are initialized, the properties for the EMEvent payload needs to be changed for every event we wish to log. The most important one here is the ‘event_type’ property as it decides the type of event logged into the execution manager. So, the next step here is to change the event_type in a content modifier to ‘START’ as we initially need to log a START event.

c) The next step in line is to log the data that matters in the form of many events showing data that is required to be presented in the Execution Manager Dashboard. That is where the ‘log data’ step comes in, which triggers the Log data integration process mentioned in the next section. This integration process can be used to fire multiple events as the iFlow carries on its execution. In this case, the iFlow is designed to log a single event of the type INFO to represent a portion of information.

d) Once, all the data/error/warnings have been logged during the execution, there is another enclosing event that needs to be logged. You guessed it right, END event it is 🙂
If we fail to log the END event for events, the process in all would have an ‘IN_PROGRESS’ state rather than the expected ‘COMPLETED’ state (please refer to my previous blog if you need a reference to different scenarios and the final process state for each).

2. Local Integration Processes

2.1 Log Data integration process

Fig.4: Generating data to log

This local integration flow is to represent a process/set of processes that execute during the lifecycle of an integration flow where the iFlow does what it is actually designed to do (pick data from source A, transform, write into destination B). This could be a single local integration flow, or a set of flows to get your job done. This is where we log the events depending on what data is really required by the business.

2.2 Execution Manager Logger integration process

Fig.5: Generating and logging payload to Execution Manager Dashboard

This local integration flow is principally responsible for 2 tasks:

• • Executing a script which generates a payload to log an event by using the external properties and body populated by components before this integration process is called.

• • Firing a HTTP call to SAP SuccessFactors Execution Manager Dashboard to log the event.

3. Global Exception Handler

Fig.6: Handling errors and exceptions

While working on the solution, there were times when the iFlow during its execution ran into errors such as errors with the script, HTTP connection issues, etc. that caused the iFlow to fail. But how does that impact our logging and why should we care?
As mentioned before, it is important to log an ‘END’ event for every execution to ensure the EMMonitoredProcess does not end up in a ‘IN_PROGRESS’ state even when the actual execution of the iFlow has stopped/failed.

Therefore, this local integration process is to take care of all the errors/exceptions/failures that occur during the execution and ensuring that proper error messages are logged in to the Execution Manager, followed by an ‘END’ event.

4. EMEvent payload generating groovy script

As simple as it should be, this script picks up data from the external properties and populates them into one single JSON payload consisting of an EMEvent and an EMMonitoringProcess.

//EMEvent payload construction
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import groovy.json.*;

def Message processData(Message message) {
    //EMEvent
    def map = message.getProperties()
    def eventInfo = message.getBody(String.class)
    def eventNameValue = map.get("eventName")
    def eventTypeValue = map.get("eventType")
    def eventDescriptionValue = map.get("eventDescription")
    def eventTimeValue = "/Date("+new Date().getTime()+"+0000)/"
    def nameValue = map.get("ea_name")
    def valueValue = map.get("ea_value")

    // EMMonitoredProcess
    def processTypeValue =  "INTEGRATION"
    def processDefinitionIdValue =  map.get("processDefinitionId")
    def processDefinitionNameValue = map.get("processDefinitionName")
    def processInstanceNameValue =  map.get("processInstanceName")
    def processInstanceIdValue =  map.get("processInstanceId")

    def json = new JsonBuilder()

    def jsonObject = json {
        eventName  eventNameValue
        eventType eventTypeValue
        eventDescription eventDescriptionValue
        eventTime eventTimeValue
        eventAttributes {
            name nameValue
            value valueValue
        }
        process {
            processDefinitionId processDefinitionIdValue
            processDefinitionName processDefinitionNameValue
            processInstanceId processInstanceIdValue
            processInstanceName processInstanceNameValue
            processType processTypeValue
        }
    }

    message.setBody(json.toString())
    //Properties
    map = message.getHeaders()
    message.setHeader("Content-Type", "application/json")
    message.setHeader("Accept", "application/json")

    return message
}

Script 1: Generating EMEvent payload

*disclaimer: Script provided for discussion purposes only, any attempt to use this script or a variation of it in your own integrations is at your own risk. No fitness for use in any particular purpose is, or should be, implied.*

Reaping the results

Just for the sake of demonstration, I have considered executing the iFlow 4 times, once for each ‘event_type’ and its impact on the ‘final process state’ of the EMMonitoredProcess/execution as mentioned below.

Table 1: EMEvent use case scenarios and their impact on final process state

And this is what the scenarios look like when executed one after the other:

Fig.7: Execution Manager Dashboard after logging the events

But how do I debug this? How do I trace a Process/execution back to the message in CPI?

If you scroll back up to Fig.3 and look at the EMMonitoredProcess payload, you will notice an attribute called ‘processInstanceId’. Process Instance ID as the name suggests, is to uniquely identify a single execution/process which clubs together several events under it.

But don’t we have something similar in SAP CPI to identify a particular execution of an artifact/iFlow?
As mentioned in the Developer’s Guide (2020), every message that travels across in an iFlow has a few header fields by default which has information about the iFlow/artifact they belong to. One such header field is ‘SAP_MessageProcessingLogID’ which uniquely identifies a particular message/execution of the iFlow and can be found under the Label MessageID in CPI’s ‘Monitor Message Processing’ page.

Let me show you below how the ‘processInstanceId’ that we populated with the field ‘SAP_MessageProcessingLogID’ in CPI can be correlated with Execution Manager Dashboard’s ‘Process Instance ID’.

Fig.8: Mapping an EMMonitoredProcess to an execution in SAP CPI

Conclusion

With everything in place, I hope you can see how this is an effective and relatively simple way to standardize the monitoring of integration flows in SAP CPI for an SAP SuccessFactors admin. It does this in a way that collates all the integrations associated with your SAP SuccessFactors instance in a single place to enable the developers and admins to monitor everything from a single dashboard.

This not only helps the developers but the business users who need to check the overall performance of the Integrations without having to access SAP CPI and learn how to navigate that platform as well as SAP SuccessFactors. 

That’s all Folks!

I hope that you enjoyed these posts, I’d like to thank my colleagues at Discovery Consulting for helping me pull these together. We recently took part in a global SAP SuccessFactors partner best practice discussion, where this concept was presented. Hopefully, sometime in the near future we may see some official documentation come from SAP about how best to implement such logging in your integrations! In the meantime, it’s been a good experience to write about this and I’m keen to hear if others have tried similar or are now inspired to do so!

Thanks for your time!

References

Developer’s Guide, 2020. Developer’s Guide: Managing Integration Content. [Online]
Available at: https://help.sap.com/doc/dd250f2e3c2645a8ae327e935071281e/Cloud/en-US/DevGuide_ManageIntContent_External.pdf
[Accessed 11 08 2020].