In the Interface Strategy one of the most obvious section which should be considered more effectively is Exception handling. How the design/mapping/runtime issues(if any) should be handled and which all business cases can be grouped to adapt similar kind of notification/resolution etc., . All this can be done with a calibrated approach of implementing the Exception Subprocess blocks either in the Integration Process or Local Integration Processes accordingly.
All the business requirements may not be the same so it depends on the case to case scenario how the simple or advanced exception handling can be implemented. Lets check out all the possible templates or ways in which Exception Handling can be done.
Template 1: Raise an Custom/Standard Exception using Content Modifier or Groovy Script
In the Content Modifier body, we can use ${exception.message},${exception.stacktrace}, ${property.CamelExceptionCaught} accordingly to log the error in MPL and send it to the sender
If we use the Groovy Script then the below sample can also be used to trigger the customized Message.
The custom status in MPL can also be set using “SAP_MessageProcessingLogCustomStatus” when an exception occurs with completing the actual message status as success or failed using the error end event accordingly in the Exception Subprocess.
Template 2: Log Payload of the source or intermediate or target payload
Apart from externalized parameter value of logging TRUE or FALSE and then using the groovy script for storing the payload it can be done using the same groovy script inside the exception subprocess in case of exception only.
This option is very much required when the externalized parameter value is not set to TRUE or its not at all used. In doing so the memory circuit breaker of 1 GB per day should be considered accordingly.
Template 3: Get Payload and Set Headers to send EMAIL
One of the mostly used exception handling approach of sending an email when there is any issue in the IFLOW steps. The error message alone or the input payload or any other required data can be set from the main flow using the Headers and the same can be retrieved into the exception subprocess. Other email related header also can be set in the Exception subprocess flow and the final email can be sent to the configured recipients.
Fancy Emails can be sent from CI as mentioned in this blog: here
Template 4: Get Payload and Set Headers to call Global Exception using Process Direct
One of the commonly used Global Exception Handling approach of sending an email when there is any issue in the IFLOW steps. Based on the workstream or functional or technical requirement we can have the global exception handling to facilitate the reusability of the artifacts.
All the required payloads/headers can be set and get in the exception subprocess. The supported templates for migration tooling which already provides the setting of Headers can also be used for the exception subprocess. Also the local headers also can be used for EMAIL attributes to form a complete required body which can be passed to the global exception handling through the process direct
Template 5: Get Payload and generate the file in Error Folder
For some of the business cases where the requirement will be to save all the payload messages in case of any issues in the Integration layer so that it can be resent altogether from the source system or can have separate IFLOW for picking those files and resending to the target system after analysing the issues occurred.
From the main flow all the required payload to be set and get in the exception subprocess along with other required details and generate a file in the cloud or onprem SFTP/FTP folders.
Maintaining a separate File system can be infrastructure burden which also needs frequent monitoring on memory outages or downtimes. Reprocessing of the failed files should also be monitored again for any issues which may again end up moving the files still to another sub error folder and so on.
Template 6: Get Payload and INSERT in the Database
The major issues which are present in maintaining a FILE system can be more or less be avoided by maintaining a cloud or onprem Database system by configuring a FLAG for the error records which loaded into the tables.
The required payload message should be set in the main flow using the Header and then get in the exception subprocess and prepare the respective JDBC structure for inserting into the database tables.
Maintaining a separate Database system also can be infrastructure burden which also needs frequent monitoring on memory outages or downtimes. Reprocessing of the failed files should also be monitored again for any issues by configuring a FLAG for each record.
Template 7: Get Payload and INSERT in the JMS Queue
One of the most used exception handler for the interface is usage of JMS queue. For all the critical interfaces which needs to have a guaranteed delivery are designed using JMS queue in the success or failure case.
The payload can be set and get in the exception subprocess and the message can be just stored in the JMS queue. Separate interface should developed to pick those messages again from JMS queue and send to the original target system.
For a landscape of having considerable amount of interfaces can be redirected to JMS queues to have the retry mechanism happen in case of any errors. But in case of thousands of interfaces landscape the redirection to JMS queue cannot be possible considering its resource limitation of 30 Queues/9 GB of total tenant space.
Template 8: Get Payload and Store in Datastore
For the cases where the payload memory is huge and the retry mechanism should be achieved this was one of the option technically. But functionally it has its own flaws.
Set the payload in the Headers and get it in the exception process and store in the Datastore. Separate flows should be configured to get the message read and processed to the actual target system.
The disadvantage of limited memory size of JMS queues can be overcome by the Datastore which share the tenants memory size of 32/35 GB but then the datastore memory is always temporary in nature meaning it cannot be longer than 180 days. Also the querying is not that appropriate when compared to a normal database.
Template 9: Get Payload and send to AMQP (MQ/RABBIT/SONIC) Queues
There can be a case if the original target system is a Queue system and if some errors during the message processing in CI should be redirected to the Error Queues on the same Queue Managers.
The required payloads can be set in the headers and get it in the exception subprocess and send to the respected queue using the AMQP channel. Ideally for reprocessing again there should be separate IFLOWs reading the failed message and process to the target system.
If the original target system is not a Queue System then maintain a separate queue system will be a infrastructure burden and this should be equally justified.
Also the external calls for storing the failed messaged unlike JMS/Datastores will always have extra risk of memory and outages.
Template 10: Set the notification body and send to SAP AIF
For the scenarios of Interfaces receiving or sending the messages to SAP S4 HANA with AIF enabled which has its own monitoring or retrying for data issues can be requirement to send the CI errors also to the AIF for common monitoring of workstream or functional based flows.
In the Exception Subprocess, the Content Modifier’s Exchange Property section should be configured with the required parameters like AIF_Namespace, AIF_Interface_Name, AIF_Interface_Version, Cloud_Integration_Tenant_Name etc., . Based on this parameters and others the body should be defined.
If the Message ID in the SOAP Header is filled, it shouled be deleted from the Message Header in the Exception Subprocess.In the Content Modifier the properties SapMessageIdEx and SapMessageId should be defined with delete action
AIF is not a default addon. It has its own subscriptions and should be managed accordingly.
Template 11: Set the notification body and send to SAP Event Mesh
Not all business process will expect the traditional approach of handling errors. If there is a valid need for continuous the further process with the required steps it should be followed in the same way as documented.
If there is a requirement to publish the error notification on to the Event Mesh so that the configured consumer can receive those error event and continue further process.
All the required notification body should be prepared in the Exception Subprocess body and send to the Event Mesh using the AMQP(websocket) channel.
SAP Event Mesh need separate subscription and is not included in Integration Suite.
Configuring the Event Mesh along with CI can be found here
Template 12: Execute the mandatory iflow steps even after an exception
There can be a case where some of the IFLOW steps to be executed compulsorily those steps can be included in the exception subprocess by calling a local integration process.
In the called local integration process all the steps of groovy script, content modifiers, routers, mappings can be defined. So that even in the case of failure these steps will be executed.
If the same steps mentioned in the local integration process should be executed even in successful case then the same steps to be also configured in the main IFLOW.
Template 13: Common Exception Handling for Integration Process and Local Integration Processes
There can be a case where the global exception handling may/may not be used and need a specific error handling which should be common for entire IFLOW meaning the main Integration Process or Local Integration Process it should be same.
Another case of reusability of Exception handling in which the exception subprocess of main integration process is configured with the process call and the local integration integration is configured with same process call in its local exception subprocess. There by for both process common steps can be executed as part of exception handling.
Template 14: Send the message to Open Connectors
If the requirement is to send the exception message to WhatsApp or any ticketing tool like JIRA then the open connectors like JIRA/JIRA On Prem or Twilio/Twilio SMS can be used.
All the prerequisite configuration in the open connectors should be done accordingly on the selected open connector like Twilio or JIRA to use those parameters in the Open connector channel.
The required payload and other properties can be set and get in the exception subprocess and send onto open connector.
Enough permission should be required for configuring the open connectors and their corresponding systems with respective API KEY, authorization etc.,
Template 15: Hybrid Exception Handling
One or more combinations from the above templates can be implemented in single interface exception subprocess as per the business requirement.
For example if the requirement is to generate a file with failed payload along with sending a email to the designated recipient then set the payload from the main flow and get it in the exception subprocess. Configure a send step with FTP channel to its receiver and another send step with mail channel which can be executed one by one in a sequential flow.
If one step fails to achieve the error handling then the other step will be sufficient in not making the error unnoticeable.
If all the steps to be executed irrespective of its own failure chances also then we can have a parallel multicast and configure all the send steps.
Desired Improvement related to Exception Subprocess:
Please refer to the Feature 2 section in this blog: here
Limitations of Exception Subprocess:
If there is an integration flow scenario with a nested multicast pattern along with an Exception Subprocess present in the same Integration Process to handle any errors.If any flowstep on any inner multicast (Ex: Inner Multicast 1 -> Branch 1 -> Error Script) branches throws an exception then this exception is not identified by the exception subprocess.This is a known limitation with nested Multicast and Exception subprocess combination.
In order to handle the exceptions on nested Multicast branches, the integration flow model has to be modified following the one-multicast-per-integration-process strategy. This means that each internal multicast step along with its branches has to be moved to a new Local Integration Process with its own Exception Subprocess to handle any errors. For the same example as above the Inner Multicast 1 has to be moved to a separate Local Integration Process with its dedicated exception subprocess. With such arrangement any error happening on Inner Multicast 1 shall be identified by Exception Subprocess 2 in Local Integration Process. Same steps must be followed until further levels of nesting in multicast.
Conclusion:
All these templates along with further customization if provisioned then with a click of a button the draft version of the Exception Subprocess will also be created.
Just sending notification mails as a Error Handling may eventually land into thousands of emails in the configured recipient’s Email BOX where tracking can be more cumbersome.
On top of email notification if there is more reliable and traceable option provided then it can be considered as a good design of the Interfaces Exception handling
Disclaimer:
This article is only for informational purpose listing all the possible templates of Exceptional handling in Exception Subprocess with Error End Event. If required Escalation End Event can also be used accordingly.
Few cases are shown as an example but no limitations in any perspective.
All the business requirements may not be the same so it depends on the case to case scenario how the simple or advanced exception handling can be implemented. Lets check out all the possible templates or ways in which Exception Handling can be done.
Template 1: Raise an Custom/Standard Exception using Content Modifier or Groovy Script
In the Content Modifier body, we can use ${exception.message},${exception.stacktrace}, ${property.CamelExceptionCaught} accordingly to log the error in MPL and send it to the sender
If we use the Groovy Script then the below sample can also be used to trigger the customized Message.
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
def Message processData(Message message) {
throw new Exception('Hey Farooq there is an exception:-' + message.getProperty('CamelExceptionCaught'))
return message}The custom status in MPL can also be set using “SAP_MessageProcessingLogCustomStatus” when an exception occurs with completing the actual message status as success or failed using the error end event accordingly in the Exception Subprocess.
Template 2: Log Payload of the source or intermediate or target payload
Apart from externalized parameter value of logging TRUE or FALSE and then using the groovy script for storing the payload it can be done using the same groovy script inside the exception subprocess in case of exception only.
This option is very much required when the externalized parameter value is not set to TRUE or its not at all used. In doing so the memory circuit breaker of 1 GB per day should be considered accordingly.
Template 3: Get Payload and Set Headers to send EMAIL
One of the mostly used exception handling approach of sending an email when there is any issue in the IFLOW steps. The error message alone or the input payload or any other required data can be set from the main flow using the Headers and the same can be retrieved into the exception subprocess. Other email related header also can be set in the Exception subprocess flow and the final email can be sent to the configured recipients.
Fancy Emails can be sent from CI as mentioned in this blog: here
Template 4: Get Payload and Set Headers to call Global Exception using Process Direct
One of the commonly used Global Exception Handling approach of sending an email when there is any issue in the IFLOW steps. Based on the workstream or functional or technical requirement we can have the global exception handling to facilitate the reusability of the artifacts.
All the required payloads/headers can be set and get in the exception subprocess. The supported templates for migration tooling which already provides the setting of Headers can also be used for the exception subprocess. Also the local headers also can be used for EMAIL attributes to form a complete required body which can be passed to the global exception handling through the process direct
Template 5: Get Payload and generate the file in Error Folder
For some of the business cases where the requirement will be to save all the payload messages in case of any issues in the Integration layer so that it can be resent altogether from the source system or can have separate IFLOW for picking those files and resending to the target system after analysing the issues occurred.
From the main flow all the required payload to be set and get in the exception subprocess along with other required details and generate a file in the cloud or onprem SFTP/FTP folders.
Maintaining a separate File system can be infrastructure burden which also needs frequent monitoring on memory outages or downtimes. Reprocessing of the failed files should also be monitored again for any issues which may again end up moving the files still to another sub error folder and so on.
Template 6: Get Payload and INSERT in the Database
The major issues which are present in maintaining a FILE system can be more or less be avoided by maintaining a cloud or onprem Database system by configuring a FLAG for the error records which loaded into the tables.
The required payload message should be set in the main flow using the Header and then get in the exception subprocess and prepare the respective JDBC structure for inserting into the database tables.
Maintaining a separate Database system also can be infrastructure burden which also needs frequent monitoring on memory outages or downtimes. Reprocessing of the failed files should also be monitored again for any issues by configuring a FLAG for each record.
Template 7: Get Payload and INSERT in the JMS Queue
One of the most used exception handler for the interface is usage of JMS queue. For all the critical interfaces which needs to have a guaranteed delivery are designed using JMS queue in the success or failure case.
The payload can be set and get in the exception subprocess and the message can be just stored in the JMS queue. Separate interface should developed to pick those messages again from JMS queue and send to the original target system.
For a landscape of having considerable amount of interfaces can be redirected to JMS queues to have the retry mechanism happen in case of any errors. But in case of thousands of interfaces landscape the redirection to JMS queue cannot be possible considering its resource limitation of 30 Queues/9 GB of total tenant space.
Template 8: Get Payload and Store in Datastore
For the cases where the payload memory is huge and the retry mechanism should be achieved this was one of the option technically. But functionally it has its own flaws.
Set the payload in the Headers and get it in the exception process and store in the Datastore. Separate flows should be configured to get the message read and processed to the actual target system.
The disadvantage of limited memory size of JMS queues can be overcome by the Datastore which share the tenants memory size of 32/35 GB but then the datastore memory is always temporary in nature meaning it cannot be longer than 180 days. Also the querying is not that appropriate when compared to a normal database.
Template 9: Get Payload and send to AMQP (MQ/RABBIT/SONIC) Queues
There can be a case if the original target system is a Queue system and if some errors during the message processing in CI should be redirected to the Error Queues on the same Queue Managers.
The required payloads can be set in the headers and get it in the exception subprocess and send to the respected queue using the AMQP channel. Ideally for reprocessing again there should be separate IFLOWs reading the failed message and process to the target system.
If the original target system is not a Queue System then maintain a separate queue system will be a infrastructure burden and this should be equally justified.
Also the external calls for storing the failed messaged unlike JMS/Datastores will always have extra risk of memory and outages.
Template 10: Set the notification body and send to SAP AIF
For the scenarios of Interfaces receiving or sending the messages to SAP S4 HANA with AIF enabled which has its own monitoring or retrying for data issues can be requirement to send the CI errors also to the AIF for common monitoring of workstream or functional based flows.
In the Exception Subprocess, the Content Modifier’s Exchange Property section should be configured with the required parameters like AIF_Namespace, AIF_Interface_Name, AIF_Interface_Version, Cloud_Integration_Tenant_Name etc., . Based on this parameters and others the body should be defined.
If the Message ID in the SOAP Header is filled, it shouled be deleted from the Message Header in the Exception Subprocess.In the Content Modifier the properties SapMessageIdEx and SapMessageId should be defined with delete action
AIF is not a default addon. It has its own subscriptions and should be managed accordingly.
Template 11: Set the notification body and send to SAP Event Mesh
Not all business process will expect the traditional approach of handling errors. If there is a valid need for continuous the further process with the required steps it should be followed in the same way as documented.
If there is a requirement to publish the error notification on to the Event Mesh so that the configured consumer can receive those error event and continue further process.
All the required notification body should be prepared in the Exception Subprocess body and send to the Event Mesh using the AMQP(websocket) channel.
SAP Event Mesh need separate subscription and is not included in Integration Suite.
Configuring the Event Mesh along with CI can be found here
Template 12: Execute the mandatory iflow steps even after an exception
There can be a case where some of the IFLOW steps to be executed compulsorily those steps can be included in the exception subprocess by calling a local integration process.
In the called local integration process all the steps of groovy script, content modifiers, routers, mappings can be defined. So that even in the case of failure these steps will be executed.
If the same steps mentioned in the local integration process should be executed even in successful case then the same steps to be also configured in the main IFLOW.
Template 13: Common Exception Handling for Integration Process and Local Integration Processes
There can be a case where the global exception handling may/may not be used and need a specific error handling which should be common for entire IFLOW meaning the main Integration Process or Local Integration Process it should be same.
Another case of reusability of Exception handling in which the exception subprocess of main integration process is configured with the process call and the local integration integration is configured with same process call in its local exception subprocess. There by for both process common steps can be executed as part of exception handling.
Template 14: Send the message to Open Connectors
If the requirement is to send the exception message to WhatsApp or any ticketing tool like JIRA then the open connectors like JIRA/JIRA On Prem or Twilio/Twilio SMS can be used.
All the prerequisite configuration in the open connectors should be done accordingly on the selected open connector like Twilio or JIRA to use those parameters in the Open connector channel.
The required payload and other properties can be set and get in the exception subprocess and send onto open connector.
Enough permission should be required for configuring the open connectors and their corresponding systems with respective API KEY, authorization etc.,
Template 15: Hybrid Exception Handling
One or more combinations from the above templates can be implemented in single interface exception subprocess as per the business requirement.
For example if the requirement is to generate a file with failed payload along with sending a email to the designated recipient then set the payload from the main flow and get it in the exception subprocess. Configure a send step with FTP channel to its receiver and another send step with mail channel which can be executed one by one in a sequential flow.
If one step fails to achieve the error handling then the other step will be sufficient in not making the error unnoticeable.
If all the steps to be executed irrespective of its own failure chances also then we can have a parallel multicast and configure all the send steps.
Desired Improvement related to Exception Subprocess:
Please refer to the Feature 2 section in this blog: here
Limitations of Exception Subprocess:
If there is an integration flow scenario with a nested multicast pattern along with an Exception Subprocess present in the same Integration Process to handle any errors.If any flowstep on any inner multicast (Ex: Inner Multicast 1 -> Branch 1 -> Error Script) branches throws an exception then this exception is not identified by the exception subprocess.This is a known limitation with nested Multicast and Exception subprocess combination.
In order to handle the exceptions on nested Multicast branches, the integration flow model has to be modified following the one-multicast-per-integration-process strategy. This means that each internal multicast step along with its branches has to be moved to a new Local Integration Process with its own Exception Subprocess to handle any errors. For the same example as above the Inner Multicast 1 has to be moved to a separate Local Integration Process with its dedicated exception subprocess. With such arrangement any error happening on Inner Multicast 1 shall be identified by Exception Subprocess 2 in Local Integration Process. Same steps must be followed until further levels of nesting in multicast.
Conclusion:
All these templates along with further customization if provisioned then with a click of a button the draft version of the Exception Subprocess will also be created.
Just sending notification mails as a Error Handling may eventually land into thousands of emails in the configured recipient’s Email BOX where tracking can be more cumbersome.
On top of email notification if there is more reliable and traceable option provided then it can be considered as a good design of the Interfaces Exception handling
Disclaimer:
This article is only for informational purpose listing all the possible templates of Exceptional handling in Exception Subprocess with Error End Event. If required Escalation End Event can also be used accordingly.
Few cases are shown as an example but no limitations in any perspective.
- SAP Managed Tags:
6 Comments
