EIPinCPI – Pipes-and-Filters
Previous – Format Indicator | Index | Next – Message Router
This week we begin studying Message Routing patterns. First Message Routing pattern is Pipes-and-Filters.
When do I use this pattern?
Use this pattern when the steps for the integration are reusable. The simplest example is reusing a flow that connects to the receiver system. For example, consider integrating a CRM system with ECC. The CRM system sends Customer and Orders to the ECC system. Here, the flow for both Customer and Order consists of 3 steps:
- Read from CRM System
- Map to IDoc
- Send IDoc to ECC
Reading the orders and customers would require separate flows. Mapping will also require separate flows as the source and target messages are different. However, sending IDoc to ECC can be designed to be a reusable step.
Pipes-and-Filters in CPI
In CPI, I’ll use JMS Adapters to demonstrate the Pipes-and-Filters pattern.
Demo of Pipes-and-Filters – Customer Integration
Let’s consider Northwind as the CRM system that creates/updates Customers. The flow will read customers from Northwind, map them to DEBMAS IDoc, and send the DEBMAS IDoc to ECC.
Step 1: Read Customer
Here, OData Adapter is used to read Customers from Northwind and JMS Receiver Adapter is used to send the read customers into ‘Customers’ queue.
Here, the Read Customer demonstrates Request-Reply pattern.
Step 2: Map Customer
Here, customers are read from the ‘Customers’ queue using JMS Sender Adapter, mapped to DEBMAS IDoc and put into ‘ECC’ queue.
Step 3: Send to ECC
This flow reads from ‘ECC’ queue and posts the IDoc to ECC.
Demo of Reusability in Pipes-and-Filters – Order Integration
When sending orders from the CRM system into ECC, the same 3 steps are applicable. Read orders from Northwind, map them to ORDERS IDoc, and send the ORDERS Idoc to ECC. However, the 3rd step of sending the IDoc to ECC is a reusable component.
Step 1: Read Order
Similar to Customer Integration, this flow reads orders and puts them into the ‘Orders’ queue.
Step 2: Map Order
Similar to Customer Integration, this flow reads orders from the ‘Orders’ queue, maps them to ORDERS IDoc, and puts them in the ‘ECC’ queue.
Step 3: Send to ECC
This is a common step between Customer integration and Order integration. It reads from ‘ECC’ queue and sends the IDocs to ECC.
Monitoring tip for Pipes-and-Filter pattern
Dividing the flow of a message into several components like this does not make message monitoring difficult. Use the SAP_ApplicationID header to effectively understand the flow of the message. Check the 5th tip in Eng Swee‘s tips for CPI development for more information. For official documentation, search for SAP_ApplicationID in Headers and Exchange Properties provided by the Integration Framework.
Pipes-and-Filters is a powerful pattern that promotes creating small components that are reusable. The dependency between these components is reduced through asynchronous processing. The Asynchronous processing is implemented by the use of JMS queues in CPI.
- Pipes-and-Filters Pattern in Enterprise Integration Patterns
- Eng Swee‘s tips for CPI development
- Search for SAP_ApplicationID in Headers and Exchange Properties provided by the Integration Framework
- CPI Components
Hope this helps,
Good one again.
I did not know this name, but I guess something that I have used often.
I guess the JMS can be replaced by process direct in many cases, depending on what you want to achieve with it.
Thanks Daniel Graversen.
I was also happy to have a name for the pattern so commonly used in CPI.
When strictly following this pattern, I wouldn’t use the ProcessDirect between flows because ProcessDirect in CPI does not support asynchronous processing. The reusable steps should be able to process the message without depending on any other step at runtime.
I believe Camel’s Direct VM component is underneath ProcessDirect. Camel’s Direct VM component has a query parameter named ‘synchronous’ to set whether synchronous processing should be strictly used, or Camel is allowed to use asynchronous processing (if supported) [Source: https://camel.apache.org/components/latest/direct-vm-component.html#_query_parameters_11_parameters). This is something for CPI to adopt from Camel. ?