Protect your EDI Pipeline with SAP APIM

[harsh_chawla]

Introduction

EDI based B2B interactions often form a communication backbone for large organizations. EDI is how large organizations receive orders from customers, procure raw materials from suppliers and track carrier shipments. In this post, I will share a reference architecture that uses APIM’s rate limiting capabilities to protect EDI messages flowing through PI. The post has the following sections:

1. EDI Pipeline as a DoS Attack Vector: Make a case for the need for traffic control in EDI pipelines


2. Traffic Control in PI: Review PI’s(limited) out of the box traffic control capabilities


3. Use SAP APIM to rate limit PI message flow: Detail an architecture that leverages APIM’s rate limiting capabilities to enable traffic control in PI

EDI Pipeline as an DoS attack vector. 

An incoming EDI document can kick off multiple business processes within a company’s ERP system and therefore an EDI document can be resource "heavy" on your backend ERP/application system. With this context in mind let’s look at a real world example:

A large organization(Acme Corp), uses SAP Transportation Management(TM) and Logistics System(SAP TM) to track thousands of global shipments in near real time. At any moment there are hundreds of shipment update EDI messages streaming into TM via the P. These shipment updates are time sensitive documents that need to be routed by PI ASAP and subsequently processed by TM. A typical carrier is sending between 10-100 updates per hour.

Carriers sending Shipment Updates to SAP TM

Let’s consider a scenario where one of the carriers' shipment tracking system has a code bug that accidentally sends a burst of 1,000 invalid/duplicate shipment updates within a short time interval(minutes)l? This extra volume may double the total shipment updates by SAP TM and grind it to a halt. 

Denial of Service Attack by Malicious Carrier

What if it’s worse? What if the cause of the volume increase is a malicious attack on the carrier’s IT infrastructure and the resulting increase in volume isn’t 10,000 but 100,000? As you think through this scenario, it becomes obvious that Acme Corp needs the ability to restrict the number of shipment updates sent by any one carrier to the backend. This limit must be configurable and enforced at the integration layer. Considering the potential for business disruption, traffic flow capabilities aren't just nice to have. Traffic flow regulation should be considered table stakes for all EDI interactions between external partners and your critical business applications.

Traffic Control in PI

Traditional middleware tools like PI are architected to process large volumes of messages as efficiently(quickly) as possible. This means that middleware tools running well designed integrations are able to deal with message volume spikes and quickly forward the messages on to backend applications. Unlike the middleware, application systems(e.g. ERPs) are more resource constrained for a host of reasons and can get backlogged when there is a sudden influx of large volumes of messages, causing performance and stability issues that can negatively impact business operations. In such scenarios, the integration team is often asked to “regulate” the flow messages into the backend systems, i.e. traffic control. SAP PI has some capabilities that allow it to "slow down" message processing. Following is a list of configurations I’ve seen used to regulate message flow:

1. Adapter Queue Threads: By limiting the number of threads per adapter type, you can limit the amount of resources that PI dedicates to a specific protocol


2. Receiver Parallelism: This configuration can be used to control the number of messages processed per receiver system, interface and namespace.


3. Sequencing(EOIO): Forcing sequencing of messages has the effect of slowing them down as the flow becomes single threaded through PI.

I’ve also come across even more extreme hacks like adding sleep commands to message mappings to hold messages in PI. While all these approaches work at slowing things down, they have less than ideal failure modes. For eg: forcing EOIO can cause valid messages to get backlogged due to a single invalid document. These options also do not provide fine grained control that is desired in our EDI example. 

Using SAP APIM to Rate Limit SAP PI

Unlike PI, SAP APIM provides multiple options to implement traffic management. You can find the details here. In our scenario we used the Quota policy in SAP APIM. The rest of the post focuses on the different ways to integrate SAP APIM with PI and elaborates the pro’s and con’s of each approach.

Option 1: Carrier > APIM > PI >TM

Use APIM to mediate calls from Carriers

The simplest design is to have PI proxy all EDI traffic via an SAP APIM proxy. APIM then uses a Quota policy to limit the amount of traffic coming from a Carrier. This would look something like this:

 

This is a suboptimal design for the following reasons:

1. Carrier Involvement: To implement this design, we would need to request 100+ carriers to change how they connect with our IT infrastructure. An ideal design change should be transparent to external partners.


2. Protocol Limitations: APIM can only implement quota policies on http based protocols, which means Carriers would have to switch away from protocols like SFTP. This would lead to extra work on ends.


3. Data Format Limitations: EDI traffic is typically batched, which means a single EDI message can contain 100 records of shipment updates. APIM doesn’t have the ability to parse EDI data and rate limit traffic on a per record level.

Option 2: Carrier>PI>APIM>TM

Send parsed EDI payloads to APIM for rate limiting

We can address the limitations of the above design by switching PI and APIM around in the pipeline as follows.:

Benefits: 

1. Transparent to Carriers: This design change can be implemented by the integration team without any involvement from the carriers. They Carriers would continue sending their EDI traffic directly to PI.


2. Protocol Independent: As all Carrier traffic is still mediated via PI, this architecture would work for SFTP and any other protocol supported by PI, independent of APIM protocol limitations.


3. Data Independent: PI can parse the EDI data and pass XML/Json versions of messages to APIM as needed

Limitations: While this design improves on option 1, it is still a suboptimal design, for several reasons:

1. APIM Outages: An SAP APIM outage will block all your EDI traffic, adding an additional point of failure to your EDI pipeline.


2. Cloud Round Trip: All EDI traffic must make a round trip from on-prem to the SAP APIM and back, adding latency.


3. All Or Nothing: A large burst of messages from one carrier will cause APIM to block messages from all carriers. Ideally we want to configure limits on a per carrier, per document basis

Option 3: Carrier>PI(APIM)>TM

Design Requirements:

Let’s document the above concerns(and more) as design requirements for the Traffic Management Architecture(TMA):

1. TMA should be re-usable across:
   
   
   
   1. External partners(customers, vendors, carrier), 
   
   
   2. Document types(PO, invoice, shipment etc) 
   
   
   3. Integration platforms(PI/CPI).
   
   
   
   


2. TMA  should implement Rate Limiting without requiring a full copy of EDI payloads to APIM


3. TMA  policy limits should be configurable outside of APIM by integration or functional teams


4. TMA should continue to run the EDI pipeline in case of APIM outages

That’s quite a list! It almost feels like we need to extract the Quota policy out of SAP APIM and push it into PI, or something close to it 🙂

After some research and strong collaboration between the PI and APIM teams we were able to achieve all our requirements via the following design:

 

Use APIM as a Throttling Engine for PI

PI Design: 

As each EDI document steps through the PI pipeline, during receiver determination PI does a GET API call to APIM for permission to process incoming shipment updates. If it receives a positive response(HTTP 200) from APIM, the documents are sent to TM. If APIM returns a rate limit response(HTTP 429), PI errors out the receiver determination step, which can be retried automatically or manually at a later time. To help APIM make a decision PI sends the following information during the GET call:

1. metering-id : unique string in the following format: <carrier_name>:<doc_type>


2. quota-limit: numeric max limit allowed before throttling


3. quota-interval: numeric time interval to be used for throttling 


4. quota-timeunit: time unit(seconds, minutes, hours etc) to be used for throttling


5. message-weight: number of records(used incase of batch files)

APIM Design: 

In the APIM proxy we dynamically map the PI HTTP header parameters to a Quota policy. The PI parameters listed above are mapped to quota attributes(link) as follows:

PI Parameter	APIM Quota Attribute
metering-id	Identifier
quota-limit	Allow
quota-interval	Interval
quota-timeunit	TimeUnit
message-weight	MessageWeight

By implementing dynamic mapping we effectively extract the Quota policy engine out to APIM and give PI developers control over when and how it gets used.

Sample Usage:

We needed to restrict each carrier to only send upto 1000 shipment updates per day of type shipment status, and you get an EDI message from a Carrier with 50 updates PI can call APIM as follows:

PI Parameter	Value
metering-id	ShipmentStatus:CarrierXYZ
quota-limit	1000
quota-interval	24
quota-timeunit	hours
message-weight	50

By implementing dynamic mapping we effectively extract the Quota policy engine out to APIM and give PI developers control over when and how it gets used. We can easily repurpose this functionality for different use cases. For eg: If you need rate limit supplier’s sending upto 100 invoices per day you could do that as follows:

PI Parameter	Value
metering-id	SupplierInvoice:Supplier123
quota-limit	100
quota-interval	1
quota-timeunit	day
message-weight	1(or number of invoices)

What if SAP APIM is down?

We can make this design resilient against SAP APIM outages by ensuring that PI only rate limits messages when it gets a HTTP 429(Too Many Requests) response from SAP APIM. In case of SAP APIM outages, PI will receive HTTP 408(Timeout) or HTTP 500(Error) responses to the GET call. We can program PI to ignore these response codes and continue processing of messages.

Summary

We can enhance SAP PI’s traffic regulation capabilities by leveraging SAP APIM in several ways. By leveraging a GET API that dynamically maps HTTP headers to Quota attributes in SAP APIM, we can create a reusable architecture. This architecture can then be leveraged to regulate any/all traffic through PI. This architecture also optimizes on performance by minimizing the amount of On-Prem to Cloud data transfers. The architecture is resilient against SAP APIM outages.

We’ve been running PI interfaces with this architecture in production environments for a few years and have been satisfied with its performance. 

How do you regulate your EDI traffic? I'd appreciate your feedback in the comments.