Integration Advisor – Extended semantic validation...

joerg_ackermann · ‎02-09-2022

We have extended our features for semantic message payload validation in Integration Advisor and Cloud Integration such that you can now also validate cross-field dependencies via so called Assertions.

In this blog post I would like to give you not only a short overview of this new feature but also put it into context of our overall strategy and features for message payload validation.

The picture before depicts the overall building blocks of Integration Advisor. This blog post relates to the Message Implementation Guideline area, but also touches Runtime Artifacts and Runtime Support.

Message Payload Validation based on Integration Advisor MIGs

In real-life integration projects, it is often necessary to validate an incoming message payload to check if the payload is according to the specification/expectations. In many traditional integration projects, such consistency checks are encoded within the mapping implementation.

Instead of this, the Integration Advisor approach allows you to clearly distinguish between message payload validation and mapping.

In Message Implementation Guidelines (MIGs) you can specify in detail how you want to exchange messages with your Business Partner, or what do you want to use within your company for A2A integration. Based on such a customized MIG structure definition, you can export Cloud Integration Runtime Artifacts that automatically include an XML Schema Definition (XSD) file (file name <Your MIG Name>_RD.xsd).

Your message payload processing in Cloud Integration can include a flow step for XML Validation where this XSD file can be directly consumed. Please see screenshot below for an exemplary integration flow.

By the way, SAP Integration Advisor provides pre-packaged integration flow templates (like the one shown above) on the SAP API Business Hub (Link). They demonstrate how Integration Advisor artifacts can be consumed in Cloud Integration.

More details and options how to use the XML Validation in an Integration Flow can be found in this blog post: Link

In the next chapters I want to give you an overview of available features for MIG structure definition and payload validation.

Qualification feature for MIGs

You probably have already heard about the so-called Qualification feature in Integration Advisor. Qualification means explicitly modeling the precise semantic meaning of a generic node in a message structure. By doing so, you reap the following key benefits:

More precise specification / documentation (e.g., in communication with your B2B Partner)

Simplified and extended payload validation at runtime

Significantly simplified mapping implementation (save up to 30-50% of otherwise required mapping functions)

You can find an excellent introduction to the qualification concept in this blog post: Link

This powerful qualification feature will help you to define the MIG structure more precisely. To illustrate this, let us have a look at the example in the next screenshot.

Say you need to specify your message structure based on an ORDERS message from UN/EDIFACT. Within this message, the SG2 node is used to send information about the involved business partners. Please note that the UN/EDIFACT standard specifies this node only in a generic way: Node SG2 can be repeated up to 99 times and there are ~530 different business partner roles available.

But in your specific case, you want to express that you expect exactly one SG2 node for the Buyer role (BY) and exactly one SG2 node for the Seller role (SE).You achieve this goal by qualifying the SG2 node for both BY and SE and by setting the cardinality of both nodes to [1..1].

Moreover, the qualification also allows you to define the substructures differently for the qualified variants. In our example, the Buyer must provide address information (required for shipping the goods) while it is sufficient for the Supplier to provide country information (for tax and duty purposes).

Node properties in MIGs

You can provide additional details such as documentation or properties for each node of your MIG. The following details are relevant for message payload validation:

Checkbox to select/deselect nodes: specify if a node is allowed/used at a particular place or not

Cardinality: specifies how often a node is expected and how often it can be repeated.
- Use min=0 for optional and min=1 for mandatory (or min>1 for mandatory repetitions)
- Use max=1 for non-repeating and max>1 for repeatable nodes.
- Please note that cardinality is always in the context of its parent node. For example, parent node C082 has cardinality [0..1] and its child node 3039 has cardinality [1..1]. Therefore, C082 is optional while node 3039 is mandatory if node C082 is present. However, if node C082 is missing then 3039 will be missing as well.

Length (for text): specifies permitted length of the value in the node

Number of integer and fractional digits (for numbers): specifies the permitted size of a number

Code Values: specifies allowed code values
- You can reduce the allowed code values in your MIG. For example, for Buyer country you could specify that you only allow country codes BE, DE, FR and LU.

XSD Patterns and Assertions: see the following chapters for more details.

All these settings will be included in the generated MIG XSD file and therefore automatically considered as part of your message payload validation (in the XML Validation flow step, as described above).

Value patterns via XSD Patterns

There might be some special situations where the value of a node should follow a particular format or pattern. For example, say you have agreed with your business partners that the value of the Party Identifier of the Buyer (node C082 > 3039) shall look like "BUY-FR-12345678" where BUY is a fixed string, FR the country code of the buyer and 12345678 an 8-digit number representing the partner identifier.

To document such a requirement, Integration Advisor offers you the special field Format. This is a free text field for documentation purposes (but has no consequences on the runtime). Moreover, it is recommended to define one or more Example Values which might help a reader of your Message Guideline to better understand the requirement.

But you can also include this requirement into your message payload validation at runtime. To do so, you can define one or more so called XSD Patterns. Such an XSD Pattern precisely describes the expected pattern in a machine-readable way. For this, you need to follow the conventions of XSD Patterns as defined by the XSD standard. You’ll find some good Integration Advisor independent examples at: http://www.datypic.com/books/defxmlschema/chapter09.html

In our example, we defined an XSD Pattern "BUY-[A-Z]{2}-[0-9]{8}" which enforces that BUY is followed by two letters and then by 8 digits (separated by -). If the list of country codes you support is known and not too long, you could alternatively write your pattern as "BUY-(BE|DE|FR|LU)-[0-9]{8}" - by doing this you enforce that one of the (country code) values BE, DE, FR or LU must be used instead of arbitrary 2 letters.

Please note that you can define multiple XSD Patterns for the same field. If you specify more than one pattern, the field value must match one of the given patterns. In our example, you could alternatively define for each of the 4 countries a separate XSD Pattern (like "BUY-FR-[0-9]{8}" for France) - this would have the same result at runtime validation.

NEW feature: cross-field dependencies via Assertions

Last but not least, let us look at our newest, recently delivered feature extension.

There are situations where the message structure assumes dependencies between different nodes within the same substructure, like a conditional occurrence of a node depending on another node.

Coming back to our previous example, there might be the following requirements:

Buyer must identify itself either via Party identifier (node C082 > 3039) or via Party name (node C080 > 3036). That is, at least one of these values must be provided (both would be allowed as well).

In Buyer address information, City (node 3164) and Postal code (node 3251) must come as a pair, that is, either both nodes or none of the nodes shall be populated.

Please note that cardinality can be used to express that a node is either always required (min=1) or always optional (min=0), but it can't capture the semantics of such conditional occurrences.

MIG Editor now supports the definition of so called Assertions on the level of group nodes. Such an assertion expresses a Boolean condition that the payload must fulfil. As part of the message payload validation, these assertions must evaluate to true - otherwise the validation will throw an error. If you specify more than one Assertion, then all the assertions must evaluate to true.

In our example, we have defined two Assertions for the NAD node of the Buyer that perfectly express the requirements from above. As you can see, such straightforward conditions are simple enough to be expressed without a very detailed XPath knowledge. But in complex cases, a business expert might ask his/her integration developers for assistance.

This XPath is assigned to a group node (NAD in our example) and considered in the context of this node. Direct children of this node can be accessed directly (like D_3164), while deeper nested subnodes can be addressed with the help of / (like C_C082/D_3039).

Technically each such Assertion must be a valid Boolean XPath 2.0 expression and will be used within an xsd:assert statement from the XSD standard (which requires the use of XML Schema V1.1). You’ll find some general examples on how to define XSD assertions at: http://www.datypic.com/books/defxmlschema/chapter14.html. Please note that XSD offers three different features that are related to the overall Assertion concept: xsd:assertion at leaf node level, xsd:assert at group node level and xsd:alternative for providing alternative types based on specific rules. Integration Advisor currently only supports xsd:assert at group node level.

The text you enter in the Assertion field in MIG Editor is directly included in the XSD (within an xsd:assert statement). Therefore, you need to address the nodes exactly as they are named in an XML Payload. Please note that MIG Editor shows nodes in a business-expert friendly way (like C082 or 3039), while in the XML Payload these nodes might follow slightly different technical conventions (like C_C082 or D_3039). This is particularly the case for old EDI standards like UN/EDIFACT and ASC X12. How to find this out? Simply export the Cloud Integration Runtime Artefacts for your MIG - the generated zip includes the XML Schema Definition (XSD) file (file name <Your MIG Name>_RD.xsd). There you find the XML node names as required in the Assertions.

Conclusion

Message Payload Validation is one of the key features of Integration Advisor that helps to simplify and improve your overall integration project.Following the outlined Integration Advisor approach for message payload validation gives you the following major benefits:

Clear separation of concerns between message payload validation and mapping possible

MIG structure definition can be done by a business expert in a no-code/low-code fashion and does not require deep development skills / capacity

Significant simplification of the mapping implementation as exception handling on incoming message payload is not necessary anymore (for most cases).

With our new feature increment, we have now added the option to express dependencies between different nodes and include them into the validation. This allows you to make the semantic payload validation even more expressive.