Queues & contexts concept of graphical mapping: understand it to turn it into your advantage
The more I work with graphical mappings in PI, the more I get convinced that the queues & contexts concept is one of the most difficult things to understand there. This is also confirmed by the SDN forum, since most of the mapping problems described by users are more or less connected with queues or context handling difficulties. Luckily, our Process Integration (PI) & SOA Middleware Community is active enough to answer every mapping problems thread. But last days I started to wonder if I can do anything more, if I can help better. To give them a rod instead of a fish, as they say. Why? Because I strongly believe that understanding the queues & contexts concept deeply, becomes a key to firstly: using graphical mapping effectively, and secondly: even starting to like it (well, for those who don’t like it yet)! This is how I decided to give it a try and explain it all myself, with examples and links to other useful sources. And for those of you who already know it all, let this be a nice review, reminder or a trip back to the roots.
Normally when working with XML, data is analyzed and processed by one of XML processors (or parsers), two of which are by far most popular: DOM (Document Object Model) and SAX (Simple API for XML). You can read a really nice description of DOM and SAX here and a comparison of their strengths and weaknesses here.
Unfortunately both models have significant disadvantages (time and memory consumption of DOM and limited access to nodes in SAX) aren’t really acceptable while processing a message in an integration platform like PI, which made SAP invent their own way of working. How PI works with XML is somewhere in the middle between DOM and SAX. Every single message received by PI is parsed into a collection of queues. For every source field, a queue is created that contains all values that appear in the whole dataset: every time a node occurs, its value is placed into a dedicated queue for that node. When a structure element occurs multiple times, it simply creates multiple values in the queue. Later on, the whole PI mapping processing is performed on these queues. Every mapping step and mapping function (including UDFs) operates on queues: receives one or more queues as input, and add its results to an output queue, which in turn can be an input for another mapping step. Every target field also receives its own queue, based on which the XML structure is built after the mapping.
While queues depict the list of values for some particular element, the contexts show their place in the whole structure. Values having the same parent node on a certain level are within one context, and these having separate parents are also in separate contexts. For each source field in mapping, you can also set, at which level the context should be placed. Let’s see how a queue is built for an example XML data:
And here is what happens when you change the context of the quantity field to the “order” node:
The queues are an important tool for testing and debugging: you can display a queue in the message mapping for each source field and every processing step (once you provide test data).
Ok, now we are through the process of creating queues from source data. Does it really help in anything? Yes it does, because same thing is done on the target side. Want several occurrences of a node under single parent? Put them in one context. Want nodes under separate parents? Add context changes. Now that you know how it works, ANY source to target conversion should become piece a cake, your creativity is your only limitation.
In addition to functions that manipulate field value, which are quite intuitive to use, PI also offers you some dedicated standard functions to manipulate the queues and context, that are also worth knowing:
- removeContext that removes all context changes from the given queue
- collapseContext that leaves only one value per context and only then removes context changes (see their detailed comparison here)
- splitByValue that inserts context changes after each value or each value change in the queue
- formatByExample that adjusts the contexts of one queue to the “example” of another queue
- And the most underestimated one: createIf, that creates one special value that I did not mention so far: SUPPRESS.
The SUPPRESS is important whenever you want to create a whole node and its subnodes under some certain condition. Going back to the initial example with orders, you might want to create items only when their quantity is greater than zero. You could achieve this with the following mappings:
quantity(context: order) -> greater (Constant 0.0) -> createIf -> item
Here is the “Display queue” result for the “item” result node, for some example data:
Thanks to this, you don’t need any further conditions for “material” and “quantity” fields, since their creation will be controlled by their parent node existence. You can simply map them both 1:1, and the whole item for material B with quantity = 0 will be skipped when creating target XML.
Let me conclude my explanation with providing a few useful links to further master your mapping skills:
An overview of the mapping functions in SAP Help
And a Wiki with examples
A long list of the mapping concept-related Wikis and Blogs
Especially this one can be interesting as an addition to my description: Introduction to queues in message mapping
Now that you have all that knowledge, do you still want to post your question on “how to map X to Y”, or is it obvious now? 🙂
At First, thank you for this blog and all the links compilation.
I agree with you, that the context graphical mapping understanding is a key objetive and very confusing sometimes.
In my opinion, one problem with this concept is the existence of certain peculiarities a bit weird like the target ocurrency . Also, one of the most frustrating things is the "Display Queue" help, because you check the target-values there and later they dissapear in the test tab and the most times are an occurrency-context problem.
Brilliant introduction to contexts and queues, thanks Greg.
Good blog Greeg !!
Perfect explain !
Thanks Ricardo! I'm glad you like it 🙂
When I start with SAP PI was hard to understand the context, change context and queue process.
With your blog, wow !! Now the SAP PI beginner´s has the smooth way to understand rsrsr
Excellent Blog for Beginners!
Indeed, this is very much useful to SAP PI beginner 🙂
I like your way of putting all relevant links/ resources - kind of one page to meet all refresh/ get concept of queues/ context.
Well that was pretty much my intention: a perfect starting point for beginners or those who want to review the basics. I'm happy to hear you like it that way 🙂