Technical Articles
EIPinCPI – Message Sequence
Previous – Correlation Identifier | Index | Next – Message Expiration
This week, we’re going to study complementary pattern to Message known as Message Sequence.
When do I use this pattern?
In Messaging, Message is used to send data from sender to receiver. The message body is used to store this data. Often, the size of the message is restricted to adhere to performance standards. How would you send the data if it exceeds the restriction on size?
The answer is simple, you break up the data into parts and send each part as a message. However, as data is broken up at the sender side, it needs to be reassembled at the receiver side. For reassembling, three additional parameters are required when sending the data:
- Sequence identifier to uniquely identify one piece of information from another. Here, a piece of information is one full message that was broken up to adhere to message size restriction.
- Position identifier to uniquely identify the position of the current message in the sequence.
- Size or End indicator to notify the receiver to start assembling all pieces back into the whole message.
Message Sequence in CPI
To demonstrate the Message Sequence in CPI, I am going to use Anaplan APIs. In short, Anaplan is a planning platform and accepts data as files. Planning is based on a huge amount of information, so the files are uploaded in chunks.
Here are the steps to upload products to Anaplan:
- Set the chunkCount to the number of products. For simplicity, one product will be sent in a chunk. The chunkCount is set for a given file, for example, here we will set the chunkCount for ‘Products.txt’ file. Technically, a file is given a numeric fileId, let’s consider ‘Products.txt’ has fileId ‘111000000001’.
- Read the products from Northwind service in pages and send them to Anaplan in a chunk. A chunkId URL path parameter is used to identify the position of a given chunk.
- Once Anaplan receives chunkCount number of chunks, it knows that file upload is complete.
In the above example,
- fileId ‘111000000001’ is a Sequence identifier, it uniquely identifies which file/message the chunks belong to.
- chunkId is a Position identifier, it uniquely identifies the location of the chunk in a given file.
- chunkCount is a Size or End indicator, once chunkCount number of chunks are received, file upload is complete.
Integration Flow
The integration flow will start immediately upon deployment. It will get all the products from Northwind service and post them one by one in Anaplan as chunks.
Steps
Start Immediately
This Timer Start step starts the flow immediately after deployment.
Get Products
Get Products is a Request-Reply step that gets products from Northwind service using OData Adapter. The configuration of the OData Adapter is as follows:
Tab | Parameter | Value |
Connection | Address | https://services.odata.org/V2/Northwind/Northwind.svc |
Processing | Resource Path | Products |
This step is an example of the Request-Reply pattern.
General Splitter
This General Splitter step splits the products for sending them one by one. Here’s the configuration of General Splitter:
Tab | Parameter | Value |
Processing | XPath Expression | /Products/Product |
Processing | Grouping | 1 |
XML to CSV Converter
This XML to CSV Converter converts products data from XML into CSV. Here’s the configuration of XML to CSV Converter:
Tab | Parameter | Value |
Processing | Path to Source Element in XSD | /Products/Product |
Is First Chunk?
This Router step checks whether the current chunk is the first one and routes accordingly. Here are the conditions under the Processing tab of the Router:
Order | Route Name | Condition Expression | Default Route |
1 | Yes | ${property.CamelSplitIndex} = ‘0’ | No |
2 | No | Yes |
The path for the First chunkId
Sequential Multicast
This Sequential Multicast step allows us to perform steps in sequence. Firstly, chunkCount will be sent to Anaplan and then Header row will be added to the first chunk.
Post Chunk Count
Prepare Request Message
This Content Modifier step prepares the request message for posting chunkCount. Here’s the body:
{
"id": "111000000001",
"name": "Products.txt",
"chunkCount": ${property.CamelSplitSize},
"delimiter": "",
"encoding": "UTF-8",
"firstDataRow": 2,
"format": "txt",
"headerRow": 1,
"separator": ","
}
Post Chunk Count
This Request-Reply step posts the chunkCount to Anaplan using HTTP Adapter. Here’s the configuration for HTTP Adapter:
Tab | Parameter | Value |
Connection | Address | https://api.anaplan.com/1/3/workspaces/{{Workspace Id}}/models/{{Model Id}}/files/{{File Id}} |
Connection | Authentication | Basic |
Connection | Credential Name | Anaplan Credentials |
This step is an example of the Request-Reply pattern.
Add Header Row
Add Header Row
This Content Modifier step will add a header row to the body. Here’s the body configuration in Content Modifier:
CategoryID,Discontinued,SupplierID,UnitPrice,ProductName,QuantityPerUnit,UnitsOnOrder,ProductID,ReorderLevel,UnitsInStock
${in.body}
The path for other chunkIds
In the path for other chunks, there are no additional steps.
Post Chunk
Finally, either after sending the chunkCount and setting header row for the first chunkId or after doing nothing for other chunkIds, the chunk itself is sent to Anaplan using Request-Reply step using HTTP Adapter. Here’s the configuration for HTTP Adapter:
Tab | Parameter | Value |
Connection | Address | https://api.anaplan.com/1/3/workspaces/{{Workspace Id}}/models/{{Model Id}}/files/{{File Id}}/chunks/${property.CamelSplitIndex} |
Connection | Authentication | Basic |
Connection | Credential Name | Anaplan Credentials |
This step is an example of the Request-Reply pattern.
Output
Once all chunks have been uploaded, the ‘Products.txt’ file can be seen in Anaplan as:
CategoryID,Discontinued,SupplierID,UnitPrice,ProductName,QuantityPerUnit,UnitsOnOrder,ProductID,ReorderLevel,UnitsInStock
1,false,1,18.0000,Chai,10 boxes x 20 bags,0,1,10,39
1,false,1,19.0000,Chang,24 – 12 oz bottles,40,2,25,17
2,false,1,10.0000,Aniseed Syrup,12 – 550 ml bottles,70,3,25,13
2,false,2,22.0000,Chef Anton’s Cajun Seasoning,48 – 6 oz jars,0,4,0,53
2,true,2,21.3500,Chef Anton’s Gumbo Mix,36 boxes,0,5,0,0
2,false,3,25.0000,Grandma’s Boysenberry Spread,12 – 8 oz jars,0,6,25,120
7,false,3,30.0000,Uncle Bob’s Organic Dried Pears,12 – 1 lb pkgs.,0,7,10,15
2,false,3,40.0000,Northwoods Cranberry Sauce,12 – 12 oz jars,0,8,0,6
6,true,4,97.0000,Mishi Kobe Niku,18 – 500 g pkgs.,0,9,0,29
8,false,4,31.0000,Ikura,12 – 200 ml jars,0,10,0,31
4,false,5,21.0000,Queso Cabrales,1 kg pkg.,30,11,30,22
4,false,5,38.0000,Queso Manchego La Pastora,10 – 500 g pkgs.,0,12,0,86
8,false,6,6.0000,Konbu,2 kg box,0,13,5,24
7,false,6,23.2500,Tofu,40 – 100 g pkgs.,0,14,0,35
2,false,6,15.5000,Genen Shouyu,24 – 250 ml bottles,0,15,5,39
3,false,7,17.4500,Pavlova,32 – 500 g boxes,0,16,10,29
6,true,7,39.0000,Alice Mutton,20 – 1 kg tins,0,17,0,0
8,false,7,62.5000,Carnarvon Tigers,16 kg pkg.,0,18,0,42
3,false,8,9.2000,Teatime Chocolate Biscuits,10 boxes x 12 pieces,0,19,5,25
3,false,8,81.0000,Sir Rodney’s Marmalade,30 gift boxes,0,20,0,40
3,false,8,10.0000,Sir Rodney’s Scones,24 pkgs. x 4 pieces,40,21,5,3
5,false,9,21.0000,Gustaf’s Knäckebröd,24 – 500 g pkgs.,0,22,25,104
5,false,9,9.0000,Tunnbröd,12 – 250 g pkgs.,0,23,25,61
1,true,10,4.5000,Guaraná Fantástica,12 – 355 ml cans,0,24,0,20
3,false,11,14.0000,NuNuCa Nuß-Nougat-Creme,20 – 450 g glasses,0,25,30,76
3,false,11,31.2300,Gumbär Gummibärchen,100 – 250 g bags,0,26,0,15
3,false,11,43.9000,Schoggi Schokolade,100 – 100 g pieces,0,27,30,49
7,true,12,45.6000,Rössle Sauerkraut,25 – 825 g cans,0,28,0,26
6,true,12,123.7900,Thüringer Rostbratwurst,50 bags x 30 sausgs.,0,29,0,0
8,false,13,25.8900,Nord-Ost Matjeshering,10 – 200 g glasses,0,30,15,10
4,false,14,12.5000,Gorgonzola Telino,12 – 100 g pkgs,70,31,20,0
4,false,14,32.0000,Mascarpone Fabioli,24 – 200 g pkgs.,40,32,25,9
4,false,15,2.5000,Geitost,500 g,0,33,20,112
1,false,16,14.0000,Sasquatch Ale,24 – 12 oz bottles,0,34,15,111
1,false,16,18.0000,Steeleye Stout,24 – 12 oz bottles,0,35,15,20
8,false,17,19.0000,Inlagd Sill,24 – 250 g jars,0,36,20,112
8,false,17,26.0000,Gravad lax,12 – 500 g pkgs.,50,37,25,11
1,false,18,263.5000,Côte de Blaye,12 – 75 cl bottles,0,38,15,17
1,false,18,18.0000,Chartreuse verte,750 cc per bottle,0,39,5,69
8,false,19,18.4000,Boston Crab Meat,24 – 4 oz tins,0,40,30,123
8,false,19,9.6500,Jack’s New England Clam Chowder,12 – 12 oz cans,0,41,10,85
5,true,20,14.0000,Singaporean Hokkien Fried Mee,32 – 1 kg pkgs.,0,42,0,26
1,false,20,46.0000,Ipoh Coffee,16 – 500 g tins,10,43,25,17
2,false,20,19.4500,Gula Malacca,20 – 2 kg bags,0,44,15,27
8,false,21,9.5000,Rogede sild,1k pkg.,70,45,15,5
8,false,21,12.0000,Spegesild,4 – 450 g glasses,0,46,0,95
3,false,22,9.5000,Zaanse koeken,10 – 4 oz boxes,0,47,0,36
3,false,22,12.7500,Chocolade,10 pkgs.,70,48,25,15
3,false,23,20.0000,Maxilaku,24 – 50 g pkgs.,60,49,15,10
3,false,23,16.2500,Valkoinen suklaa,12 – 100 g bars,0,50,30,65
7,false,24,53.0000,Manjimup Dried Apples,50 – 300 g pkgs.,0,51,10,20
5,false,24,7.0000,Filo Mix,16 – 2 kg boxes,0,52,25,38
6,true,24,32.8000,Perth Pasties,48 pieces,0,53,0,0
6,false,25,7.4500,Tourtière,16 pies,0,54,10,21
6,false,25,24.0000,Pâté chinois,24 boxes x 2 pies,0,55,20,115
5,false,26,38.0000,Gnocchi di nonna Alice,24 – 250 g pkgs.,10,56,30,21
5,false,26,19.5000,Ravioli Angelo,24 – 250 g pkgs.,0,57,20,36
8,false,27,13.2500,Escargots de Bourgogne,24 pieces,0,58,20,62
4,false,28,55.0000,Raclette Courdavault,5 kg pkg.,0,59,0,79
4,false,28,34.0000,Camembert Pierrot,15 – 300 g rounds,0,60,0,19
2,false,29,28.5000,Sirop d’érable,24 – 500 ml bottles,0,61,25,113
3,false,29,49.3000,Tarte au sucre,48 pies,0,62,0,17
2,false,7,43.9000,Vegie-spread,15 – 625 g jars,0,63,5,24
5,false,12,33.2500,Wimmers gute Semmelknödel,20 bags x 4 pieces,80,64,30,22
2,false,2,21.0500,Louisiana Fiery Hot Pepper Sauce,32 – 8 oz bottles,0,65,0,76
2,false,2,17.0000,Louisiana Hot Spiced Okra,24 – 8 oz jars,100,66,20,4
1,false,16,14.0000,Laughing Lumberjack Lager,24 – 12 oz bottles,0,67,10,52
3,false,8,12.5000,Scottish Longbreads,10 boxes x 8 pieces,10,68,15,6
4,false,15,36.0000,Gudbrandsdalsost,10 kg pkg.,0,69,15,26
1,false,7,15.0000,Outback Lager,24 – 355 ml bottles,10,70,30,15
4,false,15,21.5000,Flotemysost,10 – 500 g pkgs.,0,71,0,26
4,false,14,34.8000,Mozzarella di Giovanni,24 – 200 g pkgs.,0,72,0,14
8,false,17,15.0000,Röd Kaviar,24 – 150 g jars,0,73,5,101
7,false,4,10.0000,Longlife Tofu,5 kg pkg.,20,74,5,4
1,false,12,7.7500,Rhönbräu Klosterbier,24 – 0.5 l bottles,0,75,25,125
1,false,23,18.0000,Lakkalikööri,500 ml,0,76,20,57
2,false,12,13.0000,Original Frankfurter grüne Soße,12 boxes,0,77,15,32
Conclusion
Message Sequence is a complementary pattern to Message pattern. When message size is too huge to be sent in one go, the Message Sequence pattern is used to divide the data into multiple parts and reassemble at the receiver end.
References/Further Readings
- Message Sequence Pattern in Enterprise Integration Patterns
- Anaplan APIs
- CPI Components
- EIPinCPI
Hope this helps,
Bala
Previous – Correlation Identifier | Index | Next – Message Expiration
Hi Bhalchandra,
Thanks for the explanation. Regarding the connectivity with Anaplan API, what are all the other kind of adapters other than http adapter?
Regards,
Sindhuja.
Hi Sindhuja Jayapandiyan,
Currently, in CPI, there is only the HTTP Adapter to connect to Anaplan. We (Syniti) are working on a dedicated Anaplan Adapter.
Kind regards,
Bala
Hi Bhalchandra,
Thanks for the reply.
Regards,
Sindhuja.
Hi BalaChandra,
Any idea , is there any trial accounts possible with the ANAPLAN. Please let us know if possible.
Thanks and Regards,
Vijay.