Previous - Correlation Identifier | Index | Next - Message Expiration
This week, we're going to study complementary pattern to
Message known as
Message Sequence.
When do I use this pattern?
In Messaging, Message is used to send data from sender to receiver. The message body is used to store this data. Often, the size of the message is restricted to adhere to performance standards. How would you send the data if it exceeds the restriction on size?
The answer is simple, you break up the data into parts and send each part as a message. However, as data is broken up at the sender side, it needs to be reassembled at the receiver side. For reassembling, three additional parameters are required when sending the data:
- Sequence identifier to uniquely identify one piece of information from another. Here, a piece of information is one full message that was broken up to adhere to message size restriction.
- Position identifier to uniquely identify the position of the current message in the sequence.
- Size or End indicator to notify the receiver to start assembling all pieces back into the whole message.
Message Sequence in CPI
To demonstrate the Message Sequence in CPI, I am going to use
Anaplan APIs. In short, Anaplan is a planning platform and accepts data as files. Planning is based on a huge amount of information, so the files are uploaded in chunks.
Here are the steps to upload products to Anaplan:
- Set the chunkCount to the number of products. For simplicity, one product will be sent in a chunk. The chunkCount is set for a given file, for example, here we will set the chunkCount for 'Products.txt' file. Technically, a file is given a numeric fileId, let's consider 'Products.txt' has fileId '111000000001'.
- Read the products from Northwind service in pages and send them to Anaplan in a chunk. A chunkId URL path parameter is used to identify the position of a given chunk.
- Once Anaplan receives chunkCount number of chunks, it knows that file upload is complete.
In the above example,
- fileId '111000000001' is a Sequence identifier, it uniquely identifies which file/message the chunks belong to.
- chunkId is a Position identifier, it uniquely identifies the location of the chunk in a given file.
- chunkCount is a Size or End indicator, once chunkCount number of chunks are received, file upload is complete.
Integration Flow
The integration flow will start immediately upon deployment. It will get all the products from Northwind service and post them one by one in Anaplan as chunks.
Steps
Start Immediately
This
Timer Start step starts the flow immediately after deployment.
Get Products
Get Products is a
Request-Reply step that gets products from Northwind service using
OData Adapter. The configuration of the OData Adapter is as follows:
This step is an example of the
Request-Reply pattern.
General Splitter
This
General Splitter step splits the products for sending them one by one. Here's the configuration of General Splitter:
Tab |
Parameter |
Value |
Processing |
XPath Expression |
/Products/Product |
Processing |
Grouping |
1 |
XML to CSV Converter
This
XML to CSV Converter converts products data from XML into CSV. Here's the configuration of XML to CSV Converter:
Tab |
Parameter |
Value |
Processing |
Path to Source Element in XSD |
/Products/Product |
Is First Chunk?
This
Router step checks whether the current chunk is the first one and routes accordingly. Here are the conditions under the Processing tab of the Router:
Order |
Route Name |
Condition Expression |
Default Route |
1 |
Yes |
${property.CamelSplitIndex} = '0' |
No |
2 |
No |
|
Yes |
The path for the First chunkId
Sequential Multicast
This Sequential Multicast step allows us to perform steps in sequence. Firstly,
chunkCount will be sent to Anaplan and then Header row will be added to the first chunk.
Post Chunk Count
Prepare Request Message
This
Content Modifier step prepares the request message for posting
chunkCount. Here's the body:
{
"id": "111000000001",
"name": "Products.txt",
"chunkCount": ${property.CamelSplitSize},
"delimiter": "",
"encoding": "UTF-8",
"firstDataRow": 2,
"format": "txt",
"headerRow": 1,
"separator": ","
}
Post Chunk Count
This
Request-Reply step posts the
chunkCount to Anaplan using
HTTP Adapter. Here's the configuration for HTTP Adapter:
This step is an example of the
Request-Reply pattern.
Add Header Row
Add Header Row
This
Content Modifier step will add a header row to the body. Here's the body configuration in Content Modifier:
CategoryID,Discontinued,SupplierID,UnitPrice,ProductName,QuantityPerUnit,UnitsOnOrder,ProductID,ReorderLevel,UnitsInStock
${in.body}
The path for other chunkIds
In the path for other chunks, there are no additional steps.
Post Chunk
Finally, either after sending the
chunkCount and setting header row for the first
chunkId or after doing nothing for other
chunkIds, the chunk itself is sent to Anaplan using
Request-Reply step using
HTTP Adapter. Here's the configuration for HTTP Adapter:
This step is an example of the
Request-Reply pattern.
Output
Once all chunks have been uploaded, the 'Products.txt' file can be seen in Anaplan as:
CategoryID,Discontinued,SupplierID,UnitPrice,ProductName,QuantityPerUnit,UnitsOnOrder,ProductID,ReorderLevel,UnitsInStock
1,false,1,18.0000,Chai,10 boxes x 20 bags,0,1,10,39
1,false,1,19.0000,Chang,24 - 12 oz bottles,40,2,25,17
2,false,1,10.0000,Aniseed Syrup,12 - 550 ml bottles,70,3,25,13
2,false,2,22.0000,Chef Anton's Cajun Seasoning,48 - 6 oz jars,0,4,0,53
2,true,2,21.3500,Chef Anton's Gumbo Mix,36 boxes,0,5,0,0
2,false,3,25.0000,Grandma's Boysenberry Spread,12 - 8 oz jars,0,6,25,120
7,false,3,30.0000,Uncle Bob's Organic Dried Pears,12 - 1 lb pkgs.,0,7,10,15
2,false,3,40.0000,Northwoods Cranberry Sauce,12 - 12 oz jars,0,8,0,6
6,true,4,97.0000,Mishi Kobe Niku,18 - 500 g pkgs.,0,9,0,29
8,false,4,31.0000,Ikura,12 - 200 ml jars,0,10,0,31
4,false,5,21.0000,Queso Cabrales,1 kg pkg.,30,11,30,22
4,false,5,38.0000,Queso Manchego La Pastora,10 - 500 g pkgs.,0,12,0,86
8,false,6,6.0000,Konbu,2 kg box,0,13,5,24
7,false,6,23.2500,Tofu,40 - 100 g pkgs.,0,14,0,35
2,false,6,15.5000,Genen Shouyu,24 - 250 ml bottles,0,15,5,39
3,false,7,17.4500,Pavlova,32 - 500 g boxes,0,16,10,29
6,true,7,39.0000,Alice Mutton,20 - 1 kg tins,0,17,0,0
8,false,7,62.5000,Carnarvon Tigers,16 kg pkg.,0,18,0,42
3,false,8,9.2000,Teatime Chocolate Biscuits,10 boxes x 12 pieces,0,19,5,25
3,false,8,81.0000,Sir Rodney's Marmalade,30 gift boxes,0,20,0,40
3,false,8,10.0000,Sir Rodney's Scones,24 pkgs. x 4 pieces,40,21,5,3
5,false,9,21.0000,Gustaf's Knäckebröd,24 - 500 g pkgs.,0,22,25,104
5,false,9,9.0000,Tunnbröd,12 - 250 g pkgs.,0,23,25,61
1,true,10,4.5000,Guaraná Fantástica,12 - 355 ml cans,0,24,0,20
3,false,11,14.0000,NuNuCa Nuß-Nougat-Creme,20 - 450 g glasses,0,25,30,76
3,false,11,31.2300,Gumbär Gummibärchen,100 - 250 g bags,0,26,0,15
3,false,11,43.9000,Schoggi Schokolade,100 - 100 g pieces,0,27,30,49
7,true,12,45.6000,Rössle Sauerkraut,25 - 825 g cans,0,28,0,26
6,true,12,123.7900,Thüringer Rostbratwurst,50 bags x 30 sausgs.,0,29,0,0
8,false,13,25.8900,Nord-Ost Matjeshering,10 - 200 g glasses,0,30,15,10
4,false,14,12.5000,Gorgonzola Telino,12 - 100 g pkgs,70,31,20,0
4,false,14,32.0000,Mascarpone Fabioli,24 - 200 g pkgs.,40,32,25,9
4,false,15,2.5000,Geitost,500 g,0,33,20,112
1,false,16,14.0000,Sasquatch Ale,24 - 12 oz bottles,0,34,15,111
1,false,16,18.0000,Steeleye Stout,24 - 12 oz bottles,0,35,15,20
8,false,17,19.0000,Inlagd Sill,24 - 250 g jars,0,36,20,112
8,false,17,26.0000,Gravad lax,12 - 500 g pkgs.,50,37,25,11
1,false,18,263.5000,Côte de Blaye,12 - 75 cl bottles,0,38,15,17
1,false,18,18.0000,Chartreuse verte,750 cc per bottle,0,39,5,69
8,false,19,18.4000,Boston Crab Meat,24 - 4 oz tins,0,40,30,123
8,false,19,9.6500,Jack's New England Clam Chowder,12 - 12 oz cans,0,41,10,85
5,true,20,14.0000,Singaporean Hokkien Fried Mee,32 - 1 kg pkgs.,0,42,0,26
1,false,20,46.0000,Ipoh Coffee,16 - 500 g tins,10,43,25,17
2,false,20,19.4500,Gula Malacca,20 - 2 kg bags,0,44,15,27
8,false,21,9.5000,Rogede sild,1k pkg.,70,45,15,5
8,false,21,12.0000,Spegesild,4 - 450 g glasses,0,46,0,95
3,false,22,9.5000,Zaanse koeken,10 - 4 oz boxes,0,47,0,36
3,false,22,12.7500,Chocolade,10 pkgs.,70,48,25,15
3,false,23,20.0000,Maxilaku,24 - 50 g pkgs.,60,49,15,10
3,false,23,16.2500,Valkoinen suklaa,12 - 100 g bars,0,50,30,65
7,false,24,53.0000,Manjimup Dried Apples,50 - 300 g pkgs.,0,51,10,20
5,false,24,7.0000,Filo Mix,16 - 2 kg boxes,0,52,25,38
6,true,24,32.8000,Perth Pasties,48 pieces,0,53,0,0
6,false,25,7.4500,Tourtière,16 pies,0,54,10,21
6,false,25,24.0000,Pâté chinois,24 boxes x 2 pies,0,55,20,115
5,false,26,38.0000,Gnocchi di nonna Alice,24 - 250 g pkgs.,10,56,30,21
5,false,26,19.5000,Ravioli Angelo,24 - 250 g pkgs.,0,57,20,36
8,false,27,13.2500,Escargots de Bourgogne,24 pieces,0,58,20,62
4,false,28,55.0000,Raclette Courdavault,5 kg pkg.,0,59,0,79
4,false,28,34.0000,Camembert Pierrot,15 - 300 g rounds,0,60,0,19
2,false,29,28.5000,Sirop d'érable,24 - 500 ml bottles,0,61,25,113
3,false,29,49.3000,Tarte au sucre,48 pies,0,62,0,17
2,false,7,43.9000,Vegie-spread,15 - 625 g jars,0,63,5,24
5,false,12,33.2500,Wimmers gute Semmelknödel,20 bags x 4 pieces,80,64,30,22
2,false,2,21.0500,Louisiana Fiery Hot Pepper Sauce,32 - 8 oz bottles,0,65,0,76
2,false,2,17.0000,Louisiana Hot Spiced Okra,24 - 8 oz jars,100,66,20,4
1,false,16,14.0000,Laughing Lumberjack Lager,24 - 12 oz bottles,0,67,10,52
3,false,8,12.5000,Scottish Longbreads,10 boxes x 8 pieces,10,68,15,6
4,false,15,36.0000,Gudbrandsdalsost,10 kg pkg.,0,69,15,26
1,false,7,15.0000,Outback Lager,24 - 355 ml bottles,10,70,30,15
4,false,15,21.5000,Flotemysost,10 - 500 g pkgs.,0,71,0,26
4,false,14,34.8000,Mozzarella di Giovanni,24 - 200 g pkgs.,0,72,0,14
8,false,17,15.0000,Röd Kaviar,24 - 150 g jars,0,73,5,101
7,false,4,10.0000,Longlife Tofu,5 kg pkg.,20,74,5,4
1,false,12,7.7500,Rhönbräu Klosterbier,24 - 0.5 l bottles,0,75,25,125
1,false,23,18.0000,Lakkalikööri,500 ml,0,76,20,57
2,false,12,13.0000,Original Frankfurter grüne Soße,12 boxes,0,77,15,32
Conclusion
Message Sequence is a complementary pattern to Message pattern. When message size is too huge to be sent in one go, the Message Sequence pattern is used to divide the data into multiple parts and reassemble at the receiver end.
References/Further Readings
Hope this helps,
Bala
Previous - Correlation Identifier | Index | Next - Message Expiration