Previous – Correlation Identifier | Index | Next – Message Expiration

This week, we’re going to study complementary pattern to Message known as Message Sequence.

When do I use this pattern?

In Messaging, Message is used to send data from sender to receiver. The message body is used to store this data. Often, the size of the message is restricted to adhere to performance standards. How would you send the data if it exceeds the restriction on size?

The answer is simple, you break up the data into parts and send each part as a message. However, as data is broken up at the sender side, it needs to be reassembled at the receiver side. For reassembling, three additional parameters are required when sending the data:

1. Sequence identifier to uniquely identify one piece of information from another. Here, a piece of information is one full message that was broken up to adhere to message size restriction.
2. Position identifier to uniquely identify the position of the current message in the sequence.
3. Size or End indicator to notify the receiver to start assembling all pieces back into the whole message.

Message Sequence in CPI

To demonstrate the Message Sequence in CPI, I am going to use Anaplan APIs. In short, Anaplan is a planning platform and accepts data as files. Planning is based on a huge amount of information, so the files are uploaded in chunks.

Here are the steps to upload products to Anaplan:

1. Set the chunkCount to the number of products. For simplicity, one product will be sent in a chunk. The chunkCount is set for a given file, for example, here we will set the chunkCount for ‘Products.txt’ file. Technically, a file is given a numeric fileId, let’s consider ‘Products.txt’ has fileId ‘111000000001’.
2. Read the products from Northwind service in pages and send them to Anaplan in a chunk. A chunkId URL path parameter is used to identify the position of a given chunk.
3. Once Anaplan receives chunkCount number of chunks, it knows that file upload is complete.

In the above example,

• fileId ‘111000000001’ is a Sequence identifier, it uniquely identifies which file/message the chunks belong to.
• chunkId is a Position identifier, it uniquely identifies the location of the chunk in a given file.
• chunkCount is a Size or End indicator, once chunkCount number of chunks are received, file upload is complete.

Integration Flow

The integration flow will start immediately upon deployment. It will get all the products from Northwind service and post them one by one in Anaplan as chunks.

Steps

Start Immediately

This Timer Start step starts the flow immediately after deployment.

Get Products

Get Products is a Request-Reply step that gets products from Northwind service using OData Adapter. The configuration of the OData Adapter is as follows:

This step is an example of the Request-Reply pattern.

General Splitter

This General Splitter step splits the products for sending them one by one. Here’s the configuration of General Splitter:

XML to CSV Converter

This XML to CSV Converter converts products data from XML into CSV. Here’s the configuration of XML to CSV Converter:

Is First Chunk?

This Router step checks whether the current chunk is the first one and routes accordingly. Here are the conditions under the Processing tab of the Router:

The path for the First chunkId

Sequential Multicast

This Sequential Multicast step allows us to perform steps in sequence. Firstly, chunkCount will be sent to Anaplan and then Header row will be added to the first chunk.

Post Chunk Count

Prepare Request Message

This Content Modifier step prepares the request message for posting chunkCount. Here’s the body:

 {
      "id": "111000000001",
      "name": "Products.txt",
      "chunkCount": ${property.CamelSplitSize},
      "delimiter": "",
      "encoding": "UTF-8",
      "firstDataRow": 2,
      "format": "txt",
      "headerRow": 1,
      "separator": ","
  }

Post Chunk Count

This Request-Reply step posts the chunkCount to Anaplan using HTTP Adapter. Here’s the configuration for HTTP Adapter:

This step is an example of the Request-Reply pattern.

Add Header Row

Add Header Row

This Content Modifier step will add a header row to the body. Here’s the body configuration in Content Modifier:

CategoryID,Discontinued,SupplierID,UnitPrice,ProductName,QuantityPerUnit,UnitsOnOrder,ProductID,ReorderLevel,UnitsInStock
${in.body}

The path for other chunkIds

In the path for other chunks, there are no additional steps.

Post Chunk

Finally, either after sending the chunkCount and setting header row for the first chunkId or after doing nothing for other chunkIds, the chunk itself is sent to Anaplan using Request-Reply step using HTTP Adapter. Here’s the configuration for HTTP Adapter:

This step is an example of the Request-Reply pattern.

Output

Once all chunks have been uploaded, the ‘Products.txt’ file can be seen in Anaplan as:

CategoryID,Discontinued,SupplierID,UnitPrice,ProductName,QuantityPerUnit,UnitsOnOrder,ProductID,ReorderLevel,UnitsInStock
1,false,1,18.0000,Chai,10 boxes x 20 bags,0,1,10,39
1,false,1,19.0000,Chang,24 – 12 oz bottles,40,2,25,17
2,false,1,10.0000,Aniseed Syrup,12 – 550 ml bottles,70,3,25,13
2,false,2,22.0000,Chef Anton’s Cajun Seasoning,48 – 6 oz jars,0,4,0,53
2,true,2,21.3500,Chef Anton’s Gumbo Mix,36 boxes,0,5,0,0
2,false,3,25.0000,Grandma’s Boysenberry Spread,12 – 8 oz jars,0,6,25,120
7,false,3,30.0000,Uncle Bob’s Organic Dried Pears,12 – 1 lb pkgs.,0,7,10,15
2,false,3,40.0000,Northwoods Cranberry Sauce,12 – 12 oz jars,0,8,0,6
6,true,4,97.0000,Mishi Kobe Niku,18 – 500 g pkgs.,0,9,0,29
8,false,4,31.0000,Ikura,12 – 200 ml jars,0,10,0,31
4,false,5,21.0000,Queso Cabrales,1 kg pkg.,30,11,30,22
4,false,5,38.0000,Queso Manchego La Pastora,10 – 500 g pkgs.,0,12,0,86
8,false,6,6.0000,Konbu,2 kg box,0,13,5,24
7,false,6,23.2500,Tofu,40 – 100 g pkgs.,0,14,0,35
2,false,6,15.5000,Genen Shouyu,24 – 250 ml bottles,0,15,5,39
3,false,7,17.4500,Pavlova,32 – 500 g boxes,0,16,10,29
6,true,7,39.0000,Alice Mutton,20 – 1 kg tins,0,17,0,0
8,false,7,62.5000,Carnarvon Tigers,16 kg pkg.,0,18,0,42
3,false,8,9.2000,Teatime Chocolate Biscuits,10 boxes x 12 pieces,0,19,5,25
3,false,8,81.0000,Sir Rodney’s Marmalade,30 gift boxes,0,20,0,40
3,false,8,10.0000,Sir Rodney’s Scones,24 pkgs. x 4 pieces,40,21,5,3
5,false,9,21.0000,Gustaf’s Knäckebröd,24 – 500 g pkgs.,0,22,25,104
5,false,9,9.0000,Tunnbröd,12 – 250 g pkgs.,0,23,25,61
1,true,10,4.5000,Guaraná Fantástica,12 – 355 ml cans,0,24,0,20
3,false,11,14.0000,NuNuCa Nuß-Nougat-Creme,20 – 450 g glasses,0,25,30,76
3,false,11,31.2300,Gumbär Gummibärchen,100 – 250 g bags,0,26,0,15
3,false,11,43.9000,Schoggi Schokolade,100 – 100 g pieces,0,27,30,49
7,true,12,45.6000,Rössle Sauerkraut,25 – 825 g cans,0,28,0,26
6,true,12,123.7900,Thüringer Rostbratwurst,50 bags x 30 sausgs.,0,29,0,0
8,false,13,25.8900,Nord-Ost Matjeshering,10 – 200 g glasses,0,30,15,10
4,false,14,12.5000,Gorgonzola Telino,12 – 100 g pkgs,70,31,20,0
4,false,14,32.0000,Mascarpone Fabioli,24 – 200 g pkgs.,40,32,25,9
4,false,15,2.5000,Geitost,500 g,0,33,20,112
1,false,16,14.0000,Sasquatch Ale,24 – 12 oz bottles,0,34,15,111
1,false,16,18.0000,Steeleye Stout,24 – 12 oz bottles,0,35,15,20
8,false,17,19.0000,Inlagd Sill,24 – 250 g jars,0,36,20,112
8,false,17,26.0000,Gravad lax,12 – 500 g pkgs.,50,37,25,11
1,false,18,263.5000,Côte de Blaye,12 – 75 cl bottles,0,38,15,17
1,false,18,18.0000,Chartreuse verte,750 cc per bottle,0,39,5,69
8,false,19,18.4000,Boston Crab Meat,24 – 4 oz tins,0,40,30,123
8,false,19,9.6500,Jack’s New England Clam Chowder,12 – 12 oz cans,0,41,10,85
5,true,20,14.0000,Singaporean Hokkien Fried Mee,32 – 1 kg pkgs.,0,42,0,26
1,false,20,46.0000,Ipoh Coffee,16 – 500 g tins,10,43,25,17
2,false,20,19.4500,Gula Malacca,20 – 2 kg bags,0,44,15,27
8,false,21,9.5000,Rogede sild,1k pkg.,70,45,15,5
8,false,21,12.0000,Spegesild,4 – 450 g glasses,0,46,0,95
3,false,22,9.5000,Zaanse koeken,10 – 4 oz boxes,0,47,0,36
3,false,22,12.7500,Chocolade,10 pkgs.,70,48,25,15
3,false,23,20.0000,Maxilaku,24 – 50 g pkgs.,60,49,15,10
3,false,23,16.2500,Valkoinen suklaa,12 – 100 g bars,0,50,30,65
7,false,24,53.0000,Manjimup Dried Apples,50 – 300 g pkgs.,0,51,10,20
5,false,24,7.0000,Filo Mix,16 – 2 kg boxes,0,52,25,38
6,true,24,32.8000,Perth Pasties,48 pieces,0,53,0,0
6,false,25,7.4500,Tourtière,16 pies,0,54,10,21
6,false,25,24.0000,Pâté chinois,24 boxes x 2 pies,0,55,20,115
5,false,26,38.0000,Gnocchi di nonna Alice,24 – 250 g pkgs.,10,56,30,21
5,false,26,19.5000,Ravioli Angelo,24 – 250 g pkgs.,0,57,20,36
8,false,27,13.2500,Escargots de Bourgogne,24 pieces,0,58,20,62
4,false,28,55.0000,Raclette Courdavault,5 kg pkg.,0,59,0,79
4,false,28,34.0000,Camembert Pierrot,15 – 300 g rounds,0,60,0,19
2,false,29,28.5000,Sirop d’érable,24 – 500 ml bottles,0,61,25,113
3,false,29,49.3000,Tarte au sucre,48 pies,0,62,0,17
2,false,7,43.9000,Vegie-spread,15 – 625 g jars,0,63,5,24
5,false,12,33.2500,Wimmers gute Semmelknödel,20 bags x 4 pieces,80,64,30,22
2,false,2,21.0500,Louisiana Fiery Hot Pepper Sauce,32 – 8 oz bottles,0,65,0,76
2,false,2,17.0000,Louisiana Hot Spiced Okra,24 – 8 oz jars,100,66,20,4
1,false,16,14.0000,Laughing Lumberjack Lager,24 – 12 oz bottles,0,67,10,52
3,false,8,12.5000,Scottish Longbreads,10 boxes x 8 pieces,10,68,15,6
4,false,15,36.0000,Gudbrandsdalsost,10 kg pkg.,0,69,15,26
1,false,7,15.0000,Outback Lager,24 – 355 ml bottles,10,70,30,15
4,false,15,21.5000,Flotemysost,10 – 500 g pkgs.,0,71,0,26
4,false,14,34.8000,Mozzarella di Giovanni,24 – 200 g pkgs.,0,72,0,14
8,false,17,15.0000,Röd Kaviar,24 – 150 g jars,0,73,5,101
7,false,4,10.0000,Longlife Tofu,5 kg pkg.,20,74,5,4
1,false,12,7.7500,Rhönbräu Klosterbier,24 – 0.5 l bottles,0,75,25,125
1,false,23,18.0000,Lakkalikööri,500 ml,0,76,20,57
2,false,12,13.0000,Original Frankfurter grüne Soße,12 boxes,0,77,15,32

Conclusion

Message Sequence is a complementary pattern to Message pattern. When message size is too huge to be sent in one go, the Message Sequence pattern is used to divide the data into multiple parts and reassemble at the receiver end.

References/Further Readings

• Message Sequence Pattern in Enterprise Integration Patterns
• Anaplan APIs
• CPI Components
  • Timer Start
  • Request-Reply
  • OData Adapter
  • General Splitter
  • XML to CSV Converter
  • Router
  • Content Modifier
  • HTTP Adapter
• EIPinCPI
  • Request-Reply

Hope this helps,
Bala

Previous – Correlation Identifier | Index | Next – Message Expiration