Technical Articles
SAP Cloud Integration(CPI) – Split records by field value and sending with dynamic file name.
Dear Reader,
This blog describes how to split a message into multiple output messages and send them with a dynamic file name. I am using “dynamic” because the file name to be used in the target server are determined at the runtime. And because I could not find any resource that perfectly applies to my scenario, I am writing this blog. If you have any questions or suggestions to improve my solution, please kindly write in the comment. Thank you for reading in advance.
Scenario:
Prerequisites
・Create a CDS view and a OData service in the eclipse and in the S4 respectively.
iFlow components:
Steps:
These are the important steps to achieve our goal: splitting a message by the field value and sending with a dynamic file name.
1.Sort the data in the OData adopter by the “Type” field.
<Root>
<Record>
<Number>0000000001</Number>
<Type>A</Type>
</Record>
<Record>
<Number>0000000002</Number>
<Type>A</Type>
</Record>
<Record>
<Number>0000000003</Number>
<Type>A</Type>
</Record>
<Record>
<Number>0000000004</Number>
<Type>B</Type>
</Record>
<Record>
<Number>0000000005</Number>
<Type>B</Type>
</Record>
<Record>
<Number>0000000006</Number>
<Type>C</Type>
</Record>
</Root>
If you use this code, you need to modify the closures like <Root></Root>, <Record></Record>, and <Type></Type> according to your data. And you need to change the length of the field value. In the line 42 ‘type.length()!=1‘, I set 1 because the ‘Type’ field value must have the length 1.
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
def Message processData(Message message) {
def body = message.getBody(java.lang.String) as String;
def properties = message.getProperties();
def messageLog = messageLogFactory.getMessageLog(message);
//split the message body by </Record> to take a list of records
def String[] ref_records = body.split("</Record>", -1);
//split the message body by <Type> to take a list of records which starts with the type field value
def String[] ref_types1 = body.split("<Type>", -1);
//counter to iterate ref_records
def counter = 0;
//string buffer to ouput as a property
def records = new StringBuffer();
//this helps us to see if the type is changed through the iteration of records
def pre_type = "";
//ref_records and ref_types shuold have the same length, otherwise, it is a cardinality error
if ( ref_records.length != ref_types1.length ) {
messageLog.addAttacmentAsString("Error:", "cardinality error", "text/plain");
}
for(int i=1; i < ref_types1.length; i++){
//split the ref_types1 by </Type> to take only the type itself.
String[] ref_types2 = ref_types1[i].split("</Type>", 2);
String type = ref_types2[0];
//initial case. Do nothing but note the 'pre_type' for later use
if (counter == 0) {
pre_type = type;
}
//general case (i.e. 'counter' > 0)
else {
//the type must have length = 1. otherwise, it's an error.
if (type.length()!=1) {
messageLog.addAttachmentAsString("Length error:", type, "text/plain");
}
else {
//if current type is different from the previous type, split records by a line break
if (pre_type!=type) {
records.append("</Record></Root>\n<Root>");
pre_type = type;
//if current type is the same as the previous type, do not split
} else {
records.append("</Record>");
}
}
}
//append the current record
records.append(ref_records[counter]);
//increment the counter by 1
counter += 1;
}
//the last line needs to get closed.
records.append("</Record>");
//</Root> should be appended as well
records.append(ref_records[counter]);
//set the records with line breaks to the message body
message.setBody(records);
return message;
}
This script should output the message like below.
<?xml version="1.0" encoding="UTF-8"?>
<Root><Record><Number>0000000001</Number><Type>A</Type></Record><Record><Number>0000000002</Number><Type>A</Type></Record><Record><Number>0000000003</Number><Type>A</Type></Record></Root>
<Root><Record><Number>0000000004</Number><Type>B</Type></Record><Record><Number>0000000005</Number><Type>B</Type></Record></Root>
<Root><Record><Number>0000000006</Number><Type>C</Type></Record></Root>
Select the line break and the parallel processing mode. The number of concurrent processes should be more than the number of types. In this case, we only have three types: A, B, C, but if the data contains D and E as well and the maximum number of types is 5, you have to set 5. For now, I set 3 in this demonstration.
<?xml version="1.0" encoding="UTF-8"?>
<Root><Record><Number>0000000001</Number><Type>A</Type></Record><Record><Number>0000000002</Number><Type>A</Type></Record><Record><Number>0000000003</Number><Type>A</Type></Record></Root>
<?xml version="1.0" encoding="UTF-8"?>
<Root><Record><Number>0000000004</Number><Type>B</Type></Record><Record><Number>0000000005</Number><Type>B</Type></Record></Root>
<?xml version="1.0" encoding="UTF-8"?>
<Root><Record><Number>0000000006</Number><Type>C</Type></Record></Root>
As you can see, you could split records by the ‘Type’ field. The sequence of the messages may change(the first message could be the one having the type ‘C’), but it does not matter at all as long as all three messages are saved in the target server.
4. Configure a content modifier with the exchange property.
You can specify the field in two ways: //Type or Root/Record/Type. This sets the current ‘Type’ value to the one of the properties named ‘CurrentType’.
Note: you can also use this expression in the directory name as required.
@Akira Tachibana: Nice Blog, Thanks for sharing your experience.
Small confusion, As per my understanding the option, "Number of Concurrent processes" is like how many messages should the tool processed parallelly. and i believe It is nothing to do with type here.
Please correct me if my understanding is wrong.
Thanks and Regards,
Hari
Hi, Hari Kotha
Thank you so much for your comment.
Yes, you are right. That is actually the reason why I wrote "The number of concurrent processes should be more than the number of types". If the number of concurrent processes is less than the number of 'types'(in my case), the messages split by 'type' would no longer be processed in parallel. Because the number of messages is always equal to the number of 'types' the data contains, we have to consider the number of 'types'.
I should have explained this as well, so your comment definitely helps me and other readers.
Thanks and Regards,
Akira
Hi Akira Tachibana
Thanks a lot for this wonderful blog.
I also wanted to simulate the same scenario without using Groovy Script and found another great resource by Sunil Chandra :XPath beyond filtering. https://blogs.sap.com/2020/10/08/xpath-beyond-filtering/
And found that case 3 from this blog can be used to achieve the results we are expecting.
Here is the XPATH expression which can be used in a Content Modifier as a property. Now, this we can use when our payload size is small.
Result
This can be wrapped around another Root node and then we can use Iterating Splitter.
Thanks
Prachetas
Hi, prachetas singh
Thank you so much for your comment!
I wanted to try your solution, but I got this error. It seems that the data in XML cannot contain multiple roots. Have you gotten this kind of error?
This is how the payload looks like after separating the data by 'Type' in the content modifier and sending it as a new message body.
Error was raised before the Iterating Splitter.
Error message
Thanks and Regards,
Akira