Technical Articles
How to read/write SAP CPI Datastore from Groovy
A few days ago I stumbled upon Daniel’s question whether it is possible to address the SAP CPI Datastore via Groovy Script. I did not want to deal with the answer that there was no possibility and the standard building blocks had to be used. So I did another deep dive into SAP CPI’s internals and found two solutions.
In the following article I would like to show you how it is possible to write to and read from the SAP CPI Datastore using Groovy Script. Before we start, please read the following disclaimer.
Disclaimer: The functions shown below are not officially documented. Although they work perfectly and are partly used by the official Datastore modules, you should always keep in mind that these are not official functions. So use them responsibly.
Table of contents
- Solution 1: Access via DataStoreService-class
- Solution 2: Access via DataStore-class
- When to use which solution?
- Class/function descriptions
- DataStoreService-class
- DataBean-class
- DataConfig-class
- DataStore-class
- Data-class
- Conclusion
Solution 1: Access via DataStoreService-class
The first option to access the Datastore via Groovy script is via the DataStoreService-class from the com.sap.it.api.asdk.datastore-package. This class gives simple and easy access but comes with less flexibility and options than the second solution. The function call itself is similiar to the use of the official ValueMappingApi.
Read from Datastore
To read from the Datastore the following code is needed. Please check the source code comments for further description.
import com.sap.gateway.ip.core.customdev.util.Message
//Imports for DataStoreService-class
import com.sap.it.api.asdk.datastore.*
import com.sap.it.api.asdk.runtime.*
Message processData(Message message) {
//Get service instance
def service = new Factory(DataStoreService.class).getService()
//Check if valid service instance was retrieved
if( service != null) {
//Read data store entry via id
def dsEntry = service.get("DatastoreName","EntryId")
def result = new String(dsEntry.getDataAsArray())
message.setBody(body)
}
return message
}
Restrictions:
- You can only read from Datastore that are either “Global” or belong to your IFlow. (Same behaviour like the official Datastore-blocks in the IFlow editor.)
Write to Datastore
To write to the Datastore via the DataStoreService-class you need two more “helper” classes: DataBean and DataConfig. The DataBean holds the actual data you want to store and the DataConfig holds the target Datastore name as also some metadata.
import com.sap.gateway.ip.core.customdev.util.Message
//Imports for DataStoreService-class
import com.sap.it.api.asdk.datastore.*
import com.sap.it.api.asdk.runtime.*
Message processData(Message message) {
//Data to be stored in datatore
def payload = "This is sample data"
//Get service instance
def service = new Factory(DataStoreService.class).getService()
//Check if valid service instance was retrieved
if( service != null) {
def dBean = new DataBean()
dBean.setDataAsArray(payload.getBytes("UTF-8"))
//Class model offers headers, but for me it didn't work out
//Map<String, Object> headers = ["headerName1":"me", "anotherHeader": false]
//dBean.setHeaders(headers)
//Define datatore name and entry id
def dConfig = new DataConfig()
dConfig.setStoreName("DatastoreFromGroovyASDK")
dConfig.setId("TestEntry")
dConfig.setOverwrite(true)
//Write to data store
result = service.put(dBean,dConfig)
}
return message
}
Please note, that we use a string as payload here, but you can write anything to the store, as long as you pass it as byte[] to the DataBean instance. To keep the example above simple and clear, I didn’t use all parameters (like expiration period, etc.). To see full list of parameters/function, check the paragraph with the class description at the end of this article.
The entry, created by the above code looks as follows:
Restrictions:
- Passing headers when storing entries doesn’t work. (No error is thrown, but at least in my tests the headers never arrived in the Datastore.)
- You can’t set a context, thus entries/datastore created via the DataStoreService-class will be global entries and can be read by all other IFlows!
- You can’t set a message id/MPL id, so the “Message ID” column in SAP CPI’s datastore viewer will always be empty. (See screenshot above.)
Solution 2: Access via DataStore-class
The second way to access the data storage is the so-called DataStore-class. As I have seen, it seems to be the underlying class of the DataStoreService. (I’m not sure, however.) But I can say with certainty that this class is much more flexible, but also at the expense of ease of use. Let’s have a look how it works.
Read from Datastore
Please check the source code comments. By use of the DataStore class you can read from any Datastore (regardless of local/global modifiers). You can also select set of entries via the select-function.
import com.sap.gateway.ip.core.customdev.util.Message
//Imports for the DataStore-class handling/access
import com.sap.esb.datastore.DataStore
import com.sap.esb.datastore.Data
import org.osgi.framework.*
Message processData(Message message) {
//Get CamelContext and from that the DataStore instance
def camelCtx = message.exchange.getContext()
DataStore dataStore = (DataStore)camelCtx.getRegistry().lookupByName(DataStore.class.getName())
//Read from Datastore params => (DatastoreName, EntryId)
def dsEntry = dataStore.get("DatastoreFromGroovyServiceImpl", "TestEntry")
//Get datastore entry payload as String
def payload = new String(dsEntry.getDataAsArray())
//NOTE: If you pass the name context-name ad third parameter, you can read from
// any datastore, regardless if it's global or local! The context name usually
// equals the id of the IFlow a Datastore belongs to.
//def dsEntry = dataStore.get("DatastoreName", "ContextName", "EntryId")
//If you want to select multiple entry, use select-method and pass the
//following params => (DatastoreName, ContextName, NumberOfEntriesToPull)
//def dsEntryArray = dataStore.select("DatastoreName", "ContextName", 10)
message.setBody(payload)
return message
}
As you can see there are two major differences between the DataStore- and the DataStoreService implementation.
- You can read from any Datastore by passing the Datastore-context as third parameter.
- You can use a select-function, to read multiple entries, without passing entries ids as parameter.
Write to DataStore
When writing to the Datastore via the DataStore-class you can also add headers and set the context and thus create “local”/IFlow-related datastores.
import com.sap.gateway.ip.core.customdev.util.Message
//Imports for the DataStore-class handling/access
import com.sap.esb.datastore.DataStore
import com.sap.esb.datastore.Data
import org.osgi.framework.*
Message processData(Message message) {
//Get CamelContext and from that the DataStore instance
def camelCtx = message.exchange.getContext()
DataStore dataStore = (DataStore)camelCtx.getRegistry().lookupByName(DataStore.class.getName())
//Define headers and payload/body as byte[]
Map<String, Object> headers = ["headerName1":"me", "anotherHeader": false]
def payload = "This is sample data".getBytes("UTF-8")
//Create datastore payload/data with the following parameters
//params => (DatastoreName, ContextName, EntryId, Body, Headers, MessageId, Version)
//Note: Setting ContextName to null, will create a global Datastore
Data dsData = new Data("DatastoreFromGroovyServiceImpl", null,
"TestEntry", payload, headers, "life-is-hard", 0)
//Write dsData element to the data store
//params => (DataInstance, overwriteEntry, encrypt, alertPeriodInMs, expirePeriodInMs)
dataStore.put(dsData, true, false, 13824000000, 90552000000)
return message
}
Let’s check the result in SAP CPI’s WebIDE now. Do wonder about the same things I did, when I saw the result?
There are two interesting things in the Datastore entry shown above:
- It seems like the MessageID is just a string field and we can pass any string (like “life-is-hard”)
- The expiration period is in 2022! Yes, there is no input validation and thus we can create entries without the 180 days lifespan restriction (which we face when dealing with the regular Datastore elements in the IFlow editor).
Note: Since it is only a couple of days ago when I figured out how to use this classes, I can’t say if the longer lifespan is just displayed, but entries are deleted nevertheless after 180 days latest, or if this “hack” really works. But time will show. In 180 days we know if it works.
When to use which solution?
Now that you have an insight into the two classes and their possibilities, let’s take a look at when you use which class best. (Note: This are just my suggestions. I really would like to discuss my point of view with you in the comment section.)
Use the DataStoreService-class if…
- you want a small code footprint.
- you don’t need to save headers.
- you don’t want to create a “local” datastore.
Use the DataStore-class if…
- you want to store headers.
- read from private/local Datastores of other IFlows.
- you want to bypass the expiration restriction.
Class/function descriptions
As mentioned in the introduction, I tried to keep the code examples as slim as possible. However, the classes presented today offer some more methods and overloads.
However, presenting each one of them would make the article too confusing, which is why I would like to show only the (public) methods of the classes and their overloads below. That should be enough as a starting point so that you can gain your own experience with it.
DataStoreService-class
//DataStoreService.class
public int delete(String storeName, String id) throws DataStoreException
public DataBean get(String storeName, String id) throws DataStoreException
public Class<DataStoreService> getServiceInterface()
public void put(DataBean data, DataConfig config) throws DataStoreException
DataBean-class
//DataBean.class
public byte[] getDataAsArray()
public InputStream getDataAsStream()
public Map<String, Object> getHeaders()
public void setDataAsArray(byte[] dataAsArray)
public void setDataAsStream(InputStream dataAsStream)
public void setHeaders(Map<String, Object> headers)
DataConfig-class
//DataConfig.class
public Boolean doEncrypt()
public Boolean doOverwrite()
public long getExpires()
public String getId()
public String getStoreName()
public void setEncrypt(Boolean encrypt)
public void setExpires(long expires)
public void setId(String id)
public void setOverwrite(Boolean overwrite)
public void setStoreName(String storeName)
DataStore-class
//DataStore.class
public int countEntries(String storeName) throws DataStoreException
public int countStores(boolean alertOnly, String excludeName) throws DataStoreException
public int delete(Long tid) throws DataStoreException
public int delete(String storeName, String id) throws DataStoreException
public int delete(String storeName, String qualifier, String id) throws DataStoreException
public int deleteExpired(int blocksize) throws DataStoreException
public int deleteStore(String storeName, String qualifier, int blocksize) throws DataStoreException
public ExtendedData get(Long tid) throws DataStoreException, MessageNotFoundException
public ExtendedData get(String storeName, String id) throws DataStoreException, MessageNotFoundException
public ExtendedData get(String storeName, String qualifier, String id) throws DataStoreException, MessageNotFoundException
public DataStoreTable getDataStoreTable()
public PlatformTransactionManager getLocalTransactionManager()
public ExtendedData getLock(Long tid) throws DataStoreException, MessageNotFoundException
public ExtendedData getLock(String storeName, String qualifier, String id) throws DataStoreException, MessageNotFoundException
public Long getTid(String storeName, String qualifier, String id) throws DataStoreException, MessageNotFoundException
public int move(String storeName, String qualifier, String oldStoreName, String oldQualifier, List<String> ids) throws DataStoreException
public void put(Data data, BaseData oldData, boolean encrypt, long alert, long expires) throws DataStoreException
public void put(Data data, boolean overwrite, boolean encrypt, long alert, long expires) throws DataStoreException
public void put(Data data, boolean encrypt, long alert, long expires) throws DataStoreException
public List<Data> select(String storeName, String qualifier, int numRows) throws DataStoreException
public List<Data> select(String storeName, int numRows) throws DataStoreException
public List<String> selectIds(String storeName, String qualifier) throws DataStoreException
public List<MetaData> selectMetaData(String storeName, String id, Date alertAt, int numRows, Long excludeRowBefore) throws DataStoreException
public List<MetaData> selectMetaData(String storeName, String qualifier, String id, Date alertAt, int numRows, Long excludeRowBefore) throws DataStoreException
public List<MetaData> selectMetaData(String storeName, String qualifier, String id, Date alertAt, int numRows, Long excludeRowBefore, boolean excludeNonRetry) throws DataStoreException
public List<DataStoreAggregate> selectStores(boolean alertOnly) throws DataStoreException
public List<DataStoreAggregate> selectStoresInternal(boolean alertOnly) throws DataStoreException
public List<Long> selectTids(String storeName, String qualifier) throws DataStoreException
public void setCipherStreamFactory(CipherStreamFactory cipherStreamFactory)
public void setDataSource(DataSource dataSource)
public void setDatabaseProperties(DatabaseProperties databaseProperties)
public void setPlatformTransactionManager(PlatformTransactionManager platformTransactionManager)
public boolean supportsSaneLocking()
public void updateRetry(Long tid, Date retryAt) throws MessageNotFoundException, DataStoreException
public void updateRetry(Long tid, Date retryAt, String mplId) throws MessageNotFoundException, DataStoreException
Data-class
//Data.class
public Data(String storeName, String id, InputStream data)
public Data(String storeName, String id, InputStream data, Map<String, Object> headers)
public Data(String storeName, String id, InputStream data, Map<String, Object> headers, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data)
public Data(String storeName, String qualifier, String id, InputStream data, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data, Map<String, Object> headers)
public Data(String storeName, String qualifier, String id, InputStream data, Map<String, Object> headers, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data, Map<String, Object> headers, String mplId, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data, String mplId, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data)
public Data(String storeName, String qualifier, String id, byte[] data, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data, Map<String, Object> headers)
public Data(String storeName, String qualifier, String id, byte[] data, Map<String, Object> headers, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data, Map<String, Object> headers, String mplId, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data, String mplID, Integer version)
public Data(String storeName, String id, byte[] data)
public Data(String storeName, String id, byte[] data, Map<String, Object> headers)
public Object getData()
public byte[] getDataAsArray()
public InputStream getDataAsStream()
public Map<String, Object> getHeaders()
public void setHeaders(Map<String, Object> headers)
Conclusion
Now another article is coming to an end. I hope you enjoyed reading and maybe learned something new. I would like to know from you whether you find the classes presented above interesting and whether you can see practical applications for them. (It may also be that you say: “No, they are not official and will therefore never be used by me.”)
For my part, I will try to write a DataStore viewer for the RealCore SAP CPI dashboard. (Unless someone of you wants to do it – the source code is free and cooperation is always welcome.)
Hi Raffael,
You are using internal classes that can anytime change which will cause your code to break.
Also transaction handling is not considered in your approach which might cause data inconsistencies.
The CPI development team therefore strongly recommends to not use the described approach in productive integration flows.
But we would like to understand your use case behind it in order to understand what functionality you're missing.
best regards,
CPI dev
Hi Axel,
thanks for your feedback. First things first. I know that this classes can change anytime and therefore shouldn't be used in productive environments. To make other readers aware, I added the disclaimer right at the beginning of the article. If I can improve the disclaimer or if you got the feeling it's not clear enough, please let me know how I could/should re-write it. 🙂
Regarding the use-cases:
Since we, as CPI (interface) developers, should take care to keep the Exchange's footprint as small as possible, I would like to access data store directly from Groovy script, instead of reading data via DataStore component, bloating up the Exchange's properties, just to get data accessible in the Groovy scripts.
Maybe Vadim Klimov, Eng Swee Yeoh, Ariel Bravo Ayala, Morten Wittrock or Daniel Graversen can add more/other valid points to the list.
If I could wish for something, I would love to see a DataStoreService class analogous to the ValueMappingService class in the future.
Thanks again for listening and taking the time to read the blog.
Regards,
Raffael
Me as well would love to easily access data stores via groovy.
This makes it much easier to access and write information, where building this within the iflow would be a pain with persisting the original body, getting the stored information, writing it to a variable and then restoring the original body.
Usecase: Get information, process, get last run body from data store, process the last run body vs. current run with groovy and exchange body to groovy result.
My use case is set in the context of 3rd party integration.
Maybe a reason to consider this.
Thanks a lot!
Kind regards,
Gandalf
Hi Gandalf,
so you mainly need reading capabilities?
regards,
Axel
Reading and Writing.
If I want to read something from a datastore, which I only need in a script, I have to first persist the original body message, call the Data Store, which overwrites the message body. Then I need to run the script and set a property to the message and update the body to the message, which I would like to write back.
In the iFlow the Data Store Write takes place and then the original Message or previously set property needs to be set as body.
Well, it works, but blows up the iflow in complexity.
Hi Raffael Herrmann ,
Its really a helpful post .However for development in local need download the relevant jar files to
access the com.sap.it.api.asdk.datastore-package which I didnt find yet. Need your assistance to know place where from I can get the relevant jar files.
Regards
R Adhikary
Hi Axel,
has since Raffael posted this blog something been changed with the status of the mentioned APIs (e.g. officially documented now) or is that planned soon?
I would currently have the urgent need to store records inside a GroovyScript from a GroovyScript. That would highly simplify the change.
Best regards,
Malte
Hi Malte Schluenz ,
so far there is no script API for the datastore available.
best regards,
Axel
Hi Raffael Herrmann & All
I am just trying to store my custom object into data store and retrieve it . I tried but i am getting null always .
my custom class is .:
package src.main.resources.script
class Employee implements Serializable{
private String name;
private Long id;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public Employee( Long id, String name) {
super();
this.name = name;
this.id = id;
}
@Override
public String toString() {
return “Employee [name=” + name + “, id=” + id + “]”;
}
@Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + ((id == null) ? 0 : id.hashCode());
return result;
}
@Override
public boolean equals(Object obj) {
if (this == obj)
return true;
if (obj == null)
return false;
if (getClass() != obj.getClass())
return false;
Employee other = (Employee) obj;
if (id == null) {
if (other.id != null)
return false;
} else if (!id.equals(other.id))
return false;
return true;
}
}
create script :
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import com.sap.it.api.asdk.datastore.*
import com.sap.it.api.asdk.runtime.*
import src.main.resources.script.*
def Message processData(Message message) {
def messageLog = messageLogFactory.getMessageLog(message);
//Data to be stored in datatore
Employee emp = new Employee(1l, “name”);
messageLog.addAttachmentAsString(“emp object”, emp.toString(), “text/plain”)
//Get service instance
def service = new Factory(DataStoreService.class).getService()
//Check if valid service instance was retrieved
if( service != null) {
def dBean = new DataBean()
dBean.setDataAsArray(serialize(emp))
//Class model offers headers, but for me it didn’t work out
//Map<String, Object> headers = [“headerName1″:”me”, “anotherHeader”: false]
//dBean.setHeaders(headers)
//Define datatore name and entry id
def dConfig = new DataConfig()
dConfig.setStoreName(“TestStore”)
dConfig.setId(“TestId”)
dConfig.setOverwrite(true)
dConfig.setExpires(1)
dConfig.getExpires();
messageLog.addAttachmentAsString(“getExpires”, dConfig.getExpires().toString(), “text/plain”)
//Write to data store
result = service.put(dBean,dConfig)
}
return message;
}
def byte[] serialize(Object obj) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(out);
os.writeObject(obj);
return out.toByteArray();
}
read script:
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import src.main.resources.script.*
import com.sap.it.api.asdk.datastore.*
import com.sap.it.api.asdk.runtime.*
import java.io.*
def Message processData(Message message) {
//Get service instance
def service = new Factory(DataStoreService.class).getService()
def messageLog = messageLogFactory.getMessageLog(message);
//Check if valid service instance was retrieved
if( service != null) {
//Read data store entry via id
def dsEntry = service.get(“TestStore”,”TestId”)
messageLog.addAttachmentAsString(“ds Entry”,new String( dsEntry.getDataAsArray()), “text/plain”)
Employee emp = toObject(dsEntry.getDataAsArray())
messageLog.addAttachmentAsString(“Employee “, emp.toString(), “text/plain”)
message.setBody(emp.toString())
}
return message
}
private static Object toObject(byte[] bytes) {
Object obj = null;
try {
ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
ObjectInputStream ois = new ObjectInputStream(bis);
obj = ois.readObject();
} catch (Exception ex) {
ex.printStackTrace(); // ClassNotFoundException
}
return obj;
}
Can you please help me what i am doing wrong. Thanks in Advance.
Hi Raffael Herrmann ,
great blog!!! Hope your entries are still available.. refer to the hack over 180 days:D
is there a possiblity , to read and write the global variable through groovy scripting?
Best Regards,
Said
Hi Raffael Herrmann,
That's a great information but, can you please provide another information that,
How to use if, else condition in the read script.
I am using "def dsEntry = dataStore.get('datastorename', 'datadtoreID')" . For datastoreID I am defining a value and if that value not exists as a entryid I wanted to create another possible ID using if else condition.
so 'if' fails 'else' should work.
The error I am getting is " No message ID found for the ID "" in store "" and at this point I want to use else condition.
I am unable to understand which variable is getting effected for this message ID not found.
Kindly help.
Regards
Vinod
Hi Vinod,
you could simply wrape the dataStore.get(...)-call in a try-catch block. The check in the catch block if the catched Exception reads out as "No message ID..." and react on it. (https://www.tutorialspoint.com/groovy/groovy_exception_handling.htm)
Regards,
Raffael
Hi Raffael Herrmann
can you do i flow design ,to fetch a message based on JMS Message ID from JMS Queue using groovy script.
To fetch a message based on JMS Message ID from Data store using groovy script.