How to read/write SAP CPI Datastore from Groovy

A few days ago I stumbled upon Daniel’s question whether it is possible to address the SAP CPI Datastore via Groovy Script. I did not want to deal with the answer that there was no possibility and the standard building blocks had to be used. So I did another deep dive into SAP CPI’s internals and found two solutions.

In the following article I would like to show you how it is possible to write to and read from the SAP CPI Datastore using Groovy Script. Before we start, please read the following disclaimer.

Disclaimer: The functions shown below are not officially documented. Although they work perfectly and are partly used by the official Datastore modules, you should always keep in mind that these are not official functions. So use them responsibly.

Table of contents

• Solution 1: Access via DataStoreService-class
• Solution 2: Access via DataStore-class
• When to use which solution?
• Class/function descriptions
  • DataStoreService-class
  • DataBean-class
  • DataConfig-class
  • DataStore-class
  • Data-class
• Conclusion

Solution 1: Access via DataStoreService-class

The first option to access the Datastore via Groovy script is via the DataStoreService-class from the com.sap.it.api.asdk.datastore-package. This class gives simple and easy access but comes with less flexibility and options than the second solution. The function call itself is similiar to the use of the official ValueMappingApi.

Read from Datastore

To read from the Datastore the following code is needed. Please check the source code comments for further description.

import com.sap.gateway.ip.core.customdev.util.Message

//Imports for DataStoreService-class
import com.sap.it.api.asdk.datastore.*
import com.sap.it.api.asdk.runtime.*

Message processData(Message message) {
	
	//Get service instance
	def service = new Factory(DataStoreService.class).getService()
	
	//Check if valid service instance was retrieved
	if( service != null) {			
		//Read data store entry via id
		def dsEntry = service.get("DatastoreName","EntryId")
		def result = new String(dsEntry.getDataAsArray())
		message.setBody(body)
	}		
    return message
}

Restrictions: 

• You can only read from Datastore that are either “Global” or belong to your IFlow. (Same behaviour like the official Datastore-blocks in the IFlow editor.)

Write to Datastore

To write to the Datastore via the DataStoreService-class you need two more “helper” classes: DataBean and DataConfig. The DataBean holds the actual data you want to store and the DataConfig holds the target Datastore name as also some metadata.

import com.sap.gateway.ip.core.customdev.util.Message

//Imports for DataStoreService-class
import com.sap.it.api.asdk.datastore.*
import com.sap.it.api.asdk.runtime.*

Message processData(Message message) {
	
	//Data to be stored in datatore
	def payload = "This is sample data"
	//Get service instance
	def service = new Factory(DataStoreService.class).getService()
	
	//Check if valid service instance was retrieved
	if( service != null) {		
		def dBean = new DataBean()
		dBean.setDataAsArray(payload.getBytes("UTF-8"))		
		//Class model offers headers, but for me it didn't work out
		//Map<String, Object> headers = ["headerName1":"me", "anotherHeader": false]
		//dBean.setHeaders(headers)
		
		//Define datatore name and entry id
		def dConfig = new DataConfig()
		dConfig.setStoreName("DatastoreFromGroovyASDK")
		dConfig.setId("TestEntry")
		dConfig.setOverwrite(true)
		
		//Write to data store
		result = service.put(dBean,dConfig)
	}		
    return message
}

Please note, that we use a string as payload here, but you can write anything to the store, as long as you pass it as byte[] to the DataBean instance. To keep the example above simple and clear, I didn’t use all parameters (like expiration period, etc.). To see full list of parameters/function, check the paragraph with the class description at the end of this article.

The entry, created by the above code looks as follows:

Restrictions:

• Passing headers when storing entries doesn’t work. (No error is thrown, but at least in my tests the headers never arrived in the Datastore.)
• You can’t set a context, thus entries/datastore created via the DataStoreService-class will be global entries and can be read by all other IFlows!
• You can’t set a message id/MPL id, so the “Message ID” column in SAP CPI’s datastore viewer will always be empty. (See screenshot above.)

Solution 2: Access via DataStore-class

The second way to access the data storage is the so-called DataStore-class. As I have seen, it seems to be the underlying class of the DataStoreService. (I’m not sure, however.) But I can say with certainty that this class is much more flexible, but also at the expense of ease of use. Let’s have a look how it works.

Read from Datastore

Please check the source code comments. By use of the DataStore class you can read from any Datastore (regardless of local/global modifiers). You can also select set of entries via the select-function.

import com.sap.gateway.ip.core.customdev.util.Message

//Imports for the DataStore-class handling/access
import com.sap.esb.datastore.DataStore
import com.sap.esb.datastore.Data
import org.osgi.framework.*

Message processData(Message message) {
	
	//Get CamelContext and from that the DataStore instance
	def camelCtx = message.exchange.getContext()
	DataStore dataStore = (DataStore)camelCtx.getRegistry().lookupByName(DataStore.class.getName())
	
	//Read from Datastore params => (DatastoreName, EntryId)
	def dsEntry = dataStore.get("DatastoreFromGroovyServiceImpl", "TestEntry")
	//Get datastore entry payload as String
	def payload = new String(dsEntry.getDataAsArray())
	
	//NOTE: If you pass the name context-name ad third parameter, you can read from 
	//      any datastore, regardless if it's global or local! The context name usually 
	//      equals the id of the IFlow a Datastore belongs to.
	//def dsEntry = dataStore.get("DatastoreName", "ContextName", "EntryId")
	
	//If you want to select multiple entry, use select-method and pass the 
	//following params => (DatastoreName, ContextName, NumberOfEntriesToPull)
	//def dsEntryArray = dataStore.select("DatastoreName", "ContextName", 10)
	
    message.setBody(payload)
    return message
}

As you can see there are two major differences between the DataStore- and the DataStoreService implementation.

1. You can read from any Datastore by passing the Datastore-context as third parameter.
2. You can use a select-function, to read multiple entries, without passing entries ids as parameter.

Write to DataStore

When writing to the Datastore via the DataStore-class you can also add headers and set the context and thus create “local”/IFlow-related datastores.

import com.sap.gateway.ip.core.customdev.util.Message

//Imports for the DataStore-class handling/access
import com.sap.esb.datastore.DataStore
import com.sap.esb.datastore.Data
import org.osgi.framework.*

Message processData(Message message) {
	
	//Get CamelContext and from that the DataStore instance
	def camelCtx = message.exchange.getContext()
	DataStore dataStore = (DataStore)camelCtx.getRegistry().lookupByName(DataStore.class.getName())
	
	//Define headers and payload/body as byte[]
	Map<String, Object> headers = ["headerName1":"me", "anotherHeader": false]
	def payload = "This is sample data".getBytes("UTF-8")
	
	//Create datastore payload/data with the following parameters
	//params => (DatastoreName, ContextName, EntryId, Body, Headers, MessageId, Version)
	//Note: Setting ContextName to null, will create a global Datastore
	Data dsData = new Data("DatastoreFromGroovyServiceImpl", null, 
	                       "TestEntry", payload, headers, "life-is-hard", 0)
	
	//Write dsData element to the data store
	//params => (DataInstance, overwriteEntry, encrypt, alertPeriodInMs, expirePeriodInMs)
	dataStore.put(dsData, true, false, 13824000000, 90552000000)
	
    return message
}

Let’s check the result in SAP CPI’s WebIDE now. Do wonder about the same things I did, when I saw the result?

There are two interesting things in the Datastore entry shown above:

1. It seems like the MessageID is just a string field and we can pass any string (like “life-is-hard”)
2. The expiration period is in 2022! Yes, there is no input validation and thus we can create entries without the 180 days lifespan restriction (which we face when dealing with the regular Datastore elements in the IFlow editor).

Note: Since it is only a couple of days ago when I figured out how to use this classes, I can’t say if the longer lifespan is just displayed, but entries are deleted nevertheless after 180 days latest, or if this “hack” really works. But time will show. In 180 days we know if it works.

When to use which solution?

Now that you have an insight into the two classes and their possibilities, let’s take a look at when you use which class best. (Note: This are just my suggestions. I really would like to discuss my point of view with you in the comment section.)

Use the DataStoreService-class if…

• you want a small code footprint.
• you don’t need to save headers.
• you don’t want to create a “local” datastore.

Use the DataStore-class if…

• you want to store headers.
• read from private/local Datastores of other IFlows.
• you want to bypass the expiration restriction.

Class/function descriptions

As mentioned in the introduction, I tried to keep the code examples as slim as possible. However, the classes presented today offer some more methods and overloads.

However, presenting each one of them would make the article too confusing, which is why I would like to show only the (public) methods of the classes and their overloads below. That should be enough as a starting point so that you can gain your own experience with it.

DataStoreService-class

//DataStoreService.class
public int delete(String storeName, String id) throws DataStoreException
public DataBean get(String storeName, String id) throws DataStoreException
public Class<DataStoreService> getServiceInterface()
public void put(DataBean data, DataConfig config) throws DataStoreException

DataBean-class

//DataBean.class
public byte[] getDataAsArray()
public InputStream getDataAsStream()
public Map<String, Object> getHeaders()
public void setDataAsArray(byte[] dataAsArray)
public void setDataAsStream(InputStream dataAsStream)
public void setHeaders(Map<String, Object> headers)

DataConfig-class

//DataConfig.class
public Boolean doEncrypt()
public Boolean doOverwrite()
public long getExpires()
public String getId()
public String getStoreName()
public void setEncrypt(Boolean encrypt)
public void setExpires(long expires)
public void setId(String id)
public void setOverwrite(Boolean overwrite)
public void setStoreName(String storeName)

DataStore-class

//DataStore.class
public int countEntries(String storeName) throws DataStoreException
public int countStores(boolean alertOnly, String excludeName) throws DataStoreException
public int delete(Long tid) throws DataStoreException
public int delete(String storeName, String id) throws DataStoreException
public int delete(String storeName, String qualifier, String id) throws DataStoreException
public int deleteExpired(int blocksize) throws DataStoreException
public int deleteStore(String storeName, String qualifier, int blocksize) throws DataStoreException
public ExtendedData get(Long tid) throws DataStoreException, MessageNotFoundException
public ExtendedData get(String storeName, String id) throws DataStoreException, MessageNotFoundException
public ExtendedData get(String storeName, String qualifier, String id) throws DataStoreException, MessageNotFoundException
public DataStoreTable getDataStoreTable()
public PlatformTransactionManager getLocalTransactionManager()
public ExtendedData getLock(Long tid) throws DataStoreException, MessageNotFoundException
public ExtendedData getLock(String storeName, String qualifier, String id) throws DataStoreException, MessageNotFoundException
public Long getTid(String storeName, String qualifier, String id) throws DataStoreException, MessageNotFoundException
public int move(String storeName, String qualifier, String oldStoreName, String oldQualifier, List<String> ids) throws DataStoreException
public void put(Data data, BaseData oldData, boolean encrypt, long alert, long expires) throws DataStoreException
public void put(Data data, boolean overwrite, boolean encrypt, long alert, long expires) throws DataStoreException
public void put(Data data, boolean encrypt, long alert, long expires) throws DataStoreException
public List<Data> select(String storeName, String qualifier, int numRows) throws DataStoreException
public List<Data> select(String storeName, int numRows) throws DataStoreException
public List<String> selectIds(String storeName, String qualifier) throws DataStoreException
public List<MetaData> selectMetaData(String storeName, String id, Date alertAt, int numRows, Long excludeRowBefore) throws DataStoreException
public List<MetaData> selectMetaData(String storeName, String qualifier, String id, Date alertAt, int numRows, Long excludeRowBefore) throws DataStoreException
public List<MetaData> selectMetaData(String storeName, String qualifier, String id, Date alertAt, int numRows, Long excludeRowBefore, boolean excludeNonRetry) throws DataStoreException
public List<DataStoreAggregate> selectStores(boolean alertOnly) throws DataStoreException
public List<DataStoreAggregate> selectStoresInternal(boolean alertOnly) throws DataStoreException
public List<Long> selectTids(String storeName, String qualifier) throws DataStoreException
public void setCipherStreamFactory(CipherStreamFactory cipherStreamFactory)
public void setDataSource(DataSource dataSource)
public void setDatabaseProperties(DatabaseProperties databaseProperties)
public void setPlatformTransactionManager(PlatformTransactionManager platformTransactionManager)
public boolean supportsSaneLocking()
public void updateRetry(Long tid, Date retryAt) throws MessageNotFoundException, DataStoreException
public void updateRetry(Long tid, Date retryAt, String mplId) throws MessageNotFoundException, DataStoreException

Data-class

//Data.class
public Data(String storeName, String id, InputStream data)
public Data(String storeName, String id, InputStream data, Map<String, Object> headers)
public Data(String storeName, String id, InputStream data, Map<String, Object> headers, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data)
public Data(String storeName, String qualifier, String id, InputStream data, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data, Map<String, Object> headers)
public Data(String storeName, String qualifier, String id, InputStream data, Map<String, Object> headers, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data, Map<String, Object> headers, String mplId, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data, String mplId, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data)
public Data(String storeName, String qualifier, String id, byte[] data, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data, Map<String, Object> headers)
public Data(String storeName, String qualifier, String id, byte[] data, Map<String, Object> headers, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data, Map<String, Object> headers, String mplId, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data, String mplID, Integer version)
public Data(String storeName, String id, byte[] data)
public Data(String storeName, String id, byte[] data, Map<String, Object> headers)
public Object getData()
public byte[] getDataAsArray()
public InputStream getDataAsStream()
public Map<String, Object> getHeaders()
public void setHeaders(Map<String, Object> headers)

Conclusion

Now another article is coming to an end. I hope you enjoyed reading and maybe learned something new. I would like to know from you whether you find the classes presented above interesting and whether you can see practical applications for them. (It may also be that you say: “No, they are not official and will therefore never be used by me.”)

For my part, I will try to write a DataStore viewer for the RealCore SAP CPI dashboard. (Unless someone of you wants to do it – the source code is free and cooperation is always welcome.)