Skip to Content
Technical Articles

How to read/write SAP CPI Datastore from Groovy

A few days ago I stumbled upon Daniel’s question whether it is possible to address the SAP CPI Datastore via Groovy Script. I did not want to deal with the answer that there was no possibility and the standard building blocks had to be used. So I did another deep dive into SAP CPI’s internals and found two solutions.

In the following article I would like to show you how it is possible to write to and read from the SAP CPI Datastore using Groovy Script. Before we start, please read the following disclaimer.

Disclaimer: The functions shown below are not officially documented. Although they work perfectly and are partly used by the official Datastore modules, you should always keep in mind that these are not official functions. So use them responsibly.

Table of contents

  • Solution 1: Access via DataStoreService-class
  • Solution 2: Access via DataStore-class
  • When to use which solution?
  • Class/function descriptions
    • DataStoreService-class
    • DataBean-class
    • DataConfig-class
    • DataStore-class
    • Data-class
  • Conclusion

Solution 1: Access via DataStoreService-class

The first option to access the Datastore via Groovy script is via the DataStoreService-class from the com.sap.it.api.asdk.datastore-package. This class gives simple and easy access but comes with less flexibility and options than the second solution. The function call itself is similiar to the use of the official ValueMappingApi.

Read from Datastore

To read from the Datastore the following code is needed. Please check the source code comments for further description.

import com.sap.gateway.ip.core.customdev.util.Message

//Imports for DataStoreService-class
import com.sap.it.api.asdk.datastore.*
import com.sap.it.api.asdk.runtime.*

Message processData(Message message) {
	
	//Get service instance
	def service = new Factory(DataStoreService.class).getService()
	
	//Check if valid service instance was retrieved
	if( service != null) {			
		//Read data store entry via id
		def dsEntry = service.get("DatastoreName","EntryId")
		def result = new String(dsEntry.getDataAsArray())
		message.setBody(body)
	}		
    return message
}

Restrictions: 

  • You can only read from Datastore that are either “Global” or belong to your IFlow. (Same behaviour like the official Datastore-blocks in the IFlow editor.)

Write to Datastore

To write to the Datastore via the DataStoreService-class you need two more “helper” classes: DataBean and DataConfig. The DataBean holds the actual data you want to store and the DataConfig holds the target Datastore name as also some metadata.

import com.sap.gateway.ip.core.customdev.util.Message

//Imports for DataStoreService-class
import com.sap.it.api.asdk.datastore.*
import com.sap.it.api.asdk.runtime.*

Message processData(Message message) {
	
	//Data to be stored in datatore
	def payload = "This is sample data"
	//Get service instance
	def service = new Factory(DataStoreService.class).getService()
	
	//Check if valid service instance was retrieved
	if( service != null) {		
		def dBean = new DataBean()
		dBean.setDataAsArray(payload.getBytes("UTF-8"))		
		//Class model offers headers, but for me it didn't work out
		//Map<String, Object> headers = ["headerName1":"me", "anotherHeader": false]
		//dBean.setHeaders(headers)
		
		//Define datatore name and entry id
		def dConfig = new DataConfig()
		dConfig.setStoreName("DatastoreFromGroovyASDK")
		dConfig.setId("TestEntry")
		dConfig.setOverwrite(true)
		
		//Write to data store
		result = service.put(dBean,dConfig)
	}		
    return message
}

Please note, that we use a string as payload here, but you can write anything to the store, as long as you pass it as byte[] to the DataBean instance. To keep the example above simple and clear, I didn’t use all parameters (like expiration period, etc.). To see full list of parameters/function, check the paragraph with the class description at the end of this article.

The entry, created by the above code looks as follows:

Restrictions:

  • Passing headers when storing entries doesn’t work. (No error is thrown, but at least in my tests the headers never arrived in the Datastore.)
  • You can’t set a context, thus entries/datastore created via the DataStoreService-class will be global entries and can be read by all other IFlows!
  • You can’t set a message id/MPL id, so the “Message ID” column in SAP CPI’s datastore viewer will always be empty. (See screenshot above.)

Solution 2: Access via DataStore-class

The second way to access the data storage is the so-called DataStore-class. As I have seen, it seems to be the underlying class of the DataStoreService. (I’m not sure, however.) But I can say with certainty that this class is much more flexible, but also at the expense of ease of use. Let’s have a look how it works.

Read from Datastore

Please check the source code comments. By use of the DataStore class you can read from any Datastore (regardless of local/global modifiers). You can also select set of entries via the select-function.

import com.sap.gateway.ip.core.customdev.util.Message

//Imports for the DataStore-class handling/access
import com.sap.esb.datastore.DataStore
import com.sap.esb.datastore.Data
import org.osgi.framework.*

Message processData(Message message) {
	
	//Get CamelContext and from that the DataStore instance
	def camelCtx = message.exchange.getContext()
	DataStore dataStore = (DataStore)camelCtx.getRegistry().lookupByName(DataStore.class.getName())
	
	//Read from Datastore params => (DatastoreName, EntryId)
	def dsEntry = dataStore.get("DatastoreFromGroovyServiceImpl", "TestEntry")
	//Get datastore entry payload as String
	def payload = new String(dsEntry.getDataAsArray())
	
	//NOTE: If you pass the name context-name ad third parameter, you can read from 
	//      any datastore, regardless if it's global or local! The context name usually 
	//      equals the id of the IFlow a Datastore belongs to.
	//def dsEntry = dataStore.get("DatastoreName", "ContextName", "EntryId")
	
	//If you want to select multiple entry, use select-method and pass the 
	//following params => (DatastoreName, ContextName, NumberOfEntriesToPull)
	//def dsEntryArray = dataStore.select("DatastoreName", "ContextName", 10)
	
    message.setBody(payload)
    return message
}

As you can see there are two major differences between the DataStore- and the DataStoreService implementation.

  1. You can read from any Datastore by passing the Datastore-context as third parameter.
  2. You can use a select-function, to read multiple entries, without passing entries ids as parameter.

Write to DataStore

When writing to the Datastore via the DataStore-class you can also add headers and set the context and thus create “local”/IFlow-related datastores.

import com.sap.gateway.ip.core.customdev.util.Message

//Imports for the DataStore-class handling/access
import com.sap.esb.datastore.DataStore
import com.sap.esb.datastore.Data
import org.osgi.framework.*

Message processData(Message message) {
	
	//Get CamelContext and from that the DataStore instance
	def camelCtx = message.exchange.getContext()
	DataStore dataStore = (DataStore)camelCtx.getRegistry().lookupByName(DataStore.class.getName())
	
	//Define headers and payload/body as byte[]
	Map<String, Object> headers = ["headerName1":"me", "anotherHeader": false]
	def payload = "This is sample data".getBytes("UTF-8")
	
	//Create datastore payload/data with the following parameters
	//params => (DatastoreName, ContextName, EntryId, Body, Headers, MessageId, Version)
	//Note: Setting ContextName to null, will create a global Datastore
	Data dsData = new Data("DatastoreFromGroovyServiceImpl", null, 
	                       "TestEntry", payload, headers, "life-is-hard", 0)
	
	//Write dsData element to the data store
	//params => (DataInstance, overwriteEntry, encrypt, alertPeriodInMs, expirePeriodInMs)
	dataStore.put(dsData, true, false, 13824000000, 90552000000)
	
    return message
}

Let’s check the result in SAP CPI’s WebIDE now. Do wonder about the same things I did, when I saw the result?

There are two interesting things in the Datastore entry shown above:

  1. It seems like the MessageID is just a string field and we can pass any string (like “life-is-hard”)
  2. The expiration period is in 2022! Yes, there is no input validation and thus we can create entries without the 180 days lifespan restriction (which we face when dealing with the regular Datastore elements in the IFlow editor).

Note: Since it is only a couple of days ago when I figured out how to use this classes, I can’t say if the longer lifespan is just displayed, but entries are deleted nevertheless after 180 days latest, or if this “hack” really works. But time will show. In 180 days we know if it works.

When to use which solution?

Now that you have an insight into the two classes and their possibilities, let’s take a look at when you use which class best. (Note: This are just my suggestions. I really would like to discuss my point of view with you in the comment section.)

Use the DataStoreService-class if…

  • you want a small code footprint.
  • you don’t need to save headers.
  • you don’t want to create a “local” datastore.

Use the DataStore-class if…

  • you want to store headers.
  • read from private/local Datastores of other IFlows.
  • you want to bypass the expiration restriction.

Class/function descriptions

As mentioned in the introduction, I tried to keep the code examples as slim as possible. However, the classes presented today offer some more methods and overloads.

However, presenting each one of them would make the article too confusing, which is why I would like to show only the (public) methods of the classes and their overloads below. That should be enough as a starting point so that you can gain your own experience with it.

DataStoreService-class

//DataStoreService.class
public int delete(String storeName, String id) throws DataStoreException
public DataBean get(String storeName, String id) throws DataStoreException
public Class<DataStoreService> getServiceInterface()
public void put(DataBean data, DataConfig config) throws DataStoreException

DataBean-class

//DataBean.class
public byte[] getDataAsArray()
public InputStream getDataAsStream()
public Map<String, Object> getHeaders()
public void setDataAsArray(byte[] dataAsArray)
public void setDataAsStream(InputStream dataAsStream)
public void setHeaders(Map<String, Object> headers)

DataConfig-class

//DataConfig.class
public Boolean doEncrypt()
public Boolean doOverwrite()
public long getExpires()
public String getId()
public String getStoreName()
public void setEncrypt(Boolean encrypt)
public void setExpires(long expires)
public void setId(String id)
public void setOverwrite(Boolean overwrite)
public void setStoreName(String storeName)

DataStore-class

//DataStore.class
public int countEntries(String storeName) throws DataStoreException
public int countStores(boolean alertOnly, String excludeName) throws DataStoreException
public int delete(Long tid) throws DataStoreException
public int delete(String storeName, String id) throws DataStoreException
public int delete(String storeName, String qualifier, String id) throws DataStoreException
public int deleteExpired(int blocksize) throws DataStoreException
public int deleteStore(String storeName, String qualifier, int blocksize) throws DataStoreException
public ExtendedData get(Long tid) throws DataStoreException, MessageNotFoundException
public ExtendedData get(String storeName, String id) throws DataStoreException, MessageNotFoundException
public ExtendedData get(String storeName, String qualifier, String id) throws DataStoreException, MessageNotFoundException
public DataStoreTable getDataStoreTable()
public PlatformTransactionManager getLocalTransactionManager()
public ExtendedData getLock(Long tid) throws DataStoreException, MessageNotFoundException
public ExtendedData getLock(String storeName, String qualifier, String id) throws DataStoreException, MessageNotFoundException
public Long getTid(String storeName, String qualifier, String id) throws DataStoreException, MessageNotFoundException
public int move(String storeName, String qualifier, String oldStoreName, String oldQualifier, List<String> ids) throws DataStoreException
public void put(Data data, BaseData oldData, boolean encrypt, long alert, long expires) throws DataStoreException
public void put(Data data, boolean overwrite, boolean encrypt, long alert, long expires) throws DataStoreException
public void put(Data data, boolean encrypt, long alert, long expires) throws DataStoreException
public List<Data> select(String storeName, String qualifier, int numRows) throws DataStoreException
public List<Data> select(String storeName, int numRows) throws DataStoreException
public List<String> selectIds(String storeName, String qualifier) throws DataStoreException
public List<MetaData> selectMetaData(String storeName, String id, Date alertAt, int numRows, Long excludeRowBefore) throws DataStoreException
public List<MetaData> selectMetaData(String storeName, String qualifier, String id, Date alertAt, int numRows, Long excludeRowBefore) throws DataStoreException
public List<MetaData> selectMetaData(String storeName, String qualifier, String id, Date alertAt, int numRows, Long excludeRowBefore, boolean excludeNonRetry) throws DataStoreException
public List<DataStoreAggregate> selectStores(boolean alertOnly) throws DataStoreException
public List<DataStoreAggregate> selectStoresInternal(boolean alertOnly) throws DataStoreException
public List<Long> selectTids(String storeName, String qualifier) throws DataStoreException
public void setCipherStreamFactory(CipherStreamFactory cipherStreamFactory)
public void setDataSource(DataSource dataSource)
public void setDatabaseProperties(DatabaseProperties databaseProperties)
public void setPlatformTransactionManager(PlatformTransactionManager platformTransactionManager)
public boolean supportsSaneLocking()
public void updateRetry(Long tid, Date retryAt) throws MessageNotFoundException, DataStoreException
public void updateRetry(Long tid, Date retryAt, String mplId) throws MessageNotFoundException, DataStoreException

Data-class

//Data.class
public Data(String storeName, String id, InputStream data)
public Data(String storeName, String id, InputStream data, Map<String, Object> headers)
public Data(String storeName, String id, InputStream data, Map<String, Object> headers, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data)
public Data(String storeName, String qualifier, String id, InputStream data, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data, Map<String, Object> headers)
public Data(String storeName, String qualifier, String id, InputStream data, Map<String, Object> headers, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data, Map<String, Object> headers, String mplId, Integer version)
public Data(String storeName, String qualifier, String id, InputStream data, String mplId, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data)
public Data(String storeName, String qualifier, String id, byte[] data, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data, Map<String, Object> headers)
public Data(String storeName, String qualifier, String id, byte[] data, Map<String, Object> headers, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data, Map<String, Object> headers, String mplId, Integer version)
public Data(String storeName, String qualifier, String id, byte[] data, String mplID, Integer version)
public Data(String storeName, String id, byte[] data)
public Data(String storeName, String id, byte[] data, Map<String, Object> headers)
public Object getData()
public byte[] getDataAsArray()
public InputStream getDataAsStream()
public Map<String, Object> getHeaders()
public void setHeaders(Map<String, Object> headers)

Conclusion

Now another article is coming to an end. I hope you enjoyed reading and maybe learned something new. I would like to know from you whether you find the classes presented above interesting and whether you can see practical applications for them. (It may also be that you say: “No, they are not official and will therefore never be used by me.”)

For my part, I will try to write a DataStore viewer for the RealCore SAP CPI dashboard. (Unless someone of you wants to do it – the source code is free and cooperation is always welcome.)

6 Comments
You must be Logged on to comment or reply to a post.
  • Hi Raffael,

    You are using internal classes that can anytime change which will cause your code to break.

    Also transaction handling is not considered in your approach which might cause data inconsistencies.

    The CPI development team therefore strongly recommends to not use the described approach in productive integration flows.

    But we would like to understand your use case behind it in order to understand what functionality you’re missing.

    best regards,
    CPI dev

    • Hi Axel,

      thanks for your feedback. First things first. I know that this classes can change anytime and therefore shouldn’t be used in productive environments. To make other readers aware, I added the disclaimer right at the beginning of the article. If I can improve the disclaimer or if you got the feeling it’s not clear enough, please let me know how I could/should re-write it. 🙂

      Regarding the use-cases:

      • It would be helpful, if we could pass the “qualifier” argument via the DataStore components in the IFlow editor. (I guess by choosing “Global” or “Local” we already set this qualifiers with “Local” = IFlowId.) If we could set the qualifier in the IFlows we could create private/local DataStores for groups of IFlows. Background: We’re trying to modularize interfaces where it is possible. So we may have one sender flow and multiple receiver flows. It would be great if we could share a data store between those interfaces, without setting it to global and allow access to every iflow.
      • It would be helpful to have general acccess to datastores from Groovy scripts. When it comes to complex or non-XML mappings, Groovy is the choice. When dealing with value mappings we can use the ValueMappingService, but if we want to use datastores we have to add a lot of “overhead” before/after the Groovy scripts to access data from datastores in Groovy scripts.
        Since we, as CPI (interface) developers, should take care to keep the Exchange’s footprint as small as possible, I would like to access data store directly from Groovy script, instead of reading data via DataStore component, bloating up the Exchange’s properties, just to get data accessible in the Groovy scripts.

      Maybe Vadim Klimov, Eng Swee Yeoh, Ariel Bravo Ayala, Morten Wittrock or Daniel Graversen can add more/other valid points to the list.

      If I could wish for something, I would love to see a DataStoreService class analogous to the ValueMappingService class in the future.

      Thanks again for listening and taking the time to read the blog.

      Regards,
      Raffael

       

    • Me as well would love to easily access data stores via groovy.

      This makes it much easier to access and write information, where building this within the iflow would be a pain with persisting the original body, getting the stored information, writing it to a variable and then restoring the original body.

      Usecase: Get information, process, get last run body from data store, process the last run body vs. current run with groovy and exchange body to groovy result.

      My use case is set in the context of 3rd party integration.

      Maybe a reason to consider this.

      Thanks a lot!

       

      Kind regards,

      Gandalf

        • Reading and Writing.

          If I want to read something from a datastore, which I only need in a script, I have to first persist the original body message, call the Data Store, which overwrites the message body. Then I need to run the script and set a property to the message and update the body to the message, which I would like to write back.

          In the iFlow the Data Store Write takes place and then the original Message or previously set property needs to be set as body.

          Well, it works, but blows up the iflow in complexity.

  • Hi Raffael Herrmann ,

    Its really a helpful post .However for development in local need download the relevant jar files to
    access the com.sap.it.api.asdk.datastore-package which I didnt find yet. Need your assistance to know    place where from I can get the relevant jar files.

    Regards
    R Adhikary