Technical Articles
Automatic Data Store cleanup using SAP API
Introduction:
With the increasing level of interface complexity, it’s very important to log payloads at different stages of build. Payloads can be logged using groovy scripts in message processing logs (MPL), data store and persist message. SAP has added a circuit breaker to stop MPL attachments if 1 GB data is stored in 24hrs. Currently SAP doesn’t provide any UI to download Persisted messages. Logging payloads using data store is a better option, but with the increase in payload logging the memory consumption will also be high. So, data store entries should be cleared regularly, especially in production tenant. SAP has provided standard API to extract data store entries. In this blog I will explain how to delete data store payloads/entries using SAP API, groovy script and delete data store operation.
Scenario:
Retrieve the data store entries from SAP API, filter the entries to be deleted using groovy script and delete the data store entries using delete operation.
Integration Artifact Details:
Step 1:
StorePayload
Main iFlow
- Start Timer: Schedule the interface to recur daily.
- Set Header: Configure the header details as below
- Request Reply SAP API call:
https://tenantID-tmn.xxxxx.ondemand.com/api/v1/DataStores(DataStoreName=’DS_Payloads’,IntegrationFlow=”,Type=”)/Entries
- Format API Response: Refer the below xslt code to format API response
<?xml version="1.0"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:output indent="yes" method="xml" encoding="utf-8" omit-xml-declaration="yes"/> <!-- Stylesheet to remove all namespaces from a document --> <!-- template to copy elements --> <xsl:template match="*"> <xsl:element name="{local-name()}"> <xsl:apply-templates select="@* | node()"/> </xsl:element> </xsl:template> <!-- template to copy attributes --> <xsl:template match="@*"> <xsl:attribute name="{local-name()}"> <xsl:value-of select="."/> </xsl:attribute> </xsl:template> <!-- template to copy the rest of the nodes --> <xsl:template match="comment() | text() | processing-instruction()"> <xsl:copy/> </xsl:template> </xsl:stylesheet>
- Filter Entries ToBeDeleted: Refer the below groovy script to filter data store entries to be deleted and create a deletion entries list – EntryID. Here I am deleting the payloads which were saved 3 days before the current time. Payload’s retention period can be changed based on the business requirement.
import com.sap.gateway.ip.core.customdev.util.Message import java.util.HashMap import com.sap.it.api.ITApiFactory import com.sap.it.api.mapping.ValueMappingApi import groovy.xml.MarkupBuilder import groovy.time.*; def Message processData(Message message) { def a = ITApiFactory.getApi(ValueMappingApi.class, null); def body = message.getBody(java.lang.String) as String; def Test = new XmlParser().parseText(body); def writer = new StringWriter(); def builder = new MarkupBuilder(writer); def ToBeDeleted = [] def ID = [] def TimeDuration = [] String Create = ''; builder.Target { 'Record' { Test.entry.each { it.properties.each{ def mappedValue = "${it.Id[0].text()}"; if ( mappedValue.equals("") || mappedValue ==null ) { ToBeDeleted.push("No") } else { Create = "${it.CreatedAt[0].text()}" def CreatedAt = Create.substring(0,19) CreatedAt = CreatedAt.replace("T", " ") def Newdate = Date.parse("yyyy-MM-dd HH:mm:ss",CreatedAt) def Currentdate = new Date() TimeDuration duration = TimeCategory.minus(Currentdate, Newdate); TimeDuration.push(duration) if(duration.getDays() > 3){ ToBeDeleted.push("Yes") ID.push(mappedValue) } } } } } 'Id'(ID) } // Generate output message.setBody(writer.toString()) message.setProperty("ToBeDeleted", ToBeDeleted) message.setProperty("EntryID", ID) message.setProperty("TimeDuration", TimeDuration) return message; }
- Looping Process Call: Process call to delete messages. This process repeats for each entry in the deletion list.
- Set DelEntryID: Refer the below groovy script to set delete EntryID and set indicator if any valid entries are present to delete.
import com.sap.gateway.ip.core.customdev.util.Message; import java.util.HashMap; import com.google.gson.Gson; import com.google.gson.GsonBuilder; import groovy.json.*; def Message processData(Message message) { def body = message.getBody(java.lang.String) as String def DelEntryID = message.getProperties().get("EntryID")[0] def ToBeDeleted if ( DelEntryID.equals("") || DelEntryID ==null ) { ToBeDeleted= 'No' } else{ ToBeDeleted= 'Yes' } message.setProperty("DelEntryID", DelEntryID) message.setProperty("ToBeDeleted", ToBeDeleted) return message; }
- Router: Decides if there is any valid entry to delete. If not ends the call.
- Delete: Deletes data store entry:
- Remove Deleted Entry Id: Refer the below groovy script to remove the deleted EntryID from the deletion list.
import com.sap.gateway.ip.core.customdev.util.Message; import java.util.HashMap; import com.google.gson.Gson; import com.google.gson.GsonBuilder; import groovy.json.*; def Message processData(Message message) { def body = message.getBody(java.lang.String) as String //Remove EntryID from lookup list def packages = message.getProperties().get("EntryID").reverse() as Stack packages.pop() packages = packages.reverse() as Stack message.setProperty("EntryID", packages) return message; }
Step 3:
Configure the above iflow, schedule it as per business requirement and deploy it.
Conclusion:
We just saw how to configure the interface to automatically delete the logged payload in data store. The interface schedule can be altered based on payload retention period as per the business requirement.
References:
https://blogs.sap.com/2020/01/29/sap_cpi-demonstrating-the-datastoreoperations-feature/
Great blog Arundhati..Nicely captured all processing steps within Iflow.
Regards
Deepak
Thank you Deepak 🙂
Fantastic, thank you!
Sorry for late response. 🙂
Thank you Wilson!!
Very well explained. Thank you 🙂