Skip to Content
Technical Articles
Author's profile photo Arundathi Sarala

Automatic Data Store cleanup using SAP API

Introduction:

With the increasing level of interface complexity, it’s very important to log payloads at different stages of build. Payloads can be logged using groovy scripts in message processing logs (MPL), data store and persist message. SAP has added a circuit breaker to stop MPL attachments if 1 GB data is stored in 24hrs. Currently SAP doesn’t provide any UI to download Persisted messages. Logging payloads using data store is a better option, but with the increase in payload logging the memory consumption will also be high. So, data store entries should be cleared regularly, especially in production tenant. SAP has provided standard API to extract data store entries. In this blog I will explain how to delete data store payloads/entries using SAP API, groovy script and delete data store operation.

Scenario:

Retrieve the data store entries from SAP API, filter the entries to be deleted using groovy script and delete the data store entries using delete operation.

Integration Artifact Details:

Step 1:

Log/Store payloads using write data store operation in any required integration process. I will not cover many details on this topic. In the current scenario I am storing the payloads in data store: DS_Payloads:

StorePayload

StorePayload

Step 2: Create an iFlow:
Main%20iFlow

Main iFlow

  1. Start Timer: Schedule the interface to recur daily.
  2. Set Header: Configure the header details as below
  3. Request Reply SAP API call:
    https://tenantID-tmn.xxxxx.ondemand.com/api/v1/DataStores(DataStoreName=’DS_Payloads’,IntegrationFlow=”,Type=”)/Entries
  4. Format API Response: Refer the below xslt code to format API response
    <?xml version="1.0"?>
    <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    
    	<xsl:output indent="yes" method="xml" encoding="utf-8" omit-xml-declaration="yes"/>
    
    	<!-- Stylesheet to remove all namespaces from a document -->
    	<!-- template to copy elements -->
    	<xsl:template match="*">
    		<xsl:element name="{local-name()}">
    			<xsl:apply-templates select="@* | node()"/>
    		</xsl:element>
    	</xsl:template>
    
    	<!-- template to copy attributes -->
    	<xsl:template match="@*">
    		<xsl:attribute name="{local-name()}">
    			<xsl:value-of select="."/>
    		</xsl:attribute>
    	</xsl:template>
    
    	<!-- template to copy the rest of the nodes -->
    	<xsl:template match="comment() | text() | processing-instruction()">
    		<xsl:copy/>
    	</xsl:template>
    </xsl:stylesheet>​
  5. Filter Entries ToBeDeleted: Refer the below groovy script to filter data store entries to be deleted and create a deletion entries list – EntryID. Here I am deleting the payloads which were saved 3 days before the current time. Payload’s retention period can be changed based on the business requirement.
    import com.sap.gateway.ip.core.customdev.util.Message
    import java.util.HashMap
    import com.sap.it.api.ITApiFactory
    import com.sap.it.api.mapping.ValueMappingApi
    import groovy.xml.MarkupBuilder
    import groovy.time.*;
    
    def Message processData(Message message)
    {
        def a = ITApiFactory.getApi(ValueMappingApi.class, null);
        def body = message.getBody(java.lang.String) as String;
        def Test = new XmlParser().parseText(body);
        def writer = new StringWriter();
        def builder = new MarkupBuilder(writer);
        
        def ToBeDeleted = []
        def ID = []
        def TimeDuration = []
        String Create = '';
        
        builder.Target 
        {
            'Record' 
            {
    
               Test.entry.each { 
                  
    
                it.properties.each{
    
                 def mappedValue = "${it.Id[0].text()}";
                 
            
                 if ( mappedValue.equals("") || mappedValue ==null )
                    {
    
                          ToBeDeleted.push("No")
                    }
                 else
                    {
           
                            Create = "${it.CreatedAt[0].text()}"
                            def CreatedAt = Create.substring(0,19)
                            CreatedAt = CreatedAt.replace("T", " ")
                            def Newdate = Date.parse("yyyy-MM-dd HH:mm:ss",CreatedAt)
            
                            def Currentdate = new Date()
    
                            TimeDuration duration = TimeCategory.minus(Currentdate, Newdate);
                                                        TimeDuration.push(duration)
                            if(duration.getDays() > 3){
                                ToBeDeleted.push("Yes")
                                ID.push(mappedValue)
                             }
    
                    } 
                }    
                }
    			}
    			 'Id'(ID)
    		}
    	// Generate output
        message.setBody(writer.toString())
        message.setProperty("ToBeDeleted", ToBeDeleted)
        message.setProperty("EntryID", ID)
        message.setProperty("TimeDuration", TimeDuration)
        
        return message;	
    
    }​
  6. Looping Process Call:  Process call to delete messages. This process repeats for each entry in the deletion list.

    Delete Message Logs: Sub process to delete data store entries

  7. Set DelEntryID: Refer the below groovy script to set delete EntryID and set indicator if any valid entries are present to delete.
    import com.sap.gateway.ip.core.customdev.util.Message;
    import java.util.HashMap;
    import com.google.gson.Gson;
    import com.google.gson.GsonBuilder;
    
    import groovy.json.*;
    
    def Message processData(Message message) {
      
        def body = message.getBody(java.lang.String) as String
    
        def DelEntryID = message.getProperties().get("EntryID")[0]
        def ToBeDeleted
                
                 if ( DelEntryID.equals("") || DelEntryID ==null )
                    {
    
                          ToBeDeleted= 'No'
                    }
                    else{
                        ToBeDeleted= 'Yes'
                    }
    
        message.setProperty("DelEntryID", DelEntryID)
        message.setProperty("ToBeDeleted", ToBeDeleted)
    
        return message;
    }​
  8. Router: Decides if there is any valid entry to delete. If not ends the call.
  9. Delete: Deletes data store entry:
  10. Remove Deleted Entry Id: Refer the below groovy script to remove the deleted EntryID from the deletion list.
    import com.sap.gateway.ip.core.customdev.util.Message;
    import java.util.HashMap;
    import com.google.gson.Gson;
    import com.google.gson.GsonBuilder;
    
    import groovy.json.*;
    
    def Message processData(Message message) {
      
        def body = message.getBody(java.lang.String) as String
    
        //Remove EntryID from lookup list
        def packages = message.getProperties().get("EntryID").reverse() as Stack
        packages.pop()
        packages = packages.reverse() as Stack
        message.setProperty("EntryID", packages)
    
        return message;
    }​

Step 3:
Configure the above iflow, schedule it as per business requirement and deploy it.

Conclusion:
We just saw how to configure the interface to automatically delete the logged payload in data store. The interface schedule can be altered based on payload retention period as per the business requirement.

References:

https://blogs.sap.com/2020/01/29/sap_cpi-demonstrating-the-datastoreoperations-feature/

https://help.sap.com/viewer/368c481cd6954bdfa5d0435479fd4eaf/Cloud/en-US/8c35f3fa3b9c42c5b810332eccbc5a2f.html

 

Assigned Tags

      2 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Deepak Jaiswal
      Deepak Jaiswal

      Great blog Arundhati..Nicely captured all processing steps within Iflow.

      Regards

      Deepak

      Author's profile photo Arundathi Sarala
      Arundathi Sarala
      Blog Post Author

      Thank you Deepak 🙂