Automatic Data Store cleanup using SAP API

Introduction:

With the increasing level of interface complexity, it’s very important to log payloads at different stages of build. Payloads can be logged using groovy scripts in message processing logs (MPL), data store and persist message. SAP has added a circuit breaker to stop MPL attachments if 1 GB data is stored in 24hrs. Currently SAP doesn’t provide any UI to download Persisted messages. Logging payloads using data store is a better option, but with the increase in payload logging the memory consumption will also be high. So, data store entries should be cleared regularly, especially in production tenant. SAP has provided standard API to extract data store entries. In this blog I will explain how to delete data store payloads/entries using SAP API, groovy script and delete data store operation.

Scenario:

Retrieve the data store entries from SAP API, filter the entries to be deleted using groovy script and delete the data store entries using delete operation.

Integration Artifact Details:

Step 1:

Log/Store payloads using write data store operation in any required integration process. I will not cover many details on this topic. In the current scenario I am storing the payloads in data store: DS_Payloads:

StorePayload

Step 2: Create an iFlow:

Main iFlow

1. Start Timer: Schedule the interface to recur daily.
2. Set Header: Configure the header details as below
   
3. Request Reply SAP API call:
   https://tenantID-tmn.xxxxx.ondemand.com/api/v1/DataStores(DataStoreName=’DS_Payloads’,IntegrationFlow=”,Type=”)/Entries
   
4. Format API Response: Refer the below xslt code to format API response

   <?xml version="1.0"?>
   <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
   
   	<xsl:output indent="yes" method="xml" encoding="utf-8" omit-xml-declaration="yes"/>
   
   	<!-- Stylesheet to remove all namespaces from a document -->
   	<!-- template to copy elements -->
   	<xsl:template match="*">
   		<xsl:element name="{local-name()}">
   			<xsl:apply-templates select="@* | node()"/>
   		</xsl:element>
   	</xsl:template>
   
   	<!-- template to copy attributes -->
   	<xsl:template match="@*">
   		<xsl:attribute name="{local-name()}">
   			<xsl:value-of select="."/>
   		</xsl:attribute>
   	</xsl:template>
   
   	<!-- template to copy the rest of the nodes -->
   	<xsl:template match="comment() | text() | processing-instruction()">
   		<xsl:copy/>
   	</xsl:template>
   </xsl:stylesheet>​

5. Filter Entries ToBeDeleted: Refer the below groovy script to filter data store entries to be deleted and create a deletion entries list – EntryID. Here I am deleting the payloads which were saved 3 days before the current time. Payload’s retention period can be changed based on the business requirement.

   import com.sap.gateway.ip.core.customdev.util.Message
   import java.util.HashMap
   import com.sap.it.api.ITApiFactory
   import com.sap.it.api.mapping.ValueMappingApi
   import groovy.xml.MarkupBuilder
   import groovy.time.*;
   
   def Message processData(Message message)
   {
       def a = ITApiFactory.getApi(ValueMappingApi.class, null);
       def body = message.getBody(java.lang.String) as String;
       def Test = new XmlParser().parseText(body);
       def writer = new StringWriter();
       def builder = new MarkupBuilder(writer);
       
       def ToBeDeleted = []
       def ID = []
       def TimeDuration = []
       String Create = '';
       
       builder.Target 
       {
           'Record' 
           {
   
              Test.entry.each { 
                 
   
               it.properties.each{
   
                def mappedValue = "${it.Id[0].text()}";
                
           
                if ( mappedValue.equals("") || mappedValue ==null )
                   {
   
                         ToBeDeleted.push("No")
                   }
                else
                   {
          
                           Create = "${it.CreatedAt[0].text()}"
                           def CreatedAt = Create.substring(0,19)
                           CreatedAt = CreatedAt.replace("T", " ")
                           def Newdate = Date.parse("yyyy-MM-dd HH:mm:ss",CreatedAt)
           
                           def Currentdate = new Date()
   
                           TimeDuration duration = TimeCategory.minus(Currentdate, Newdate);
                                                       TimeDuration.push(duration)
                           if(duration.getDays() > 3){
                               ToBeDeleted.push("Yes")
                               ID.push(mappedValue)
                            }
   
                   } 
               }    
               }
   			}
   			 'Id'(ID)
   		}
   	// Generate output
       message.setBody(writer.toString())
       message.setProperty("ToBeDeleted", ToBeDeleted)
       message.setProperty("EntryID", ID)
       message.setProperty("TimeDuration", TimeDuration)
       
       return message;	
   
   }​

6. Looping Process Call:  Process call to delete messages. This process repeats for each entry in the deletion list.
   

   Delete Message Logs: Sub process to delete data store entries
   

7. Set DelEntryID: Refer the below groovy script to set delete EntryID and set indicator if any valid entries are present to delete.

   import com.sap.gateway.ip.core.customdev.util.Message;
   import java.util.HashMap;
   import com.google.gson.Gson;
   import com.google.gson.GsonBuilder;
   
   import groovy.json.*;
   
   def Message processData(Message message) {
     
       def body = message.getBody(java.lang.String) as String
   
       def DelEntryID = message.getProperties().get("EntryID")[0]
       def ToBeDeleted
               
                if ( DelEntryID.equals("") || DelEntryID ==null )
                   {
   
                         ToBeDeleted= 'No'
                   }
                   else{
                       ToBeDeleted= 'Yes'
                   }
   
       message.setProperty("DelEntryID", DelEntryID)
       message.setProperty("ToBeDeleted", ToBeDeleted)
   
       return message;
   }​

8. Router: Decides if there is any valid entry to delete. If not ends the call.
   
9. Delete: Deletes data store entry:
   
10. Remove Deleted Entry Id: Refer the below groovy script to remove the deleted EntryID from the deletion list.

    import com.sap.gateway.ip.core.customdev.util.Message;
    import java.util.HashMap;
    import com.google.gson.Gson;
    import com.google.gson.GsonBuilder;
    
    import groovy.json.*;
    
    def Message processData(Message message) {
      
        def body = message.getBody(java.lang.String) as String
    
        //Remove EntryID from lookup list
        def packages = message.getProperties().get("EntryID").reverse() as Stack
        packages.pop()
        packages = packages.reverse() as Stack
        message.setProperty("EntryID", packages)
    
        return message;
    }​

Step 3:
Configure the above iflow, schedule it as per business requirement and deploy it.

Conclusion:
We just saw how to configure the interface to automatically delete the logged payload in data store. The interface schedule can be altered based on payload retention period as per the business requirement.

References:

https://blogs.sap.com/2020/01/29/sap_cpi-demonstrating-the-datastoreoperations-feature/

https://help.sap.com/viewer/368c481cd6954bdfa5d0435479fd4eaf/Cloud/en-US/8c35f3fa3b9c42c5b810332eccbc5a2f.html