Introduction:

This blog post is the continuation of my previous blog post: Part 1

Scenario:

Get files from Commerce Azure Blob storage.

Prerequisite Setup:

1. Set up an Integration suite trial. Help Link: Setup.
2. Azure Storage Explorer. Help Link: Azure Storage Explorer.

Design Solution in Cloud Integration:

Integration Flow

Step 1:

Configure timer to pick up files at a regular interval.

Step 2:

Use Content Modifier to set properties and header for version, as shown below.

Set Version Header

Set Properties

Step 3:

Use Groovy Script to set up the headers for calling REST API to get the name of files starting with ‘order’.

import com.sap.gateway.ip.core.customdev.util.Message
import com.sap.it.api.ITApiFactory
import com.sap.it.api.securestore.SecureStoreService
import com.sap.it.api.securestore.UserCredential
import com.sap.it.api.securestore.exception.SecureStoreException
import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec
import java.security.InvalidKeyException
def Message processData(Message message) 
{
    //Set Current Time
    TimeZone.setDefault(TimeZone.getTimeZone('GMT'))
    def now = new Date().format("EEE, dd MMM yyyy HH:mm:ss") + " GMT"
    //Set header for datetime
    message.setHeader("x-ms-date", now)
    //Get container name
    String container = message.getProperties().get("container")
    //Get folder path
    String folderPath = message.getProperties().get("folderPath")
    //Get File Name Prefix
    String filenamePrefix = message.getProperties().get("filenamePrefix")
    //Set Prefix name
    String prefix = folderPath + '/' + filenamePrefix + '_'
    message.setProperty("prefix", prefix)
    //Get Account Name
    String account = message.getProperties().get("accountName")
    // Set canonicalized Resource
    String canonicalizedResource = '/'+ account + '/'+ container + '\n'+ 'comp:list' + '\n' + 'prefix:' + prefix + '\n'+ 'restype:container'
    // set verb as requested method
    String verb = 'GET'
    //Get version
    String version = message.getHeaders().get("x-ms-version")
    //Set Signature String
    String StringToSign = verb +'\n'+'\n'+'\n'+ '\n' +'\n'+ '\n' + '\n' +'\n'+'\n'+'\n'+'\n'+'\n'+'x-ms-date:'+ now +'\n' +'x-ms-version:' + version + '\n'+ canonicalizedResource
    //Get Account Key from Secure Parameter
    String accountKeyAlias = message.getProperties().get("accountKeyAlias")
    def accountKey = getAccountKey(accountKeyAlias)
    // Decode Account Key
    def decodedKey = accountKey.decodeBase64()
    //Get Hash Value
    String hash = hmac_sha256(decodedKey, StringToSign)
    //Set Authorization header
    String auth = 'SharedKey'+ ' ' + account + ':' + hash
    message.setHeader("Authorization", auth)
    return message
}

String getAccountKey(String accountKeyAlias)
{
   def secureStorageService =  ITApiFactory.getService(SecureStoreService.class, null)
    try
    {
        def secureParameter = secureStorageService.getUserCredential(accountKeyAlias)
        return secureParameter.getPassword().toString()
    } 
    catch(Exception e)
    {
        throw new SecureStoreException("Secure Parameter not available")
    }
}

String hmac_sha256(byte[] secretKey, String data) 
{
    try 
    {
        Mac sha256_HMAC = Mac.getInstance("HmacSHA256")
        SecretKeySpec secret_key = new SecretKeySpec(secretKey, "HmacSHA256")
        sha256_HMAC.init(secret_key)
        byte[] digest = sha256_HMAC.doFinal(data.getBytes())
        return digest.encodeBase64()

    } catch (InvalidKeyException e) 
    {
        throw new RuntimeException("Invalid key exception while converting to HMac SHA256")
    }
}

Step 4:

Use Request-Reply step to get the list of files. Configure HTTP receiver channel as below.

HTTP Receiver Channel

Step 5:

Use XSLT mapping to sort the list of filenames in ascending order based on last modified time.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="yes"/>
  <xsl:template match="EnumerationResults/Blobs">
    <xsl:copy>
      <xsl:apply-templates select="Blob">
        <xsl:sort select="Properties/Last-Modified"/>
      </xsl:apply-templates>
    </xsl:copy>
  </xsl:template>
  <xsl:template match="@* | node()">
    <xsl:copy>
      <xsl:apply-templates select="@* | node()"/>
    </xsl:copy>
  </xsl:template>
</xsl:stylesheet>

Step 6:

Use General Splitter step to retrieve each file entry as shown below.

General Splitter

Step 7:

Use Content Modifier step to save last modified time and filename as properties.

Content Modifier – Properties

Step 8:

Use Groovy Script to check whether the file is new based on the comparison between last poll time and last modified time.

import com.sap.gateway.ip.core.customdev.util.Message
import java.text.SimpleDateFormat
def Message processData(Message message) {
    SimpleDateFormat customFormat = new SimpleDateFormat("E, dd MMM yyyy HH:mm:ss z");
    def lastmodified = message.getProperties().get("lastmodified")
    def lastpoll = message.getProperties().get("lastpoll")
    def dateTime1 = customFormat.parse(lastmodified)
    def dateTime2 = customFormat.parse(lastpoll)
    def getFile = "No"

   if (dateTime1 > dateTime2)
     {
         getFile = "Yes"
     }
    message.setProperty("getFile", getFile)
    return message
    }

Step 9:

Use Router step as shown below.

Router

Step 10:

Use Groovy Script to set up the headers for calling REST API to get the specific file.

import com.sap.gateway.ip.core.customdev.util.Message
import com.sap.it.api.ITApiFactory
import com.sap.it.api.securestore.SecureStoreService
import com.sap.it.api.securestore.UserCredential
import com.sap.it.api.securestore.exception.SecureStoreException
import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec
import java.security.InvalidKeyException
def Message processData(Message message) 
{
    //Set Current Time
    TimeZone.setDefault(TimeZone.getTimeZone('GMT'))
    def now = new Date().format("EEE, dd MMM yyyy HH:mm:ss") + " GMT"
    //Set header for datetime
    message.setHeader("x-ms-date", now)
    // Get file name
    String filename = message.getProperties().get("fileName")
    //Get container name
    String container = message.getProperties().get("container")
    //Get Account Name
    String account = message.getProperties().get("accountName")
    // Set canonicalized Resource
    String canonicalizedResource = '/'+ account + '/'+ container +'/' + filename
    // set verb as requested method
    String verb = 'GET'
    //Get version
    String version = message.getHeaders().get("x-ms-version")
    //Set Signature String
    String StringToSign = verb +'\n'+'\n'+'\n'+ '\n' +'\n'+ '\n' + '\n' +'\n'+'\n'+'\n'+'\n'+'\n'+'x-ms-date:'+ now +'\n' +'x-ms-version:' + version + '\n'+ canonicalizedResource
    //Get Account Key from Secure Parameter
    String accountKeyAlias = message.getProperties().get("accountKeyAlias")
    def accountKey = getAccountKey(accountKeyAlias)
    // Decode Account Key
    def decodedKey = accountKey.decodeBase64()
    //Get Hash Value
    String hash = hmac_sha256(decodedKey, StringToSign)
    //Set Authorization header
    String auth = 'SharedKey'+ ' ' + account + ':' + hash
    message.setHeader("Authorization", auth)
    return message
}

String getAccountKey(String accountKeyAlias)
{
   def secureStorageService =  ITApiFactory.getService(SecureStoreService.class, null)
    try
    {
        def secureParameter = secureStorageService.getUserCredential(accountKeyAlias)
        return secureParameter.getPassword().toString()
    } 
    catch(Exception e)
    {
        throw new SecureStoreException("Secure Parameter not available")
    }
}

String hmac_sha256(byte[] secretKey, String data) 
{
    try 
    {
        Mac sha256_HMAC = Mac.getInstance("HmacSHA256")
        SecretKeySpec secret_key = new SecretKeySpec(secretKey, "HmacSHA256")
        sha256_HMAC.init(secret_key)
        byte[] digest = sha256_HMAC.doFinal(data.getBytes())
        return digest.encodeBase64()

    } catch (InvalidKeyException e) 
    {
        throw new RuntimeException("Invalid key exception while converting to HMac SHA256")
    }
}

Step 11:

Use Request-Reply step to get the file. Configure HTTP receiver channel as below.

HTTP Receiver Channel

Step 12:

Use Write Variables to save the last poll time.

Write Variables

Conclusion:

Using the above explained steps, the new files can be retrieved based on last modified datetime. Please note, to avoid getting a large list of filenames during 1st call, an archiving strategy can be devised at Azure Blob storage end to move the old files to archive directory after N days.

Thank you for reading this blog post. Please feel free to share your feedback or thoughts or ask questions in the Q&A tag below.

QA link

Reference Links:

• Get a list of files: List Containers
• Get file: Get Blob
• Script Logic: Authorize with shared Key

Regards,

Priyanka Chakraborti

   Previous – Part 1