Skip to Content
Technical Articles
Author's profile photo Priyanka Chakraborti

Cloud Integration with Commerce Azure Blob Storage using REST API – Part 2

Introduction:

This blog post is the continuation of my previous blog post: Part 1

Scenario:

Get files from Commerce Azure Blob storage.

Prerequisite Setup:

  1. Set up an Integration suite trial. Help Link: Setup.
  2. Azure Storage Explorer. Help Link: Azure Storage Explorer.

Design Solution in Cloud Integration:

Integration%20Flow

Integration Flow

Step 1:

Configure timer to pick up files at a regular interval.

Step 2:

Use Content Modifier to set properties and header for version, as shown below.

Set%20Version%20Header

Set Version Header

Set%20Properties

Set Properties

Step 3:

Use Groovy Script to set up the headers for calling REST API to get the name of files starting with ‘order’.

import com.sap.gateway.ip.core.customdev.util.Message
import com.sap.it.api.ITApiFactory
import com.sap.it.api.securestore.SecureStoreService
import com.sap.it.api.securestore.UserCredential
import com.sap.it.api.securestore.exception.SecureStoreException
import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec
import java.security.InvalidKeyException
def Message processData(Message message) 
{
    //Set Current Time
    TimeZone.setDefault(TimeZone.getTimeZone('GMT'))
    def now = new Date().format("EEE, dd MMM yyyy HH:mm:ss") + " GMT"
    //Set header for datetime
    message.setHeader("x-ms-date", now)
    //Get container name
    String container = message.getProperties().get("container")
    //Get folder path
    String folderPath = message.getProperties().get("folderPath")
    //Get File Name Prefix
    String filenamePrefix = message.getProperties().get("filenamePrefix")
    //Set Prefix name
    String prefix = folderPath + '/' + filenamePrefix + '_'
    message.setProperty("prefix", prefix)
    //Get Account Name
    String account = message.getProperties().get("accountName")
    // Set canonicalized Resource
    String canonicalizedResource = '/'+ account + '/'+ container + '\n'+ 'comp:list' + '\n' + 'prefix:' + prefix + '\n'+ 'restype:container'
    // set verb as requested method
    String verb = 'GET'
    //Get version
    String version = message.getHeaders().get("x-ms-version")
    //Set Signature String
    String StringToSign = verb +'\n'+'\n'+'\n'+ '\n' +'\n'+ '\n' + '\n' +'\n'+'\n'+'\n'+'\n'+'\n'+'x-ms-date:'+ now +'\n' +'x-ms-version:' + version + '\n'+ canonicalizedResource
    //Get Account Key from Secure Parameter
    String accountKeyAlias = message.getProperties().get("accountKeyAlias")
    def accountKey = getAccountKey(accountKeyAlias)
    // Decode Account Key
    def decodedKey = accountKey.decodeBase64()
    //Get Hash Value
    String hash = hmac_sha256(decodedKey, StringToSign)
    //Set Authorization header
    String auth = 'SharedKey'+ ' ' + account + ':' + hash
    message.setHeader("Authorization", auth)
    return message
}

String getAccountKey(String accountKeyAlias)
{
   def secureStorageService =  ITApiFactory.getService(SecureStoreService.class, null)
    try
    {
        def secureParameter = secureStorageService.getUserCredential(accountKeyAlias)
        return secureParameter.getPassword().toString()
    } 
    catch(Exception e)
    {
        throw new SecureStoreException("Secure Parameter not available")
    }
}

String hmac_sha256(byte[] secretKey, String data) 
{
    try 
    {
        Mac sha256_HMAC = Mac.getInstance("HmacSHA256")
        SecretKeySpec secret_key = new SecretKeySpec(secretKey, "HmacSHA256")
        sha256_HMAC.init(secret_key)
        byte[] digest = sha256_HMAC.doFinal(data.getBytes())
        return digest.encodeBase64()

    } catch (InvalidKeyException e) 
    {
        throw new RuntimeException("Invalid key exception while converting to HMac SHA256")
    }
}

Step 4:

Use Request-Reply step to get the list of files. Configure HTTP receiver channel as below.

HTTP%20Receiver%20Channel

HTTP Receiver Channel

Step 5:

Use XSLT mapping to sort the list of filenames in ascending order based on last modified time.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="yes"/>
  <xsl:template match="EnumerationResults/Blobs">
    <xsl:copy>
      <xsl:apply-templates select="Blob">
        <xsl:sort select="Properties/Last-Modified"/>
      </xsl:apply-templates>
    </xsl:copy>
  </xsl:template>
  <xsl:template match="@* | node()">
    <xsl:copy>
      <xsl:apply-templates select="@* | node()"/>
    </xsl:copy>
  </xsl:template>
</xsl:stylesheet>

Step 6:

Use General Splitter step to retrieve each file entry as shown below.

General%20Splitter

General Splitter

 

Step 7:

Use Content Modifier step to save last modified time and filename as properties.

Content%20Modifier%20-%20Properties

Content Modifier – Properties

Step 8:

Use Groovy Script to check whether the file is new based on the comparison between last poll time and last modified time.

import com.sap.gateway.ip.core.customdev.util.Message
import java.text.SimpleDateFormat
def Message processData(Message message) {
    SimpleDateFormat customFormat = new SimpleDateFormat("E, dd MMM yyyy HH:mm:ss z");
    def lastmodified = message.getProperties().get("lastmodified")
    def lastpoll = message.getProperties().get("lastpoll")
    def dateTime1 = customFormat.parse(lastmodified)
    def dateTime2 = customFormat.parse(lastpoll)
    def getFile = "No"

   if (dateTime1 > dateTime2)
     {
         getFile = "Yes"
     }
    message.setProperty("getFile", getFile)
    return message
    }

 

Step 9:

Use Router step as shown below.

Router

Router

Step 10:

Use Groovy Script to set up the headers for calling REST API to get the specific file.

import com.sap.gateway.ip.core.customdev.util.Message
import com.sap.it.api.ITApiFactory
import com.sap.it.api.securestore.SecureStoreService
import com.sap.it.api.securestore.UserCredential
import com.sap.it.api.securestore.exception.SecureStoreException
import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec
import java.security.InvalidKeyException
def Message processData(Message message) 
{
    //Set Current Time
    TimeZone.setDefault(TimeZone.getTimeZone('GMT'))
    def now = new Date().format("EEE, dd MMM yyyy HH:mm:ss") + " GMT"
    //Set header for datetime
    message.setHeader("x-ms-date", now)
    // Get file name
    String filename = message.getProperties().get("fileName")
    //Get container name
    String container = message.getProperties().get("container")
    //Get Account Name
    String account = message.getProperties().get("accountName")
    // Set canonicalized Resource
    String canonicalizedResource = '/'+ account + '/'+ container +'/' + filename
    // set verb as requested method
    String verb = 'GET'
    //Get version
    String version = message.getHeaders().get("x-ms-version")
    //Set Signature String
    String StringToSign = verb +'\n'+'\n'+'\n'+ '\n' +'\n'+ '\n' + '\n' +'\n'+'\n'+'\n'+'\n'+'\n'+'x-ms-date:'+ now +'\n' +'x-ms-version:' + version + '\n'+ canonicalizedResource
    //Get Account Key from Secure Parameter
    String accountKeyAlias = message.getProperties().get("accountKeyAlias")
    def accountKey = getAccountKey(accountKeyAlias)
    // Decode Account Key
    def decodedKey = accountKey.decodeBase64()
    //Get Hash Value
    String hash = hmac_sha256(decodedKey, StringToSign)
    //Set Authorization header
    String auth = 'SharedKey'+ ' ' + account + ':' + hash
    message.setHeader("Authorization", auth)
    return message
}

String getAccountKey(String accountKeyAlias)
{
   def secureStorageService =  ITApiFactory.getService(SecureStoreService.class, null)
    try
    {
        def secureParameter = secureStorageService.getUserCredential(accountKeyAlias)
        return secureParameter.getPassword().toString()
    } 
    catch(Exception e)
    {
        throw new SecureStoreException("Secure Parameter not available")
    }
}

String hmac_sha256(byte[] secretKey, String data) 
{
    try 
    {
        Mac sha256_HMAC = Mac.getInstance("HmacSHA256")
        SecretKeySpec secret_key = new SecretKeySpec(secretKey, "HmacSHA256")
        sha256_HMAC.init(secret_key)
        byte[] digest = sha256_HMAC.doFinal(data.getBytes())
        return digest.encodeBase64()

    } catch (InvalidKeyException e) 
    {
        throw new RuntimeException("Invalid key exception while converting to HMac SHA256")
    }
}

Step 11:

Use Request-Reply step to get the file. Configure HTTP receiver channel as below.

HTTP%20Receiver%20Channel

HTTP Receiver Channel

Step 12:

Use Write Variables to save the last poll time.

Write%20Variables

Write Variables

 

Conclusion:

Using the above explained steps, the new files can be retrieved based on last modified datetime. Please note, to avoid getting a large list of filenames during 1st call, an archiving strategy can be devised at Azure Blob storage end to move the old files to archive directory after N days.

Thank you for reading this blog post. Please feel free to share your feedback or thoughts or ask questions in the Q&A tag below.

QA link

Reference Links:

 

Regards,

Priyanka Chakraborti

 

   Previous – Part 1

 

 

Assigned Tags

      9 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Ganesh Chandorkar
      Ganesh Chandorkar

      Does it mean, we can use the same approach to connect with any azure blob storage?

      Author's profile photo Priyanka Chakraborti
      Priyanka Chakraborti
      Blog Post Author

      Hi Ganesh,

      Yes, it should work.

      Regards,

      Priyanka

      Author's profile photo Saurabh Kumar
      Saurabh Kumar

      Thanks Priyanka,

      It helped me a lot.

      Clearly explained with attachment .

      Author's profile photo Priyanka Chakraborti
      Priyanka Chakraborti
      Blog Post Author

      Hi Saurabh,

      I am glad that it helped,

      Regards,

      Priyanka

      Author's profile photo Saurabh Kumar
      Saurabh Kumar

      Hi Priyanka,

      One quick question :

      I am sending files to SFTP ,Since files are being picked based on last poll.

      Which retry mechanism would you suggest if SFTP server fails to respond. As files will not get picked in the next poll if it is not modified.

       

      Thanks and Regards

      Saurabh Kumar

      Author's profile photo Priyanka Chakraborti
      Priyanka Chakraborti
      Blog Post Author

      Hi Saurabh,

      You can set the variable value to the last successful one in case of failure using exception subprocess block.

      Regards,

      Priyanka

       

       

      Author's profile photo Saurabh Kumar
      Saurabh Kumar

      Hi Priyanka,

       

      Yeah thank you we can do this ,but I have a scenario where multiple files are having same modified date and if among them one is successful and later SFTP gets down , variable is updated with successful one and in next run those having same date will be missed.

      Thanks and regards

      Saurabh

      Author's profile photo Priyanka Chakraborti
      Priyanka Chakraborti
      Blog Post Author

      Hi Saurabh,

      Based on your business requirement, you have to modify the logic. If the modified timestamp is not unique, use combination of modified timestamp and filename as criteria for picking up files. Similarly, if you don't expect old files to be modified, use filename only as unique identifier.

       

      Regards,

      Priyanka

      Author's profile photo Saurabh Kumar
      Saurabh Kumar

      Hi Priyanka,

      Business requirements is picking the files based on last modified ,which is working fine.

      But when considering the target failure scenario, Thought of above given resolution and it conflicts for the files having same modified date.