Integrating Amazon Simple Storage Service (Amazon S3) and SAP ECC v6.0 via SAP PI v7.5 using AWS Signature v5 and Signing Algorithm (HMAC-SHA256)

[former_member607993]

It is not at all surprising that more than a million active customers, from Airbnb to GE, use AWS Cloud solutions to deliver flexibility, fast, scalability, reliability and inexpensive data storage infrastructure. Companies like Netflix, Airbnb, Disney, NASA, BWM and many more are all using AWS to make business decisions in real time. These companies use data collection systems for nearly everything from business analytics to near-real-time operations, to executive reporting, computing and storage.

As part of AWS Storage, Amazon Simple Storage Service (S3) provides scalable object storage for data backup, archival and analytics and used to store and retrieve any amount of data, at any time, from anywhere on the web.

Benefits

Amazon Simple Storage Service (S3) is low cost, 99.99% availability, secure by default, transfer a large amount of data and easy to handle.

Conceptualizes

Amazon Simple Storage Service (S3) Conceptualizes of buckets, objects, regions, keys and Amazon S3 data consistency model.

Data is stored as objects within resources called “buckets”, and a single object can be up to 5 terabytes in size. S3 features include capabilities to append metadata tags to objects, move and store data across the S3 Storage Classes, configure and enforce data access controls, secure data against unauthorized users, run big data analytics, and monitor data at the object and bucket levels.

Amazon S3 is designed for 99.999999999% (11 9's) of durability, and stores data for millions of applications for companies all around the world.

Integration:

Amazon S3 supports the 'REST API'. Support for SOAP over HTTP is deprecated, but it is still available over HTTPS. However, new Amazon S3 features will not be supported for SOAP. Amazon recommends that you use either the REST API or the AWS SDKs.



Recently I had developed a unidirectional interface integrating Amazon Simple Storage Service (Amazon S3) and SAP ECC 6.0 via SAP PI 7.5 using AWS Signature v5 and Signing Algorithm (HMAC-SHA256). This integration scenario defines IDOC to REST by means iDocument-CSV conversion using REST Adapter.

In the server side i.e. Amazon Simple Storage Service (Amazon S3), file should be delivered in the form of comma separated value(.csv) using AWS Signature version 5 and Signing Algorithm (HMAC-SHA256). Amazon has provided authentication methods and signing requests to calculate the Signature process. Below are the methods to generate Header values:

1. Generate Signature - Authorization


2. Generate Content Hash - X-Amz-Content-Sha256


3. Generate Date Stamp -X-Amz-Date


4. Generate Content Type -Content-Type


5. Dynamically generate HTTP Headers

Basically, Amazon S3 expects all the above mentioned mandatory header values to authenticate the client and it looks like below:

Common Request Headers:

The following table describes headers that can be used by various types of Amazon S3 REST requests.

Header Name	Description
Authorization	The information required for request authentication. It starts with AWS4-HMAC-SHA256 and value looks like: AWS4-HMAC-SHA256 Credential=access-key-id/date/aws-region/aws-service/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=256-bit signature expression where <date> value is specified using YYMMDD format and <aws-service> value is s3 when sending request to Amazon S3.
Content-Type	The content type of the resource in case the request content in the body. Example: text/plain
Content-MD5	The base64 encoded 128-bit MD5 digest of the message (without the headers) according to RFC 1864. This header can be used as a message integrity check to verify that the data is the same data that was originally sent.
Host	For path-style requests, the value is s3.amazonaws.com. For virtual-style requests, the value is BucketName.s3.amazonaws.com.
x-amz-content-sha256	When using signature version 4 to authenticate request, this header provides a hash of the request payload. For more information see Signature Calculations for the Authorization Header: Transferring Payload in a Single Chunk (AWS Signature Version 4). When uploading object in chunks, you set the value to STREAMING-AWS4-HMAC-SHA256-PAYLOAD to indicate that the signature covers only headers and that there is no payload.
x-amz-date	The current date and time according to the requester. Example: Wed, 01 Mar 2006 12:00:00 GMT. When you specify the Authorization header, you must specify either the x-amz-date or the Date header. If you specify both, the value specified for the x-amz-date header takes precedence.

 

Calculating a Signature:

To calculate a signature, you first need a string to sign. You then calculate a HMAC-SHA256 hash of the string to sign by using a signing key. The following diagram illustrates the process, including the various components of the string that you create for signing.

The process of putting a request in an agreed-upon form for signing is called 'canonicalization'.

Deriving the Header values Using Java (User defined functions)

1. Generate Signature - Authorization


2. Generate Content Hash - X-Amz-Content-Sha256


3. Generate Date Stamp -X-Amz-Date


4. Generate Content Type -Content-Type


5. Generate Payload


6. Dynamically generate HTTP Headers

At first to define the individual global methods for above header parameters.

In the Enterprise service repository:

Step 1: Create a new function library and specify the attributes and methods(global variables) as below:

String dateStamp ="";

String signature ="";

String method = "PUT";

String FileName="";

 

Step 2: Create a user defined function to Generate Signature

public String generateSignature(String lcl_filePath, String lcl_dateTimeStamp, String contentType, String awsAccessKeyId, String awsSecretKey, String payload, Container container) throws StreamTransformationException{

{

AbstractTrace trace = container.getTrace();

String authorization = "";

try {

String algorithm = "HmacSHA256";

Mac mac = Mac.getInstance(algorithm);

SimpleDateFormat dt1 = new SimpleDateFormat("yyyyMMdd'T'HHmmss'Z'");

Date parsedDate = dt1.parse(lcl_dateTimeStamp.toString());

SimpleDateFormat dt2 = new SimpleDateFormat("yyyyMMdd");

String lcl_dateStamp =                         dt2.format(parsedDate);

trace.addWarning("Date:" + lcl_dateStamp);

MessageDigest md = MessageDigest.getInstance("SHA-256");

byte[] hashPayloadInBytes = md.digest(payload.getBytes("UTF-8"));

StringBuilder payloadSb = new StringBuilder();

for (byte b : hashPayloadInBytes) {

payloadSb.append(String.format("%02x", b));

}

String hashPayload = payloadSb.toString();

trace.addWarning(hashPayload);

trace.addWarning(lcl_dateTimeStamp);

StringBuffer canonicalRequest = new StringBuffer();

canonicalRequest.append("PUT").append("\n");

canonicalRequest.append(lcl_filePath).append("\n\n");

canonicalRequest.append("content-type:" + contentType).append("\n");

canonicalRequest.append("host:bucketName.s3.amazonaws.com").append("\n");

canonicalRequest.append("x-amz-content-sha256:" + hashPayload).append("\n");

canonicalRequest.append("x-amz-date:" + lcl_dateTimeStamp).append("\n\n");

canonicalRequest.append("content-type;host;x-amz-content-sha256;x-amz-date").append("\n");

canonicalRequest.append(hashPayload);

byte[] hashCanonicalReqInBytes = md.digest(canonicalRequest.toString().getBytes("UTF-8"));

StringBuilder hashCanonicalSb = new StringBuilder();

for (byte b : hashCanonicalReqInBytes) {

hashCanonicalSb.append(String.format("%02x", b));

}

trace.addWarning(hashCanonicalSb.toString());

StringBuffer sringToSignSb = new StringBuffer();

sringToSignSb.append("AWS4-HMAC-SHA256").append("\n");

sringToSignSb.append(lcl_dateTimeStamp).append("\n");

sringToSignSb.append(lcl_dateStamp + "/" + "ap-south-1/s3/aws4_request").append("\n");

sringToSignSb.append(hashCanonicalSb.toString());

String stringToSign = sringToSignSb.toString();

System.out.println(stringToSign);

trace.addWarning(stringToSign);

byte[] kSecret = ("AWS4" + awsSecretKey).getBytes("UTF-8");

mac.init(new SecretKeySpec(kSecret, algorithm));

byte[] kDate = mac.doFinal(lcl_dateStamp.getBytes("UTF-8"));

mac.init(new SecretKeySpec(kDate, algorithm));

byte[] kRegion = mac.doFinal("ap-south-1".getBytes("UTF-8"));

mac.init(new SecretKeySpec(kRegion, algorithm));

byte[] kService = mac.doFinal("s3".getBytes("UTF-8"));

mac.init(new SecretKeySpec(kService, algorithm));

byte[] kSigning = mac.doFinal("aws4_request".getBytes("UTF-8"));

mac.init(new SecretKeySpec(kSigning, algorithm));

byte[] kSignature = mac.doFinal(stringToSign.getBytes("UTF-8"));

String signature = Hex.encodeHexString(kSignature);

authorization = "AWS4-HMAC-SHA256 Credential=" + awsAccessKeyId + "/" + lcl_dateStamp

+ "/ap-south-1/s3/aws4_request,SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date,Signature="

+ signature;

} catch (Exception e) {

e.printStackTrace();

}

return authorization;

}

}

 

Step 3: Create a user defined function to Generate  DateStamp

public String generateDateTimeStamp(Container container) throws StreamTransformationException{

AbstractTrace trace = container.getTrace();

SimpleDateFormat dt1 = new SimpleDateFormat("yyyyMMdd'T'HHmmss'Z'");

dt1.setTimeZone(TimeZone.getTimeZone("GMT"));

dateStamp = dt1.format(new Date());

trace.addWarning(dateStamp);

return dateStamp;

}

 

Step 4: Create a user defined function to Generate Content Hash

public String generateContentHashing(String payload, Container container) throws StreamTransformationException{

AbstractTrace trace = container.getTrace();

StringBuilder payloadSb = new StringBuilder();

try {

MessageDigest md = MessageDigest.getInstance("SHA-256");

byte[] hashPayloadInBytes = md.digest(payload.getBytes());

for (byte b : hashPayloadInBytes) {

payloadSb.append(String.format("%02x", b));

}

} catch (NoSuchAlgorithmException e) {

e.printStackTrace();

}

trace.addWarning(payload);

return payloadSb.toString();

}

 

Step 5: Create a user defined function to Generate csv Payload

public void generatePayload(String[] SKUId, String[] EANNumber, String[] Warehouse, String[] Quantity, String[] UOM, String[] Cost, String[] Entity, String[] TransactionType, ResultList rs, Container container) throws StreamTransformationException{

AbstractTrace trace = container.getTrace();
try
{
String header = "SKUId,EANNumber,Warehouse,Quantity,UOM,Cost,Entity,TransactionType"; // field names from your first structure
String content = header + "\n";
for(int i =0; i< SKUId.length; i++)
{

// adjust the below line with your field names from first structure
content = content + SKUId[i] +"," +EANNumber[i] + ","+ Warehouse[i] + "," + Quantity[i] + "," + UOM[i] + "," + Cost[i] + "," + Entity[i] + "," + TransactionType[i] + "\n";

}
trace.addInfo(content);
rs.addValue(content);
}

//Create attachment with CSV data
catch (Exception e)
{
e.toString();
}

 

Step 6: Create a user defined function to Generate Dynamic HTTP Headers

public String HTTPHeaders(String dateStamp, String signature, String contentHash, String fileName, Container container) throws StreamTransformationException{

DynamicConfiguration conf2 = (DynamicConfiguration) container.getTransformationParameters().get(StreamTransformationConstants.DYNAMIC_CONFIGURATION);

DynamicConfigurationKey key3 = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/REST", "XAmzDate");

conf2.put(key3,dateStamp);

DynamicConfigurationKey key4 = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/REST", "Authorization");

conf2.put(key4,signature); 

DynamicConfigurationKey key5 = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/REST", "XAmzContentSha256");

conf2.put(key5,contentHash);

DynamicConfigurationKey key6 = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/REST", "FileName");

conf2.put(key6,fileName);

return "";

 

Below are the Detailed steps which explains in reading the API headers parameters i.e. the key and value and sending the file name dynamically :

In the first graphical mapper(IDOC to XML) declare the Adapter Specific Message Attributes as an User-defined Functions for deriving the file name scheme as below and map to the target field ‘fileName’ and also pass the respective file path to the target field 'filePath'.

User defined function to generate file name dynamically:

public String getASMAFileName(String CREDAT, String CRETIM, Container container) throws StreamTransformationException{

String filename = “INV_” + CREDAT + “_” + CRETIM + “.csv”;

return filename;

1^st Mapper:



In the second graphical mapper(XML to CSV) use the function libraries mentioned in above steps.

Hard coded values (like file path, content type, aws access key id and aws secret key can be moved to value mapping appropriately.

2nd Mapper:



 

In the Integration Directory:

Coming to the REST Receiver communication channel, under the REST URL tab the header variables defined in pattern variables are replaced by the respective values in the request message dynamically. For each part, I use an adapter specif attribute to read dynamically  the respective values from the Adapter specific attributes.

The URL Pattern describes the full URL produced by this channel by using named placeholders for dynamic parts. Placeholder variable names must be enclosed in curly braces.

Here value source is Adapter specific attribute which retrieves the value from an Adapter-Specific Attribute by name. The predefined names are: service, resource, id, resource2, id2, operation.



Switching to tab REST Operation . Here, I have set the HTTP Operation Source equals PUT which is a static value.



Now defining the format of the messages of the RESTful service. Switch to tab Data Format here the format of the request is JSON and response is expected to be in XML.



Finally in the HTTP Headers define the header and value pattern appropriately.These are dynamically generated using user defined functions.The header value may contain all placeholders defined on the REST URL tab.



 

Run the Scenario:

Background job is scheduled in ECC and subsequently an iDOc is generated and delivered to PI for transformation and exchange of message.

 

SAP PI Middleware Server:



Amazon S3 target server:



comma separated value(.csv) looks like:



 

Conclusion

However, in this blog, I had accomplished this integration using Java mapping as per the recommendations provided by AWS. Below are the references:

https://docs.aws.amazon.com/general/latest/gr/Welcome.html

https://docs.aws.amazon.com/index.html

Other benefits include low cost, 99.99% availability, secure by default, transfer a large amount of data and easy to handle.

 

In next blogs I will be briefing on the integration with MS Azure and Kafka Applications using external adapters.

Thank you!