Technical Articles
File Upload in AWS S3 using REST API
Introduction
Recently I have come across a new requirement where we need to replace an Oracle DB with AWS setup. So we will drop data in CSV format into AWS S3 and from there we use AWS GLUE crawlers and ETL job to transform data to parquet format and share it with Amazon Redshift Spectrum to query the data using standard SQL or Apache Hive. There are multiple AWS connectors available in market for uploading data to AWS S3 from middleware like SAP PO or SAP CPI. But here I want to share a simple UDF which can help you with uploading any data to AWS S3 from SAP PO using REST API of AWS.
Main
AWS REST API usage require following header parameters which needs to be passed every time whenever you are calling to the endpoint. For more details regarding the headers, you can read AWS documentation.
- Authorization
- x-amz-content-sha256
- content-length
- x-amz-date
- Host
- x-amz-storage-class
All these headers can only be generated by some scripts as they are dependent on runtime variables like payload, date and time etc. you can go through the java classes which can generate these headers.
AWS S3 Rest API has certain format for endpoint as well. So we will generate endpoint using the same UDF.
We have below input parameters for the UDF.
- bucketName: AWS S3 Bucket name as provided by the admin
- regionName: AWS S3 bucket region (eg. us-east-1)
- awsAccessKey: AWS IAM user Access key
- awsSecretKey: AWS IAM user Scecret Key
- objectContent: Payload which should be passed as REST Body
- path: Folder details with filename name (/Folder_Name/Filename.txt)
Deasign object Development:
Now we can start ESR design object development. Firstly, we need to create a Function library which can be used as UDF in the mapping.
Function Library
Create a function Signature with following input arguments and add below entries under Import Instructions.
com.sap.aii.mapping.api.* , com.sap.aii.mapping.lookup.* , com.sap.aii.mappingtool.tf7.rt.* , java.io.* , java.lang.reflect.* , java.util.* , java.net.* , javax.crypto.spec.SecretKeySpec , javax.crypto.Mac , java.security.MessageDigest , java.text.SimpleDateFormat , javax.crypto.* , java.nio.charset.Charset
Then Add below java code to function Signature.
URL endpointUrl;
try {
if (regionName.equals("us-east-1")) {
endpointUrl = new URL("https://s3.amazonaws.com/" + bucketName + path);
} else {
endpointUrl = new URL("https://s3-" + regionName + ".amazonaws.com/" + bucketName + path);
}
} catch (MalformedURLException e) {
throw new RuntimeException("Unable to parse service endpoint: " + e.getMessage());
}
String Content = objectContent;
// precompute hash of the body content
byte[] contentHash = AWS4SignerBase.hash(Content);
String contentHashString = BinaryUtils.toHex(contentHash);
Map<String, String> headers = new HashMap<String, String>();
headers.put("x-amz-content-sha256", contentHashString);
headers.put("content-length", "" + objectContent.getBytes(Charset.forName("UTF-8")).length);
headers.put("x-amz-storage-class", "REDUCED_REDUNDANCY");
AWS4SignerForAuthorizationHeader signer = new AWS4SignerForAuthorizationHeader(
endpointUrl, "PUT", "s3", regionName);
String authorization = signer.computeSignature(headers,
null, // no query parameters
contentHashString,
awsAccessKey,
awsSecretKey);
// express authorization for this as a header
headers.put("Authorization", authorization);
String var1 = headers.get("x-amz-content-sha256");
String var2 = headers.get("x-amz-date");
String var3 = endpointUrl.toString();
String var4 = headers.get("content-length");
DynamicConfiguration conf1 = (DynamicConfiguration) container.getTransformationParameters().get(StreamTransformationConstants.DYNAMIC_CONFIGURATION);
//DynamicConfigurationKey key1 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/XI/System/File","FileName");
DynamicConfigurationKey key2 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/XI/System/REST","Auth");
conf1.put(key2,authorization);
DynamicConfigurationKey key3 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/XI/System/REST","xamzcontent");
conf1.put(key3,var1);
DynamicConfigurationKey key4 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/XI/System/REST","xamzdate");
conf1.put(key4,var2);
DynamicConfigurationKey key5 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/XI/System/REST","endpoint");
conf1.put(key5,var3);
DynamicConfigurationKey key6 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/XI/System/REST","len");
conf1.put(key6,var4);
return(Content);
And then add below required nested classes in the Attributes and Methods area.
public abstract static class AWS4SignerBase {
/** SHA256 hash of an empty request body **/
public static final String EMPTY_BODY_SHA256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
public static final String UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD";
public static final String SCHEME = "AWS4";
public static final String ALGORITHM = "HMAC-SHA256";
public static final String TERMINATOR = "aws4_request";
/** format strings for the date/time and date stamps required during signing **/
public static final String ISO8601BasicFormat = "yyyyMMdd'T'HHmmss'Z'";
public static final String DateStringFormat = "yyyyMMdd";
protected URL endpointUrl;
protected String httpMethod;
protected String serviceName;
protected String regionName;
protected final SimpleDateFormat dateTimeFormat;
protected final SimpleDateFormat dateStampFormat;
/**
* Create a new AWS V4 signer.
*
* @param endpointUri
* The service endpoint, including the path to any resource.
* @param httpMethod
* The HTTP verb for the request, e.g. GET.
* @param serviceName
* The signing name of the service, e.g. 's3'.
* @param regionName
* The system name of the AWS region associated with the
* endpoint, e.g. us-east-1.
*/
public AWS4SignerBase(URL endpointUrl, String httpMethod,
String serviceName, String regionName) {
this.endpointUrl = endpointUrl;
this.httpMethod = httpMethod;
this.serviceName = serviceName;
this.regionName = regionName;
dateTimeFormat = new SimpleDateFormat(ISO8601BasicFormat);
dateTimeFormat.setTimeZone(new SimpleTimeZone(0, "UTC"));
dateStampFormat = new SimpleDateFormat(DateStringFormat);
dateStampFormat.setTimeZone(new SimpleTimeZone(0, "UTC"));
}
/**
* Returns the canonical collection of header names that will be included in
* the signature. For AWS4, all header names must be included in the process
* in sorted canonicalized order.
*/
protected static String getCanonicalizeHeaderNames(Map<String, String> headers) {
List<String> sortedHeaders = new ArrayList<String>();
sortedHeaders.addAll(headers.keySet());
Collections.sort(sortedHeaders, String.CASE_INSENSITIVE_ORDER);
StringBuilder buffer = new StringBuilder();
for (String header : sortedHeaders) {
if (buffer.length() > 0) buffer.append(";");
buffer.append(header.toLowerCase());
}
return buffer.toString();
}
/**
* Computes the canonical headers with values for the request. For AWS4, all
* headers must be included in the signing process.
*/
protected static String getCanonicalizedHeaderString(Map<String, String> headers) {
if ( headers == null || headers.isEmpty() ) {
return "";
}
// step1: sort the headers by case-insensitive order
List<String> sortedHeaders = new ArrayList<String>();
sortedHeaders.addAll(headers.keySet());
Collections.sort(sortedHeaders, String.CASE_INSENSITIVE_ORDER);
// step2: form the canonical header:value entries in sorted order.
// Multiple white spaces in the values should be compressed to a single
// space.
StringBuilder buffer = new StringBuilder();
for (String key : sortedHeaders) {
buffer.append(key.toLowerCase().replaceAll("\\s+", " ") + ":" + headers.get(key).replaceAll("\\s+", " "));
buffer.append("\n");
}
return buffer.toString();
}
/**
* Returns the canonical request string to go into the signer process; this
consists of several canonical sub-parts.
* @return
*/
protected static String getCanonicalRequest(URL endpoint,
String httpMethod,
String queryParameters,
String canonicalizedHeaderNames,
String canonicalizedHeaders,
String bodyHash) {
String canonicalRequest =
httpMethod + "\n" +
getCanonicalizedResourcePath(endpoint) + "\n" +
queryParameters + "\n" +
canonicalizedHeaders + "\n" +
canonicalizedHeaderNames + "\n" +
bodyHash;
return canonicalRequest;
}
/**
* Returns the canonicalized resource path for the service endpoint.
*/
protected static String getCanonicalizedResourcePath(URL endpoint) {
if ( endpoint == null ) {
return "/";
}
String path = endpoint.getPath();
if ( path == null || path.isEmpty() ) {
return "/";
}
String encodedPath = HttpUtils.urlEncode(path, true);
if (encodedPath.startsWith("/")) {
return encodedPath;
} else {
return "/".concat(encodedPath);
}
}
/**
* Examines the specified query string parameters and returns a
* canonicalized form.
* <p>
* The canonicalized query string is formed by first sorting all the query
* string parameters, then URI encoding both the key and value and then
* joining them, in order, separating key value pairs with an '&'.
*
* @param parameters
* The query string parameters to be canonicalized.
*
* @return A canonicalized form for the specified query string parameters.
*/
public static String getCanonicalizedQueryString(Map<String, String> parameters) {
if ( parameters == null || parameters.isEmpty() ) {
return "";
}
SortedMap<String, String> sorted = new TreeMap<String, String>();
Iterator<Map.Entry<String, String>> pairs = parameters.entrySet().iterator();
while (pairs.hasNext()) {
Map.Entry<String, String> pair = pairs.next();
String key = pair.getKey();
String value = pair.getValue();
sorted.put(HttpUtils.urlEncode(key, false), HttpUtils.urlEncode(value, false));
}
StringBuilder builder = new StringBuilder();
pairs = sorted.entrySet().iterator();
while (pairs.hasNext()) {
Map.Entry<String, String> pair = pairs.next();
builder.append(pair.getKey());
builder.append("=");
builder.append(pair.getValue());
if (pairs.hasNext()) {
builder.append("&");
}
}
return builder.toString();
}
protected static String getStringToSign(String scheme, String algorithm, String dateTime, String scope, String canonicalRequest) {
String stringToSign =
scheme + "-" + algorithm + "\n" +
dateTime + "\n" +
scope + "\n" +
BinaryUtils.toHex(hash(canonicalRequest));
return stringToSign;
}
/**
* Hashes the string contents (assumed to be UTF-8) using the SHA-256
* algorithm.
*/
public static byte[] hash(String text) {
try {
MessageDigest md = MessageDigest.getInstance("SHA-256");
md.update(text.getBytes("UTF-8"));
return md.digest();
} catch (Exception e) {
throw new RuntimeException("Unable to compute hash while signing request: " + e.getMessage(), e);
}
}
/**
* Hashes the byte array using the SHA-256 algorithm.
*/
public static byte[] hash(byte[] data) {
try {
MessageDigest md = MessageDigest.getInstance("SHA-256");
md.update(data);
return md.digest();
} catch (Exception e) {
throw new RuntimeException("Unable to compute hash while signing request: " + e.getMessage(), e);
}
}
protected static byte[] sign(String stringData, byte[] key, String algorithm) {
try {
byte[] data = stringData.getBytes("UTF-8");
Mac mac = Mac.getInstance(algorithm);
mac.init(new SecretKeySpec(key, algorithm));
return mac.doFinal(data);
} catch (Exception e) {
throw new RuntimeException("Unable to calculate a request signature: " + e.getMessage(), e);
}
}
}
public static class AWS4SignerForAuthorizationHeader extends AWS4SignerBase {
public AWS4SignerForAuthorizationHeader(URL endpointUrl, String httpMethod,
String serviceName, String regionName) {
super(endpointUrl, httpMethod, serviceName, regionName);
}
/**
* Computes an AWS4 signature for a request, ready for inclusion as an
* 'Authorization' header.
*
* @param headers
* The request headers; 'Host' and 'X-Amz-Date' will be added to
* this set.
* @param queryParameters
* Any query parameters that will be added to the endpoint. The
* parameters should be specified in canonical format.
* @param bodyHash
* Precomputed SHA256 hash of the request body content; this
* value should also be set as the header 'X-Amz-Content-SHA256'
* for non-streaming uploads.
* @param awsAccessKey
* The user's AWS Access Key.
* @param awsSecretKey
* The user's AWS Secret Key.
* @return The computed authorization string for the request. This value
* needs to be set as the header 'Authorization' on the subsequent
* HTTP request.
*/
public String computeSignature(Map<String, String> headers,
Map<String, String> queryParameters,
String bodyHash,
String awsAccessKey,
String awsSecretKey) {
// first get the date and time for the subsequent request, and convert
// to ISO 8601 format for use in signature generation
Date now = new Date();
String dateTimeStamp = dateTimeFormat.format(now);
// update the headers with required 'x-amz-date' and 'host' values
headers.put("x-amz-date", dateTimeStamp);
String hostHeader = endpointUrl.getHost();
int port = endpointUrl.getPort();
if ( port > -1 ) {
hostHeader.concat(":" + Integer.toString(port));
}
headers.put("Host", hostHeader);
// canonicalize the headers; we need the set of header names as well as the
// names and values to go into the signature process
String canonicalizedHeaderNames = getCanonicalizeHeaderNames(headers);
String canonicalizedHeaders = getCanonicalizedHeaderString(headers);
// if any query string parameters have been supplied, canonicalize them
String canonicalizedQueryParameters = getCanonicalizedQueryString(queryParameters);
// canonicalize the various components of the request
String canonicalRequest = getCanonicalRequest(endpointUrl, httpMethod,
canonicalizedQueryParameters, canonicalizedHeaderNames,
canonicalizedHeaders, bodyHash);
System.out.println("--------- Canonical request --------");
System.out.println(canonicalRequest);
System.out.println("------------------------------------");
// construct the string to be signed
String dateStamp = dateStampFormat.format(now);
String scope = dateStamp + "/" + regionName + "/" + serviceName + "/" + TERMINATOR;
String stringToSign = getStringToSign(SCHEME, ALGORITHM, dateTimeStamp, scope, canonicalRequest);
System.out.println("--------- String to sign -----------");
System.out.println(stringToSign);
System.out.println("------------------------------------");
// compute the signing key
byte[] kSecret = (SCHEME + awsSecretKey).getBytes();
byte[] kDate = sign(dateStamp, kSecret, "HmacSHA256");
byte[] kRegion = sign(regionName, kDate, "HmacSHA256");
byte[] kService = sign(serviceName, kRegion, "HmacSHA256");
byte[] kSigning = sign(TERMINATOR, kService, "HmacSHA256");
byte[] signature = sign(stringToSign, kSigning, "HmacSHA256");
String credentialsAuthorizationHeader =
"Credential=" + awsAccessKey + "/" + scope;
String signedHeadersAuthorizationHeader =
"SignedHeaders=" + canonicalizedHeaderNames;
String signatureAuthorizationHeader =
"Signature=" + BinaryUtils.toHex(signature);
String authorizationHeader = SCHEME + "-" + ALGORITHM + " "
+ credentialsAuthorizationHeader + ", "
+ signedHeadersAuthorizationHeader + ", "
+ signatureAuthorizationHeader;
return authorizationHeader;
}
}
public static class BinaryUtils {
/**
* Converts byte data to a Hex-encoded string.
*
* @param data
* data to hex encode.
*
* @return hex-encoded string.
*/
public static String toHex(byte[] data) {
StringBuilder sb = new StringBuilder(data.length * 2);
for (int i = 0; i < data.length; i++) {
String hex = Integer.toHexString(data[i]);
if (hex.length() == 1) {
// Append leading zero.
sb.append("0");
} else if (hex.length() == 8) {
// Remove ff prefix from negative numbers.
hex = hex.substring(6);
}
sb.append(hex);
}
return sb.toString().toLowerCase(Locale.getDefault());
}
/**
* Converts a Hex-encoded data string to the original byte data.
*
* @param hexData
* hex-encoded data to decode.
* @return decoded data from the hex string.
*/
public static byte[] fromHex(String hexData) {
byte[] result = new byte[(hexData.length() + 1) / 2];
String hexNumber = null;
int stringOffset = 0;
int byteOffset = 0;
while (stringOffset < hexData.length()) {
hexNumber = hexData.substring(stringOffset, stringOffset + 2);
stringOffset += 2;
result[byteOffset++] = (byte) Integer.parseInt(hexNumber, 16);
}
return result;
}
}
public static class HttpUtils {
/**
* Makes a http request to the specified endpoint
*/
public static String invokeHttpRequest(URL endpointUrl,
String httpMethod,
Map<String, String> headers,
String requestBody) {
HttpURLConnection connection = createHttpConnection(endpointUrl, httpMethod, headers);
try {
if ( requestBody != null ) {
DataOutputStream wr = new DataOutputStream(
connection.getOutputStream());
wr.writeBytes(requestBody);
wr.flush();
wr.close();
}
} catch (Exception e) {
throw new RuntimeException("Request failed. " + e.getMessage(), e);
}
return executeHttpRequest(connection);
}
public static String executeHttpRequest(HttpURLConnection connection) {
try {
// Get Response
InputStream is;
try {
is = connection.getInputStream();
} catch (IOException e) {
is = connection.getErrorStream();
}
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer response = new StringBuffer();
while ((line = rd.readLine()) != null) {
response.append(line);
response.append('\r');
}
rd.close();
return response.toString();
} catch (Exception e) {
throw new RuntimeException("Request failed. " + e.getMessage(), e);
} finally {
if (connection != null) {
connection.disconnect();
}
}
}
public static HttpURLConnection createHttpConnection(URL endpointUrl,
String httpMethod,
Map<String, String> headers) {
try {
HttpURLConnection connection = (HttpURLConnection) endpointUrl.openConnection();
connection.setRequestMethod(httpMethod);
if ( headers != null ) {
System.out.println("--------- Request headers ---------");
for ( String headerKey : headers.keySet() ) {
System.out.println(headerKey + ": " + headers.get(headerKey));
connection.setRequestProperty(headerKey, headers.get(headerKey));
}
}
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);
return connection;
} catch (Exception e) {
throw new RuntimeException("Cannot create connection. " + e.getMessage(), e);
}
}
public static String urlEncode(String url, boolean keepPathSlash) {
String encoded;
try {
encoded = URLEncoder.encode(url, "UTF-8");
} catch (UnsupportedEncodingException e) {
throw new RuntimeException("UTF-8 encoding is not supported.", e);
}
if ( keepPathSlash ) {
encoded = encoded.replace("%2F", "/");
}
return encoded;
}
}
Then we can import this FL to any of the Message mapping and pass required input parameters to the UDF. This UDF will automatically set required headers and endpoint URL in the ASMA of REST adapter which can be accessed at runtime.
UDF
This UDF will return whatever parameter you have passed in the input parameter “objectContent”.
So you can create a XSLT mapping to fetch only that field and pass it to REST body. (you can also use module configuration to accomplish the same.)
XSLT Mapping to convert XML to text
Configuration object Development
Now we can develop Configuration Objects in ID. I am just explaining what are required changes in REST receiver channel.
Firstly, we have to access the ASMA parameters from runtime and set them as REST headers. Please follow the below screenshot and add as per that. Also add PUT operation in REST Operation tab.
Rest Endpoint configuration
Rest Headers configuration
Rest request format configuration
Channel Module configuration
You need to set CRLF to LF converter module if you are using XSLT mapping to set payload as REST body as some CR will be added in the conversion process. If you use xml to flat file converter module, then you do not have to add that module. Other modules are simply used for logging purposes.
Test Results
You can find 200 response code in the audit logs.
Audit logs
AWS S3 Console
Conclusion
All the java codes are provided by AWS documentation. I have implemented the same with some modification. It’s a custom code so it requires much maintenance effort to fix if any issue occurs so I suggest to use AWS S3 connectors but you can always explore and it’s a bit less costly than the connectors. You can also use the same java code and make a jar file and use it in SAP CPI and use one groovy script to make a function.
Hi Ashutosh
I am doing similar thing for my project.
however ur bolg is very interesting. thanks for sharing the knowledge.
Dear Juhi,
I don’t recall specifically what udf I have written for file name but you can use your own code as per your requirements. If you only want uniqueness then you can go for adding timestamps to your filename.
Thanks,
Asutosh