Technical Articles
How to download a file from Amazon S3 Buckets
- Update April 15, 2019:
Include a enhancement to the UDF to work with HTTP and HTTPS simultaneously;
Include a enhancement to the UDF to work with URL encoded and NOT encoded (it’s necessary comment/uncomment the line).
Today I’ll explain step-by-step how to calculate the signature to authenticate and download a file from the Amazon S3 Bucket service without third-party adapters.
Step-by-step Interface Flow
Request
In summary this interface receive download URL, Bucket, AccessKeyID, SecretAccessKey, Token and AWSRegion, a mapping calculate the signature with this information and sent to REST Adapter, the signature and anothers parameters are insert in HTTP header.
Some information for calculate the signature are provide another service, this post explain only how to calculate, but is possible implemented enhancements, for example, create a rest/soap lookup to get a Token and SecretAccessKey.
Response
The response is a file, and the REST Adapter don’t work with format different of XML or JSON, then you will need convert the file to binary, and this content are insert in a tag of XML. For this conversion I recommend a module adapter FormatConversionBean developed by @engswee.yeoh
Development and Configuration the Interface
Request mapping
For the request mapping you need create a two structures, one for inbound and another for outbound.
Inbound
Outbound
After create the structures for the request mapping (data type, message type, etc), you need create a message mapping.
Now you need to map the fields, pay attention to the next steps for configurations the roles.
Roles for Message Mapping
- Fields XAmzSecurityTokenand Url are mapped directly….
- Field XAmzSha256 is mapped with a constant value e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 (this string is a hash of a null value)
- Field XAmzDate is mapped with a CurrentDate (format yyyyMMdd’T’HHmmss’Z’) function…
- Field ContentType is mapped with a constant value application/x-www-form-urlencoded…
- Field Host is mapped with a UDF or ConstantValue.
The Host is a result of concatenation of the Bucket + “.s3.amazonaws.com“,
so you can use a ConstantValue (eu01-s3-store.s3.amazonaws.com for example), which receives the bucket and returns the Host
public String Host(String inBucket, Container container) throws StreamTransformationException{
String vServiceName = "s3";
String vS3Host = vServiceName + ".amazonaws.com";
String vHost = inBucket + "."+ vS3Host;
return vHost;
}
- Field Authorizathion is mapped with a UDF.
In field Authorization you have insert the signature calculated with the UDF below.
public String AWS4Signer(String inURL, String inDate, String inBucket, String inAccessKeyId, String inSecretAccessKey, String inToken, String inAWSRegion, Container container) throws StreamTransformationException{
String vSignedHeaders="content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token";
String vHashedCannonicalRequest="";
String vScope= inDate.substring(0,8) + "/" + inAWSRegion + "/s3/aws4_request";
String vCredential="";
String vSignature="";
String vSignatureKey="";
String vServiceName="s3";
String vS3Host=vServiceName+".amazonaws.com";
String vHost=(inBucket+"."+vS3Host).toLowerCase();
// Only URL NOT encoded
String vCanonicalURI = urlEncode(inURL.replace("https://"+vHost,"").replace("http://"+vHost,"").trim(),true);
// Only URL encoded
//String vCanonicalURI = inURL.replace("https://"+vHost,"").replace("http://"+vHost,"").trim();
String vCanonicalQueryString = "";
String vHTTPRequestMethod = "GET";
String vContentSHA256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
String vContentType = "application/x-www-form-urlencoded";
String cannonicalRequest="";
String stringToSign="";
String outAutorization="";
MappingTrace importanttrace;
importanttrace = container.getTrace();
// Canonical Request
importanttrace.addInfo("***** Canonical Request ***** ");
cannonicalRequest =
vHTTPRequestMethod.trim() +"\n"+
vCanonicalURI.trim()+"\n"+
vCanonicalQueryString.trim()+"\n"+
"content-type:"+vContentType.trim()+"\n"+
"host:" +vHost.trim()+"\n"+
"x-amz-content-sha256:"+vContentSHA256.toLowerCase().trim()+"\n"+
"x-amz-date:"+inDate.trim()+"\n"+
"x-amz-security-token:"+inToken.trim()+"\n"+
"\n"+
vSignedHeaders.trim()+"\n"+
vContentSHA256.toLowerCase().trim();
importanttrace.addInfo(cannonicalRequest);
//Canonical Request Hashed
importanttrace.addInfo("***** Hashed Canonical Request ***** ");
vHashedCannonicalRequest = hash(cannonicalRequest);
importanttrace.addInfo( vHashedCannonicalRequest);
//String to Sign
importanttrace.addInfo("***** String to Sign ***** ");
stringToSign =
"AWS4-HMAC-SHA256"+"\n"+
inDate.trim()+"\n"+
vScope.trim()+"\n"+
vHashedCannonicalRequest.trim();
importanttrace.addInfo(stringToSign);
//Signing
importanttrace.addInfo("***** Signing ***** ");
try{
byte[] kSigning = getSignatureKey(inSecretAccessKey, inDate.substring(0,8), inAWSRegion, vServiceName);
byte[] signature = HmacSHA256(stringToSign.toString(), kSigning);
vSignatureKey = bytesToHex (kSigning).toLowerCase(Locale.getDefault());
vSignature = bytesToHex (signature).toLowerCase(Locale.getDefault());
} catch (Exception e) {
throw new RuntimeException("Unable to compute hash while signing request: " + e.getMessage(), e);
}
outAutorization = "AWS4-HMAC-SHA256 Credential=" + inAccessKeyId + "/" + vScope + ", SignedHeaders=" + vSignedHeaders + ", Signature=" + vSignature;
importanttrace.addInfo("Siging Key: " + vSignatureKey);
importanttrace.addInfo("Signature: " + vSignature);
importanttrace.addInfo("Authorization: " + outAutorization);
importanttrace.addInfo(outAutorization);
return outAutorization;
}
You also need to create some methods, which will be used by UDF in signing.
public static String hash(String text) {
String hashedString = "";
try {
MessageDigest md = MessageDigest.getInstance("SHA-256");
byte[] hashedBytes = md.digest(text.getBytes("UTF-8"));
for (int i = 0; i < hashedBytes.length; i++){
hashedString += Integer.toHexString((hashedBytes[i] >> 4) & 0xf);
hashedString += Integer.toHexString(hashedBytes[i] & 0xf);
}
return hashedString;
} catch (Exception e) {
throw new RuntimeException("Unable to compute hash while signing request: " + e.getMessage(), e);
}
}
private static String bytesToHex(byte[] hashInBytes) {
StringBuilder sb = new StringBuilder();
for (byte b : hashInBytes) {
sb.append(String.format("%02x", b));
}
return sb.toString();
}
public static byte[] HmacSHA256(String data, byte[] key) throws Exception {
String algorithm="HmacSHA256";
Mac mac = Mac.getInstance(algorithm);
mac.init(new SecretKeySpec(key, algorithm));
return mac.doFinal(data.getBytes("UTF8"));
}
public static byte[] getSignatureKey(String key, String dateStamp, String regionName, String serviceName) throws Exception {
byte[] kSecret = ("AWS4" + key).getBytes("UTF8");
byte[] kDate = HmacSHA256(dateStamp, kSecret);
byte[] kRegion = HmacSHA256(regionName, kDate);
byte[] kService = HmacSHA256(serviceName, kRegion);
byte[] kSigning = HmacSHA256("aws4_request", kService);
return kSigning;
}
public static String urlEncode(String url, boolean keepPathSlash) {
String encoded;
try {
encoded = URLEncoder.encode(url, "UTF-8");
} catch (UnsupportedEncodingException e) {
throw new RuntimeException("UTF-8 encoding is not supported.", e);
}
if ( keepPathSlash ) {
encoded = encoded.replace("%2F", "/");
}
return encoded;
}
and import the packages…
com.sap.aii.mapping.api.*
com.sap.aii.mapping.lookup.*
com.sap.aii.mappingtool.tf7.rt.*
java.io.*
java.lang.reflect.*
java.util.*
java.lang.String.*
java.lang.Object
java.util.Locale
java.security.MessageDigest
java.security.NoSuchAlgorithmException
java.net.URLEncoder
java.io.UnsupportedEncodingException
java.math.BigInteger
javax.crypto.Mac
javax.crypto.spec.SecretKeySpec
After developed the UDF, its necessary configure with the inbound values.
Note: the format of CurrentDate is yyyyMMdd’T’HHmmss’Z’.
Now save and Activate the Request mapping.
Response mapping
The response mapping it’s simple and not necessary many explanation.
Configure the interface normally …
After created Request/Response Mapping, build the Operation Mapping and a Integrated Configuration normally. The Communication Channel can be of any type that is synchronous, but the Receiver must be to type rest and configured as below.
Receiver Communication Channel
Now you need configure the Receiver Channel, for this the values generates in request message mapping are storage in variables, and this variables are used in the communication channel.
Now the variables storage are used in the HTTP Header, here you configure how the canonical request is create.
It’s necessary configure the REST Operation, for this case the operation is GET.
And finally configure the module adapter FormatConversionBean to converte the file in b64string.
IMPORTANTE: The module adapter FormatConversionBean isn’t standard, and you need deploy if you have not already, for more information and download of module you can access here.
Save and active all objects, now we going test!
Running a test
Fill in all the fields correctly in the interface and call the created service, the response should be the file in b64string format.
If you analyze the log of request messages, the parameters are populated in the HTTP header and communication has succeeded (HTTP 200)
and the response (the file) is converted to b64 string.
That’s all! I hope I have collaborated and I am waiting for your feedback on this post.
References
How to Calculate AWS Signature Version 4
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
Module Adapter FormatConversionBean
https://blogs.sap.com/2015/03/25/formatconversionbean-one-bean-to-rule-them-all/
PI REST Adapter – Define custom http header elements
https://blogs.sap.com/2015/04/14/pi-rest-adapter-define-custom-http-header-elements/
Hello Alexander,
Very Nice Blog. This is really helpful. This is exactly what I am trying to implement.
However, I have a question. In the response, When you decode the file from the b64string in the response. Will it be possible to retain the original filename?
Thanks,
Chan
Hi Chan, I'm very happy to have helped.
Usually the file name is already in the url, for example: "https://eu01-s3-store.s3.amazonaws.com/bucket/Example123.jpg", the file name is "Example123.jpg".
In this case the request mapping can copy the name of the file, then use it in the response mapping.
I found the post below that explains this in more detail.
https://blogs.sap.com/2015/08/28/copy-value-from-request-message-to-response-message-using-dynamicconfigurationbean-and-dynamic-header-fields/
Hello Alexander,
I followed your blog and all good !
However I am unable to convert this Base64 message to image at ECC.
A normal text file is all good, however when I convert Base64 to image or PDF... at ECC, it complains that the downloaded files are in corrupted.
I am using below functions to base64 to image convertion at ECC
Could you suggest and help !
Thank you
Satya
Thank you Alexander for your help. This helped me solve my requirement to download the files from S3 Buckets.
Just to summarize Here are a few issues I faced during my testing
Thanks,
Chan
Hi Alexandre,
How do I get the Token? As long as I see, when I've created the S3 bucket and then the IAM user I get the access key ID and secret Key (of course, the bucket name and region too).
Could you help us to obtain the Token?
In addition, is there any way to protect this information?
Thanks in advance.
Best regards,
Emi