Recently I had a task to upload a few media files into the SCP Object Store based on AWS IaaS. I was building my web service on Go and, hence, was looking for a few code snippets on how to work with S3 media buckets.
I found a few great
Java code samples and a generic
Java application to work with the Object Store, but there was nothing written in Go. So I’m just sharing a few basic lines I’ve got so that someone could reuse them as a starting point in the future.
You can find the full code sample
here.
In order to familiarise yourself more with the concept of SAP's media buckets, please have a look at the following
FAQ section.
Create an Object Store instance
So the first thing you need to do is to create an Object Store instance provided by AWS IaaS layer. Unfortunately, it doesn’t look to be offered by an SCP trial account as it is a paid service.
As a result you should have a bucket id and a set of secret keys to access the instance from the code. As you aware, the secret keys should not be hardcoded in a productive application and instead stored in environment variables.
The values listed below are enough for our purpose. Be careful with the region value as it depends on your SCP account location and may vary.
const (
s3_region = "us-east-1"
s3_bucket = "hcp-<YOUR-S3-BUCKET>"
aws_access_key_id = "<YOUR-AWS-ACCESS-KEY-ID>"
aws_secret_access_key = "<YOUR-AWS-SECRET-ACCESS-KEY>"
)
Create an S3 client instance
Now we can use the credentials above to create a configuration and instantiate an S3 Client provided by AWS Go SDK:
token := ""
creds := credentials.NewStaticCredentials(aws_access_key_id, aws_secret_access_key, token)
_, err := creds.Get()
if err != nil {
// error handling
}
cfg := aws.NewConfig().WithRegion(s3_region).WithCredentials(creds)
svc := s3.New(session.New(), cfg)
The created S3 client opens possibilities to access the full set of S3
APIs provided by the SDK. I’ve put the code above into a separate method called
getS3Client.
Prepare and upload a file
I’ve put a small pdf file for testing purposes into the folder named '
static' inside of the project. I’m using it as input payload for the created Object Store service instance. Firstly, I open it locally by the Go OS utilities and make a 'defer' instruction to close it once the current function finishes execution.
filePath := "./static/" + fileName
file, err := os.Open(filePath)
if err != nil {
// error handling
}
defer file.Close()
If you were creating a web service instead of a desktop application like in our case, you would probably call and external service and work with input byte stream received from it. Here we are working with the byte stream from the input file as per below:
fileInfo, _ := file.Stat()
size := fileInfo.Size() // determine the file size
buffer := make([]byte, size) // prepare a buffer
file.Read(buffer) // read bytes into the buffer
reader := bytes.NewReader(buffer) // create a Reader instance
fileType := http.DetectContentType(buffer) // determing the file type (e.g. application/pdf)
path := "/media/" + key // construct a key
Here we are basically reading the contents of the whole file into memory of our PC. This is not the best approach if we were developing a highly loaded web service working with large files. For that case it would be more effective to buffer the input into separate chunks of bytes as they arrive and upload them one by one via
multipart upload API.
Finally, we assemble the request and pass the byte stream into the S3 instance:
params := &s3.PutObjectInput{
Bucket: aws.String(s3_bucket),
Key: aws.String(path),
Body: reader,
ContentLength: aws.Int64(size),
ContentType: aws.String(fileType),
}
resp, err := s3Client.PutObject(params)
if err != nil {
// error handling
}
Download a file
In order to complete a reverse operation and download the same file, we create a new file on disk:
path := "/media/" + key
file, err := os.Create("./static/" + "downloaded_pdf.pdf")
if err != nil {
// error handling
}
defer file.Close()
After this we create a new session and use a new download manager instance to get the file from the S3:
cfg := s3Client.Config
sess, _ := session.NewSession(&cfg)
params := &s3.GetObjectInput{
Bucket: aws.String(s3_bucket),
Key: aws.String(path),
}
downloader := s3manager.NewDownloader(sess)
_, err = downloader.Download(file, params)
if err != nil {
// error handling
}
So now if you execute the app by the command below, you should see that a copy of the test file is downloaded into the local ‘static’ folder.
$ go run main.go
List files in the bucket
In order to print the files already existing in the bucket, use the code below. It's basic to be explained.
params := &s3.ListObjectsV2Input{
Bucket: aws.String(s3_bucket),
}
resp, err := s3Client.ListObjectsV2(params)
if err != nil {
// error handling
}
for _, item := range resp.Contents {
log.Println(item.String())
}
The full code can be found
here. I hope it was useful for a quick reference.
Thank you.
