Skip to Content
Technical Articles
Author's profile photo Pierre Dominique

Working with files in CAP (and Amazon S3)

Working with media resources (i.e files) is well covered in the CAP documentation so let’s jump into a very simple example.

1 – Defaults handlers

First we need to create a new CAP project (Node.js OData v4) and a cds file to define our data model (db\schema.cds):

namespace media;

entity Pictures {
  key ID : UUID;
  @Core.MediaType: 'image/png'
  content : LargeBinary;
}

 

Then we can create a simple service (srv\media-service.cds):

using media as db from '../db/schema';

service MediaService {
  entity Pictures as projection on db.Pictures;
}

 

As we are using OData v4, we can store an image by sending one request to create the object:

POST: https://host/media/Pictures

Request Headers:
Content-Type: application/json

Request Body : {}

Note: the request body is an empty object in this case because the ID is generated by the framework (type UUID) and we only have one mandatory property (we still need to send some application/json payload for some reason). Thanks Uwe Fetzer for pointing this out.

 

And then a second request to upload the image (using the id returned by the first request):

PUT: https://host/media/Pictures(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)/content

Request Headers:
Content-Type: image/png

Request Body : <MEDIA>

 

We can then get the image back with the following request:

GET: https://host/media/Pictures(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)/content

 

Pretty simple, right? However using the default handlers means that the files are stored in the database which is often not a good idea. We can for instance store the files in Amazon S3.

2 – Storing files in Amazon S3

SAP Cloud Platform provides an ObjectStore service so let’s see how we can leverage it. You’ll need an SCP subaccount on Cloud Foundry (AWS) for this. Or you can also run it locally thanks to Gregor Wolf contribution.

First let’s create a service instance of the objectstore service:

> cf create-service objectstore s3-standard s3-pictures

 

Then update the mta.yaml file to add a new resource and dependency:

modules:
...
  - name: cap-media-node-srv
   ...
    requires:
      - name: cap-media-node-db-hdi-container
      - name: s3-pictures
resources:
...
  - name: s3-pictures
    type: objectstore
    parameters:
      service: objectstore
      service-plan: s3-standard
      service-name: s3-pictures

 

Then add a dependency in the srv\package.json file for the AWS SDK :

...
    "dependencies": {
        "@sap/cds": "^3.18.1",
        "aws-sdk": "^2.559.0",
        "express": "^4.17.1",
        "hdb": "^0.17.1"
    },
...

 

After building the srv module, we can then use the AWS SDK to interact with S3. We just need to implement the UPDATE handler to store the file in S3 using the upload method and the READ handler to retrieve the file from S3 using the getObject method :

module.exports = srv => {

	const vcap_services = JSON.parse(process.env.VCAP_SERVICES)
	const AWS = require('aws-sdk')
	const credentials = new AWS.Credentials(
		vcap_services.objectstore[0].credentials.access_key_id,
		vcap_services.objectstore[0].credentials.secret_access_key)
	AWS.config.update({
		region: vcap_services.objectstore[0].credentials.region,
		credentials: credentials
	})
	const s3 = new AWS.S3({
		apiVersion: '2006-03-01'
	})

	srv.on('UPDATE', 'Pictures', async req => {
		const params = {
			Bucket: vcap_services.objectstore[0].credentials.bucket,
			Key: req.data.ID,
			Body: req.data.content,
			ContentType: "image/png"
		};
		s3.upload(params, function (err, data) {
			console.log(err, data)
		})
	})

	srv.on('READ', 'Pictures', (req, next) => {
		if (!req.data.ID) {
			return next()
		}

		return {
			value: _getObjectStream(req.data.ID)
		}
	})

	/* Get object stream from S3 */
	function _getObjectStream(objectKey) {
		const params = {
			Bucket: vcap_services.objectstore[0].credentials.bucket,
			Key: objectKey
		};
		return s3.getObject(params).createReadStream()
	}
}

 

Note: the VCAP_SERVICES environment variable has to be parsed to retrieve the parameters for the AWS config (region, bucket id and credentials). These parameters can also be listed with the cf env command :

> cf env <APP_NAME>

 

Now when we send the same PUT request we used earlier, we can see in the console that the file is stored on S3:

PUT /media/Pictures(<ID>)/content
null { ETag: '"<ETAG>"',
ServerSideEncryption: 'AES256',
Location: '<LOCATION>',
key: '<ID>',
Key: '<ID>',
Bucket: '<BUCKET>' }

 

3 – Next steps

Now that we have a basic example, we could improve it in many ways:

  • add some error handling
  • handle different MIME types
  • implement DELETE handler
  • create generic service to handle interactions with S3 and re-use this service in other projects
  • investigate pros and cons of other methods to access S3 (service broker, user provided services)

Note: the source code is available on GitHub .

Cheers,

Pierre

Edit: add some details about the first POST request and how to run the project locally.

Assigned Tags

      16 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Uwe Fetzer
      Uwe Fetzer

      Hi Pierre,

      nice one.

      Haven't played around a lot with CAP yet. Isn't it possible to store the picture with the initial POST? Why do we need the PUT here?

      Author's profile photo Pierre Dominique
      Pierre Dominique
      Blog Post Author

      Hi Uwe,

      I don’t know if this is a CAP limitation but according to the documentation we have to send 2 requests (PUT then POST) for OData V4 and only one POST for OData V2: https://cap.cloud.sap/docs/guides/generic-providers#media-data

      Maybe someone from SAP can give use more details: Daniel Hutzel or Christian Georgi maybe?

      Edit: typos...

      Author's profile photo Christian Georgi
      Christian Georgi

      Or even Vitaly Kozyura? 🙂

      Author's profile photo Pierre Dominique
      Pierre Dominique
      Blog Post Author

      Yeah right. I didn't want to mention all of you guys. 🙂

      Author's profile photo Maximilian Streifeneder
      Maximilian Streifeneder

      awesome work, Pierre Dominique!

      keep the momentum going 🙂

      Author's profile photo Pierre Dominique
      Pierre Dominique
      Blog Post Author

      I haven't contributed on SCN for years and then 2 posts in a week! 😉

      https://blogs.sap.com/2019/11/06/instant-realtime-graphql-engine-on-cloud-foundry-with-postgresql-and-docker/

      Author's profile photo Uwe Fetzer
      Uwe Fetzer

      As promised here's the corresponding Twitter conversation:

      Uwe Fetzer @se38
      Does someone know why we need two requests (POST+PUT) here for #OData v4? #CAP

      Tobias Hofmann @tobiashofmann
      First he creates the entity, then uploads the picture and intercepts the PUT request to store the data on S3. For storing, he needs the ID of the entity. ID is available after the entity was created.

      Volker Buzek @vobu
      there's no deserializer for multipart messages (yet) in #CAP, so the two-step "workaround" is necessary

      Uwe Fetzer @se38
      where do you see a multipart message? It would be simply a single post (like in OData v2)

      Volker Buzek @vobu
      JSON payload + binary stream (image)

      Uwe Fetzer @se38
      hmm, and why do we need the empty JSON in the first POST?

      Volker Buzek @vobu
      not sure if @pdominique's example is 100% correct - given the model https://github.com/pdominique/cap-media-node/blob/master/db/schema.cds there'd be a POST JSON payload necessary like { ID: <guid> }

      Pierre Dominique @pdominique
      The ID is generated automatically by the framework because it’s a UUID. However in this case we still need to send an empty json payload because some application/json content is expected.

      Maybe someone from SAP can explain, why we need two requests and if this will change in the future eventually.

      Author's profile photo Bhuvan Mehta
      Bhuvan Mehta

      Hello Pierre,

      Thanks for the blog. Have you tried the same with CAP Java also?

      Similar application with CAP Java doesn't succeed with reporting error for unsupported content type.

      --

      Thanks and Regards,

      Bhuvan Mehta

      Author's profile photo Kangana Dash
      Kangana Dash

      Hi Pierre Dominique! ,

       

      Thanks for the good blog. I have followed your blog and am able to PERFORM PUT ,POST requests. But when I am sending a GET request using post man request i am seeing this error . Can you please help me undersand what could be the problem...

       

      What is the expected outcome? Should I see IMAGE as output?

      Author's profile photo Pankaj Yadav
      Pankaj Yadav

      Hi Pierre,

      Thanks for this blog.. can this object service work with BTP, ABAP?

      Author's profile photo Cesar Felce
      Cesar Felce

      Hi Pierre,

      Awesome work! we could replicate your blog! thank you!

      We have a doubt on how to use the SAPUI5 controls like for example. FileUploader, because this one use only one request, and also the uploadUrl attribute doesn't have the ID and the .../content of the entity.

      Do you have any ideas o documentation on how to use it in a fiori sapui5 app?

       

      regards,

      Author's profile photo Phillip Phiri
      Phillip Phiri

      Thanks for the great example Pierre Dominique!.

      I'm having trouble getting the READ portion to work.

      The upload works but the read returns an empty response for anything other than tiny text files.

      So far I've confirmed that the stream coming from s3.getObject(params).createReadStream() contains a valid stream. I confirmed by converting the stream to a buffer and saving the file with fs.writeFile()

      This is what my read looks like:

      srv.on("READ", "Attachment", async (req, next) => {
          if (!req.data.ID) {
            return next();
          }
      
          const readableStream = _getObjectStream(req.data.ID);
          const buff = await getRawBody(readableStream);
          fs.writeFile("output.png", buff, "binary", function () {
            console.log("saved to file");
          });
          return readableStream;
        });
      
        /* Get object stream from S3 */
        function _getObjectStream(objectKey) {
          const params = {
            Bucket: services.storage.bucket,
            Key: objectKey,
          };
          return s3.getObject(params).createReadStream();
        }
      
      Author's profile photo Cesar Felce
      Cesar Felce

      Hi Pierre,

      We have implemented this and it's working on our service, but we found a code with an extended control of the FileUploader, because none of the standard controls, that we tested, works properly.

      Do you any ideas on how to use a SAP UI5 control with this service?

       

      kind regards,
      Cesar

      Author's profile photo Abin Mathew
      Abin Mathew

      Hi Pierre,

       

      Great blog!. I tried a similar approach with CAP Java but facing issues for GET call as OdataJsonParser throws exception while serializing. Do you know any standard for implementing this on Java

      Author's profile photo Kanika Malhotra
      Kanika Malhotra

      Hi @Pierre Dominique!

      While deploying the CAP project, I am getting below error in the deployment log

      "[Error] Cannot find module aws-sdk".

      Even though I have mentioned the dependency in the package.json. Also did an npm install aws-sdk. But nothing helped.

      I can see the aws-sdk folder in my node_modules folder.

      Kindly help with the same.

      Thanks

      Kanika

      Author's profile photo CESAR FELCE AXITY
      CESAR FELCE AXITY

      Hi Pierre,

      This blog was really useful thanks for sharing, but now for node 18 I'm having the following errors:

      1. Is necessary migrate your code to use AWS SDK for JavaScript (v3).
      2. In the second request in the PUT, not is doesn't work the req.data.content
      [cds] - ❗️Uncaught TypeError: Converting circular structure to JSON

       

      Any ideas on how to get the media stream of the file and put it in the Body.

       

      Regards,
      Cesar