SAP Cloud Platform Backend service: Tutorial [17]: CDS: ETag

This blog is part of a series of tutorials explaining the usage of SAP Cloud Platform Backend service in detail.

Quicklinks:
Definition
Usage
Reference

If you’ve never heard about ETag: this is a good blog for you
If you’ve always wanted to know about these ETags: continue reading
If you’re familiar with ETags and only need to know the CDS syntax: do this hop

What is it, the ETag?

This name is so mystic…
But the meaning is quite down-to-earth

Abbreviation: entity tag

It is used for optimistic concurrency control
What is it, optimistic concurrency control?
It is not pessimistic concurrency control.
What is it, pessimistic concurrency control?

OK, in case of “pessimistic” we want to be 100% on the safe side.
Whenever a data entry in a backend (database) has to be modified, then first of all, a lock is applied.
Like that, only the current user (or application) is allowed to modify the data
Then the data is changed and safely written to the database.
Afterwards the lock is released.
This procedure is not very fast (and might cause deadlocks).
Especially, if the lock isn’t really required, because the resource is not likely to be accessed by multiple users at the same time.
Furthermore, an OData service is anyways “stateless”.

That’s why “pessimistic” doesn’t apply in our case, we have to look at “optimistic concurrency control”:
When a resource, an entity, is requested, it doesn’t get locked.
So it can be modified.
Nevertheless, it could be requested and modified by a second user at the same time.
If that occurs, changes can be overwritten or requests can fail, etc
To prevent such failure, we have the mighty ETag.
Being so great solution, it should be bold
To prevent such failure, we have the all-mighty ETAG.

The wonderful ETag provides concurrency control but doesn’t lock
With other words, the ETag indicates the version of an entity.
With even other words, the ETag is used for conditional requests

Why do we need it, the ETag ?

Because it prevents concurrent modifications, without overhead.

Example:
In a web application, a user opens the details of whatever
e.g. a product?
Yes, e.g. a product
The UI allows to edit, so the user changes some details, but doesn’t submit
At the same time a second users decides to delete that same product and submits the deletion.
When the first user finally decides to submit, the changed product doesn’t exist anymore, so a crash is unavoidable
So what now?
To prevent such failure, we have the all-mighty ETAG.
Nicely bold, but being so great solution, it should be colorful
To prevent such failure, we have the wonderful ETAG.

Who needs it, the ETag?

ETag is not mandatory for a resource requested via HTTP.
It is not required for services used in demo scenarios

But it should be used in productive APIs

How does it work, the ETag?

Being an electric tag, it works magically.
Kidding.

We have to add a tag to the entity.
That tag can be anything.
For example an ETag?
Whenever an entity is changed, the tag is changed as well.
Furthermore, the HTTP protocol has few status codes reserved for ETag handling.

Example flow:

An OData service implementation decides to support ETag
Whenever such an entity is sent to the client (e.g. single GET), the ETag-header is sent
If the client sends a modification request (e.g. PATCH or DELETE) for that entity, it has to send the ETag value in a header called If-Match
Taking it literally: “I want to modify this entity, but only if the ETag matches the given value.”
As such, the service implementation can check if the modification request is allowed.
If the client doesn’t send that header, the server responds with an error Precondition required
If the ETag is sent, but doesn’t match (entity has changed in the meantime), then the error is Precondition failed
If this is the case, then the client fetches the entity with GET and takes the new ETag to send a valid modification request

Another example flow:

Until now we’ve talked about concurrency control, I guess this scenario is now clear.
Quite clear, thanks
There’s a second advantage which comes with ETag:
OData services are stateless, but nevertheless it would be an advantage to avoid redundant calls

Like this:
A client application calls an entity. A big entity.
Even a huge entity
The client reads and remembers the ETag
When the client app does another request to that entity, it will send that ETag
If the (huge) entity is NOT modified, then the server indicates it by returning Not Modified
And the good thing: the (huge) entity is NOT sent again.
Means I send the following wish to the server:
“Please send me that huge entity only, IF the current ETag does NOT MATCH the ETag which I’m sending”
This is the actual use case of conditional GET request.

We’ll go through these examples in the Test-section below

How is it expressed in CDS, the ETag?

An entity has one property which is marked as ETag-relevant by adding the annotation

@odata.etag

to that property
As such, whenever the entity changes, then this property will be changed as well.
Usually, such property can be a timestamp,indicating when it was last changed. Such ETag value can be automatically generated

How do we express our CDS, the ETag?w

Would be nice to see it in action
Sure, we’re going to verify it all in an example.
Note:
To make life easier, let’s use a version-property, a number which we manually change when we change the entity.
Below there will be a second example with timestamp

To create an example, I’ve once again thought of a Pet shop, because some pets are known to change their mind often, so if they liked their food today, it might turn to be absolutely uneatable the next day. So for such pets we need a version property…?

Example CDS:

service PetService {
    entity Pets {
        key id: UUID;
        name : String;
        species : String;
        food : String;
        @odata.etag
        version: Integer;
    }
}

Have you seen it, the mighty ETag?
Yes

How does it look in OData, the ETag?

Well, it looks like an Excellent Tag, the ETag.
Kidding.
The contract of an OData service is written in stone, it is the metadata document and it has to show if and how ETag is used.
Because a client application has to be aware of it.
As such, the property which is used for ETag, has to be flagged.

In OData V2, it looks like this:
The ETag-property doesn’t have an attribute called ETag, even not etag, not even et, the attribute is called: ConcurrencyMode=”Fixed”
In OData V4, it doesn’t look at all, currently, but I’m sure it will look as described in the OData specification, once it is supported

Our example:

Have you seen it, the mighty ETag?
Yes

How do we use it at runtime, the ETag?

With other words: how do we test it, the Endless Tag?
Kidding.

Preparation

If not already done, create an API with the above sample CDS
Go to the cockpit and enable the OData V2 checkbox
Scroll down to API REFERENCE and select OData V2 in the drop down field

Create a sample entry
(See here for a description of the cockpit test tool   )

Example payload:
{
“name”: “kitty”,
“species”: “cat”,
“food”: “fish”,
“version”: 1
}

Test 1: PATCH request with ETag

First perform a single GET request to the created entry

Note:
Remember, the Guid-key in v2 has to be composed like this:
guid’dc2c61cb-a855-4958-a4c9-188a4f501357′

Scroll down to see the response headers and find the ETag value, sent by the server:

As expected, the ETag contains the value of the version property which we’ve given at creation: 1

Now try to modify the entry:
Go to the PATCH section, ignore the payload, enter the key like above
Note:
while pasting the guid, make sure you don’t add a blank, as it would be interpreted as invalid character and would cause the request to fail

Execute the request and look at the expected error message.
Like described above, the HTTP response status is 428 Precondition Required

Good.
Good?
Yes, if we create an ETag-enabled service, then we cannot be surprised if it behaves like an ETag-enabled service
OK, we MUST send the ETag header in the PATCH request
So please go ahead and enter the header…
ohhhh, I’ve searched one hour and still don’t know how to do
Yes, OK, OK, that was mean….. the cockpit doesn’t allow to enter custom headers.
But don’t be frustrated, luckily you have this great series of tutorials, you can go to this blog to learn how to send requests from REST client to Backend service API
Read it, like it, and follow the explanation to call the service from external REST client
Oh, so much work
Yes, it is a bit tedious, but we’re happy that we can do it.
Actually, with Postman it is just few clicks to do
True.

OK, our service works as designed, so we need a REST client to send a PATCH request with custom headers

Now let’s go ahead

First execute a GET request in order to take a note of the ETag in the response-header

Example URL:
https://backend-service-api.<…>/odatav2/DEFAULT/PETSERVICE;v=1/Pets(guid’dc2c61cb-a855-4958-a4c9-188a4f501357′)

We knew that ETag already…
Anyways, take a note of it

Now compose the PATCH request

URL: the same
HTTP verb: PATCH
Request body:

{
“food” : “mouse”,
“version” : 2
}

Request headers (raw format, copy&paste ready):

Content-Type:  application/json
If-Match:      W/”1″

Execute the request and verify the Result:
We should get the correct 204 status code for PATCH request
And in the response-header section, we can see the new ETag (it is 2, like specified it in the request body)

See screenshot:

You can verify with another GET request:
The picky cat has now a different food: currently it prefers mouse over fish.
That’s life…
<Sigh>

Next steps:
Verify the different cases of scenarios with ETag.
Please don’t ask for a new screenshot every time
No?
No –  I don’t want to take so many screenshots
Really?
Yes –  only cats are allowed to be so picky….

Test 2: PATCH with invalid ETag

Send the same request like above, but this time with invalid ETag.
E.g.

Header:
If-Match:   W/”99″

Result:
Status Code 412 Precondition Failed

Response Body:
Error message: The Data Services Request could not be completed due to a conflict with the current state of the resource

Response Headers:
no ETag

That error was expected, since we sent a wrong ETag
In a productive scenario this would mean that the client application would issue another GET request to obtain the newest ETag value. Then the user might need to re-thing his modifications to the data, because the data has changed in the meantime. Then the PATCH request would be re-sent with new ETag and potentially modified request body

Test 3: PATCH with *

Send the same request like above, but this time with ETag value as *, meaning everything is acceptable.

Header:
If-Match:  *

Result: success

So if you’re too lazy to deal with correct values, e.g. in a test or demo scenario, then you can always just send a *
(If supported by the service)

Test 4: PATCH with no ETag

Same request, this time with no ETag at all

Result:
Status Code 428 Precondition Required

Response Body:
Error message: Precondition required

Response Headers:
no ETag

Test 5: GET with no ETag

Send a GET request to the same URL like above,
E.g.
https://backend-service-api.cfapps.eu10.hana.ondemand.com/odatav2/DEFAULT/PETSERVICE;v=1/Pets(guid’dc2c61b-18…8a47′)

Request headers:
No special header, no ETag header

Result:
Just normal result of single GET request

Response headers:
Contain the current ETag

Test 6: GET with If-Match

Send a GET request to the same URL like above
This time, specify an ETag header with valid ETag
E.g.

Request header:
If-Match:W/”2″

Result:
Since the condition is met, we receive a normal result of a normal GET request, Status 200 and same ETag in the response headers

Test 7: GET with invalid ETag

GET request to same URL like above, this time with invalid ETag

E.g.:
If-Match:W/”99″

Result:
As expected: 412

Test 8: GET with If-None-Match, invalidly-valid

A GET request with if-none-match is the actual use case of conditional GET request, as mentioned in the introduction
Send GET request to the same URL like above, this time with If-None-Match and invalid ETag
E.g.:

Header:
If-None-Match: W/”99″

Result:
Normal result like a normal GET request,
Status 200,
response payload in body,
valid ETag in response header

Test 9 : GET with If-None-Match, validly-invalid

Send GET request to the same URL like above, this time with If-None-Match and currently valid ETag
E.g.:

Header:
If-None-Match:W/”2″

Result:
NOT normal result like a normal GET request,
Status 304 Not Modified

Response body:
<empty>

Response header:
Valid ETag

This is the interesting scenario:
We can see that NO payload is sent.
Good, because it is anyways not modified
The client is informed about it: via the HTTP Status Code 304 Not Modified
Means, the client would not replace the current data in the UI-Table with the new – empty – payload.
No, no that would be wrong, obviously
But since the client does always interpret the response status code, everything’s good:
The UI remains unchanged
Exactly

Note:
Since the response status is not an error status code, the current ETag is sent along with the response

As you can see, this is the actual use case of conditional GET request:
Fetch data only if it has changed.
With other words:
If the data has NOT changed, we don’t need to fetch it.
Save time.
Save network resources
Save money
Save the planet
As a consequence:
More time for playing with cat
More bandwidth for cat videos
More money to buy fish (or mice)
More greenery to entertain cats

Test 10: DELETE

Nothing new, to be short:

• With no ETag:
  All the same, the result is as expected, 428
• With invalid ETag:
  All the same, the result is as expected, 412
• With valid ETag:
  Deletion successful, status 204 as expected, no ETag in the response, obviously
• With If-None-Match and valid Etag
  412
• With If-None-Match and invalid Etag:
  Successful deletion

Test 11: PATCH with If-None-Match

• If-None-Match and valid ETag: 412
• If-None-Match and invalid ETag: success
• If-None-Match with value null: success
• If-None-Match with value * : 412

Test 12: GET ALL

At the end, let’s have a look at a new case:
As result of a QUERY operation, we get a list, so all the ETags cannot be sent as header
As such, they are included in the payload
Try it:

URL:
https://backend-service-api.cfapps.eu10.hana.ondemand.com/odatav2/DEFAULT/PETSERVICE;v=1/Pets

Response:
<entry m:etag=”W/”2″”>
<id>https://<..>-backend-<…>/odatav2/DEF/SRV/Pets(guid’00…’)</id>
<title type=”text”>Pets</title>
…

Are we done with it, the ETag?

No, we have missed a few scenarios with the Exhausting Tag…
No, please not, I’m tired…
OK OK, I was kidding, don’t worry, we’ve gone through all relevant scenarios…
Thanks that sounds better…pouf
For me it has just started being great fun….
Sorry, I’m a bit picky sometimes….
Sometimes?
BTW, why do we write a W/ in the ETag value?
Not tired anymore?
No, not at all, why tired???
Sigh…

What is the syntax for it, for the ETag?

There are 2 possibilities:

1. The ETag value with quotes:
   “2”
2. The ETag value with quotes and preceeding W/:
   W/”2”

Why are there 2 of them, of the ETags?

One is the so-called “weak” ETag. As the name indicates, it is less strict.
I’m a bit tired, so let me quote the OData spec:

“As OData allows multiple formats for representing the same structured information, services SHOULD use weak ETags that only depend on the representation-independent entity state. A strong ETag MUST change whenever the representation of an entity changes, so it has to depend on the Content-Type, the Content-Encoding, and potentially other characteristics of the response.”

I don’t get it
Sigh…

The background is that data can be represented as xml or json. As such, the value of an ETag-property can look different. But still the content is the same.
Therefore it makes sense to use weak ETags in OData
(See your next question for an example)
You can try a strong ETag as follows:

If-Match: “3”

How does a more meaningful ETag look like, the ETag?

A still very simple example, but this time using a meaningful ETag property:
Data type timestamp for a property like “last-modified”, “changed-at” or similar:

service PetService {
    entity Pets {
        key id: UUID;
        name : String;
        species : String;
        @(odata.etag)
        modifiedAt : DateTime;
    }
}

Note:
Alternative notation: @(odata.etag)
It allows to add multiple annotations between the brackets

At runtime, do single GET request and try with different representations, by adding $format=json or $format=xml, which is the default in OData V2
You see that the value of the property “modifiedAt” looks different in json, compared to xml
However, have a look at the ETag header: it is the same in both cases

Try different requests

E.G.

Weak:
If-Match: W/”2019-05-15T15:51:04Z”

Strict
If-Match: “2019-05-15T15:51:04Z”

Change a second
If-Match: W/”2019-05-15T15:51:05Z”

Now try all 12 tests like before
Oh… I’m too tired
Hehe…

Summary

An ETag is like little version information, attached to an entity
More concrete, a property of an entity definition is annotated, to mark it as ETAg relevant
Whenever the value of this property changes, it means that the complete entity is different
When doing service requests, the value of ETag is sent in a header

With the help of ETag,

1. a client can make sure not to fetch data which he already has
2. a server can make sure that 2 clients don’t do concurrent modifications to an entity

Links

ETag in HTTP/1.1 specification

ETag in the OData specification:
Data modification with ETag
Response header with ETag
Request header If-Match
Request header If-None-Match

Optimistic Concurrency Control in Wikipedia

SAP Help Portal: ETag definition in CDS

Appendix 1: Quick Reference

CDS:
@odata.etag

Runtime:
The ETag value is taken from response header of a GET request
Required header names:
If-Match
If-None-Match
Workaround: Use * to avoid entering correct ETag

Appendix 2: The pet, the picky