Netweaver Gateway & OData: URI conventions in practice

Over the last year I’ve had various exposure to Netweaver Gateway; the usual “so what is this all about?” tinkering and self-teaching, actual project use and even training of junior consultants.

Prior to this I had been working on some R&D with a custom REST dispatch layer, i.e. doing RESTful SAP access without Gateway. The main bone of contention in the abstract design was the URI scheme – just what was the best way to structure resource paths and pass key values, parameters, etc. to the backend? In this respect REST is open to interpretation since it is not a protocol.

When it was announced that Gateway would use OData we were still some way off of getting a useable product, but it got me thinking: if I were using the OData protocol, how would I incorporate it into my dispatcher? It was pretty clear that an OData parser would be required to turn the URI scheme into something that the average non-webhead ABAP guy could understand.

With the arrival of Gateway Builder, it is now much less of an effort to put together the implementation layer of the data provider, and the model provider classes can pretty much be left alone. Now we have more time to delve into the OData interface and discover exactly what OData can do with ‘standard’ method results and, which OData scenarios require some extra understanding and ABAP effort.

To this end, I’ve been working through the OData URI conventions and comparing these to the Gateway OData implementation.

Basic entity addressing

OK let’s clear something up first. Always keep the cardinality 0:n in mind. Even if you know that an entity has a 1:1 cardinality, the OData protocol is taking  0:n as a baseline.  This makes sense as 0:n has to encapsulate all the other cardinalities.

The practicality of this means I should be referring to all entities in plural terms.

resource_path/service/foos

could return no foos, one foo or a whole basketful. That’s the point, it’s meant to work that way. Suppose I get several foos and want to reference one  (foo9) and navigate into it?

Well, I’ve actually got a navigation link to it in the result feed, so I don’t need to worry about that. Assuming I didn’t have that link, the logical addressing would be:

resource_path/service/foo(‘foo9’)

I’m referencing a single resource so I don’t need to bother with the entity set syntax now, do I…

WRONG!

The above would be thrown back by the OData parser telling me that the resource ‘foo’ does not exist.

I have to sideline English and logic here and return to my 0:n way of thinking. The correct syntax is:

resource_path/service/foos(‘foo9’)

Dammit – so I’ve got to go through the get_entityset method again? No, I don’t.

Although I am using the plural syntax, the simple addition of the key specifier section – adding  (‘foo9’) after the resource name – tells the parser that it needs to send this request to the get_entity method in my data provider.

Intrinsic System Query Options

The OData standard specifies a set of system query options that you can apply to any resource request; that is, the result will be often be different to the vanilla request on the same node.

Gateway implements some of these system query options intrinsically; provided that you have working GET methods for the entity and entityset you can expect the OData parser to transform the result for you.

Here are the intrinsic options.

Property extraction – single

Assume that the ‘foo’ entity has these simplified properties:

<EntityType Name=”foo” sap:content-version=”1″>

<Key>

<PropertyRef Name=”id” />

</Key>

<Property Name=”id” Type=”Edm.String” MaxLength=”10″ sap:label=”Foo ID” />

<Property Name=”name” Type=”Edm.String” MaxLength=”30″ sap:label=”Name” />

<Property Name=”category” Type=”Edm.String” MaxLength=”10″ sap:label=”Category” />

<Property Name=”price” Type=”Edm.decimal” Precision=”9″ Scale=”2” sap:label=”Price” />

<Property Name=”currency” Type=”Edm.String” MaxLength=”5″ sap:label=”Foo ID” />

</EntityType>

It is possible to extract just one of these properties as a result. The option can only be applied to a single entity URI.

resource_path/service/foos(‘foo9’)/category               //obtains the category of an entity identified as ‘foo9’

N.B. there are further formats of URI that can access single values at different levels of navigation – here I am just presenting the basic form against a simple primary entity address. I hope to revisit these later.

Property extraction – SETS ($select)

Sets of properties can be extracted, these are obtained with the $select query option. The option can be applied to any URI addressing an entity or a set.

resource_path/service/foos?$select=name,price,currency

resource_path/service/foos(‘foo9’)?$select=name,price,currency

// property section only contains request properties

<m:properties>
  <d:name>foo-nine</d:name>
  <d:price>380.00</d:price>
  <d:currency>gbp</d:currency>
</m:properties>

Instance count – ($count)

The $count option is similar to the Open SQL count in that it will return a count of the query results and no data (note that in the URI, $count sits behind a final ‘/’, not a ‘?’ operator like other options do!)

resource_path/service/foos/$count 

The $count option only makes sense when applied to an entity set address, but it will still work (sort of) on the access to a single entity, since it operates on the assumption of an overall 0:n cardinality.  I say ‘sort of’ because if the entity does not exist for the requested key, the count is still 1. That is semantically incorrect, since a client should be able to run an existence check on a key and determine true or false from the count result.

Why do we get a count of 1 when the key is wrong? It is technically correct because the response feed contains an empty property set after it comes back from the GET method. There doesn’t seem to be a way to make it return zero; if you modify the DPC class method to not transfer the data, you get an inner error reporting the resource is not found. That is “correct” but I’d rather not have inner errors used to report non-existence since they are logged as server errors.

One point to note on the above regarding performance : the property extraction/counting is done after the method call, applied to the results. Your method has no inkling that a reduced set of properties is being requested therefore it will always obtain the full property set. Until SAP change the design to transfer this level of request information to the backend, the ABAP layer will not be able to provide faster access to the data.  This is technically very similar to WDA contexts that are based on Dictionary structures – the whole set of columns are present in the background and the meta-model maps to those attributes used in the context, but in DB terms it’s always a full width access.

There are some further intrinsic query options; $expand and $links –  however they only have any value once you start navigating between entities. I’ll cover these in the next installment.

System Query Options requiring ABAP extensions

Now we come to the system query options that won’t work unless you provide some ABAP that anticipates and executes them.

Entity Set Paging ($top and $skip)

These options are at once the simplest to understand and possibly the hardest to implement within an enterprise service.

$top limits the amount of results from a set. This is a GOOD THING and should be mandatory in my opinion. Having an open query could be disastrous where the potential datasource is massive.

resource_path/service/foos?$top=200

Developing an entityset feed should always involve some kind of $top consideration. The value for the $top option is fed into the ABAP method via IS_PAGING-TOP.

You could design the ABAP logic to assign a default value for $top if the requester doesn’t set it, rather than having an unlimited retrieval, but I don’t believe that’s a good idea. How would the recipient know we had limited the results?  In short, the client should be required to send a $top value  and the acceptable maximum top value should have a limit . How this is implemented and enforced  is up to you guys

Thorny issues don’t stop with $top. So I got the top x entities, I want to get the next bunch.  In order to get a new collection, I can ask the server to skip n entries using the $skip option.

resource_path/service/foos?$skip=200

However that takes us back to the ‘how many?’ question posed by $top. There’s no point limiting the first query to 1,000 entries then letting the client skip past those and get a million.

No, $skip by itself is pretty useless, dangerous and other nasty things. Therefore another of Ron’s Rules is that any $skip still requires a $top value to be sent with it. Yes, you can do this – this is the first mention of the fact that you can combine some of these options. It also matches the paging concept as you expect a discrete number of rows per page.

resource_path/service/foos?$skip=200&$top=200

By the way, the value for the $skip option is fed into the ABAP method via IS_PAGING-SKIP – in case you hadn’t figured it out when you checked out its cousin ’TOP’.

As if that wasn’t bad enough, things are now complicated by the architecture. The requests are – by design – stateless as far as the server is concerned. A $top option in a request is not too bad and easily ABAPed.

$skip on the other hand – ewww! It is stateless, which means that the only way you can relocate your “logical cursor” to satisfy the request is to go back to entity 1 and read your way forward $skip entries. Other than adding a stateful component on the server side that is linked to the GET method I cannot see how this mechanism for traversing through paged data can be good for performance.

On top of this, the ordering of any logical pages has to be consistent so that you get the same instances in the same location in each page. That’s one for individual implementers to decide on…

…which brings us to $orderby. This allows us (via IT_ORDER in the method interface) to specify a sort order. Since OPEN SQL has an ‘order by’ clause you will probably be tempted to use that however I’d be really worried that I’d be missing something in these disconnected queries. I never use that clause anyway as I was warned it would place too much burden on the DB server.

Since there are other features of OData that can satisfy queries with better performing results, I would limit the use of $top and $skip to “document model”  type resources that would typically have a few dozen ‘pages’ but could still have a few thousand and work well.

Update June 2013: if you are serving your data from HANA, you do not have to worry so much about performance and reselection. In fact, $top and $skip are delivered out-of-the-box by standard data services that you create in HANA – Nice!.

Well I think I’ve gone on long enough about those three – mainly to get you thinking about the pitfalls those options can open up. There seems to be a pattern there – the URI scheme OData suggests that the backend requirements for something like $top are quite lightweight and therefore the implementation should be light. Quite often it’s the case that the simplest options invoke the heavier load on development and resources.

In my next blog I intend to look at progressing to more complex and advanced topics such as navigations and better query options.