Delta Queries (Part 1)
This is to my mind the greatest new features for OData and for Gateway 7.4 SP7 is the support for OData 4.0 Delta Queries.
Recently I was running one of the partner enablement bootcamps for SMP 3.0 and found out there was little documentation on the greatest feature of OData: Delta Queries. This feature is incredibly useful for providing support for synchronisation of OData tables with the minimum amount of changes being transferred to the client. The other feature that has got me excited is the ability to side load data. Sideloading is where the database on the device is populated through an SD card or a zip file downloaded from somewhere. By using this the time taken to initialise the tables on a synchronisation project is reduced.
What are Delta Queries?
To efficiently replicate a source table (on for example a server) to a client table (on a device) you need to synchronise the full table and then it would be helpful if you could ask the source table for a list of changes that have occurred since the last time you checked the source. In many scenarios it is often the case that the table changed slightly and so a small number of changes need to be transferred to the client. Now whilst it is easy to keep track of a changes yourself by adding a timestamp column to an individual collection, OData V4 provides a simple infrastructure to allow this to happen without you needing to be concerned with the mechanics of the happening on the server. Other use cases around paging, filtering and inlining are also specified in the specification but we will not cover these advanced cases, however if the need arises the OASIS website can be accessed to get the full delta token specification.
Of course there are also some caveats. The first is this only works on collections and not individual entities or values of the entities for obvious reasons. So querying an individual entity in a collection will only return the entity. The second is that in Netweaver 4.7 SP7 there is only delta query support for ATOM feeds, JSON is not yet supported and finally if OData paging is being used then the last page will contain the delta element and finally if the $select is being used and a field outside of the selected fields is changed the record will be returned though it may look unmodified.
How do they work in theory:
When a query is made to the OData service to a Delta Query enabled collection the first time the call will return all the OData plus a link to that encapsulates a token. This token is a an opaque blob that only has meaning for the server but will need to be stored on the client. The next call to the collection should use this token and a list of changes will be returned by the server. These will include new entities, modified entities and tombstones for deleted entities. In addition a new timestamp will be returned which will need to be cached for the next call. This is repeated continuously to replicate changes in the collection on the server efficiently to the client.
You make a call to the OData server (in case this I am using Netweaver gateway 2.3 SP7) to get a collection of entities from a delta query enabled collection (gateway server:port/service path/collection_DQ), by convention collections that are delta query enabled have a prefix of DQ though this is not mandatory. The exact same content as would be returned from a data query on the collection will be returned but at the end of the document is an additional link element with a relative reference of “delta” and a link to a token to be supplied in a subsequent call.
The element looks as follows:
<link rel=”delta” href=”FlightCollection_DQ?!deltatoken=’22000A0D85A71EE3A7F0EAC255580924_20140227081542′”/>
One of the questions I get a lot is how to populate the device with a lot of data quickly. This is particularly true of products such as CRM or SALES where there is a large table volume that is initially needed and then updated irregularly. To do this using Delta Queries is now easy. On a high bandwidth, low latency network you do a query to the data collection on a service and then save and compress the data locally together with the token. This is then put on the device in any optimally efficient manner. For example creating a local SQLite database and then copying the database onto the device via a memory card or usb mass storage connection. Once on the device the token can be used to query the backend OData service collection for a list of changes that have occurred since the table was first synced and the local copy updated with the changes. This process should be orders of magnitude smaller than the original collection.
A delta query example:
In the part 2 an example of a typical use case for Delta Queries will be shown.