Skip to Content

Updates

  • 04.03.2014     –> Added a link to document that describes how to consume a service that desribes the OData Delta Query mechanism

Objectives

In this how-to guide, you will learn how to implement Delta Token capabilities with SAP NetWeaver Gateway to enable offline functionality. This how-to guide assumes that you have already worked your way through the following how-to guides:
  1. “How to Develop a Gateway Service using Code based Implementation” https://scn.sap.com/docs/DOC-43030 and
  2. “How to Develop Query Options for an OData Service Using Code-Based Implementation” http://scn.sap.com/docs/DOC-43000
These guides describe the basic implementation steps of the service that we are now going to enhance. The sample service uses demo data from the Enterprise Procurement Model, which is part of every SAP NetWeaver ABAP server as of 7.02.
After completing these steps, you will be able to:
• Enhance the GET_ENTITYSET and GET_ENTITYSET_DELTA methods of an entity set to support the delta query functionality.
Also check out the following document How To Enable Delta Queries using Syclo Exchange Framework and SAP NetWeaver Gateway that explains how delta queries can leverage the Syclo Exchange Framework.

Business Example

You want to build an application that shows a list of products. The application shall be able to retrieve only those products that have been newly created, changed or deleted after having retrieved a complete list of products as an initial load.

Example of a consumer Application

New: Check out the following document: Going Offline With Kapsel Now

Short Introduction to Delta Token Functionality

OData exposes collections of records as EntitySets. These EntitySets support various options like sorting, filtering or paging to minimize the amount of data that is sent accross the wire.
Paging is a very powerful option to reduce the size of the payload and improve performance. To use paging on client side, the OData request needs to specify the $skip and $top tokens. For server-driven paging you have to use the $skiptoken.
Some applications, however, prefer to use client side caching rather than paging. In such cases, SAP NetWeaver Gateway provides Delta Tokens that help to reduce resource consumption on the server and the wire when the same request is sent several times from the same client. The OData Delta Token enables clients to use caching when retrieving entity sets. Consequently, the client gets only the changed records after an initial retrieval of all records. Therefore, the initial request will return all records and a Delta Token. The subsequent request (with exactly the same query options) sends the Delta Token from the first request. The response will then contain only the changed records. If no records have been changed since the first request then the response will not contain any records at all.
You can implement Delta Token in several ways and these options can be grouped into two main approaches:
  1. The first approach, which is based on Syclo Exchange framework, calculates deltas at modification time: The ABAP system tracks relevant changes when they occur. At request time, the deltas are already prepared and thus available. On the one hand this approach requires more development. On the other hand it’s more scalable and has an optimized overall performance.
  2. The second approach is based on delta determination at request time where the system compares old and new state to find out which records have been changed/deleted. The implementation effort is rather small but it does not optimize the performance of the backend. That means, the more records you have in the full collection, the longer the response time of the request.
With both approaches, the payload of the response is reduced but as described only the first approach is also able to optimize the performance in the backend.
In this guide we will look at the basic implementation, using the Delta Request Log Component. The Delta Request Log Component is based on the second approach. Therefore the implementation effort is rather small but the performance of the backend is not optimized.

Prerequisites

You have implemented an OData service as described in the two how-to guides mentioned above.
If you are using NetWeaver 7.02, then you need at least software component IW_BEP Support Package 7 and software component IW_FND Support Package 7.
If you are on NetWeaver 7.40, then you need at least software component SAP_GWFND Support Package 4.
You have to implement following SAP notes: 1926946, 1932247 and 1970229.

Implementation Overview

In this how-to guide, you will create a new Entity Set Products_DQ that shall support delta queries in the existing sample service that you have created earlier. You will redefine the methods GET_ENTITYSET and GET_ENTITYSET_DELTA to support the delta query functionality by calling method /IWBEP/CL_QUERY_RESULT_LOG-> CREATE_UPDATE_LOG_ENTRY_HASH.

Task 1: Create an additional Entity Set Products_DQ

We will enhance the service you created earlier to support delta queries for product data. For this we create a new entity set called Products_DQ where we implement the delta query specific coding while leaving the original Products Entity Set unchanged.

Task 2: Enhance method ZCL_GW_PRODUCT_DPC_EXT->PRODUCTS_DQ_GET_ENTITYSET to support the delta query functionality

In this task you will enhance the existing GET_ENTITYSET method of the newly created entity set Products_DQ. Like the existing GET_ENTITYSET method of the Products entity set it should retrieve the application data containing product entries.
First you need to create a local data provider facade which will be needed later by the Delta Token functionality.
In the next step you will need to call method /IWBEP/CL_QUERY_RESULT_LOG-> CREATE_UPDATE_LOG_ENTRY_HASH for the delta query functionality by passing the local table containing the product entries and the local data provider facade. The method is the central method for the Delta Token functionality. It calculates a hash value for each product and stores the hash values in some application tables, so the entries can be later checked for changes. The method will also return a delta token which you need to export.

Step 1: Get the data provider facade (you need to pass the facade later to the Delta Token method)

* get the data provider facade
 
TRY.
      lo_dp_facade
= /iwbep/if_mgw_conv_srv_runtime~get_dp_facade( ).
   
CATCH /iwbep/cx_mgw_tech_exception.
     
RAISE EXCEPTION TYPE /iwbep/cx_mgw_tech_exception.
 
ENDTRY.

Step 2: Call method /IWBEP/CL_QUERY_RESULT_LOG-> CREATE_UPDATE_LOG_ENTRY_HASH for the delta query functionality by passing local table et_entityset containing the product entries

* call the delta token functionality
  TRY.
  
CALL METHOD /iwbep/cl_query_result_log=>create_update_log_entry_hash
       
EXPORTING
          io_tech_request_context 
= io_tech_request_context
          io_dp_facade            
= lo_dp_facade
          ir_service_document_name
= mr_service_document_name
          ir_service_version      
= mr_service_version
          it_entityset            
= et_entityset
       
CHANGING
          ev_delta_token          
= lv_delta_token.
   
CATCH /iwbep/cx_qrl_locked.
     
RAISE EXCEPTION TYPE /iwbep/cx_qrl_locked.
   
CATCH /iwbep/cx_qrl_delta_unavailabl.
     
RAISE EXCEPTION TYPE /iwbep/cx_qrl_delta_unavailabl.
 
ENDTRY.

Step 3: Export the delta token which was calculated by the method in step (2)

     es_response_contextdeltatoken = lv_delta_token.

Task 2: Re-define method ZCL_GW_PRODUCT_DPC_EXT-> /IWBEP/IF_MGW_APPL_SRV_RUNTIME~GET_ENTITYSET_DELTA to support the delta query functionality

In this task you will re-define method GET_ENTITYSET_DELTA. First, you need to retrieve the application data containing product entries, just like in GET_ENTITYSET method.
Then you need to create a local data provider facade which will be needed later by the Delta Token functionality.
In the next step you will need to call method /IWBEP/CL_QUERY_RESULT_LOG-> CREATE_UPDATE_LOG_ENTRY_HASH for the delta query functionality by passing the local table containing the product entries and the local data provider facade.
It calculates a hash value for each product. Those hash values are then compared with the hash values of the prior call (related to the same delta token ID) to determine which values have changed in the meantime. The method will return a new delta token which you need to export. The method will also return any entries that were changed in the meantime (please store these in local table lt_product) and any entries that ware deleted (please store these in local table lt_deleted_product).

Step 1: Re-define method /IWBEP/IF_MGW_APPL_SRV_RUNTIME~GET_ENTITYSET_DELTA

Step 2: Retrieve all relevant entries for this entity set (like in GET_ENTITYSET above)

Step 3: Get the data provider facade (you need to pass the facade later to the Delta Token method)

* get the data provider facade
 
TRY.
      lo_dp_facade
= /iwbep/if_mgw_conv_srv_runtime~get_dp_facade( ).
   
CATCH /iwbep/cx_mgw_tech_exception.
     
RAISE EXCEPTION TYPE /iwbep/cx_mgw_tech_exception.
 
ENDTRY.

Step 4: Call method /IWBEP/CL_QUERY_RESULT_LOG-> CREATE_UPDATE_LOG_ENTRY_HASH

Call method /IWBEP/CL_QUERY_RESULT_LOG-> CREATE_UPDATE_LOG_ENTRY_HASH for the delta query functionality by passing local table et_entityset containing the product entries – just like in task 1, step 2 but this time take over the changed objects (and store them in local table lt_product) as well as the deleted objects (and store them in local table lt_deleted_product):
call the delta token functionality
  TRY.
     CALL METHOD /iwbep/cl_query_result_log=>create_update_log_entry_hash
         
EXPORTING
            io_tech_request_context 
= io_tech_request_context
            io_dp_facade            
= lo_dp_facade
            ir_service_document_name
= mr_service_document_name
            ir_service_version      
= mr_service_version
            it_entityset            
= lt_product
         
IMPORTING
            et_deleted_entityset    
= lt_deleted_product
            et_entityset            
= lt_product
         
CHANGING
            ev_delta_token          
= lv_delta_token.
 
CATCH /iwbep/cx_qrl_locked.
       
RAISE EXCEPTION TYPE /iwbep/cx_qrl_locked.
 
CATCH /iwbep/cx_qrl_delta_unavailabl.
       
RAISE EXCEPTION TYPE /iwbep/cx_qrl_delta_unavailabl.
ENDTRY.

Step 5: Export the delta token which was calculated by the method in step (4)

        
  es_response_contextdeltatoken = lv_delta_token.

Step 6:  Export the deleted entity set

Export the deleted entity set – add deleted records (from table lt_deleted_product) to the deleted entries list er_deleted_entityset
*   export the deleted entity set
    copy_data_to_ref
( EXPORTING
                       is_data
= lt_deleted_product
                     
CHANGING
                       cr_data
= er_deleted_entityset ).

Step 7: Export the changed entity set – add changed records from lt_product to er_entityset

*   export the changed entity set
    copy_data_to_ref
( EXPORTING
                       is_data
= lt_product
                     
CHANGING
                       cr_data
= er_entityset ).

Implementation

ZCL_GW_PRODUCT_DPC_EXT->PRODUCTS_DQ_GET_ENTITYSET

The coding of the method:
DATA:
        lt_return         
TYPE TABLE OF bapiret2,
        lv_delta_token    
TYPE string,
        lo_dp_facade      
TYPE REF TO /iwbep/if_mgw_dp_facade.

* retrieve all relevant entries for this entityset
 
CALL FUNCTION ‘BAPI_EPM_PRODUCT_GET_LIST’
   
TABLES
      headerdata
= et_entityset
     
return     = lt_return.
* get the data provider facade
 
TRY.
      lo_dp_facade
= /iwbep/if_mgw_conv_srv_runtime~get_dp_facade( ).
   
CATCH /iwbep/cx_mgw_tech_exception.
     
RAISE EXCEPTION TYPE /iwbep/cx_mgw_tech_exception.
 
ENDTRY.
* call the delta token functionality
 
TRY.
     
CALL METHOD /iwbep/cl_query_result_log=>create_update_log_entry_hash
       
EXPORTING
          io_tech_request_context 
= io_tech_request_context
          io_dp_facade            
= lo_dp_facade
          ir_service_document_name
= mr_service_document_name
          ir_service_version      
= mr_service_version
          it_entityset            
= et_entityset
       
CHANGING
          ev_delta_token          
= lv_delta_token.
   
CATCH /iwbep/cx_qrl_locked.
     
RAISE EXCEPTION TYPE /iwbep/cx_qrl_locked.
   
CATCH /iwbep/cx_qrl_delta_unavailabl.
     
RAISE EXCEPTION TYPE /iwbep/cx_qrl_delta_unavailabl.
 
ENDTRY.

* export the delta token
  es_response_context
deltatoken = lv_delta_token.

ZCL_GW_PRODUCT_DPC_EXT-> /IWBEP/IF_MGW_APPL_SRV_RUNTIME~GET_ENTITYSET_DELTA

The coding of the method:
DATA:
         lt_return            
TYPE TABLE OF bapiret2,
         lv_entity_set_name   
TYPE string,
         lt_product           
TYPE zcl_gw_product_mpc=>tt_product,
         ls_product           
TYPE zcl_gw_product_mpc=>ts_product,
         lt_deleted_product   
TYPE zcl_gw_product_mpc=>tt_product,
         ls_deleted_product   
TYPE zcl_gw_product_mpc=>ts_product,
         lv_delta_token       
TYPE string,
         lo_dp_facade         
TYPE REF TO /iwbep/if_mgw_dp_facade.

  FIELD-SYMBOLS:
         <lt_deleted_product> 
TYPE zcl_gw_product_mpc=>tt_product,
         <lt_product>         
TYPE zcl_gw_product_mpc=>tt_product.

  lv_entity_set_name = io_tech_request_context->get_entity_set_name( ).

  IF lv_entity_set_name = ‘Products_DQ’.

*   retrieve all relevant entries for this entityset
   
CALL FUNCTION ‘BAPI_EPM_PRODUCT_GET_LIST’
     
TABLES
        headerdata
= lt_product
       
return     = lt_return.
*   get the data provider facade
   
TRY.
        lo_dp_facade
= /iwbep/if_mgw_conv_srv_runtime~get_dp_facade( ).
     
CATCH /iwbep/cx_mgw_tech_exception.
       
RAISE EXCEPTION TYPE /iwbep/cx_mgw_tech_exception.
   
ENDTRY.
*   call the delta token functionality
   
TRY.
      
CALL METHOD /iwbep/cl_query_result_log=>create_update_log_entry_hash
         
EXPORTING
            io_tech_request_context 
= io_tech_request_context
            io_dp_facade            
= lo_dp_facade
            ir_service_document_name
= mr_service_document_name
            ir_service_version      
= mr_service_version
            it_entityset            
= lt_product
         
IMPORTING
            et_deleted_entityset    
= lt_deleted_product
            et_entityset            
= lt_product
         
CHANGING
            ev_delta_token          
= lv_delta_token.
     
CATCH /iwbep/cx_qrl_locked.
       
RAISE EXCEPTION TYPE /iwbep/cx_qrl_locked.
     
CATCH /iwbep/cx_qrl_delta_unavailabl.
       
RAISE EXCEPTION TYPE /iwbep/cx_qrl_delta_unavailabl.
   
ENDTRY.
*   export the delta token
    es_response_context
deltatoken = lv_delta_token.
*   export the deleted entity set
    copy_data_to_ref
( EXPORTING
                       is_data
= lt_deleted_product
                     
CHANGING
                       cr_data
= er_deleted_entityset ).

*   export the changed entity set
    copy_data_to_ref
( EXPORTING
                       is_data
= lt_product
                     
CHANGING
                       cr_data
= er_entityset ).

  ENDIF.

Test the service

  1. Start the Gateway Client by calling transaction /IWFND/GW_CLIENT
  2. Enter the following URI to test your implementation:  

    /sap/opu/odata/sap/ZGW_PRODUCT_SRV/Products_DQ

  3. You should get a response containing all products. Almost at the end of the XML
    document, you will get your delta token (of course in your response you will
    get a different ID) 

    <link rel=”deltahref=”Products_DQ?!deltatoken= ‘005056A2025C1EE2BFE687AFDC2FAAF4_20130807073741’” />

  4. Now enter the URI from above again, but this time add your delta token (copy and paste it from your response). You URI should look something like this (but will have a different ID):

    /sap/opu/odata/sap/ZGW_PRODUCT_SRV/Products_DQ?!deltatoken=’005056A2025C1EE2BFE687AFDC2FAAF4_20130807073741′

  5. After you click on execute, you will notice that the response does not contain any products at all. This is because the Delta Token functionality returns only changed records and no products were changed in the meantime.
  6. You could now change a product and execute exactly the same URI as in step (4). The response should contain exactly one product.

Technical Restrictions

Key Fields of Entity Type

There are strict restrictions regarding the key fields of Entity Types that are used with the basic delta query functionality:
  • The maximum legth (output length) of a key field must not exceed 200 characters
  • If several key fields were defined for one Entity Type then the combined legth (output length) of those key fields must not exceed 200 characters
  • The key field type must have a fixed legth; therefore types of variable legth (e.g. type String) are not permitted

JSON Format

Currently the JSON format is not supported in combination with the delta query functionality.

Expand System Query Option ($expand)

Currently the Expand System Query Option ($expand) is not supported in combination with the delta query functionality.

Paging

Both the server-driven paging and the client-driven paging are not supported in combination with the delta query functionality. As a result, the following query options are not supported in combination with the delta query functionality:
  • Skip Token System Query Option ($skiptoken)
  • Top System Query Option ($top)
  • Skip System Query Option ($skip)

Further Notes

Clean-Up Report

The Delta Token functionality based on Delta Request Log Component uses two database tables to store hash values: /IWBEP/D_QRL_HDR and /IWBEP/D_QRL_ITM. Once this offline functionality is used productively the size of these tables will increase with each request. We offer areport that can be run to clean up outdated entries: /WBEP/R_CLEAN_UP_QRL. We suggest that you schedule a job for this report so that it is executed periodically, for instance on a daily basis.
To report this post you need to login first.

23 Comments

You must be Logged on to comment or reply to a post.

  1. Derek Wichmann

    Thanks for the write up.  I’m happy to see there is a low effort generic method for performing deltas.  I have a couple questions regarding performance of this method:

    ** Regarding the modification time (syclo exchange tables) vs. request time delta calculation, can you provide actual numbers on the performance difference?  Perhaps number of seconds required to perform calculation on X number of Products?

    ** I assume the cost to calculate the delta at request time (create_update_log_entry_hash) depends on the complexity of the business object.  Can you tell me for how many records (say for this example, Product) this basic delta calculation is no longer performant?

    (0) 
    1. Rajesh Dadwal

      Thanks Andre for informative blog,

      As we are implementing entity set method.. can we use filter option along with Delta query. When i tired to GET data using filter option – I received the delta token but it was not working when I used delta token to fetch changed data..

      Can we use filter option along with Deltatoken in URI

      Regards,

      Rajesh

      (0) 
    1. Pradeep Gudipati

      We seriously need Delta Token support for JSON data, It doesn’t make sense to keep it waiting for more than 2 years now.

      We do a lot of data interchange and XML is causing us Huge performance problems.

      Anybody has an update ?

      Thanks

      Pradeep Gudipati

      (0) 
        1. Óscar Yuste Romero

          Currently we have the same problem and we would need JSON support, our API is developed under JSON and delta token in XML is difficult to change now..

          Any workaround in hybrid apps?

          (0) 
    1. Jan Rychtar

      Hello Vinish,

      Yes it’s possible, data are filtered correctly. Filter atribut and other Odata URL attributes are  filled to get that use class attribut mr_request_details in method GET_ENTITYSET_DELTA. Like this see code:

      /wp-content/uploads/2015/04/filter_from_request_676794.png

      Remark: But we found bug that filter is missing in response url tag see screen when you comunicate throw SAP Mobile Platform. This error I reported to SAP month ago and SAP still fighting with fix that…

      /wp-content/uploads/2015/04/filter_missing_676766.png

      Best Regards,

      Jan

      (0) 
      1. Vinish Chilumuru

        Hi Jan,

        Thank you so much for the reply!!..

        Could you please clarify if i am testing it from Sap Gateway does combination of filter and delta url look something like following.

        /sap/opu/odata/sap/ZGW_PRODUCT_SRV/Products_DQ?$filter= doctype eq ‘XXX’ and !deltatoken=’005056A2025C1EE2BFE687AFDC2FAAF4_20130807073741′

        Best Regards,

        C Vinish

        (0) 
  2. Antoni Macia

    Hello,

    We have implemented Basic Delta Query Support in SAP NetWeaver Gateway, following the guidelines explained in http://scn.sap.com/docs/DOC-47043

    But there is a problem with $filter in a Delta Query scenario, as you can see below:

    1st Request:

    /sap/opu/odata/SAP/ZCONTRACTS_SRV/HierarchySet()?$filter=IvUserwin%20eq%20’MUNUEAL’&$format=xml

    In response:

    <link rel=”delta” href=”HierarchySet()?$filter=IvUserwin%20eq%20’MUNUEAL’&amp;$format=xml?!deltatoken=’005056A326E01EE586B29DDCF362D39E_20150623104524′”/>

    !deltatoken parameter is included, but with “?” operator, the correct one is “&”

    If I manually change it with & it works ok.

    Is there any known issue about it?

    Product version:

    SAP_GWFND 740 0010 SAPK-74010INSAPGWFND SAP Gateway Foundation 7.40

    Thank you,

    (0) 
    1. Jan Rychtar

      Hello Antoni,

      I tried with previous version

      SAP_GWFND 740 0009 SAPK-74009INSAPGWFND SAP Gateway Foundation

      and it works delta token is separated by “&”.  Create OSS message and “fight” with SAP or downgrade Gateway.

      Remark: btw I reported similar OSS message, when I communicate with Gateway by SMP 3.0 SP 7 PL 2 (3.0.7.2) and SAP replaces key word !deltatoken => $deltatoken !!! 🙁 see attachment

      DATA.png

      Best Regards,

      Jan

      (0) 
  3. Hemendra Sabharwal

    Very nice blog, thanks for sharing Andre Fischer,


    But I am stuck. Could you please suggest how to implement “Delta Token” when the WCF service from Microsoft SQL server is exposed as OData service and consumed through Integration Gateway/SMP 3.0 SP05 in Android Native Application.


    Thanks,

    Warm Regards

    Hemendra

    (0) 
    1. Andre Fischer Post author

      Hi Mike,

      it is planned to support delta queries with our implementation of OData V4.

      For OData V2 delta queries were not part of the OData specification. Therefore we added support for the OData V2 protocol by using a custom query option “!deltatoken” rather than “$deltatoken”.

      Here we only support ATOM/XML because only in ATOM/XML we were able to use the concept of tombstones.

      Regards,

      Andre

       

      (0) 

Leave a Reply