Skip to Content

What happened?


Today I tried to make a request to one of my SAP NetWeaver Gateway services that looked like this:

http://<gateway-url>:<gateway-port>/sap/opu/odata/<my-service>/<my-entityset>?%21deltatoken=%27token%27

But when I saw the response I wondered why the delta mechanism of my service wasn’t working. So I started to debug the request and I noticed that in the method `/iwbep/if_mgw_appl_srv_runtime~get_entityset` of my data provider class `io_tech_request_context->get_deltatoken( )` returned an empty deltatoken (`rv_deltatoken` was initial).


First I thought I made a mistake and it’s not allowed to escape URL query parameter names, but when I started research I came across an useful answer on a github issue which pointed me to IETF‘s RFC (Request for Comments) 3986 named “Uniform Resource Identifier (URI): Generic Syntax). Section 2.2 of this RFC points out:

2.2. Reserved Characters

  If data for a URI component would conflict with a reserved character’s purpose as a delimiter, then the conflicting data must be percent-encoded before the URI is formed.

reserved = gen-delims / sub-delims

gen-delims = “:” / “/” / “?” / “#” / “[” / “]” / “@”

sub-delims = “!” / “$” / “&” / “‘” / “(” / “)” / “*” / “+” / “,” / “;” / “=”

quotation from IETF RFC 3986

What does that mean?

As SAP Gateway officially uses/supports the OData Version 2.0 protocol and OData Version 2.0 is based on HTTP, SAP NetWeaver Gateway also has to respect and implement the IETF RFCs according HTTP. Instead today it’s not possible to use e.g. correctly encoded OData system query options (which are URL query parameters) like %24top ($top) or %24skip ($skip). %21deltatoken (!deltatoken).

How to fix this NOW?

The best way I can think of now to fix this (until SAP hopefully will fix this!) is to make the following enhancement at the end of the private method `INIT_REQUEST` in class `/IWFND/CL_SODATA_PROCESSOR`:


ENHANCEMENT 1  /Z_ENH_FIX_PARSING.    "active version
  DATA: lt_uri_parameter TYPE TIHTTPNVP.
   FIELD-SYMBOLS: <fs_ls_uri_parameter> LIKE LINE OF lt_uri_parameter.

   CALL METHOD mo_context->get_parameter
   EXPORTING
     iv_name = /iwfnd/if_sodata_types=>gcs_iwf_context_parameters-query_parameters
   IMPORTING
     ev_value = lt_uri_parameter.

 * ENHANCEMENT for url query parameter unencoding bug (fixing only for deltatoken parameter by now):
 * =================================================================================================
 * The problem is that all parameter names (and values) in `mo_context->get_parameter(
 * /iwfnd/if_sodata_types=>gcs_iwf_context_parameters-query_parameters` are
 * still encoded and therefore parameters with correctly encoded parameter names don't
 * get used by SAP Gateway. Example of SAP Gateway's URL query parameter parsing code
 * from above:
 *
 * | READ TABLE lt_uri_query_parameters INTO ls_dollar_parameter
 * |   WITH KEY name = /iwfnd/if_sodata_types=>gcs_uri_parameters-delta_token_tmp.
 *
 * Problem in code above:
 * ----------------------
 * `/iwfnd/if_sodata_types=>gcs_uri_parameters-delta_token_tmp` is '!deltatoken'
 * and the parameter name in `lt_uri_query_parameter` is '%21deltatoken' (still encoded).

   READ TABLE lt_uri_parameter ASSIGNING <fs_ls_uri_parameter> WITH TABLE KEY name = '%21deltatoken'.
   IF sy-subrc = 0.
     FIELD-SYMBOLS: <fs_ls_export_parameter> LIKE LINE OF et_parameter.

     APPEND INITIAL LINE TO et_parameter ASSIGNING <fs_ls_export_parameter>.

     <fs_ls_export_parameter>-name = /iwfnd/if_sodata_types=>gcs_uri_parameters-delta_token.
     <fs_ls_export_parameter>-value = cl_http_utility=>unescape_url(
         escaped = <fs_ls_uri_parameter>-value
       ).
     SHIFT <fs_ls_export_parameter>-value RIGHT DELETING TRAILING `'`.
     SHIFT <fs_ls_export_parameter>-value LEFT DELETING LEADING ` '`.

   ENDIF
ENDENHANCEMENT..






Now you just need to add all the other the parameter names you need to be parsed or come up with a generic implementation for this. No generic approach is needed as only `!deltatoken` is affected by now. Take a closer look at my first update to this post if you want to know why only `!deltatoken` is affected.

Looking forward to your thoughts on this in the comments. Bye! 🙂

Update 1 on 15.04.2015:

As Andre pointed out in his comment, not all system query options are affected. I just debugged some requests and found out that this is because not all system query options are treated equally:

The first time during the processing of an request by SAP NetWeaver Gateway the url query parameters get extracted into the context object, is in method `DISPATCH` in class `/IWFND/CL_SODATA_ROOT_HANDLER` starting line 167:


*-set parameters - not exposed in read_entry method therefore tunnel via context
   lt_query_parameters = io_request->get_uri_query_parameters( ).
   io_context->set_parameter(
     EXPORTING
       iv_name  = /iwfnd/if_sodata_types=>gcs_iwf_context_parameters-query_parameters
       iv_value = lt_query_parameters
   ).


Now the `io_context`‘s attribute `mt_parameter` looks as follows:

NAME VALUE
IWFND/QUERY_PARAMETERS all query parameters encoded in a standard table type TIHTTPNVP
<other parameters set to the context>

Some lines of code later in method `DISPATCH` in class `/IWCOR/CL_DS_HDLR_ROOT` starting line 43 :


lt_parameter = io_request->get_uri_query_parameters( iv_encoded = abap_false ).
lo_uri = /iwcor/cl_ds_uri_facade=>parse_uri(
         io_edm = lo_edm
         iv_resource_path = lv_resource_path
         it_query_parameter = lt_parameter
       ).


Now the `io_context`‘s attribute `mt_parameter` looks as follows:

NAME VALUE
IWFND/QUERY_PARAMETERS all query parameters encoded in a standard table type TIHTTPNVP
~uri

all query parameters decoded in a structure containing only one component:

It’s named `INSTANCE` and is a `REF TO /IWCOR/CL_DS_URI`.

<other parameters set to the context>

The processing is going on and then in method `/IWCOR/IF_DS_PROC_ENTITY_SET~READ` in class `/IWFND/CL_SODATA_PROCESSOR` starting line 63 the method `INIT_REQUEST` mentioned in my orginal post yesterday gets called:


* Initialization
       me->init_request(
         EXPORTING
           iv_operation         = mcs_operations-read
           iv_operation_type    = mcs_operation_types-feed
           io_entity_set        = io_entity_set
           it_key               = it_key
           it_navigation_path   = it_navigation_path
           it_expand            = it_expand
           it_select            = it_select
           io_filter            = io_filter
           io_orderby           = io_orderby
           iv_skip              = iv_skip
           iv_top               = iv_top
           iv_format            = iv_format
           iv_for_ds_operation  = iv_for
           iv_skiptoken         = iv_skiptoken
           iv_inlinecount       = iv_inlinecount
         IMPORTING
           es_request           = ls_odata_request
           et_parameter         = lt_dolar_parameter
           eo_target_entity_set = lo_target_entity_set
       ).


Looking up the stacktrace the variables `iv_skip`, `iv_top`, … get populated using the corresponding attributes from the instance of `/IWCOR/CL_DS_URI` stored in the `io_context`‘c `mt_parameter` table. Because the parameters handed to the constructor of `/IWCOR/CL_DS_URI_FACADE` (the class actually referenced to in `mt_parameters`, subclass of `/IWCOR/CL_DS_URI`) were correctly decoded, the constructor could detect them as system query options and stored the values in it’s corresponding attributes.

Now we take a look in method `INIT_REQUEST` in class `/IWFND/CL_SODATA_PROCESSOR` if there is any code accessing the still encoded parameters stored in `mt_parameters`‘s `IWFND/QUERY_PARAMETERS` (Note: I added comments including the line numbers at which the code snippets can be found in the method):


* NOTE: line 455
* Get uri parameters
     mo_context->get_parameter(
       EXPORTING
         iv_name  = /iwfnd/if_sodata_types=>gcs_iwf_context_parameters-query_parameters     " Name
       IMPORTING
         ev_value = lt_uri_query_parameters
     ).

 * Search
     CLEAR ls_dollar_parameter.
     READ TABLE lt_uri_query_parameters INTO ls_dollar_parameter
       WITH KEY name = /iwfnd/if_sodata_types=>gcs_uri_parameters-search.
     IF sy-subrc EQ 0.
* NOTE: Setup search
     ENDIF.
* NOTE: line 478
* startIndex for search
     CLEAR ls_dollar_parameter.
     READ TABLE lt_uri_query_parameters INTO ls_dollar_parameter
       WITH KEY name = /iwfnd/if_sodata_types=>gcs_uri_parameters-startindex.
     IF sy-subrc EQ 0.
       INSERT ls_dollar_parameter INTO TABLE et_parameter.
     ENDIF.
* delta token
     CLEAR ls_dollar_parameter.
     READ TABLE lt_uri_query_parameters INTO ls_dollar_parameter
       WITH KEY name = /iwfnd/if_sodata_types=>gcs_uri_parameters-delta_token_tmp.
     IF sy-subrc EQ 0.
* NOTE: Setup delta token
     ENDIF.
* NOTE: line 512
* totals
     CLEAR: es_request-technical_request-totals.
     READ TABLE lt_uri_query_parameters INTO ls_dollar_parameter
         WITH KEY name = /iwfnd/if_sodata_types=>gcs_uri_parameters-totals.
     IF sy-subrc = 0.
* NOTE: Setup totals
     ENDIF.


So the following four parameters wouldn’t be recognized if their encoded representation is different to the decoded one:

NAME VALUE
/iwfnd/if_sodata_types=>gcs_uri_parameterssearch search

/iwfnd/if_sodata_types=>gcs_uri_parametersstartindex

startIndex
/iwfnd/if_sodata_types=>gcs_uri_parameterstotals totals
/iwfnd/if_sodata_types=>gcs_uri_parametersdelta_token_tmp !deltatoken

Now we can clearly see why only the parameter `!deltatoken` is affected. It has nothing to do with the fact that it’s using a ‘!’ instead of the other system query options which are using a ‘$’. It’s just a coincidence 🙂

How would I fix this, working at SAP?

I would add `deltatoken` as variable to `/IWCOR/CL_DS_URI` which gets extracted from the decoded parameters during object creation. Then add the import parameter `iv_deltatoken` to the method `INIT_REQUEST` in class `/IWFND/CL_SODATA_PROCESSOR` and use this variable in there instead of implement my own parsing using `READ TABLE`.

I would NOT change the value of `io_context`‘s `mt_parameter`‘s `IWFND/QUERY_PARAMETERS` from encoded parameters to decoded parameters as it can’t be definitely said that this wouldn’t cause side effects in other SAP coding or already existing customer coding!

Note: All line numbers were fetched on SAP_GWFND 740 SPS 09

To report this post you need to login first.

3 Comments

You must be Logged on to comment or reply to a post.

  1. Andre Fischer

    Hi Manfred,

    I checked the behavior shortly and found that there seems to be only a problem with encoding the “!” in the URL where a deltatoken is used whereas the encoding of the ” ‘ ” works fine. Also the encoding of the ” $ ” sign in a normal filter statement works.

    I tested the following statements with the service described in my blog How to Implement Basic Delta Query Support in SAP NetWeaver Gateway

    /sap/opu/odata/SAP/ZPRODUCT_SRV/ProductSet_DQ?%21deltatoken=’005056B22F131EE4B8E4B82C6A6B1559_20150415041326′

    –> doesn’t work

    /sap/opu/odata/SAP/ZPRODUCT_SRV/ProductSet_DQ?!deltatoken=%27005056B22F131EE4B8E4B82C6A6B1559_20150415041326%27

    –> does work

    /sap/opu/odata/SAP/ZPRODUCT_SRV/ProductSet?%24filter=Price eq 1249.0000

    –> does work as well

    So what you found is that in one (special) case the encoding doesn’t work correctly.

    So it seems rather to be a bug that should be fixed rather than a problem with correctly encoded URL parameters in general.

    Best Regards,

    Andre

    (0) 
    1. Manfred Scheiner Post author

      Hi Andre,

      thank you for your tests! I took a deeper dive into this, refined the title of the blog post and updated my post with more details 🙂

      (0) 

Leave a Reply