SAP NetWeaver Gateway violates IETF RFC 3986 => How to fix parsing of correctly encoded OData system query option '!deltatoken'

[scheinem]

What happened?

Today I tried to make a request to one of my SAP NetWeaver Gateway services that looked like this:

http://<gateway-url>:<gateway-port>/sap/opu/odata/<my-service>/<my-entityset>?%21deltatoken=%27token%27

But when I saw the response I wondered why the delta mechanism of my service wasn't working. So I started to debug the request and I noticed that in the method `/iwbep/if_mgw_appl_srv_runtime~get_entityset` of my data provider class `io_tech_request_context->get_deltatoken( )` returned an empty deltatoken (`rv_deltatoken` was initial).

First I thought I made a mistake and it's not allowed to escape URL query parameter names, but when I started research I came across an useful answer on a github issue which pointed me to IETF's RFC (Request for Comments) 3986 named "Uniform Resource Identifier (URI): Generic Syntax). Section 2.2 of this RFC points out:

2.2. Reserved Characters

...

  If data for a URI component would conflict with a reserved character's purpose as a delimiter, then the conflicting data must be percent-encoded before the URI is formed.

...

reserved = gen-delims / sub-delims

gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"

sub-delims = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "="

quotation from IETF RFC 3986

What does that mean?

As SAP Gateway officially uses/supports the OData Version 2.0 protocol and OData Version 2.0 is based on HTTP, SAP NetWeaver Gateway also has to respect and implement the IETF RFCs according HTTP. Instead today it's not possible to use e.g. correctly encoded OData system query options (which are URL query parameters) like %24top ($top) or %24skip ($skip). %21deltatoken (!deltatoken).

How to fix this NOW?

The best way I can think of now to fix this (until SAP hopefully will fix this!) is to make the following enhancement at the end of the private method `INIT_REQUEST` in class `/IWFND/CL_SODATA_PROCESSOR`:

ENHANCEMENT 1  /Z_ENH_FIX_PARSING.    "active version

  DATA: lt_uri_parameter TYPE TIHTTPNVP.

   FIELD-SYMBOLS: <fs_ls_uri_parameter> LIKE LINE OF lt_uri_parameter.



   CALL METHOD mo_context->get_parameter

   EXPORTING

     iv_name = /iwfnd/if_sodata_types=>gcs_iwf_context_parameters-query_parameters

   IMPORTING

     ev_value = lt_uri_parameter.



 * ENHANCEMENT for url query parameter unencoding bug (fixing only for deltatoken parameter by now):

 * =================================================================================================

 * The problem is that all parameter names (and values) in `mo_context->get_parameter(

 * /iwfnd/if_sodata_types=>gcs_iwf_context_parameters-query_parameters` are

 * still encoded and therefore parameters with correctly encoded parameter names don't

 * get used by SAP Gateway. Example of SAP Gateway's URL query parameter parsing code

 * from above:

 *

 * | READ TABLE lt_uri_query_parameters INTO ls_dollar_parameter

 * |   WITH KEY name = /iwfnd/if_sodata_types=>gcs_uri_parameters-delta_token_tmp.

 *

 * Problem in code above:

 * ----------------------

 * `/iwfnd/if_sodata_types=>gcs_uri_parameters-delta_token_tmp` is '!deltatoken'

 * and the parameter name in `lt_uri_query_parameter` is '%21deltatoken' (still encoded).



   READ TABLE lt_uri_parameter ASSIGNING <fs_ls_uri_parameter> WITH TABLE KEY name = '%21deltatoken'.

   IF sy-subrc = 0.

     FIELD-SYMBOLS: <fs_ls_export_parameter> LIKE LINE OF et_parameter.



     APPEND INITIAL LINE TO et_parameter ASSIGNING <fs_ls_export_parameter>.



     <fs_ls_export_parameter>-name = /iwfnd/if_sodata_types=>gcs_uri_parameters-delta_token.

     <fs_ls_export_parameter>-value = cl_http_utility=>unescape_url(

         escaped = <fs_ls_uri_parameter>-value

       ).

     SHIFT <fs_ls_export_parameter>-value RIGHT DELETING TRAILING `'`.

     SHIFT <fs_ls_export_parameter>-value LEFT DELETING LEADING ` '`.



   ENDIF

ENDENHANCEMENT..

Now you just need to add all the other the parameter names you need to be parsed or come up with a generic implementation for this. No generic approach is needed as only `!deltatoken` is affected by now. Take a closer look at my first update to this post if you want to know why only `!deltatoken` is affected.

Looking forward to your thoughts on this in the comments. Bye! :smile:

Update 1 on 15.04.2015:

As andre.fischer pointed out in his comment, not all system query options are affected. I just debugged some requests and found out that this is because not all system query options are treated equally:

The first time during the processing of an request by SAP NetWeaver Gateway the url query parameters get extracted into the context object, is in method `DISPATCH` in class `/IWFND/CL_SODATA_ROOT_HANDLER` starting line 167:

*-set parameters - not exposed in read_entry method therefore tunnel via context

   lt_query_parameters = io_request->get_uri_query_parameters( ).

   io_context->set_parameter(

     EXPORTING

       iv_name  = /iwfnd/if_sodata_types=>gcs_iwf_context_parameters-query_parameters

       iv_value = lt_query_parameters

   ).

Now the `io_context`'s attribute `mt_parameter` looks as follows:

NAME	VALUE
IWFND/QUERY_PARAMETERS	all query parameters encoded in a standard table type TIHTTPNVP
<other parameters set to the context>

Some lines of code later in method `DISPATCH` in class `/IWCOR/CL_DS_HDLR_ROOT` starting line 43 :

lt_parameter = io_request->get_uri_query_parameters( iv_encoded = abap_false ).

lo_uri = /iwcor/cl_ds_uri_facade=>parse_uri(

         io_edm = lo_edm

         iv_resource_path = lv_resource_path

         it_query_parameter = lt_parameter

       ).

Now the `io_context`'s attribute `mt_parameter` looks as follows:

NAME	VALUE
IWFND/QUERY_PARAMETERS	all query parameters encoded in a standard table type TIHTTPNVP
~uri	all query parameters decoded in a structure containing only one component: It's named `INSTANCE` and is a `REF TO /IWCOR/CL_DS_URI`.
<other parameters set to the context>

The processing is going on and then in method `/IWCOR/IF_DS_PROC_ENTITY_SET~READ` in class `/IWFND/CL_SODATA_PROCESSOR` starting line 63 the method `INIT_REQUEST` mentioned in my orginal post yesterday gets called:

* Initialization

       me->init_request(

         EXPORTING

           iv_operation         = mcs_operations-read

           iv_operation_type    = mcs_operation_types-feed

           io_entity_set        = io_entity_set

           it_key               = it_key

           it_navigation_path   = it_navigation_path

           it_expand            = it_expand

           it_select            = it_select

           io_filter            = io_filter

           io_orderby           = io_orderby

           iv_skip              = iv_skip

           iv_top               = iv_top

           iv_format            = iv_format

           iv_for_ds_operation  = iv_for

           iv_skiptoken         = iv_skiptoken

           iv_inlinecount       = iv_inlinecount

         IMPORTING

           es_request           = ls_odata_request

           et_parameter         = lt_dolar_parameter

           eo_target_entity_set = lo_target_entity_set

       ).

Looking up the stacktrace the variables `iv_skip`, `iv_top`, ... get populated using the corresponding attributes from the instance of `/IWCOR/CL_DS_URI` stored in the `io_context`'c `mt_parameter` table. Because the parameters handed to the constructor of `/IWCOR/CL_DS_URI_FACADE` (the class actually referenced to in `mt_parameters`, subclass of `/IWCOR/CL_DS_URI`) were correctly decoded, the constructor could detect them as system query options and stored the values in it's corresponding attributes.

Now we take a look in method `INIT_REQUEST` in class `/IWFND/CL_SODATA_PROCESSOR` if there is any code accessing the still encoded parameters stored in `mt_parameters`'s `IWFND/QUERY_PARAMETERS` (Note: I added comments including the line numbers at which the code snippets can be found in the method):

* NOTE: line 455

* Get uri parameters

     mo_context->get_parameter(

       EXPORTING

         iv_name  = /iwfnd/if_sodata_types=>gcs_iwf_context_parameters-query_parameters     " Name

       IMPORTING

         ev_value = lt_uri_query_parameters

     ).



 * Search

     CLEAR ls_dollar_parameter.

     READ TABLE lt_uri_query_parameters INTO ls_dollar_parameter

       WITH KEY name = /iwfnd/if_sodata_types=>gcs_uri_parameters-search.

     IF sy-subrc EQ 0.

* NOTE: Setup search

     ENDIF.

* NOTE: line 478

* startIndex for search

     CLEAR ls_dollar_parameter.

     READ TABLE lt_uri_query_parameters INTO ls_dollar_parameter

       WITH KEY name = /iwfnd/if_sodata_types=>gcs_uri_parameters-startindex.

     IF sy-subrc EQ 0.

       INSERT ls_dollar_parameter INTO TABLE et_parameter.

     ENDIF.

* delta token

     CLEAR ls_dollar_parameter.

     READ TABLE lt_uri_query_parameters INTO ls_dollar_parameter

       WITH KEY name = /iwfnd/if_sodata_types=>gcs_uri_parameters-delta_token_tmp.

     IF sy-subrc EQ 0.

* NOTE: Setup delta token

     ENDIF.

* NOTE: line 512

* totals

     CLEAR: es_request-technical_request-totals.

     READ TABLE lt_uri_query_parameters INTO ls_dollar_parameter

         WITH KEY name = /iwfnd/if_sodata_types=>gcs_uri_parameters-totals.

     IF sy-subrc = 0.

* NOTE: Setup totals

     ENDIF.

So the following four parameters wouldn't be recognized if their encoded representation is different to the decoded one:

NAME	VALUE
/iwfnd/if_sodata_types=>gcs_uri_parameters-search	search
/iwfnd/if_sodata_types=>gcs_uri_parameters-startindex	startIndex
/iwfnd/if_sodata_types=>gcs_uri_parameters-totals	totals
/iwfnd/if_sodata_types=>gcs_uri_parameters-delta_token_tmp	!deltatoken

Now we can clearly see why only the parameter `!deltatoken` is affected. It has nothing to do with the fact that it's using a '!' instead of the other system query options which are using a '$'. It's just a coincidence :smile:

How would I fix this, working at SAP?

I would add `deltatoken` as variable to `/IWCOR/CL_DS_URI` which gets extracted from the decoded parameters during object creation. Then add the import parameter `iv_deltatoken` to the method `INIT_REQUEST` in class `/IWFND/CL_SODATA_PROCESSOR` and use this variable in there instead of implement my own parsing using `READ TABLE`.

I would NOT change the value of `io_context`'s `mt_parameter`'s `IWFND/QUERY_PARAMETERS` from encoded parameters to decoded parameters as it can't be definitely said that this wouldn't cause side effects in other SAP coding or already existing customer coding!

Note: All line numbers were fetched on SAP_GWFND 740 SPS 09