Conversions in SAP Gateway Foundation – Part 2
In the first blog post of this series we focused on a simple example of an alpha conversion for a property in an OData service. Before we discuss more elaborate features and issues with the built-in conversion facilities of SAP Gateway Foundation we need to understand a few details of the stack.
This post will shed light to some aspects of the service model and the data transfer between the different layers of SAP Gateway Foundation. While we stick to the feature of conversions the explanations foster an understanding of the runtime behavior of SAP Gateway Foundation in other contexts, too.
OData services mainly expose their model information in two resources:
- the service document –
- the metadata document –
The service document describes addressable entity sets and function imports and the metadata document provides details about the entity types / complex types and their (navigation) properties including their defining attributes, and including annotations. A service consumer uses this information to interact with the service.
SAP Gateway Foundation assumes the task to process OData requests translating them into appropriate calls to the data provider (see stack overview below). SAP Gateway Foundation provides a number of features to support the service implementation – one of those being the execution of conversions. We have already seen that the required information (the name of the conversion exit, for example) is provided through the implementation of the model provider class. Or, it is automatically determined when using certain model APIs. This conversion information becomes part of the model, too.
To summarize, the model describes structure and behavior of the service:
- for the service consumer providing views onto the model defined by the OData specification like the service and the metadata document and
- for the SAP Gateway Foundation framework providing views onto the model required to process service requests.
It is important to understand that the service model contains a lot more data that controls the service behavior than those exposed to the service consumer.
The following data is part of the model and directly or indirectly related to conversions but not exclusively used for this purpose:
- The conversion switches discussed in the first blog post of this series
- The conversion exit on property level
- Unit or currency code semantics on property level and references to such properties
- ABAP data type information on property level
While the last bullet point is not that obvious the ABAP data type information is the most complex data set, is of huge importance to conversions, and might be the source of issues and incidents. The data type information is best understood if we look at the data transfer inside SAP Gateway Foundation which will also reveal where conversions are executed.
The framework in SAP Gateway Foundation that processes OData requests is made up of different components and layers. The so-called hub component comprises the OData library and the hub framework layer. The hub component may run on a separate system, the FIORI frontend server, for example. The backend component contains the backend framework layer and delegates the request processing to the service-specific model and data provider classes.
Outbound Data Flow
We will start with an analysis of the outbound data flow, that is, the flow from the provisioning of data in the data provider class to the final http response payload. And, we stick to the example in the first blog post:
To respond to an entity set
GET-request the data provider selects data from database table
VBAK and will probably store this data in an internal table with a similar structure. The data is provided to the framework as a reference to such an internal table. The row structure with the data types involved are defined by the data provider. Therefore, we call this internal table the provider data container.
The framework transfers the content of the provider data container into the internal data container in two steps:
- The data is moved from the provider data container to the internal data container.
- Conversions are applied to relevant fields of the provider data container writing the result of the conversion into the internal data container.
Again, the internal data container is an internal table but the row structure is generated by the backend or hub framework layer at runtime. The data types used are derived from the service model. The ABAP data types of the internal data container need to match the EDM Types of the corresponding properties representing the sales order header in our example. Later on, the OData library serializes the OData response from the internal data container.
Please observe that the internal data container is being generated both in the hub and backend framework layer in case of a hub deployment scenario – the internal data container content is serialized and transferred through an RFC connection between the systems. In case of a co-deployment scenario there is one instance of the internal data container.
The first step of moving the data between the data containers is already a transformation. The containers may be deeply structured when
$expand is used. Therefore, the data is transferred using a serialization/de-serialization with an ID transformation – this process “converts” some of the representations already. With release 7.50 the framework uses the deep move-corresponding (
move-corresponding expanding nested tables) if possible. The tricky point is that the ID transformation differs from the deep
move-corresponding in some details – the framework pre-analyzes the model and decides upon a proper usage to ensure compatibility between the different releases.
In our example, the field
VBELN of the internal data container would still carry the value with leading zeros after the first step.
The second step was the starting point of the whole discussion. So, the provider data container forms the input to the outbound conversions with the result being stored in the internal data container from which the OData response is created.
Now, the field
VBELN contains the value without leading zeros after execution of the alpha conversion.
Inbound Data Flow
For the other direction we need to distinguish between two cases:
- To-be-converted data may be part of the request URI – keys, function import parameters, filter literals
- To-be-converted data may be part of the request payload for
The data flow in the second case is very similar to what we discussed above: the request payload is de-serialized into the internal data container and then transferred to a provider data container including conversion execution on-demand. We will see this in the last chapter.
To-be-converted data from the URI is handled slightly differently. There is no internal data container. Instead, the data is forwarded to the backend framework layer in a string representation. When the data provider tries to retrieve the data it has to provide a proper data container. The string representation is then translated into the expected format including inbound conversion execution.
Retrieving Data from the Framework
Let us see how the data provider accesses the data provided by the framework and how response data is being returned.
We get back to our example of the sales order header from the first blog post of this series. Suppose we need to implement the
GET_ENTITY method of the data provider to retrieve a single sales order header based on its key
The request URI may look like
/…/<service>/SD_HEADER_SET(Vbeln=‘21351’). The data provider class needs to call
data ls_vbak type vbak. io_tech_request_context->get_converted_keys( importing es_key_values = ls_vbak ).
The structure field
LS_VBAK-VBELN would contain value ‘0000021351’, that is, the value after application of the alpha conversion. The original value from the request URI is still available through method
Similarly, function import parameters can be retrieved with method
GET_PARAMETERS delivers the original values from the request URI.
$filter is a very complex and powerful system query option. To reduce complexity we focus on select options although not every filter expression can be translated into select options. A simple (not quite reasonable) example:
/…/<service>/SD_HEADER_SET?$filter=Vbeln eq ‘21351’
data: lo_filter type ref to /iwbep/if_mgw_req_filter, lt_select_option type /iwbep/t_mgw_select_option. lo_filter = io_tech_request_context->get_filter( ). lt_select_option = lo_filter->get_filter_select_options( ).
The structure of a row of
lt_select_option is made up of the name of the affected ABAP field and a range table containing sign, option, and low/high values. The latter values are typed as
STRING and contain the unconverted ABAP literals. For the above example:
|SIGN (CHAR 1)||OPTION (CHAR 2)||LOW (STRING)||HIGH (STRING)|
To get the converted information you need to pick on line in
lt_select_option, that is, the range information for one single field. Then call method
data: ltr_vbeln type range of vbeln_va. lo_filter->convert_select_option( exporting is_select_option = ls_select_option importing et_select_option = ltr_vbeln ).
ltr_vbeln is a range table for the specific document number field. Low and high values are typed with the respective data element. Using the above terminology, it is the (range) provider data container.
GET_OSQL_WHERE_CLAUSE_CONVERT are the two methods at io_request_context that would deliver the
$filter expression as an OSQL WHERE-clause, if possible. Either the literals are delivered in the ABAP format before or after applying the inbound conversions.
As we can see, when the data provider retrieves the “input” information the framework delivers the data into the provider data containers having applied the inbound conversions.
Returning Data to the Framework
The way back has already been described above. Let us focus on a very specific example to illustrate potential differences between the provider and the internal data container and to highlight the importance of the ABAP data type information in the model.
Suppose your entity type contains a language. In most cases, language fields in the ABAP Dictionary are typed with data element
SPRAS, a one-character code – see field
SPRAS in table
T001. Data element
SPRAS refers to domain
SPRAS carrying the conversion exit
ISOLA that translates the one-character language code into the corresponding two-character ISO representation. Please observe that the output length of domain
SPRAS is set to two. The language property in the entity type is probably typed with
Edm.String (max length 2) to expose the ISO language code.
The provider data container will be similar to the table definition in the ABAP Dictionary, that is, the one-character language code is provided to the framework. The internal data container constructed by the framework will contain a two-character representation that can be serialized into the OData response. The framework executes the outbound conversion
ISOLA on the way from the provider to the internal data container. If you would switch off the conversion then the one-character code would appear in the two-character language property.
Now that's what i call an informative blog post! I wish the sap press books on SAPUI5/OData would provide such detailed informations.
Keep up the great work!
Thanks, Marvin. Sometimes it is really difficult to illustrate complex implementation details although they may be required to build high-quality stuff on top of a specific framework like SAP Gateway.
I planned to publish one more article on the whole SAP Gateway Foundation conversion topic centered around currency and measure formatting. I put it off several times because my work focus changed but your comment motivates to go about it.
Hello Thomas Nitschke ,
Thank you for this blog series. This is really an amazing job!
Hope you keep writing such posts related with Gateway.
Thanks, Sergio Fraga
Still, there is one blog post open regarding currency and unit of measure conversion in SAP Gateway Foundation. Hope it gets finalized soon... and is of interest.
Thomas Nitschke : I have followed exactly what you have mentioned. somehow my conversions are not working is your blog only for the field which are referring to data based tables like VBAK as I am currently refererring a custom table type and structre however using the standard data elements .
as long as the structure is defined in the ABAP dictionary, there should be no big difference – the handling is quite similar. Or, did you define the structure as a type inside a class or interface? Here, things might become tricky.
Thomas Nitschke : Okay its a normal SE11 Table type I have realised I am using a generic Select Option as a ouput parameter .Where as i have seen you have used output parameter as RANGES of the VBELN. Thats how it didnot work for me .In this case i have to handle it writing additonal Logic.
Hi again Thomas Nitschke ,
Hope I find you well and wealthy in this strange times!!
I'm having some strange behavior with substringof and startswith options.
In my implementation I'm using the following:
This works as expected and I can receive the internal format of all my properties just fine.
Problem is in the following query (<property_name> is of type AUFNR which has a conv. rout):
and the result of method get_osql_where_clause_convert for this case will be: <abap_property_name> LIKE '911%'
So no conversion is happening here..
If I pass <my_entity>?$filter=startswith(<property_name>, '000000911') then I will have results.
More strange is if I use:
<my_entity>?$filter= not startswith(<property_name>, '911')
Result in this case is: ( NOT <abap_property_name> LIKE '%' ), so when using the NOT, we completely lose the value passed.
If I switch to get_osql_where_clause then everything works fine as long as I pass the internal value which is I want to avoid from the beginning.
Am I doing something wrong or there is an issue with get_osql_where_clause_convert implementation when using substringof, startswith and not options?
Can you advice what should we use/do for this cases?
Thanks a lot for any help you can give
Thanks a lot. Regarding the 'not startswith' use case: no idea, maybe this is worth an incident.
The other topic is not an implementation issue but rather a philosophical problem. With an alpha conversion, startswith('911') would probably translate to: LIKE '911%' OR LIKE '0911%' OR LIKE '00911%' ... and so forth depending on the length of the data type. The database does not know about the conversion exit, hence the SQL statement would need to simulate. But here, such an SQL where clause is probably not what you would like to send down to the database.
I know that it is probably not satisfying but those filter expression do not go really well with conversion exits in ABAP. I am not at work, so I cannot check for more details.
Thank you Thomas Nitschke ,
I have found SAP note 2135785 that states exactly what you have explained regarding the conversions.
For the NOT case, I have opened an Incident. I will update this post once I have some feedback.
Thank you again for your comments
Going deeper in the analysis, problem is in constructor method of class /IWBEP/CL_MGW_EXPR_LITERAL, since attribute /IWBEP/IF_MGW_EXPR_LITERAL~LITERAL_CONVERTED is not filled.
Later on the stack, method /IWBEP/IF_MGW_EXPR_VISITOR~PROCESS_LITERAL of class /IWBEP/CL_MGW_EXPR_OSQL_VISTR is called where we can find the code:
Since /IWBEP/IF_MGW_EXPR_LITERAL~LITERAL_CONVERTED is never initialized, LV_STRING will always be empty....
Thomas Nitschke Great blog! Really nice useful insight.
I have come across a combination that I just cannot find a solution for without coding and can use a little help to find a better solution.
I have implemented a SAPUI5 MultiInput where more than one value can be selected. After calling io_tech_request_context->get_osql_where_clause_convert( ) I get following:
Result: ( ( WERK LIKE '%2616%' OR TPLNR LIKE '%2616-07-2541%' ) OR EQUNR LIKE '%80987416,82198748%' )
Two equipment have been selected and gets returned as a comma string. Problem is that this gives a dump when used as where clause 🙁
Is there a way out? Other than code a method to handle splitting it.
Thanks a lot for any help you can give!
Thanks, Henrik Damhøj Andersen
I guess, this specific case would require a bit of debugging. Difficult to say what is happening here - I am a little puzzled, too.
EQUNR has an alpha conversion assigned. Alpha conversions are inherently difficult when it comes to a translation into a pattern. For example, startswith('1') could mean 1*, 01*, 001*, ...
I have no idea where the comma in the generated SQL clause comes from. Should be analyzed by an expert in support or development.
Thank for the quick reply!
Somehow I'm glad you did not just have the answer right away 🙂
I will look a little further, but it probably ends up with some code to handle it.
You might want to add the relevant piece of the OData request (filter option). Maybe we can take it from there.
Then, it is rather a UI issue. substringof(s, p) is built-in function in OData V2 and returns true when the content of property p contains s as a substring. Therefore, the SAP Gateway translates that part of the request into a LIKE '%s' correctly... s is just taken as it is: one string that contains digits and comma.
For a multi-selection, I would have expected several OR-connected clauses and not a comma-encoding.
Ok, thats also make more sense. Either way I have to convert it 🙂
Think that the json are just passed as is.
Ended up with a pretty simple solution. Now the backend works like a charm 🙂