ABAP: Fast JSON Serialization
tldr; Install https://github.com/timostark/abap-json-serialization and enjoy the fastest possible JSON serialization. The result will be a 10x faster JSON serialization and deserialization compared to /UI2/CL_JSON at the same quality. Be warned though: Read the limitation section first.
Oh no – another JSON Serialization Blog Post? Hey – At least no blog about excel exports 🙂
So why are we in need of a “new” way for JSON serialization? The reason is simple: Runtime! Especially when working with custom REST services with a big payload you will notice a lot of runtime getting lost in JSON serialization. Loosing 30% of your runtime in JSON serialization makes me very unhappy (when I just optimized my more difficult business class).
So, what are our goals:
- Support Camel-Case
- Support real booleans and numbers
- Not need to be easy or generic (I will accept a bad life as a developer if it is fast and reliable).
There are already multiple solutions out there – just to mention the most important ones:
- /UI2/CL_JSON (plain ABAP)
- CALL TRANSFORMATION (Kernel solution, using XSLT or Simple Transformation Mapping)
- CL_TREX_JSON_SERIALIZER (plain ABAP)
- XCO_CP_JSON (S/4 Only, not stable, extremely slow and crashes – therefore let’s keep that one out for the moment)
So how are they behaving from runtime perspective. Let’s take a very simple example and serialize 5.000 lines of SFLIGHT lines and a very complex and deep structure:
So what does that tell us?
Not really surprisingly the only feasible solution on a ABAP stack is the usage of CALL TRANSFORMATION – as this is executed directly in the Kernel, thus not depending on slow ABAP String concat and/or field-symbol traversal.
It might be strange but always remember: Building up strings using concats and traversing over field-symbols inside a structure is very slow in ABAP compared to native languages –> Where possible Kernel Modules like Simple Transformations are preferable performance wise.
There are however quality problems when using CALL TRANSFORMATION ID:
- No Camel-Case
- No real “booleans” (instead ‘X’ is printed.. tell that somebody outside of the SAP world)
- No real NUMC (instead leading 0s are printed)
There is one solution which was already mentioned in a blog post, using a custom ABAP transformation to at least support camel case. Unfortunately, that throws away the performance benefit as the fast kernel module has to go up to the ABAP stack for a simple “to-camel-case” transformation.
My suggested solution is, that we use CALL TRANSFORMATION for what it is actually thought: to transform data using ST transformations (Simple Transformation). Remark: CALL TRANSFORMATION can also be used for XSLT Transformation (which are much more powerful but also slower – see remark by @Sandra Rossi), but this is simply not required here. This means we are creating an own Simple transformation for the structure/table-type we want to serialize (nested structures are of course possible).
Let’s see an example transformation for the table SFLIGHT (shortened):
Nobody wants to write that code (and for sure nobody with a right mind will want to keep that transformation up to date) – but let’s first see the runtime impact.
==> The solution is around 10 times faster than /UI2/CL_JSON, while having the same quality as a result.
As already said of course nobody wants to write these ST mappings – especially for deeply nested structures this is horrible.
Therefore, I’ve published a small helper program ZJSON_TO_XSLT under MIT license to GitHub which allows you to directly create those transformation for any structure/table
Output (next to the generated transformation).
Execute the transformation using normal CALL TRANSFORMATION call:
DATA(lo_writer_json) = cl_sxml_string_writer=>create( type = if_sxml=>co_xt_json ). CALL TRANSFORMATION ZSFLIGHT SOURCE root = lt_flights RESULT XML lo_writer_json. DATA(lv_json) = cl_abap_codepage=>convert_from( lo_writer_json->get_output( ) ).
In my customer projects I am using the API called in the program in a regular job (including a mapping-table) which updates the transformations on the development system in a regular manner. If you want to spend a lot of time you could even create the transformations “live” as local objects on the first access. I personally do not like the approach of local development objects though.
==> Using JSON serialization with fixed transformations, you can get an extremely fast JSON serialization and deserialization while still having high quality.
- A big word of warning: The solution is thought for performance critical development. The solution comes with very high costs: you have to think of an additional development object (the transformation). Even if it updates automatically, it can get out-of-date, you can forget it or it can get corrupted.
- Summed up: If you do not have a problem (i.E. customer complains about slow type-ahead, where you need response times in ms) do not create additional problems using a more complex solution mentioned here.
- The solution works as long as you know the exported JSON types upfront ( i.E. have static data-types). For dynamic data-structures this solution will not work.
Love the irony of not-another-Excel blogpost 🙂
I've added some irony about "Excel read/write requirement in ABAP" in my new article on heise, too 🙂
Excellent! Your blog post is the first one I see to inform about the efficiency of Transformations for high data volume versus SAP classes.
Thank you for the tool, it will be very helpful. Until now, if the people wanted to use transformations for JSON, they had to understand both ST and JSON-XML format (elements object, str, etc.) and so had good reasons to not use them.
Note that the language you used is not XSLT, it's Simple Transformation AKA ST. XSLT is much more powerful but is much slower (ratio roughly 1 to 10 based on personal tests ~10 years ago). ST is also very powerful compared to XSLT as it's able to transform in the 2 directions (ABAP to XML/JSON and XML/JSON to ABAP) with the same code (with complex XML/JSON, some ST rules are to be indicated to apply specifically to either serialization or deserialization).
NB: both transactions STRANS or XSLT_TOOL are the same and can be used to maintain transformations in both languages, better use STRANS so that people stop confusing XSLT and Simple Transformation.
Thanks Sandra. You are absolutly right, that XSLT mapping is the wrong word. I've replaced it with "Simple-Transformation" everywhere (besides in the code - will do that over time 🙂 ), based on your feedback.
I tested XSLT initially (there are some hacks on stackoverflow to actually get camelCase running without a hard-coded transformation using plain xslt), but it is just much slower as the hardcoded approach (as you said).
thanks for sharing. To my knowledge this is the same approach when You activate an SEGW service. During that based on the model (entitytype) it is generating a transformation. I always wanted to have a feature or tool like this, but never had the possibility to understand it in the deep jungle of the standard code. This tool can be used in many context, to speed up integration developments.
This is great, thank You!