zmockup_loader: unit test data preparation flow
Hi Community,
This post continues the series about unit test tool zmockup_loader. To recap, it allows to prepare big volumes of test data in excel and then conveniently use it in abap unit tests. The main application cases from our experience are: when you have a lot of test data or when tests are prepared by users/analytics (not by developers). For more details about the tool have a look at my previous articles here, here and here.
This post is dedicated to unit test data preparation flow we use in our team. The context is “a lot of test data”. So let’s consider the example first.
The case
For most complex of our products we have quite a lot of test suites…
… each Excel may have multiple sheets, representing different categories of input and expected test data …
… the test data includes not only Excel tables but some static artifacts e.g. XML files to test output (in the image they are located in the ‘includes’ folder)
The compiler
Just to remind – the final result, consumable by the mockup loader, is a zip file of tab-delimited text files stored in MIME repository in the system. As you imagine, converting the above volume to text would be an ordeal without some automation. So I designed a tool for this, called mockup_compiler.
Here is how the tool works. You have to specify the source directory, optionally the includes directory and the target MIME object name (the MIME object must exist by that time, so the first time you have to create it manually with SMW0 and assign to a proper package).
When you run the tool, it finds all the excels in the directory, converts the relevant sheets to texts, uploads the includes (recursively), packs them all into a zip and, voila, you have ready to use zip slug uploaded as the MIME data.
Watching feature
One of the convenient feature of the tool is watching. During the active process of coding unnecessary actions could be annoying (like pressing exec buttons and waiting for compilation and any other distractions). This breaks the “flow” state, blah blah … you know of course. To address this I implemented the watch feature – after initial compilation the mockup compiler can start polling the source directories for changing and – if detected – incrementally re-upload just the changed files. Long to read, better to see:
Final words
I’d like to send huge regards to Ivan Femia for his abap2xlsx package (which is a dependency of the mockup compiler). Without it the abap implementation would probably not see the light (as well as, I guess, many other good excel-based solutions ;)).
I hope you’ll find the mockup compiler (and loader) useful for your test flows.
The code is open sourced and can be found in this github repository. The best way to install the tool would be the well known abapGit.
There are 2 prerequisite libraries to install. Already mentioned abap2xlsx and abap_w3mi_poller – another tool of mine which is can upload and poll files to MIME objects (but no Excel-to-text conversion, so it is kind of ‘backend’ of the mockup compiler).
P.S. For those who might be interested or who cannot use mockup compiler in-sap for any reason – there is also a java script (nodejs) implementation of the compiler (can be found here). It does the same job except that it cannot upload the slug to SAP. This should be done either manually (smw0) or automatically with the mentioned abap_w3mi_poller.
Thanks for sharing (This and previous posts).
Is there any special reason for using this external mockup “compiler” instead of integrating it natively in the zmockup_loader tool itself
i.e.
1. Support also XLSX files and not only TXT (tab delimited) files.
2. Read files directly from application server instead of MIME repository
?
Interesting questions. All of it has reasons and thoughts behind but I've never formulated them to anyone in a structured way, so let's try 🙂
My idea is to keep the 'loader' lightweight. Now it depends just on text2tab (text parser). The compiler brings in abap2xlsx and abap_w3mi_poller prerequisites. The compiler is a "user" tool whereas the 'loader' is kind of engine - I didn't want to hard link to any big dependencies for the 'loader' (like abap2xslx). Though idea to parse Excels directly had been coming to my mind. Maybe I'll do some optional "bridge" someday (like the test double implementation for stubber in the previous post - it was implemented but optional to use).
Excel vs text. Well... Text is more universal - there can be more "sources" in future. Maybe some web sources. Text takes less size. Text can be cleanly combined in one "catalog" (zip file) and can be easily inspected (downloaded with SMW0) without additional tools if needed. I'm going to publish in-sap viewing/editing tool soon - supporting both text and excel (writing!) would make it more complicated. It is easier to do some mess with excels e.g. sometimes there are millions on empty lines which seriously increase file size just because users are not accurate or not aware of the fact that "files have sizes ..." 🙂 So what I mean is that text is more predictable and reliable.
Files on the server. The main reason here is that test data should be a part of the package (MIME object travels with transports and also supported by abapGit). So in the long run it should not be a separate file but some object inside the system. Actually, the 'loader' supports reading from files on GUI side (see REFERENCE of create method and I_TYPE). It was created more for debugging purposes. But adding support for server files should be rather easy if required. This, however, may raise some security concerns...
Hopefully this clarifies 🙂
These are indeed valid points.
Thanks for the clarification.
P.S.
Dependent objects are a problematic issue (Due to possible syntax errors and/or "too dynamic" code).
One possible solution might be including a separate "wrapper" interface layer (while the implementation class itself is optional and should be added separately).
You mean some kind of plugin which would process specific files (e.g. by extension) differently ? With an optional external class ? Interesting idea. Thanks. I'll consider it for future features. Added it as issue at github repo. 🙂 Fill free to add thoughts (or code) there.