BPath. Lesson 06. Unstructured returns, sorting and performance considerations

[former_member193368]

<<05: Sub, String, Get-function(s), global variables

07: Model Check Mode, Exceptions and longtext>>

Last BPath lesson was presented some time ago. The performance investigations took slightly more time then expected, mainly due to other work. But let us concentrate on Bpath and the current lesson.

BPath. Lesson 05. Sub, String & Get-function(s) plus global variables brought us a view over quite a lot of functions. This time we have only limited content in respect to new functionality but a bigger chapter in respect to performance.

Unstructured Returns & Sorting

To allow easy access to the data we introduced the possibility to work on unstructured return types, such as integer values and strings. The rule is simple, if the declared return type is not structured; every write access to it is treated as writing to the value itself (irrespective of the target name used). Read access is not supported (use global vars instead).

The following example returns the index of the first passenger beginning with "S".

~INT2/SearchResFlightRel/FlightBookRel[@PASSNAME~="S*"]{!RETURN=INDEX()}
Code Example 30, ~INT2 or ~STRING, unstructured return data

Another useful feature is the possibility to sort the result table according to your needs. The feature includes the sorting of sub query result tables. Sorting is treated as feature of the structure definition. This means, in case you want to sort the result the structure has to be stated explicitly. Dynamic structure definitions as ~* are allowed. The sorting field names are added encapsulated by [ ... ], the sort-order ('A' or 'D') might be added as option triggered with a colon (default is ascending order). 'T' option can be used to sort data as text. The star operator is used to indicate the current order, this means, if a star is part of the sort order definition, stable sorting is done.

~*CRMS_BOL_UIF_TRAVEL_BOOK_ATT[!FORCURKEY,!FORCURAM:AT,*]/SearchResFlightRel{!DS:=SUB(~*[!CONNID:D]/FlightConnectionRel/*$)}/FlightBookRel/*$
Code Example 31, ~*[!FIELD:AT,*], sorting

The main sorting is done on the foreign currency (ascending, since sort order is not specified) and then the amount (in foreign currency), ascending as text. Sorting is done stable.

The inner sort order is just for display purposes - the sub query returns anyway only one entry in the table (as every flight has only one connection).

The sorting is done at the very end of the evaluation. The sorting used for filtering, assignments, etc remains as defined by the underlying relations.

Direct Indexing

Up to version 2.1 (a BPath version not released to customers) any filter condition which returned a numeric value (e.g. n= 2 or n=-1) was implicitly evaluated with a check against the current index (means [ i =index()]). This is theoretically fine, but results in a very poor performance in case the numerical value does not depend on the attribute values.

With the released version any filter returning a numeric value is directly indexed, in case it fulfills certain conditions. Directly indexed means that only the requested entity is evaluated, all other datasets are ignored and not further processed.

The conditions are:

• no access to any attribute in the filter
• no subquery in the filter
• no assignment block between relation and filter

The result of any query should not change, but the performance should be significantly better in case direct indexing can be used within large collections.

Please note that a negative indexing refers to entities counted from the end of the list, -1 is the last object in the collection. The syntax does not allow to use negative literals directly, [-1] has to be specified as [0-1].

A rather crude example:

~STRING/SearchResFlightRel{!!Dummy=0-1}/FlightBookRel[!!Dummy]/@PASSNAME$
Code Example 32, [0-1], direct indexing

Performance Evaluations

Unfortunately it is not easy to discuss performance seriously, because of the following reasons:

• Absolute performance is highly machine dependent.
• The BPath execution is dependent on the performance of the used relations, which can vary a lot.
• Caching effects have to be encountered.
• Even if the performance of a BPath statement execution might be pretty fast in terms of "doing all things it does pretty fast", but it may trigger actions which can be avoided if a different way would be chosen.
• It is difficult to judge what "an optimal" performance would be, to which process we compare to. Practically all code examples can be further optimized, especially when you do special coding ignoring standard layers.

The following simple evaluations were done in one of our systems, the goal of the investigations was to have an idea which performance you might expect and which guidelines may be considered. The machine where I did the tests was running in a solaris zone (virtual machine) with 2.6 GHz and 10Gb of RAM. It is a non-Unicode system.

See below the table of results. The first column contains the BPath query, which was always executed against a collection of 10 "flights". The "result" column contains the number of datasets returned, whereas the "touched" column also contains all entities which were touched during execution. The "Frst" column holds a checkbox whether we measure the complete execution ("X") or whether we measure without the startup time (means we reuse the parser). Only the first example is measured with startup performance, but it is anyway only about 3 ms.

The next column holds the needed time for the execution in microseconds, followed by the time required by the parser plus the stack.

The second last column contains the percentage of time elapsed in BPath compared to the time elapsed in the lower layers (GenIL, application specific coding as relation loading) whereas the last column simply holds time in BPath divided by number of touched entities. In other words the time required for BPath to evaluate an object lying on the execution path.

Performance Learnings:

1. Start up performance is about 3ms (can be avoided with parser reuse except for the first call).
2. An easy BPath statement needs about 300 micro seconds per object it accesses during the execution.
3. Complexity costs. In case the BPath statement is formulated with more complexity, e.g. with the usage of subs and functions, execution time rises, e.g. to 2500 micro seconds with the last example.
4. Typical usages which evaluate at least one real relation and assemble data into a result table and work on uncached data need about 20-40% of the complete time within BPath itself. At least within the implementation of our Flights.
5. Direct indexing can pay off as with the eighth example which needs substantially less time in BPath itself.
6. Scaling should be fine, there seem to be no effects ruining the performance with big collections.
7. Stack processing costs a significant proportion of time, there might be room for improvement (on the cost of maintainability unfortunately).

Well, all these evaluations are done against "Flights", which are kind of artificial demo object structure. It is difficult to judge, how it looks when we work on real life objects.

So I did some more tests in another system, which contains better data. System characteristica are comparable.

My demo program reads a list of contacts with the BuilContactAdvancedSearch query where with telephone covers '0*' and country is 'DE' to avoid working on irregular data. Then my BPath query simply tries to collect all addresses with the following query:

  ./BuilContactPersonAddressRel/*$

The query returns 33 entries, exactly one address per contact.

An execution of the BPath requires 683 ms against uncached data with 25.5 ms in our BPath code itself. This makes it to 0.38 ms per touched entity (we have 2*33  = 66 entities) within our BPath classes.

On cached data the whole execution time shrinks down to 32 ms, still with 25.5 ms in our BPath code. This means in this case Bpath directly causes for about 80% of the required time.

So, in the end this example fits to the results from the table above and I hope you now have an idea about the performance you can expect from a BPath query.

That's it for today. Lesson 7 will cover the model check mode and parser reuse plus some small language topics as attributed parent relations and attribute longtext.