Annotated links: Episode 45 of Hands-on SAP dev with qmacro

This is a searchable description of the content of a live stream recording, specifically “Ep.45 – Cool initiatives and projects with CAP with Tom Jung” in the “Hands-on SAP dev with qmacro” series. There are links directly to specific highlights in the video recording. For links to annotations of other episodes, please see the “Catch the replays” section of the series blog post.

This episode was streamed live on Wed 27 Nov 2019 and is approximately 60 minutes in length. The stream recording is available on YouTube.

Brief synopsis: In this episode Tom Jung joins us and show us some of the cool stuff he’s been hacking on in the CAP area. Tom has always got some fascinating tech to share, so make sure you tune in!

00:02:40 Screen sharing fail – I’ll get the hang of this live streaming eventually! What I wanted to share on my screen was Project Euler which is a great set of math-related puzzles to riff on when you want to practice programming in a new language or improve prowess in an existing language.

00:04:15 Switching to show Tom Jung‘s screen and video ready for the main part of this particular episode. Talking about timezones, and that we try to do the occasional Wednesday afternoon (for me) live stream to catch the US timezone based folks like Tom.

00:06:20 Tom begins to describe the project he’s going to show us, by starting with what it looks like from the user perspective (a new way to discover and consume the SAP product roadmap documents). A much more attractive, interactive and intelligent frontend – kudos!

It’s powered by HANA and CAP on the backend, and by React on the frontend. Internally it also uses a Fiori Elements powered UI for administration and maintenance of the data.

00:09:00 Tom learned a lot doing this project because it was a real one, with specialised security, circuit breaking, blue-green deployment, and many other moving parts. But the particular aspect Tom is about to show is the search capabilities, which are powered by HANA’s fuzzy search and text analytics, and tied together in CAP. In answer to a question from mayjay139, Tom says that the project took around 4-5 weeks all told (plus a lot of design work that was done up front).

00:12:45 Tom picks out an OData operation in the Chrome Developer Tools that represents the search that he just carried out in the UI (“SuccessFactors UX harmonization”), and digs into the query parameters. This is the magic that the team wanted to achieve – use standard OData protocol aspects but provide a rich search experience from HANA, including search scoring, snippets, and weighted columns (which means, for example, that a search for “suc” will favour results like “SuccessFactors” higher than regular words like “such”).

00:16:05 Moving from the UI to the backend, to explore how things work in this regard. Tom has also prepared a cut-down version that can be shared – this is available in his cap-fuzzy-search repo on GitHub. Thanks Tom!

00:17:00 Looking first at how the $search initiated process works, i.e. which fields are to be used when searching for the string supplied. This is done via an annotation, as Tom shows us here (@Search.defaultSearchElement).

00:18:15 There are a lot of annotations used in this project, and Tom has a separate file for annotations relating to the Fiori Elements frontend (in fiori-annotations-cat-service.cds).

00:18:30 Ordering search results by score goes hand in hand with fuzzy search, as you want to be able to rank the results and reflect that in the display. To have this work, Tom created a virtual column which means it’s not persisted in the database, but it’s available in the output and in the custom handler related to the service. There are virtual columns for score and titleSnippet properties.

There are different approaches for handling virtual or computed properties – see for example the post “Computed field example in CAP“, but this virtual property at the entity level is a nice approach because it then appears in DB views for reporting.

00:21:00 Tom points out that he’s not using the standard managed aspect from @sap/cds/common, but a custom set of properties to address some issues they had with DateTime data type quite correctly, relating to the V4 -> V2 adapter.

00:21:50 Tom now talks about the full text indices he created, and shows us some of the HANA configuration files for those (also including the turning on of text analysis and mining). HANA is also able to pull out product names and recognise them as such. What is surprising is the large number of tables that are generated as a result of this (which can be selected upon directly too, which is nice).

00:28:00 Starting to look at the custom handler, the “exit”, in srv/cat-service.js, and noting how easy it is to handle error scenarios, by just calling reject on the request object and supplying an appropriate HTTP status code, like this example:

req.reject(409, `Invalid Status Value: ${req.data.status}.`)

(By the way Tom, I’ve only just noticed, while annotating this recording, that you aren’t using semicolons – kudos! :-))

00:30:00 Noting that in handling specific operations, it’s also very simple to turn around and make one’s own extra calls to the database, using the philosophy of abstraction and myriad Domain Specific Languages (DSLs) – see the DSL section of the post “Tech Aloud podcast – an introduction” for a little more on that, too.

00:32:25 Starting to focus in on the handling of the search itself, with the code in this section:

this.on("READ", "Roadmap", async (req, next) => {
  ...
})

The fascinating thing here is that what’s going on is just a subtle massaging of the in-flight call, again using the power of CAP in the form of the mutable programmatic structure of that request. If you want to learn more about this, have a look at the tutorial “Add Service Logic to Consume Remote Address Data which does something similar (this tutorial is part of the mission “Use CAP and SAP Cloud SDK to Extend S/4HANA).

00:36:20 A short discussion on the fuzziness percentage value, and the fact that it took a while to adjust to the right value (0.7 in this case) to get the best results. Talk about 80/20 effort – the crazy thing is that it took Tom far longer to get that value right than it did to write the entirety of the on handler!

00:39:25 We turn at this stage briefly to the “TOKEN SEPARATORS” value in the fulltext indexing configuration, which had a set of default values that tripped Tom up first of all – the backslash and forward slash symbols were included by default, but this meant that the string “S/4” wasn’t being recognised as a viable result in the fuzzy search! Removal of the slashes from the list of characters fixed this. Nice tip!

00:40:33 Switching over to his cut down version on GitHub, where he has a reduced set of entities and properties, but at least it shows the query exit, which is super helpful. Noting at this point that req._ gives access to the whole raw request – which can be extremely useful.

00:43:00 Having set a breakpoint, Tom triggers a search to have a look what goes on inside a select, in debug mode, including what CDS generates for us.

00:47:00 A short discussion on CQL and the power it gives you when handling the request yourself – instead of parsing SQL you’re just manipulating objects, while Tom fights gently with his HANA backend.

00:48:20 Asking a question about Tom’s experience with building the React frontend using the OData service, and how that went. This led to a very fascinating conversation about how it was simply treaded as “yet another REST API” via the Swagger API interface documentation that Tom generated (via the library ‘odata2openapi’), by taking EDMX and converting it to Swagger, and which we look at now.

00:53:50 We approach the end of this live stream by a lovely conversation about fonts and ligatures. Tom is using Cascadia Code, and has turned on ligatures in VS Code. (I’ve been recently experimenting with ligatures too, as I was bowled over by how beautiful the code looked in this video on Elm: Advent of Code 2018: Day 1, Part One).

00:55:50 Based on a question by Srikanth Peri we confirm that Swagger also works with OData V2, which we can generate with the OData V2 Adapter Proxy, which you can also explore with another tutorial in the mission mentioned earlier: “Install an OData V2 Adapter“. In fact, Tom briefly shows the Fiori Elements based UI that is based on OData V2 via that very proxy.

That brings us to the end of this episode, thanks again so much to Tom Jung for coming on and showing us this stuff. Until next time!