Annotated links: Episode 8 of Hands-on SAP dev with qmacro

This is a searchable description of the content of a live stream recording, specifically “Episode 8 – Using Axios, ES6, promises & pure functions to grab data” in the “Hands-on SAP dev with qmacro” series. There are links directly to specific highlights in the video recording. For links to annotations of other episodes, please see the “Catch the replays” section of the series blog post.

This episode, titled “Using Axios, ES6, promises & pure functions to grab data“, was streamed live on Fri 01 Mar 2019 and is approximately one hour in length. The stream recording is available on YouTube.

Below is a brief synopsis, and links to specific highlights – use these links to jump directly to particular places of interest in the recording, based on ‘hh:mm:ss’ style timestamps.

Brief synopsis

In Episode 6 we set out creating our mini Northwind service, called Northbreeze, and started to write some code to grab the Northwind data, exploring Axios as an HTTP client library that supports promises and (therefore) dot chaining. In this episode we continue on towards completing this code to produce CSV files as input to our CAP model, exploring pure functions along the way.

Links to specific highlights

00:03:15: Reminding ourselves of what we’d been doing on this subject last time: We had created an initial Node.js project with a grab.js script to pull data from the Northwind OData service.

00:04:35: Ronnie shares a fun fact with us – Coffee Corner Radio is partially the reason why there will be an SAP Inside Track in Oslo this year, and it’s on 17 Aug 2019! See the sitOSLO wiki page for more details.

00:04:54: Noting we’d installed the Axios package, an HTTP client that supports the use of promises, and taking a look where we’d got to in the grab.js script, using Axios’s .all() function which allows us to fire off multiple HTTP requests, and which returns a promise.

00:07:30: Adding a function that does effectively nothing just so we can add a breakpoint in VS Code, to inspect what we get from the multiple HTTP calls:

.then(xs => {
  return xs
})

What we get is an array of two objects representing the responses to the HTTP requests. The object is the response data from an HTTP perspective, but also a nicely parsed data object.

00:09:27: Looking at the odata.nextLink based paging mechanism that’s used to return pages of data where the whole data set is deemed to large to return in one go (there are, for example, 77 products).

00:12:33: Noting that I’ve downloaded, into a data directory, the JSON representing the data resources for the entitysets we want to download, so we can store and serve them locally, rather than run any risk of being (rightly) rate-limited or banned for overuse of the public Northwind service.

00:13:40: Based on a question from Christian, we take a brief look at ranger, a terminal based file manager with lots of features including Vim keybindings. My ranger config, and more, is available in my scripts repo on GitHub.

00:16:28: Starting up a quick HTTP server using Python’s http.server module, which is super useful. We use this in the data/ directory to serve up the JSON resources we’ve previously downloaded. Note that the module name changed between Python versions 2 and 3 (it was SimpleHTTPServer with Python 2).

=> python -mhttp.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000)

00:18:00: Starting to edit the grab.js script, changing the baseurl to point to this local HTTP server endpoint.

00:19:15: Noting that my ultimate goal is to be able to work on this sort of stuff on an original VT100 terminal from DEC. I have (or used to have) many terminals, they’re a wonderful thing to collect.

00:19:40: Installing nodemon (which we first saw in episode 0) locally inside this project, and starting it up thus:

./node_modules/nodemon/bin/nodemon.js grab.js

00:22:40: Considering the steps in the code as a ‘dot chain’, a series of steps that are processed one after the other, where the output from one step becomes the input for the next. We explore this with:

.then(xs => xs.reduce((a, x) => a.concat(x.data.value), []))
.then(xs => xs.length)
.then(console.log)

and can see that we get the right “count” of items in the output.

00:24:40: Reminding ourselves of what the first .then(...) call is doing, with the reduce function; basically, here’s an example, from the Node.js REPL:

> stuff
[ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
> stuff.reduce((a, x) => a.concat(x), [])
[ 1, 2, 3, 4, 5, 6, 7, 8, 9 ]

00:28:00: Removing the static list of calls we’ve used so far (axios.get(baseurl + '0') etc) and writing something to generate them instead. In this part, we look again at Python, to briefly explore the difference between range and xrange, noting the difference between the two – one is lazy, the other is not. Laziness is a good thing! BTW there is range and xrange in Python 2, but Python 3’s range is the equivalent of Python 2’s xrange.

00:31:55: Now looking at the equivalent of range in JavaScript (I make a brief reference to Gary Bernhardt’s excellent and very entertaining talk titled “Wat“, on JavaScript), we create a range function like this:

range = x => [...Array(x).keys()]

Lovely!

00:37:00: Starting to replace the static list of get() calls in the all() call by generating the calls dynamically, making use of our new range function.

00:40:00: Slightly annoyed by myself for using a less than perfect Vim motion and action command to start appending at the end of the line. I used $ to get to the end of the line, then a to start appending from there. I should have simply used A which does both those things.

00:41:25: The resulting reworking of the all() call looks like this:

.all(range(4).map(x => axios.get(baseurl + (x * 20))))

00:42:30: Creating a configuration map for the entities (Products, Suppliers, Categories), where we can supply the number of skipTokens required to retrieve everything (and implicitly, via the keys, have a list of the entities that we need).

00:44:50: Starting to turn the main call into a function grab that we can then make calls to later. The function will return a promise.

00:46:48: Now we start to flesh out the call to grab like this:

grab('Products')
  .then(xs => xs.length)
  .then(console.log)

(For the observant amongst you, I didn’t notice that at this point I was making one HTTP call too many – note the call to ‘/Products-80’ in the top left pane, and there are only 77 products!).

00:47:43: Starting to wrap the call to grab in an Object.keys that works on the entities configuration map we previously defined, noting that the simple use of console.log in the map call, we get what is really passed to the function in a map call, i.e. three arguments (not just a single argument) – current value, index and array. See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/map#Parameters for details about this.

00:51:00: Chris Whealy makes the point, in a comment, that the grab function could be curried, to allow us to remove the reliance that it currently has on the global baseurl constant. This is a good idea and something we should look at if we cover this code again in the next few sessions. If you’re interested in what this means and why it’s a good idea, you might want to take a look at episode 12 (“Exploring and understanding parts of @sap/cds JS – code & style”) where we look at pure functions and partial application in the course of exploring a short JS script.

00:54:05: Using the fs module to start to write the JSON data to output files. When we run this, it causes nodemon to repeatedly restart, endlessly! This is because we’re creating files that it notices, causing it to restart, and recreate the files, causing it to notice those files, restarting, etc. Ad infinitum.

We fix this by modifying the call to nodemon to ignore files in the data directory, so it looks like this:

./node_modules/nodemon/bin/nodemon.js grab.js --ignore data

00:57:01: We notice that in the JSON files, we have [object Object] which is not quite right. This is fixed by adding another process in the dot chain, to stringify the data (the objects) into JSON. Along the way, I write this:

.then(x => JSON.stringify(x))

which I pick up as a “bad [code] smell” because actually this is a redundant use of function parameterisation. This is better, and more concisely written, as

.then(JSON.stringify)

Stare at that for a couple of minutes and you’ll see why.

00:58:35: Phone call from Michelle, which I ignore. Yikes!