Event Information
Annotated links: Episode 13 of Hands-on SAP dev with qmacro
This is a searchable description of the content of a live stream recording, specifically “Episode 13 – Stdio-ifying the CSV filter mechanism” in the “Hands-on SAP dev with qmacro” series. There are links directly to specific highlights in the video recording. For links to annotations of other episodes, please see the “Catch the replays” section of the series blog post.
This episode, titled “Stdio-ifying the CSV filter mechanism“, was streamed live on Wed 20 Mar 2019 and is approximately one hour in length. The stream recording is available on YouTube.
Below is a brief synopsis, and links to specific highlights – use these links to jump directly to particular places of interest in the recording, based on ‘hh:mm:ss’ style timestamps.
Brief synopsis
In the previous episode (Ep.12) we looked at a simple CSV filter utility. In this episode we improve it by giving it the ability to read from STDIN and write to STDOUT so it plays nicely in pipeline contexts. Then we’ll be ready to use it to finalise our data for our CAP based “Northbreeze” service.
Links to specific highlights
00:03:45: A brief glimpse behind the scenes, as it were, where I produce the annotation blog posts for the video recordings that get uploaded to the SAP Developers YouTube playlist. I write in Markdown in Vim, which is a pretty nice combination of reliable and comfortable tech.
I use a Vim macro while writing the content, which we have a look at, starting from a simple template, and converting the HH:MM:SS timestamps to a Markdown-annotated link. I don’t have this macro in my standard .vimrc
, as it’s only really relevant for this particular task, so I have it in a “project-local” .vimrc
which is possible via this configuration:
set exrc
(see my dotvim repo for more details).
This is what the macro looks like:
let @t = '0Ypihttps://www.youtube.com/watch?v=VIDEOID&list=PL6RpkC85SLQAIntm7MkNk78ysDm3Ua8t0&t=wwrhwrmAs0ys$)A:k0ys$]Jx0ys$*.'
(note the escape characters just after the t=
, As
and A:
bits).
00:10:30: Based on Srikanth Peri‘s question, we have a quick go at creating a simple macro in Vim.
00:14:30: A recap of the Unix STDIO philosophy, thinking about the three streams STDIN, STDOUT and STDERR, and the power that brings. There’s a parallel between that and the dotchains that we’re writing in JS with the Axios library.
00:15:55: Talking about Node.js debugging, based on a comment from Nabheet Madan.
00:17:04: Reminding ourselves of what we have in the project directory csvf
, including taking a quick look inside the project’s package.json
. This shows us that we have two dependencies, one on @sap/cds
and the other on the command-line-args
package.
00:17:40: A feature in the terminal file manager Ranger that I discovered accidentally is the “zoom” feature with i
, which I show here.
00:20:18: Setting out our new intentions by modifying the option definitions in the constant optionDefinitions
. Remembering how it works right now with a simple example:
=> ./cli.js -v -i tmp/Suppliers.csv --fields supplierID city
>> Processing tmp/Suppliers.csv
>> Filtering to supplierID,city
>> Written to _out.csv
What we actually want to be able to do now is something more like this:
=> cat tmp/Suppliers.csv | ./cli.js --fields supplierID city > file.csv 2> error.log
using the pipe operator (|
), the redirect operator (>
) and the redirect-to-stderr operator (2>
).
Anyway, we end up with options that look like this (from the help text):
Options:
-i, --input Input CSV file (reads from STDIN if not specified)
-o, --output Output CSV file (defaults to STDOUT)
-f, --fields List of fields to output (space separated)
-h, --help Shows this help
-v, --verbose Talkative mode
Examples:
csvf -i data.csv -f supplierID companyName city -o smaller.csv
csvf --fields supplierID city < input.csv > output.csv
cat input.csv | csvf -f supplierID city | less
00:28:45: I fix the call to process.exit()
by changing the value passed from a 1 to a 0, denoting success rather than failure (!).
00:29:10: Looking at the NPM module get-stdin which allows us to read from STDIN, which we will use. And it’s promised-based, too!
00:29:50: We install get-stdin
with:
=> npm install --save get-stdin
… write a simple test script to try it out:
const getStdin = require('get-stdin')
getStdin()
.then(xs => console.log("GOT:", xs))
… and try it out:
=> echo "hello" | node stdin.js
GOT: hello
and also:
=> node stdin.js < /etc/hosts
GOT: ...
00:32:00: Someprogrammingdude points out that a future version of JS will indeed get a pipeline operator – this is great. See https://github.com/tc39/proposal-pipeline-operator for more details. This information causes me to think of a talk on the F# language that I saw at Manchester Lambda Lounge which in turn reminded me of the Elm language.
At this point I dug out the content from a talk I gave at Lambda Lounge a couple of years ago: Discovering the beauty of pattern matching and recursion where I look at these two language features in different languages, including Elm, Elixir, Haskell, JavaScript and Clojure. This talk was slide based but the slides are mostly full of code, so that’s OK, right? 🙂
00:35:08: Starting to refactor the code a little bit, turning the main part of the script into a new function process
– this lays the ground for a really odd error which we’ll come across a bit further on!
00:37:20: Now we can refactor the final part which looks at the value of options.input
and we use util.promisify
to make things a little nicer – see the post Node.js: util.promisify() for more detail.
00:47:58: So we end up with this:
if (options.input) {
readFileAsync(options.input, { encoding: 'utf8' })
.then(process)
.then(console.log)
Notice how this is so clean and solid state, there’s nothing really “moving” in here, not even any explicit arguments passed to the functions in the then()
calls, nothing that can go awry.
Notice also, however, a glaring oversight that we’ll discover shortly!
00:49:40: We note that right now, while we can write to STDOUT already (with console.log
), both our verbose output (prefixed with “>>”) and also the real data output goes to the same place, which we see in a test run. We fix this in the log
function by changing the console.log
call to process.stderr.write
.
00:51:10: Bang! When we try the new script out, we get a nice error: “Cannot read property ‘write’ of undefined”. What the heck? Anyway, after a few brief moments of head scratching, we realise that this is because I inadvertently redefined a major feature of the Node.js runtime in that we created our own function called process
, clobbering the standard process
object which has, amongst other things, the stderr
property (to which we’re referring with process.stderr.write
).
What a fool!
00:57:30: Renaming our function from process
to something else (filtercsv
) fixes the problem 🙂
We can now finish the refactoring of the script, which we do, including writing an output
function in a way that we can partially apply it, too.
01:01:50: Trying the refactored script out, with STDIN and appending to the STDERR output using the append (>>
) operator (as opposed to the (over)write operator (>
)), and trying the write-to-file option too. All seems to work as intended. Great!
Phew!