Troubleshooting A Lumira Extension

[Former Member]

There are some great webinars and materials here on SCN that cover how to make a Lumira extension using D3.The first time I followed along with an example, they pasted in a big chunk of code from the D3 gallery, showed that it was already working, and went on to do some minor cleanup.

I pasted in the chunk of code from the D3 gallery, but mine didn’t work.  So then what? If your D3 Lumira extension example isn’t working, this post contains tools and techniques that will help you troubleshoot.  (It’s also useful for D3 visualizations in other SAP products).

This blog covers a portion of my SAP TechEd 2015 presentation.

Getting started: local copy, with logging

To get a D3 gallery example working as an extension, my first step is to copy it verbatim to my local webserver.  For this example, that gives me two files:

• c:\inetpub\wwwroot\packCirclesGallery\index.html
• c:\inetpub\wwwroot\packCirclesGallery\flare.json

And I open the visualization in the browser (Chrome) as http://127.0.0.1/packCirclesGallery/index.html

 

At this point, you should have a working copy of the D3 gallery example at that local URL.  What if you don’t?

Let’s check the browser console.  In Chrome, you can get to it by pushing the F12 key and then clicking the Console tab:

Now I have an error, I can troubleshoot.  It turns out that D3 gallery examples read a local file such as data.csv or data.json. Without a webserver, just launching the file locally, gives me a “cross origin request” error when the code tries to read that file.  The solution is to open it in a webserver as http://webservername/filename (instead of opening the file directly from the file system).

Adding logging

Once I have my working local example, I want to add logging in a key place: when the data is loaded. Why?  Because getting your extension working is about making your data structure match the one that you got from the D3 gallery.  Until you have a good matching data structure, there’s no point in trying to troubleshoot the lines, points, circles, axes, or anything you’ll see on the screen.  Focus on the data first.

To get the “expected” data structure, we add logging in the d3.json() call.  This call, or d3.csv() is responsible for pulling the data out of the target file. We add two lines right at the beginning of the call.  The first one logs the data stored in the root variable.  The second one does too, with a very important difference.

 

Let’s take a look at the output from the first logging line, console.log(root).  The statement comes immediately after the data load, so I expect it to look like the .json file being loaded, on the left.  Instead it comes out looking like the screenshot on the right.

Original .json file	console.log(root)

The data structure that I get back is much more complex than the JSON file that was loaded.  I recognize the name-children structure which is still there.  But it has acquired x-y values, r value, depth – where did all this come from? This is right at the beginning of my code! Immediately after the data is loaded!  Nothing should have happened to it yet!

This brings us to our first “gotcha” in JavasScript troubleshooting for D3.  The answer is that D3 functions, such as the pack.nodes function in line 62, modify the data structure.  JavaScript passes objects by reference, so the changes are made on the original object, not a copy.  So when I expand my object in the browser console, it shows me the current state of that object.  Not the state it was in when the call to console.log happened.

Fortunately there is a way to get a “snapshot” of your object logged to console.  By using the JSON.stringify() function, you can make a string copy of your object.  JSON.parse() turns it into a format that the console can display nicely .  Using this, we see a good representation of our data right after it is loaded.

Original .json file	console.log(JSON.parse(JSON.stringify))

Side-by-Side Compare and Transform

For this hierarchical data visualization, the D3 gallery starts with a nested JSON file as its data source. That converts easily into the JavaScript data structure I got from adding log statements to the original:

But all the data we bring into Lumira is tabular data.  So I need to transform it.  To make my data into a “tree” like the one in the sample, I need to group and stack it. The data is sales by region, quarter, and product line.  I wrote a function called “restack” to re-arrange my data but it turns out I had reinvented the wheel.  D3 has a nest() function which does the job better.

Any visualization where you group the data is a potential use case for nest function. For example, if you want several side-by-side charts in a "trellis", or a grouped bar chart, or if you’re using hierarchical data such as Region, Business Unit, Division. Those of us with a SQL or BI background will recognize it most easily as a "banded report" using GROUP BY.  [See also: Dong Pan’s webinar on making a Lumira extension for PopulationPyramid.  This is an end-to-end working example using the nest() function.]

Here's the syntax for nesting the data for this use case – nesting one layer deep, by region.

The data going in is flat, like a csv.  The data coming out of the statement is tree-shaped.  In between, the key function(s) design what the tree will look like.

This gives us an important way to start thinking with D3 – think about the shape of your data, and recognize the "d" variable as your row-level object.  "d" might be a simple row in a flat data structure, or it might be a nested structure.  Either way you step through it one at a time, just like a SQL cursor. It might help to think of the key function as the code you would put inside your cursor loop.

Before I learned the nest function, I had re-invented it by making a temporary object and pushing it into the array.  This is shown as a counterexample.  If you find yourself doing something like this, stop and think. You will be able to develop faster by using the built-in d3.nest() function.

After nesting the data, I have a tree structure kind of like the one in the visualization.  Kind of.  It's nested, but the original says "name"/"children" and mine has Region names and Product Line names instead.  So I could write some JavaScript to copy my object into another object with the names I need….

But I'm going to think twice before I make temporary objects, and check to see if D3 has a better way. It turns out that I could retrieve the data in entries format instead of map, and it really improves my situation.

Now it has consistent array keys, but the names are still a little different.  Again I resist the urge to write a “for” loop to rename “key-values” into “name-children.”  I'm thinking with D3.

The solution to this problem turned out to be in the next function down the line.  Before the visualization draws any circles, it uses a function called "pack" to calculate the radius of the circles, based on what the parents are.  The framework accepts a translator function which lets you specify that "children" is actually called "values" in the data structure you will pass in.  Just like in our nest function above, there is a short-lived variable “d” representing our row-level object.  In this case, “d” is a nested object itself.

I add a couple console.log statements – with stringify – to make sure that the structures match between the example and my new extension.  It looks good.  Children, r values, x and y values in both our new extension and the original example.

Did you notice we have no visualization yet?

Getting your data to match the expected pattern is half the battle.  Until now, there is nothing in our visualization that draws axes, rectangles, lines, text, or circles.  All we have is the data-load and the pack() function.

We get here faster and smarter by limiting the scope of our troubleshooting: just the data.  We get the data clean, using the clever trick of logging a data snapshot.  Finally, we add in the visualization statements.

In the file attachments to this blog, you’ll find several steps of making this visualization.  Paste the render_js.txt file into your IDE as the render.js file.  Upload the data_csv.txt file as CSV.  Due to restrictions on attachment types, you will have to rename the files for your own file system.