There are some great webinars and materials here on SCN that cover how to make a Lumira extension using D3.The first time I followed along with an example, they pasted in a big chunk of code from the D3 gallery, showed that it was already working, and went on to do some minor cleanup.

I pasted in the chunk of code from the D3 gallery, but mine didn’t work.  So then what? If your D3 Lumira extension example isn’t working, this post contains tools and techniques that will help you troubleshoot.  (It’s also useful for D3 visualizations in other SAP products).

This blog covers a portion of my SAP TechEd 2015 presentation.

Getting started: local copy, with logging

To get a D3 gallery example working as an extension, my first step is to copy it verbatim to my local webserver.  For this example, that gives me two files:

  • c:\inetpub\wwwroot\packCirclesGallery\index.html
  • c:\inetpub\wwwroot\packCirclesGallery\flare.json

And I open the visualization in the browser (Chrome) as http://127.0.0.1/packCirclesGallery/index.html

  TechEdCirclePackingBlog_01.jpg

At this point, you should have a working copy of the D3 gallery example at that local URL.  What if you don’t?

Let’s check the browser console.  In Chrome, you can get to it by pushing the F12 key and then clicking the Console tab:

  TechEdCirclePackingBlog_02.jpg

  TechEdCirclePackingBlog_03.jpg

Now I have an error, I can troubleshoot.  It turns out that D3 gallery examples read a local file such as data.csv or data.json. Without a webserver, just launching the file locally, gives me a “cross origin request” error when the code tries to read that file.  The solution is to open it in a webserver as http://webservername/filename (instead of opening the file directly from the file system).


Adding logging

Once I have my working local example, I want to add logging in a key place: when the data is loaded. Why?  Because getting your extension working is about making your data structure match the one that you got from the D3 gallery.  Until you have a good matching data structure, there’s no point in trying to troubleshoot the lines, points, circles, axes, or anything you’ll see on the screen.  Focus on the data first.

To get the “expected” data structure, we add logging in the d3.json() call.  This call, or d3.csv() is responsible for pulling the data out of the target file. We add two lines right at the beginning of the call.  The first one logs the data stored in the root variable.  The second one does too, with a very important difference.

  TechEdCirclePackingBlog_04.jpg

Let’s take a look at the output from the first logging line, console.log(root).  The statement comes immediately after the data load, so I expect it to look like the .json file being loaded, on the left.  Instead it comes out looking like the screenshot on the right.

Original .json file

console.log(root)

  TechEdCirclePackingBlog_05.jpg

  TechEdCirclePackingBlog_06.jpg

The data structure that I get back is much more complex than the JSON file that was loaded.  I recognize the name-children structure which is still there.  But it has acquired x-y values, r value, depth – where did all this come from? This is right at the beginning of my code! Immediately after the data is loaded!  Nothing should have happened to it yet!

This brings us to our first “gotcha” in JavasScript troubleshooting for D3.  The answer is that D3 functions, such as the pack.nodes function in line 62, modify the data structure.  JavaScript passes objects by reference, so the changes are made on the original object, not a copy.  So when I expand my object in the browser console, it shows me the current state of that object.  Not the state it was in when the call to console.log happened.

Fortunately there is a way to get a “snapshot” of your object logged to console.  By using the JSON.stringify() function, you can make a string copy of your object.  JSON.parse() turns it into a format that the console can display nicely .  Using this, we see a good representation of our data right after it is loaded.

Original .json file

console.log(JSON.parse(JSON.stringify))

  TechEdCirclePackingBlog_05.jpg

  TechEdCirclePackingBlog_07.jpg

Side-by-Side Compare and Transform

For this hierarchical data visualization, the D3 gallery starts with a nested JSON file as its data source. That converts easily into the JavaScript data structure I got from adding log statements to the original:

TechEdCirclePackingBlog_08.jpg

But all the data we bring into Lumira is tabular data.  So I need to transform it.  To make my data into a “tree” like the one in the sample, I need to group and stack it. The data is sales by region, quarter, and product line.  I wrote a function called “restack” to re-arrange my data but it turns out I had reinvented the wheel.  D3 has a nest() function which does the job better.

TechEdCirclePackingBlog_09.jpg

Any visualization where you group the data is a potential use case for nest function. For example, if you want several side-by-side charts in a “trellis”, or a grouped bar chart, or if you’re using hierarchical data such as Region, Business Unit, Division. Those of us with a SQL or BI background will recognize it most easily as a “banded report” using GROUP BY.  [See also: Dong Pan’s webinar on making a Lumira extension for PopulationPyramid.  This is an end-to-end working example using the nest() function.]

Here’s the syntax for nesting the data for this use case – nesting one layer deep, by region.

TechEdCirclePackingBlog_10.jpg

The data going in is flat, like a csv.  The data coming out of the statement is tree-shaped.  In between, the key function(s) design what the tree will look like.

This gives us an important way to start thinking with D3 – think about the shape of your data, and recognize the “d” variable as your row-level object.  “d” might be a simple row in a flat data structure, or it might be a nested structure.  Either way you step through it one at a time, just like a SQL cursor. It might help to think of the key function as the code you would put inside your cursor loop.

Before I learned the nest function, I had re-invented it by making a temporary object and pushing it into the array.  This is shown as a counterexample.  If you find yourself doing something like this, stop and think. You will be able to develop faster by using the built-in d3.nest() function.

TechEdCirclePackingBlog_11.jpg

After nesting the data, I have a tree structure kind of like the one in the visualization.  Kind of.  It’s nested, but the original says “name”/”children” and mine has Region names and Product Line names instead.  So I could write some JavaScript to copy my object into another object with the names I need….

TechEdCirclePackingBlog_12.jpg

But I’m going to think twice before I make temporary objects, and check to see if D3 has a better way. It turns out that I could retrieve the data in entries format instead of map, and it really improves my situation.

TechEdCirclePackingBlog_13.jpg

Now it has consistent array keys, but the names are still a little different.  Again I resist the urge to write a “for” loop to rename “key-values” into “name-children.”  I’m thinking with D3.

The solution to this problem turned out to be in the next function down the line.  Before the visualization draws any circles, it uses a function called “pack” to calculate the radius of the circles, based on what the parents are.  The framework accepts a translator function which lets you specify that “children” is actually called “values” in the data structure you will pass in.  Just like in our nest function above, there is a short-lived variable “d” representing our row-level object.  In this case, “d” is a nested object itself.

TechEdCirclePackingBlog_14.jpg

I add a couple console.log statements – with stringify – to make sure that the structures match between the example and my new extension.  It looks good.  Children, r values, x and y values in both our new extension and the original example.

TechEdCirclePackingBlog_15.jpg

Did you notice we have no visualization yet?

Getting your data to match the expected pattern is half the battle.  Until now, there is nothing in our visualization that draws axes, rectangles, lines, text, or circles.  All we have is the data-load and the pack() function.

We get here faster and smarter by limiting the scope of our troubleshooting: just the data.  We get the data clean, using the clever trick of logging a data snapshot.  Finally, we add in the visualization statements.

TechEdCirclePackingBlog_16.jpg

In the file attachments to this blog, you’ll find several steps of making this visualization.  Paste the render_js.txt file into your IDE as the render.js file.  Upload the data_csv.txt file as CSV.  Due to restrictions on attachment types, you will have to rename the files for your own file system.

To report this post you need to login first.

3 Comments

You must be Logged on to comment or reply to a post.

  1. Vincent Dechandon

    Hello.

    It’s really nice to see this kind of technical posts in here.

    The idea of thinking that just copy pasting a code from a d3 viz on the web into the lumira rendering process will work out of the box is of course illusory. Most of the time there is a lot of things to change.

    As you pointed out, inspecting elements that have been logged through console method sreturns the current state of an element, not the state it was in while the log occurred (although you first converted the element to string then reparsed it in of your example).

    For this matter as it can be tricky if you are not familiar with debugging through the console, the best thing to do is to put breakpoints in the code so you’ll be able to inspect the elements at the state they are while code is pausing itself. This way, it’s quite easy to see that first the root is just the parsed version of the flare.json, then that after the pack.nodes method is called upon it, it is modified.

    Most of the time the path to your extension files are under sap/bi/bundles/<your bundle name>/viz/ext/<your extension name>/<your extension name>-src/js/… under the Sources tab in the Chromium debugger.

    Br,

    Vincent

    (0) 
  2. Kristina Pereyra Post author

    Excellent point. The first few versions of Vizpacker did not play nicely with breakpoints so we developed workarounds using console.log.  SAP Web IDE has no such issues.  Vizpacker is still there, and convenient to use if you’re working offline.


    Here’s a link to an article about breakpoints in Chrome.  Debugging JavaScript – Google Chrome

    (0) 

Leave a Reply