Custom Choropleth Maps with VizPacker
- The vizpacker utility
- My vizpacker approach
- Adapting GEOJSON files
- Position/Size of the choropleth map with the ALBERS projection
- Japanese Football Players
- England and Wales Crime Data
- GEOJSON, how big can it get
- Thank you
- Code Download and Issues/features
- Copyright statements
- 5th Sept 2015 – Updated Extension for Lumira versions 1.27(and above)
I have been interested in geographic information systems (GIS) software (or simply maps : ) for a long time now and it is a part time hobby. I find it a good base example to use a variety of SAP software. When I failed to map some of the information in my blogs with Lumira, I was interested in the vizpacker announcement and the possibility to customise Lumira.
Two of my blogs that I will base custom choropleth maps on are,
- Data Geek Challenge 2: Can your birthday help you play football in Japan or England I would extend the data to include the place of birth of the footballers.
- Trying Out Lumira with the HANA Cloud Platform which contains crime data with location details.
The code for the Japan extension is at the bottom of the blog. It’s the same base code for all the maps in this blog, with relevant adjustments.
The vizpacker utility
Lumira now comes with the vizpacker utility tool that can be used to add extensions to Lumira and I used it against Lumira version 1.15. A thank you to Matt Lloyd as his blog pushed me to try out my choropleth map ideas. Check his blog for more details on vizpacker.
My vizpacker approach
First, I checked the D3.v2.js file in the Lumira libs install directory and confirmed that D3.v2 came with some standard d3.geo projections. These can be used for choropleth maps, such as d3.geo.mercator and d3.geo.albers (and others). Now I have used the Mercator projection before and will continue to use it however you can start to ask such questions as “How big is Greenland?” with the Mercator projection. I decided to use the albers projection with vizpacker as the D3 documentation stated that this projection is good for choropleth maps. It did lead to some issues that I will cover later as the albers projection has changed in D3 version 3 and all the documentation I could find is at version 3. However there are examples on the web using the Albers projection at version 2.
I learnt from Matt’s blog that the extensions use an SVG element called VIS and also information around using the fdata structure. I knew I could append (and keep on appending 🙂 ) to this VIS element with the D3 albers projection from relevant embedded GEOJSON. Then I could add code to link the Lumira fdata data array to the GEOJSON attributes. And for the choropleth map add colours to it from a range, depending on values.
A link for GEOJSON entry on Wikipedia here and to quote
D3 can use GEOJSON as a base for D3 geo projections. I already knew how to create GEOJSON files using open source software such as QGIS , from the command line with GDAL/OGR or simply find existing GEOJSON files. Also more recently I have been using the super fantastic database that is SQL Anywhere, which is spatially enabled and can generate all kinds of spatial data. A blog about how I used SQL Anywhere’s capabilities is here and it also explains a bit on how I got into maps.
A good source to create GEOJSON from is the public domain data in the Natural Earth downloads found here. I had previously used these files to create a KML file for Japan and they were also a base for my SDN points blog and ultimately my first Data Geek entry. I would use GEOJSON from my own adapted Natural Earth shapefile for Japan in my map on Japanese footballers.
Also freely available for UK based data is the Office for National Statistics ONS geographical portal which has some very useful shapefiles and lookup data that can be freely used. Freely used with acknowledgement of the source (always worth reading the license). I converted these files to GEOJSON with SQL Anywhere for the crime data choropleth maps.
Adapting GEOJSON files
I used some other utilities to specifically minimise GEOJSON. As I was embedding the file directly into the vizpacker, I minified the files. This way they end up as one line.
JStoolnpp plugin for Notepadd++ (and its best to run in administrator mode if adapting files directly in the Lumira install directory ;)). http://www.sunjw.us/jstoolnpp/
A NODE.js module json-minify for bigger files. One file I used for Local Authority Districts was approx 10mb and json-minify dealt with it no problem. https://www.npmjs.org/package/json-minify
I sometimes had issues using the minified geojson directly in vizpacker. So for the larger 10mb geojson I ended up cutting and pasting into the file after extracting the packed vizpacker extension.
Position/Size of the choropleth map with the ALBERS projection
The issue I noted earlier about the Albers projection changing from D3 v2 to v3 meant some experimentation with the parameters used as I found examples but not the original documentation. Using Solihull as an example – as I had extracted Solihull’s GEOJSON from an ONS portal shapefile mentioned earlier ( there is a choropleth map covering Solihull, where I live below).
Extract from the code.
var projection = d3.geo.albers()
.scale(1200 * 120)
.translate([width / 2, height / 2]);
I found that the .origin settings could be successfully adapted by using the centre point of the area in question. If you are wondering what the centre point of the area in question is, then I found it by using the Openstreetmap based nominatim service. By searching for the relevant place and clicking on the details then lots of extra information is provided under an open license . The outline shape of Solihull Metropolitan Borough Council can be seen in the image below and the centre point highlighted.
Depending on the size of the file then I started with a relative small scale number and increased it, as required, to zoom in.
There does appear to be clever ways to auto calculate these parameters however after starting with the centre point it didn’t take that long to get the map in position for the test HTML page. Getting it in a position in Lumira was another step though and required further adjustment and means they are adjusted purely to fit on the Lumira “page” to my preference.
The two remaining parameters “Parallels & translate”, I took these from Mike Bostock’s “Let’s Make a Map” example, that webpage is at d3 version 3, however these settings worked out fine for me.
For the settings for Japan I found this question and answers useful.
Japanese Football Players
I took the SPARQL query from my blog on Japanese footballer’s birth dates and adapted it to find the prefecture of birth. From this I generated the following map.
To add some context around football I thought I would add some locations for Football venues. So I queried Wikipedia a second time for the lat/long of the Football venues listed via DBpedia. I created a second GEOJSON array and a second D3 loop to append this to the VIS element. This GEOJSON was point based and all the black dots are football venues.
Moving the mouse pointer over the black dot and the name of the Football venue will be displayed. As shown above.
Also ZOOMING in is possible as below.
England and Wales Crime Data
When I originally found the Police crime data here for England, Wales and Northern Ireland I did think I would end up mapping the details. However as it was crime data I thought I had better research on how to present it. I started with a Google search “normalise crime data” . A number of approaches based on population or size of area and also what “story” are you trying to tell with the data. Some statements that no normalisation should take place and be left to the user. So as I am the user, my “story” is to see how the map would change based on “normalisation” on the Lower Super Output Area’s (LSOA) size. The LSOA is used with every crime record with a location (there are crimes with no location recorded and I leave those out). I could calculate the size of the LSOA using the SQL Anywhere function ST_AREA. The LSOA area which my GEOJSON is based has a mean population of 1500 and higher in London. I uploaded this data to the SAPHCP and updated the HANA analytic view. For the final step and the normalisation calculation of crime count / LSOA area km was done with Lumira.
Solihull Metropolitan Borough Council (MBC) Crime stats
The first example is a pure crime count for each LSOA area for Solihull MBC (the council is named after its largest town, Solihull). I live in an area covered by the Solihull MBC and would state it’s a great place to live, however here are some crime stats. The crime data is the entire range of data mentioned in my previous blog Trying Out Lumira with the HANA Cloud Platform
As per the map of Japan and football stadiums I wanted some extra details on the map to give some sense of place. When I investigate other areas it would be good for me to have towns, cities or village names on the map. To add Solihull place names I visited the following website which is again based on openstreetmap data.
I used the wizard on that site to query the data. From the results a geojson file can be exported. The query I ran was places=town or places=village after focusing on Solihull. The end result for my choropleth map is below and this time it is normalised by “crime count/LSOA area km”.
GEOJSON, how big can it get
GEOJSON file size, how big can it get? (As I am embedding the file in the vizpacker files). I found the Local Authority District shapefile which fits my crime data (after adding a lookup table for LSOA to Local Area Districts to my SAPHCP crime analytic view, lookup tables can be found on the ONS portal ). This file ended up just below 10mb and works fine at the default position. Zooming around can lead to drag and slow response on my computer.
First a pure count of data on a Local Authority District map.
Next up one showing the normalised data of crime count/LSOA area and a bias towards London Districts is shown.
Zooming in on London
As the crime data is based on LSOA areas and the stated mean population of 1500 is not applicable to London, then most likely I will separate these out in the future. I like the following a map of the UK distorted by population.
I would like to thank Tammy Powlas and the @SAPLumiraExperts in replying to my tweet for help in testing the original files.
Code Download and Issues/features
The original demo extension can be downloaded here.
**The current version can be downloaded from here and check the new updated install instructions at the end of the blog
I have altered it slightly from the one I sent to test. It is a zip file containing two files, one is the vizpacker zip file and the other is the test csv to generate the choropleth map.
The example is for Japan but it is exactly the same base code for all the examples above just slightly different loops depending on information in the map.
The code can be extracted in the extensions folder for Lumira
C:\Program Files\SAP Lumira\Desktop\extensions
And you should have a directory like this after extracting the zip file.
C:\Program Files\SAP Lumira\Desktop\extensions\bundles\viz\ext\rjrscnblog
Start Lumira and load the csv file that came with the download. Then select the “Japan SCN blog demo” chart extension and add the measure and dimension. The following map should display.
The code is offered in good faith and it was developed on my personal copy of Lumira and may or may not contain errors. If you can improve the code or offer any tips/hints to improve it then that would be good.
All of the above information used from external sites is free to use from my interpretation of the license (where found). Some information about attribution is required so some copyright statements below. I am more than happy to adapt anything above if required for clearer clarification of source material.
ONS Geo Portal
Contains Ordnance Survey data © Crown copyright and database right 2014
Contains Royal Mail data © Royal Mail copyright and database right 2014
Contains National Statistics data © Crown copyright and database right 2014
“© OpenStreetMap contributors”.
5th Sept 2015 – Updated Extension for Lumira versions 1.27(and above)
The original extension I developed and covered in my blog above does not work with the latest versions of Lumira due to an issue with the D3 version. Lumira now uses D3 version 3 whereas my extension requires D3 version 2 (as stated in the blog above). I did ask about this version inconsistency of d3 version on the forum but found a workaround for the local install of the vizpacker.
As SAP recommend the WEBIDE vizpacker and “all future innovations will be in the WEBIDE” version I did decide to use it even though the D3 version issue does cause me extra effort to get a working extension in both the WEBIDE and Lumira. So I have updated the blog with a link to my new updated extension that follows the SAP guidelines (my own interpretation of these guidelines and some ideas of my own) to get the extension working with Lumira version 1.27. Hopefully it will work with later versions as well ( I will have to wait and see what versions of D3 are released with Lumira in the future). I have found the new way to distribute Lumira via the SAP store extremely frustrating as the SAP store either does not allow the downloads or has even lost my order – and offered a 15% discount on my next order ( I will try and get SAP to pay me to download the next free version 🙂 ). Anyway onto the extension link.
Installing the extension
The way to install the extensions has also changed therefore a link to the help page for installing extensions.
Basically chose menu options File -> Extensions and in the Extension Manager screen select “Manual Installation“.
From the download select the robertrussellrjr.viz.ext.mapofjapan.zip file and install. Lumira will have to be restared.
Once installed the extension will appear as follows in the Extension Manager.
Importing the LUM file
To import my Japanese footballers data then take the menu options File -> Import to folder.
From the downloaded zip file select Map of Japan.lums and this Lumira document will use the Map of Japan extension.
The extension can be selected as shown below.
How I dealt with the different D3 versions in WEBIDE and Lumira
I chose to detect the D3 version as the way to allow the extension to work in the WEBIDE and Lumira – as per the code shown in the WEBIDE screenshot below. In the WEBIDE I use the albers d3 projection and for Lumira this switches to the d3 mercator projection.
Another issue I had was with my geojson map of Japan, it did not work with d3 v3 as it would not render correctly. I took an educated guess it was due to the clockwise winding order that is noted in the d3 v3 upgrade release notes. To get around that issue I used mapshaper to convert my geojson to topojson and then back to geojson. (topojson forces all polygons to be clockwise, so I new it would be the correct winding order after a successful conversion – I then converted back to geojson and guessed mapshaper would respect the winding order from topojson when converting back to geojson. It works in Lumira now, so must be true 😉 I converted back to geojson as I did not want to have to include topojson in my extension.)
There are two files in the download link above, one is my new extension and one LUMs file which is the Japanese footballers data. I have also extended the extension 🙂 to allow multiple matches on the dimensions for the prefecture name – e.g. it can successfully map against the ISO code, Kanji name or FIPS code of the prefecture to allow flexible data and ease of use.
I usually test again various computers but only tested this extension against my windows 8 machine running Lumira 1.27 but I am sure it will work on your machine, let me know…
A couple of issues that I do not like but are these issues are the same with some other (SAP and non SAP) extensions I have tried with 1.27.
1) no chart title is possible- just blank
2) if you remove the dimensions/measure an error message will be shown and not the nice(r) add more measures or dimensions message you get with the standard charts. By adding measures/dimensions the map will be shown.