Preparing custom GeoJSON/topoJSON maps for use in Lumira and Design Studio

If a built in mapping package works for you, great. But I am one of these people that likes things the way I like them, and will go an extra mile to achieve that. This is certainly the case with mapping, and further experiments with GeoJSON/topojson have shown me that the level of control can be as deep and detailed as you want it to be, as long as you are prepared to play with some command lines.

There are now quite a few articles on SCN related to GeoJSON/topojson, and you see a selection of links below, including my previous article in this:

• Creating Custom GeoJSON maps in Design Studio 1.5 – Part 1 of 2
• Creating Custom GeoJSON maps in Design Studio 1.5 – Part 2 of 2
• Design Studio Innovation Series – Topic 6: Geo Maps Part I – Feature Review
• DS15 – Create your own Map
• Lumira Geoextension with datamaps.js and topojson.js
• Mapping import-export flows between countries with GeoJSON/topoJSON in Lumira

In my previous article, the last one in the list above, I showed how the map was built and the arrows drawn, as well as how it was ported to SAP Lumira. I barely just touched on the preparation of the topoJSON map file was used, though, by referring to the famous Let’s Make a Map article, and this. Since then, though, I looked more into this, and realized that we can layer multiple map features on top of each other, even from different source files, and get things exactly the way we want to.

So, I did an experiment. And the result of which you see below: a topographical map of California, based on:

1. 19 layers of elevation data extracted from a NOAA source file (~15 used)
2. Rivers, lakes and oceans (each from a separate shape file)
3. Urban development
4. Major road network (interstates and state and federal highways)
5. Country border (with Mexico)
6. State borders (CA, NV, UT, AZ, OR and a little bit of ID)

The elevation data comes from a .tif file that basically functions as a Z-map. When you open it in a normal image viewer, it just looks like a black landmass of the world with with(ish) oceans, but it has within that “black” the actual elevation data for that point on the map. Using gdal_translate and gdal_calc (part of the GDAL tools that also includes ogr2ogr) you can select and filter from this tif file just like you can with ESRI shape files. Please realize, though that this tif file is an actual image rather than vector graphics, and therefore if you zoom in too far, you’ll see definite pixellation in your result. This particular file works best for larger land areas. For the waterways, roads and borders the information comes from Natural Earth shape files.

Using a make file to generate your topoJSON map file

This might look very complex and involved, but for the most part it is actually quite simple. The D3.js/JavaScript code is very straight-forward and you’ll probably spend more time on styling than writing the code to draw it onto the map in the first place. You can get a good idea how that works from this article on a topographical map of France, and basically every map feature is a path that is drawn in a very similar way. In your JavaScript you’re really just picking up all the work you’ve done in advance and instructing to draw the paths as defined by your topoJSON map. But clearly, if you know that getting a map of countries shapes takes a command line or two, a map with 19 layers of elevation data, waterways, roads and borders is going to use a whole lot more, and this is where using a “make” file is going to be very useful.

“Make” is a GNU program that allows you to specify a series of command lines that generate files and how these files are related to each other. It is used commonly in the Unix/Linux world to specify how programs should be compiled, for instance, when distributed as source files, rather than binaries. In this article Mike Bostock himself explains why he uses make files for his maps, but I’d argue even more basically: using a make file makes it real easy to build larger, complex, multi-layered maps, and makes it very easy to add even more features as you go along.

Following the links in the previous two paragraphs will get you lots of information on this, but I’ll give a few quick examples to show how make works and uses make files. Suppose we have source files A.txt and B.txt, want to merge the two together as C.txt, we could write that as a simple command line:

paste A.txt B.txt > C.txt

Now, if we want to put that in a make file, we get something like this (Note, the 2nd line is preceded by a TAB character):

C.txt: A.txt B.txt

     paste A.txt B.txt > C.txt

That is, we get the resulting file, followed by a colon, and then all source files needed for the command itself. What’s really good about this is that we can layer this multiple times, so if we first wanted to do some preprocessing of A.txt, we can do that by adding further lines. Say that A.txt itself is the result of pasting A1 and A2 together. We just add this to the file:

A.txt: A1.txt A2.txt A3.txt

     paste A1.txt A2.txt > A.txt

Make also allows variables, so you can pass in parameters to command lines and use the same value in multiple of them. As explained in my previous post, I like to pre-filter my maps to a particular window with -clipdst, so by setting up the coordinates in a variable I can easily change it just in one place and get the topoJSON file generated for a differentregion. (I’ve actually done this once over the phone in about 5 mins with a colleague in Europe, using the same make file and a few changes to generate a map of the Benelux).

So, let’s have a look at a real example, and how I generated the elevation data for California. I start off with a declaration of my BOUNDS:

BOUNDS = -124.80 42.20 -113.80 31.40

This is the WNES format coordinates of my viewing window. We can now use it to filter the tif file to a smaller subsection: (assuming you’ve downloaded that file and it is in the same location as the make file)

# boxing:

crop.tif: ETOPO1_Ice_g_geotiff.tif

  gdal_translate -projwin $(BOUNDS) ETOPO1_Ice_g_geotiff.tif crop.tif

This creates the crop.tif file, which we can now filter by elevation. Note that crop.tif is the source file, creating a levelxxxxx.tif file each:

# Raster slicing:

level0001.tif: crop.tif

  gdal_calc.py -A crop.tif –outfile=level0001.tif –calc=”1*(A>0)”       –NoDataValue=0

level0005.tif: crop.tif

  gdal_calc.py -A crop.tif –outfile=level0005.tif –calc=”5*(A>5)”       –NoDataValue=0

level0010.tif: crop.tif

  gdal_calc.py -A crop.tif –outfile=level0010.tif –calc=”10*(A>10)”       –NoDataValue=0

…

level6500.tif: crop.tif

  gdal_calc.py -A crop.tif –outfile=level6500.tif –calc=”6500*(A>6500)”    –NoDataValue=0

They are still tif files, though, just limited to the viewing box, and now limited to only shapes over the specified elevation (in meters). In the next step, we create an ESRI shape file of each of these generated tif files:

# Polygonize slices:

level0001.shp: level0001.tif

  gdal_polygonize.py level0001.tif -f “ESRI Shapefile” level0001.shp level_0001 elev

level0005.shp: level0005.tif

  gdal_polygonize.py level0005.tif -f “ESRI Shapefile” level0005.shp level_0005 elev

level0010.shp: level0010.tif

  gdal_polygonize.py level0010.tif -f “ESRI Shapefile” level0010.shp level_0010 elev

…

level6500.shp: level6500.tif

  gdal_polygonize.py level6500.tif -f “ESRI Shapefile” level6500.shp level_6500 elev

We then join these together into a single shape file:

# merge

levels.shp: level0001.shp level0005.shp level0010.shp level0025.shp level0050.shp level0100.shp level0250.shp level0500.shp level1000.shp level1500.shp level2000.shp level2500.shp level3000.shp level3500.shp level4000.shp level4500.shp level5000.shp level6500.shp

  ogr2ogr levels.shp level0001.shp

  ogr2ogr -update -append levels.shp level0005.shp

  ogr2ogr -update -append levels.shp level0010.shp

…

  ogr2ogr -update -append levels.shp level6500.shp

Followed by generating the GeoJSON file:

levels.json: levels.shp

  ogr2ogr -f GeoJSON -where “elev < 10000” levels.json levels.shp

And from there you can easily run the topojson command to generate the file you’d use in your visualization, Lumira extension or Design Studio.

But there is a problem. Look again at the topographical map of France: It’s only land! Where is Lake Geneva? Where are all the rivers?  When I first did this for California, Death Valley – which is below sea level and is mostly known for being extremely dry – showed up as water, instead…  Luckily, now that we have this base make file, adding such features is really easy:

# oceans

oceans.json: ../ne_10m_ocean/ne_10m_ocean.shp

  ogr2ogr -f GeoJSON -clipdst $(BOUNDS) oceans.json ../ne_10m_ocean/ne_10m_ocean.shp

# lakes

lakes.json: ../ne_10m_lakes/ne_10m_lakes.shp

  ogr2ogr -f GeoJSON -clipdst $(BOUNDS) lakes.json ../ne_10m_lakes/ne_10m_lakes.shp

# rivers

rivers.json: ../ne_10m_rivers_lake_centerlines/ne_10m_rivers_lake_centerlines.shp

  ogr2ogr -f GeoJSON -clipdst $(BOUNDS) rivers.json ../ne_10m_rivers_lake_centerlines/ne_10m_rivers_lake_centerlines.shp

And the lines for urban development, roadways and borders are the same, they just use different shape files as input.

We can tie this all together then in one topoJSON file like this, which brings in the elevation data (levels.json), oceans, lakes and rivers, urban development (urban.json), interstates, and federal and state freeways and primary roads (pulled in separately so that they can be easily styled differently), and finally the international and state borders.:

cali.json:  levels.json oceans.json lakes.json rivers.json urban.json interstates.json freeways_federal.json freeways_state.json primary_federal.json primary_state.json countries.json states.json

  topojson –id-property none -p name=elev -o cali.json — levels.json oceans.json lakes.json rivers.json urban.json interstates.json freeways_federal.json freeways_state.json primary_federal.json primary_state.json countries.json states.json

So, I hope you see how powerful this is. Suppose we wanted to overlay a rail network: no problem, just add a line like for the oceans and rivers with the correct shape file or other source file, and make sure it is included in the topojson command. Make also only does updates when the files don’t yet exist, so if you made a mistake just in one command line, rerunning make will only create what it doesn’t have yet, so you don’t have to regenerate the elevation levels just because you wanted to add secondary roads, for instance. And as mentioned before, adjusting the BOUNDS allows you to create the map for a different part of the world – just make sure that your shape file(s) have information for that other ‘window’.

And of course, you can use this for any topoJSON map, even those that are less involved. If it takes more than just a few command lines to generate your topoJSON map, you can save yourself a whole lot of trouble by putting it in a make file.

With this, you should be able to build just about any map of any level of complexity, largely just limited by the size of your source files and resulting topoJSON file, while maintaining full control of all styling and any interactivity you’d possibly like to add.

Smaller Scale: cities and towns: OpenStreetMap.org

While I am very thankful to NOAA and Natural Earth for making the data available that they do, there are limits to the amount of detail you can expext. As I mentioned with the elevation data, even doing a full map of all of Switzerland already gives quite a bit of pixellation when you make your map bigger than about 4-500 pixels. What you may want to try is openstreetmap.org, which is an open source mapping project (which you can join and contribute to) which – if you’re lucky as coverage varies – can be incredibly detailed on city and even street level. So, for such maps I strongly recommend you see if the county, city or neighborhood you want is covered to the level you want (if not, you could decide to add it in yourself, of course, to the benefit of all). In some cases it can be remarkably detailed. For my area it even includes post boxes and traffic lights. Many areas have detailed land use marked, indicating what is farmland, forest, commercial or residential. You may want to look at the list of officially endorsed Map Features to get a sense of what might be covered, and then check your area of interest. OSM files can be easily converted to ESRI shape files using ogr2ogr – just specify that the output format should be “ESRI Shapefile”.