Blog - MapBox

Retina Maps Added to the MapBox iOS SDK

MapBox — 2012-05-16T00:00:00-07:00

We’ve just added retina tile support to our worldwide street level base map, MapBox Streets, with full support in our iOS SDK. This brings high-DPI display support for devices like the iPhone 4/4S and the new iPad.

Retina support means that all of the billions of images that we create, update, and store - read more about that in Young’s post on rendering the world - are now duplicated in an additional map that’s optimized for display at twice the pixel density.

Though it’s hard to do it justice on non-retina displays (on which you are likely reading this), consider these before and after pictures:

Move your mouse over the image to toggle between before and after versions.

There are a number of tricks that our cartographer AJ used to enhance the look here, including widening streets and enlarging labels (expect a more detailed blog post soon). The resulting image on retina-capable devices is incredibly sharp. Previously we had been using client-side code to draw the 256-pixel tile images into 512-pixel space. Otherwise the details are sharp but are unable to be read since true 256-pixel images take one quarter of the screen space as on non-retina devices and are generally too tiny to read. Once we’ve got the details styled in a way that looks good at a true 256-pixel size, we can flip back to drawing at an actual 256-pixel screen space.

Here’s a side-by-side comparison up close:

As you can see, we were also able to provide more context by including more features.

If you are using the MapBox iOS SDK, you only need to change two things to take advantage of these retina tiles:

When editing in your hosting account, enable the 2x button for streets and save the changes.
In the Objective-C code, use the newly saved style’s TileJSON URL and be sure to set the map view’s adjustTilesForRetinaDisplay property to NO. Tiles will then draw at the normal 256-pixel size. If you are supporting non-retina devices, you will probably want two versions of your maps, just as you have two versions of graphical assets, and detect which map to use via [[UIScreen mainScreen] scale].

If you want a quick video overview of editing map styles and embedding them in an iOS app, watch our feature on Custom Styled Maps for iOS Apps in 5 Minutes.

We’re pleased to be able to bring these updates to our users and to our developers. Stay tuned for continued updates and enhancements to our tools for making beautiful maps.

Moldova Open Innovation Weekend 2012

MapBox — 2012-05-15T00:00:00-07:00

Moldova Open Innovation Week has kicked off this week to bring together local and international experts with civil society and further stimulate the work in open government and open data in Moldova. Discussions, training sessions, and a hackathon will all focus on how open data and Moldova’s new data portal can be leveraged by the government and local organizations to publish and share information.

I’ll be traveling to Chisinau to participate in the Open Innovations TechCamp on Friday and the “Apps for Moldova” Hackathon happening at the end of the week. As a part of the TechCamp, I’ll talk about using maps to visualize data and conduct training sessions on using TileMill to design custom maps. Creating custom maps to visualize data can be done with completely open source and free tools. In preparation for the event, I wanted to walk through the process of prepping your data and using TileMill to create a custom map.

Mapping numbers of primary health care centers

Everything from prepping your data to joining it with spatial data to designing a custom map can be quick with tools like QuantumGIS, TileMill, and MapBox. Using Ministry of Health data about the number of primary care centers per district downloaded from data.gov.md, we can prepare this to be mapped.

Prepping your data

Data downloaded from data.gov.md will need to be prepped in order to be mapped. Using an open source spreadsheet editor like Open Office (or something similar like LibreOffice), we can open the downloaded .xls file from data.gov.md.

To use this with spatial data we need to transpose the district names into one column and add a district ID column to help make a successful join. Here I’ve renamed the columns to be short, descriptive names (I used English in my case, but any language will work).

Now we can save our file as a .dbf file to use within QuantumGIS. This is the format that a shapefile uses to store data. It can be used to join within QGIS.

Using QuantumGIS to join your data

Administrative level shapefiles can be downloaded from Natural Earth, a free and public domain repository of spatial files. I downloaded the administrative-1 borders shapefile, extracted the Moldova spatial data, and added a district ID to help with the join. You can now load the admin shapefile and the health center .dbf file into QGIS

Now we can make the join between the two files by opening up the layer properties of the admin layer and clicking the joins tab. Click the add button and select the correct columns to join.

We can then save our newly joined data as a new shapefile.

Custom map design with TileMill

Using TileMill we can add our new data and style out a color scheme. I used Color Brewer to create a good looking color scheme. With just a few lines of code, we can create a shaded choropleth map that shows the number of primary care centers per district.

We’re looking forward to participating in the discussions and working with everyone in Moldova to create incredible maps from open data. Opening and allowing free access to government data creates great possibilities for civil society, governmental agencies, and the private sector to leverage and develop innovative products for the country’s growth.

MapBox Static API Private Beta

MapBox — 2012-05-15T00:00:00-07:00

We’re happy to announce that the new MapBox Static API is now available for private beta testing. The MapBox Static API gives you access to all your maps, dynamic compositing, custom styles for MapBox Streets, and POI markers powered by the Maki icon set all through an elegant URL scheme that generates static images. If you’re looking to integrate maps with your mobile or web application but don’t need the overhead of JavaScript or a native tiling API, the MapBox Static API is perfect for you.

MapBox Static API Overview

What is the MapBox Static API? The static API is the simplest way to show a map on your website or mobile application. Each static map is a unique image based on a simple URL containing a map ID, geographic coordinates, image dimensions, and some optional parameters. Adding custom maps to your website is as easy as writing the HTML <img> tag.

Custom maps from TileMill. Any custom map you create in TileMill and upload to your MapBox account is automatically integrated into the MapBox Static API. Highlight specific locations, navigate with thumbnails, paint a broad overview, and more using your custom maps.

Custom styles for MapBox Streets. Control the design of MapBox Streets and integrate it directly into our Static Maps API. All of your color settings, presets, and label options are preserved when you request a map through the Static API.

Beautiful POI markers. You can add dozens of Maki icons at three different sizes to your map to mark important locations or points of interest. Customize the colors of the Maki icons to fit your look or provide your own custom markers if you need full control.

Interested in participating?

To apply for our private beta,

Sign up for a MapBox account if you don’t already have one.
Contact us including your username and a note about your use case.
We’ll get back to you with more information about our beta program and provide access to our Static API if you’re eligible.

We’re excited to get our new API battle tested and as always please contact us if you have any questions.

iOS User Location Services for MapBox SDK

MapBox — 2012-05-14T00:00:00-07:00

We just released a new beta version of the MapBox iOS SDK with support for user location services. Watch the quick video below for an overview, or read on for all the details.

User location - showing it on the map

We’ve added an annotation class, RMUserLocation, that can be enabled on an RMMapView with a simple toggling of the showsUserLocation property. Regardless of whether you want to track and orient the map to the user’s location and heading, that data is available in the RMUserLocation so that it can be used in other portions of your app as well. And just like MapKit, you can query whether the user’s position is currently shown on screen with the isUserLocationVisible property.

User tracking - keeping the map in line

Beyond just plotting the user’s location on the map, you can enable tracking modes in order to keep the map in line with their location and compass heading. Just tweak the userTrackingMode property or, if you want more fine-grained control over the user interface, use the setUserTrackingMode:animated: method. You can choose between RMUserTrackingModeNone (the default), RMUserTrackingModeFollow to keep the map centered on the user’s location, or RMUserTrackingModeFollowWithHeading to also keep the map rotated in the direction the user is facing.

Delegate callbacks - staying informed

We’ve also implemented all of the map view delegate callbacks found in MapKit, whether you want to stay abreast of starts, stops, and potential errors when locating the user, changes to the user’s location and/or compass heading, or changes to the tracking mode in order to update your app’s user interface.

Sample app

We’ve built a sample app showcasing the SDK’s new user location services, as well as our worldwide street-level map MapBox Streets. It’s called MapBox Me and it’s fully open source on GitHub for you to download as a starting point.

Rapid development

This new addition of user location services in the MapBox iOS SDK brings us feature-for-feature on par with MapKit – and beyond. The new user location services come hot on the heels of our new, unrestricted mobile app offline caching policy and our screencast showing five minute integration of MapBox hosted maps into native iOS apps. Stay tuned as we continue to enhance our SDK and put more and more control into developers’ hands.

Improved British Rail Network Icons in MapBox Streets

MapBox — 2012-05-13T11:00:00-07:00

Today we’ve launched the first step in our efforts to improve transportation symbology in MapBox Streets - custom icons for London’s Underground and light rail systems, as well as the National Rail lines throughout Great Britain. The signs for these stations are iconic and will be better recognized on maps than our previous generic rail icons.

The map style accounts for stations that service lines on multiple rail networks, such as Underground stations with connections to National Rail routes. The stylesheet also handles clustering icons to avoid duplicate labels, since many stations are represented by multiple nodes (perhaps one for each track, or one for each entrance). At medium scales stations are grouped by name, and as you zoom in they become grouped by name then network. In most cases this makes for much cleaner cartography with minimal loss in accuracy.

Creating and assigning icons

We base the icons for a station on the contents of its ‘network’ tag. If a station services multiple networks, these will usually all be included separated by a semicolon, comma, or slash. Ordering is not necessarily consistent, so we normalize this tag by breaking it up into parts, stripping spaces, lowercasing everything, and assembling it back together in alphabetical order. This way two network tags that mean the same thing but are written differently will be made consistent.

Example network tag	Normalized
London Underground;London Overground	london-overground.london-underground
London Overground / London Underground	london-overground.london-underground

For stations that require multiple network icons, we design single images for the individual networks and then have a script check the database to generate all the necessary combinations. The individual images are merged as necessary with ImageMagick, and the names of the resulting files match the normalized network tag. The Carto code for assigning the icons is dead simple:

#train_stations_low[zoom>=15][zoom<=16],
#train_stations[zoom>=17] {
  ::icon { 
    point-file: url("img/rail/[network]-12.png");
    [zoom=17] { point-file: url("img/rail/[network]-18.png"); }
    [zoom>17] { point-file: url("img/rail/[network]-24.png"); }
  }
  /* ... label styles ... */
}

Mapnik replaces [network] in the image URL with the value of the current element’s network column, so only a few lines of code are required for potentially hundreds of different icons.

How the clustering works

We import OpenStreetMap data to a PostGIS database via Imposm. All railway stations are imported to their own table as points (even if a station exists in OSM as an area, it is converted). The import process adds two columns, groupby_high and groupby_low, that are used at render time for clustering points. The first column contains a normalized version of the stations name if it has one, or its OSM ID if it does not. The second column is the normalized name concatenated with the normalized network. As you zoom in, this accounts for stations that are near each other and have the same name, but are actually separate and require different icons.

We use these columns to group by in our PostgreSQL select statements. We generate a centroid from grouped points for a single label, and re-normalize the grouped network tags

In my presentation at FOSS4G North America last month, I talked about how we like to do as much data processing as possible at the import stage for MapBox Streets. The clustering of station points is something that has to happen at render time, however, because we only want to be clustering points that are relatively close to each other. For example, there are many ‘Union Stations’ across North America that should not be clustered together simply because they have the same name. Waiting until render time means that only points within a particular metatile will be considered for the grouping.

Check it out

Custom icons for Great Britain are live in MapBox Streets now, which you can start using with a free MapBox account. We are planning on bringing local transportation icons to more cities in the future - where should we travel next? Tweet us @MapBox.

How We Serve Faster Maps from MapBox

MapBox — 2012-05-11T00:00:00-07:00

MapBox is powered by open source, largely by Node.js, Backbone.js, Puppet, and Jekyll, and most of our stack is deployed on Amazon’s high-performance cloud infrastructure. This post describes MapBox’s infrastructure and how we’ve overcome the challenges of hosting worldwide maps at scale and created an efficient platform.

MapBox is designed as a fast, redundant, full featured mapping platform. In addition to efficiently serving millions of tile images to JavaScript clients like ModestMaps and Leaflet, MapBox manages composited maps, blends alpha transparent images, combines UTFGrid interaction on the fly, generates flexible embed codes, tracks every map view for analytics, and exposes all this functionality to our users through our website and API.

Our cloud

MapBox makes use of cloud services in order to scale quickly with demand and avoid centralization. At any time, we’re running a cluster of EC2 instances as our primary application servers. We can add more if needed within minutes. An Elastic Load Balancer divides traffic between the cluster of running servers and routes around ones that become unresponsive.

We use CloudFront as our CDN to distribute and cache tiles, interactivity grids, map embeds, and API request payloads in over twenty datacenters around the world so maps load quickly for everyone.

The custom maps that you design on your computer using TileMill are exported as MBTiles files and uploaded through S3 before being propagated to the application servers for permanent storage. EBS volumes in RAID configuration are attached to each application server and are responsible for housing the MBTiles files.

We use CloudWatch for monitoring and SES for transactional email. Finally, we use CloudFormation to manage these services, which allows us to quickly turn on new stacks for both staging and production purposes.

Inside our application servers

We have multiple application servers in operation at any given time. Each server is running a variety of processes, but the essential ones are described below.

The MapBox process

This is the primary custom Node.js application that serves as the heart and soul of our service.

The server process uses a separate port to handle requests against the MapBox API - requests for tiles, embeds, and metadata. This way we can cache these longer lived responses against the CDN and keep them super fast.

We can serve tiles fast because they’re stored efficiently on local EBS volumes attached to each application server in the form of SQLite database files, which we have standardized as the MBTiles format. Each tile server has disk-level access to the entire collection of MBTiles files. A robust sqlite3 library allows performance to rival other database formats and surpass speeds seen from loading individual map tiles off of a standard filesystem. With MBTiles we can also dramatically improve the performance of uploading, relocating, and deleting tiles versus filesystems that typically have very high per tile overheads.

The MapBox tile server is also responsible for our powerful compositing features that allow you to take multiple layers and combine them. This makes custom styles for MapBox Streets possible and results in maps that load significantly faster by reducing the number of HTTP requests and minimizing the amount of pixel information that is transferred. The results of composited tile requests are cached in our CDN as distinct tiles so subsequent requests are even faster.

The other port of this Node.js process is for serving application pages to our users, including the map page, login page, analytics pages, and the map builder features.

Both parts of the application are built with Bones, a client/server application framework for Node.js that uses Express and Backbone.js. It allows you to set up MVC structures once and reuse them both on the server and the client.

CouchDB

We store a variety of documents in CouchDB databases that are replicated among all instances in the cluster. These documents include users, sessions, tile request analytics, and map metadata.

Nginx

Nginx is used as a reverse proxy cache and for SSL termination. Certain images and pages are cached for anonymous users. Authenticated traffic on mapbox.com is handled over SSL, and we use Nginx to handle these connections.

Puppet

Puppet helps manage our server configuration: a micro EC2 instance runs as a dedicated ‘puppet master’ from which application servers pull configuration updates.

CloudWatch reporter

A custom Node.js application reports metrics to CloudWatch. It is invoked regularly via cron, loads metrics about the health of the instance, and reports the figures to CloudWatch. We can then configure alarms based on those metrics using the AWS console.

CDN log processor

Some non-critical processes only run on one application server, deemed “the supernode”. If the supernode fails, another instance in the cluster will automatically take on that role and spawn these unique processes. For instance, the supernode runs our log processor, which downloads tile access logs and feeds them to CouchDB to power our analytics pages.

This task tolerates failures because a newly crowned supernode can “catch up” with new logs if the previous node dies.

Monitoring, logging, and alerts

We use CloudWatch and Server Density for monitoring, Loggly for log analysis, and PagerDuty to send incident alerts and manage on-call schedules.

CloudWatch is responsible for monitoring system level metrics such as resource utilization (CPU, disk, memory), as well as custom metrics that we report to CloudWatch using their API. For example, we check every minute that our billing system is in sync with the application. If an interruption results in inconsistent data, an alarm is triggered in CloudWatch, which will dispatch a notification to the on-call engineer.

Server Density makes regular requests against multiple endpoints on each server and evaluates the response times. If a server doesn’t respond fast enough, an alarm is triggered.

Loggly allows us to quickly search through logs entries across the entire hosting system. We are running the same software on multiple machines and Loggly makes it incredibly easy to figure out where an error originated without having to login to each machine directly. We also have a few alerts configured in Loggly. Errors that occur infrequently are difficult to track down, but we can set up an alert and receive a notification as soon as one occurs.

Any time an alarm is triggered in CloudWatch, Server Density, or Loggly, it is sent into PagerDuty, a service that allows us to track incidents and send notifications to whoever is on-call. We have three engineers on-call 24 hours a day. PagerDuty will notify one engineer first and will automatically escalate to the other two if the primary engineer can’t respond to the situation immediately.

Development and deployment strategies

We develop MapBox entirely on our laptops - development never happens on staging server stacks. This preference affects what tools we use. We look for software, libraries, and services that can easily be set up on a computer so a new developer can get up to speed quickly. We also focus on making application bootstrap automatic, so many setup tasks are handled automatically when a new developer joining the project starts up for the first time.

We generally follow the GitHub approach to deployment where our master branch is always considered “deployable”. This means we never commit unstable code to the master branch, as it may be deployed at any time by any developer (often multiple times a day).

Instead of using Campfire and Hubot for kicking off a deploy like GitHub, we wrote a simple bash script for doing so. You pass in a few arguments, including the destination of the deploy (staging or production) and it connects to each instance (including the puppet master) and stages a clean build of the latest master branch. If the build completes on each server, the build is switched into production. It may not be as feature complete as tools like Capistrano, but it’s incredibly simple and easy to understand.

Jekyll and GitHub Pages

Static pages don’t really require an industrial strength solution like node.js - Jekyll hits the spot for making simple, fast, and reliable static sites like MapBox.com, MapBox Status, the Wax manual, and this blog.

Billing and support

We use Chargify’s secure, PCI compliant billing service to manage recurring payments and Tender for our support site.

Next up

This overview should give you a sense of how MapBox is engineered to serve custom maps quickly. Our benchmarks have shown that this architecture is able to scale incredibly well. Look forward to more posts about specific technology as well as updates to this post as our service evolves.

TileMill 0.9.1 Released

MapBox — 2012-05-08T00:00:00-07:00

We just released TileMill 0.9.1, which adds support for PostGIS 2.0, runs on the latest Node.js 0.6.17 release, and provides packages for the latest Ubuntu Long Term Support (LTS) distribution: 12.04 (Precise Pangolin). TileMill 0.9.1 is the culmination of a several month sprint on stability, with over 80 tickets closed. The full list of fixes and advances for this release can be found in the changelog. Here are a few highlights.

Large Rasters

Larger size rasters, like aerial imagery, are now viable in TileMill 0.9.1, and future releases will soon provide BigTiff support for reading geotiff’s over 4GB.

Error feedback

Error feedback (and log output) have been improved modestly throughout the application. Notably, when exports fail you will now see more details in the UI and in the TileMill logs. You can find where logs are located for each platform at the usage docs.

Map refresh

Users of large stylesheets like OSM Bright will now experience 5x faster map loading, significantly dropping the time required to see tiles rendered after saving a project. Even more speedups are in store for future releases as the Mapnik development team further optimizes.

Startup experience

Startups are smoother now on OS X and Windows. TileMill gracefully starts up even if fonts are encountered that cannot be loaded (OS X users of webfonts rejoice). The Windows startup process has been re-written and should no longer require being run as an administrator.

Tile Skipping

The export process now detects tiles representing fully solid colors and stores them only once in the mbtiles file, saving on storage space and encoding time.

Getting started with the new release

If you’re a new TileMill user, you can download the new release at TileMill.com. As always, don’t hesitate to ask us a question if you have trouble.

All upgrading users should note the items in the upgrade docs. Most notably, the meaning of marker-width and/or marker-size has changed. The sizes now properly represent pixels. To upgrade, simply edit stylesheets to make the values twice as large, which will maintain the previous look.

OS X users can simply “Check for Updates…”

Ubuntu users can type:

sudo apt-get update
sudo apt-get install tilemill libmapnik nodejs

Windows users should uninstall TileMill using the provided Uninstaller and manually delete any Start Menu items (since the 0.9.0 uninstaller was unable to clear them out).

We hope this release helps you enjoy making maps even more. We have a lot in store for the future, and don’t hesitate to get in touch with ideas or questions at support.mapbox.com.

State Of The Map USA: The Game

MapBox — 2012-05-07T00:00:00-07:00

State Of The Map USA just announced that this year’s conference will take place in Portland, Oregon the weekend of October 13th and 14th, and with the announcement launched a new online presence that is not your typical conference website. Saman and I collaborated to make a site that’s interactive in new ways: it’s a game.

The site uses Modest Maps at its core, with a lot of neat tweaks.

Since State Of The Map is for and about OpenStreetMap, the basemap is of course OpenStreetMap. But for this site we wanted a slightly different look, so I implemented a Modest Maps layer type called modLayer, which lets you define a custom function that uses HTML5 Canvas to modify each tile.

The function can pixellate tiles, recolor them, and much more, but in this case it tones down the map. Since this requires CORS support to work and the standard (in terms of image elements and canvas) has gone unimplemented in Safari, Internet Explorer, and Mobile Safari, it falls back to normal OSM tiles on those devices. The filter we use for this site is a good example of how this works.

The site uses mmg to position markers on the map and custom keyboard bindings to let the character walk towards them. It’s based off of the ThrowableHandler example from Modest Maps, but with a callback that allows the application to understand the walking direction and speed. You can check out the source to the handler, and, while this site doesn’t use a library for other keyboard events, we might adopt keymaster as the number of actions increases.

Saman created a series of walking and idle sprites to give the character a bit of life - bouncing, running, blinking, and so on. Some of the other elements, like the fish, are animated GIFs for combined using gifsicle and designed, like the sprite, in Inkscape.

Performance on the site is a matter of optimizing the many elements that could be ‘activated’ at any time - some of which are affected by location, and others more statefully by the actions of the user. Right now the site is relatively performant - it makes use of getAnimationFrame instead of setTimeout if available in-browser, but there’s a lot more work to do in making map-centered experiences like this fast and interactive.

The site has quite a few little secrets - places you can go, things you can do. It’ll probably gain quite a few more before the conference. After all, it’s all about creating new things with OpenStreetMap, open source code, and exploring.

Offline Caching for the MapBox iOS SDK

MapBox — 2012-05-04T00:00:00-07:00

We now offer unrestricted control for caching MapBox Streets on mobile devices with the MapBox iOS SDK, supporting both app developers who want fine-grained control of app storage size and performance, as well as for users who want the optimum use of bandwidth and the best performance for the maps in an app. In short, you can cache MapBox Streets custom styled maps (see how to put these into your app in five minutes in our latest screencast) as much as you’d like, for as long as you’d like, which is unprecedented compared to the policies of other map providers.

This will be huge whether you run a check-in service where users most often use maps around their home city, a role-playing game where you are defending your local turf, or a travel app where you want to select a region and take the map offline. This is all part of our larger platform push, supporting great developers by giving them total control of their maps.

Here’s the relevant section of the terms of service:

Proxying and/or intermediate caching of MapBox Maps is prohibited. Mobile clients may cache MapBox Maps on consumer devices for offline use and performance purposes. Further redistribution from a cache is prohibited.

How tiles are loaded

When a map view is shown and the user starts panning and zooming around to explore an area, the SDK requests the necessary tiles based on the layers that are shown. For example, maybe it’s just MapBox Streets that’s shown currently (check out how easy it is to get that in your app). The RMMapBoxSource tile source checks its local cache of tiles before ever trying to request data over the network, so if a tile has been downloaded recently, it is instantly recalled and used to reload into the map.

How the tile cache works

By default, the Route-Me rendering engine that our SDK is built upon stores the 1,000 most recently downloaded tile images. Once more room is needed, the oldest tiles are deleted from cache. The developer can also remove all tiles whenever desired with RMMapView’s aptly-named removeAllCachedImages method. But we’ve also added the capability to manage the tile cache based on length of time the tiles are in the cache.

For example, creating the RMTileCache with an expiryPeriod of 60 seconds/minute x 60 minutes/hour x 24 hours/day x 7 days = 604,800 will cause tiles to stay in the cache no longer than a week, without regard to the number of tiles, before the SDK will automatically delete the older tiles to keep things tidy.

Many potential uses

Allowing the developer this kind of control, as well as specifically enforcing it in our terms of service, really changes the traditional power balance in this type of scenario. The developer has complete control over giving their users offline capabilities, as well as refreshing map styles based on changes by the developer whenever they want, either with app updates or based on time constraints. The power really is in the developer’s hands.

Custom Styled Maps for iOS Apps in 5 minutes

MapBox — 2012-05-02T00:00:00-07:00

This quick screencast shows how easy it is to add totally custom designed street maps to a native iOS app with the new MapBox iOS SDK, our open source alternative to MapKit. Using the example app as a starting point, the video covers the process of styling a new custom map with your free MapBox account and loading that map into your app. Learn more about the MapBox iOS SDK at mapbox.com/mobile.

GeoDC Meetup on Crisis Mapping Tomorrow (Wednesday)

MapBox — 2012-05-01T00:00:00-07:00

The next GeoDC meetup is tomorrow (Wednesday, May 2) at 7:00 pm at Stetson’s and will focus on crisis mapping. With the Humanitarian OpenStreetMap board in town this week along with lots of international development folks for the InterAction forum, it is a great time to focus on how mapping can be used in humanitarian work, disaster response, and preparedness.

There are five lightning talks on the agenda:

Nicolas Chavent will talk about the Humanitarian OpenStreetMap team’s role in Haiti since the earthquake.
Harry Wood will walk through how anyone can help the Humanitarian OpenStreetMap Team respond in a crisis and grow the open geo data set in the impacted region.
John Crowley will talk about ongoing experiments at Camp Roberts to integrate crisis mapping into official geospatial systems of disaster response institutions.
Schuyler Erle will talk about community mapping and citizen reporting in the jungles of Mexico.
Kate Chapman will talk about her work bringing together village leaders and students to map critical buildings in Jakarta.

The meetup will start at 7:00 pm, with introductions and presentations kicking off at 7:30 pm. I recommend arriving early as it will be crowded, with 90 people RSVPing so far. We’ll be at the upstairs bar at Stetson’s near the intersection of 16th and U streets NW. For more details check out the event listing.

This meetup is just one of the events happening around this topic this week. The World Bank and InterAction are holding a free workshop on open source mapping tools on Thursday, and before that there is a panel at the Wilson Center discussing using tools like OpenStreetMap to better respond to disasters. For more information on these events, check out my Week in DC Tech roundup.

The GeoDC group meets once a month. For updates, join the meetup group or follow @geo_dc on Twitter.

Visualizing Global Forest Height

MapBox — 2012-04-27T00:00:00-07:00

At EcoHackNYC last weekend, I spent a lot of time talking with fantastic people about publishing maps from raster data using TileMill. A great post from NASA’s Earth Observatory on a new dataset inspired to make my own visualization of global forest heights, and I wanted to capture the process of working with raster data in TileMill.

Funky blue-to-fusia colors dramatically expose clear patterns of deforestation visible throughout the Amazon near Rodonia.

Raster mapping with open source tools

The high-resolution (1km) data is available for download from the project’s website. You’ll notice that it’s saved as a geotiff, an open raster format for geo-referenced images. A variety of open source tools exist for processing raster data. I used QuantumGIS to open the image and start styling. Once colors have been decided, GDAL allows you to burn your new color ramp into your image. After saving, use TileMill to bake your map to mbtiles and upload it to MapBox Hosting.

Picking out colors in QuantumGIS

After selecting the ‘style’ tab in the properties pane, set the Colormap drop down menu to ‘ColorMap’.

Now head over to the ColorMap tab. Set the ‘Number of Entries’ field to 5, and hit classify. This will classify your data into quantiles.

QGIS should default to a pretty ridiculous blue-to-red color ramp. You can edit the colors by clicking on them directly, or by specifying a pre-determined ramp from the drop down menu.

Using GDAL to color your raster data

Once you’ve picked out a color ramp you like, it’s time to use GDAL to permanently re-color your image. Grab the RGB values from the color picker, pictured above, and copy them into a .txt file with your favorite text editor. The first column will need to be your data values from qgis, followed by the red, green, and blue values, each separated by a tab. No need for column header. Here’s the ramp I used.

0   0   0   0
18.25   0   17  73
36.5    84  0   167
54.7    186 0   219
65  255 4   248

I tweaked the data values to exaggerate the color ramp, but the differences should be minor. Save the file as simard-forest-ramp.txt. Now you can run the following command from your command line:

gdaldem color-relief ~/<path-to>/Simard_Pinto_3DGlobalVeg_JGR.tif simard-forest-ramp.txt output-forest.tif

TileMill

Open a new project in TileMill and load your .tiff as a new layer. Get rid of the #countries layer, and change the map background so that your Carto looks like this:

Map { background-color: #000}

#simardforests {
  raster-opacity:1;
  raster-scaling: bilinear;
  }

Your map should appear. Just hit upload in the export menu to start sharing your map online. Read our documentation for more info on reprojecting geotiffs and working with terrain data. If you want more inspiration, check out some of previous work like AJ’s map of gridded population density

Thanks to Javier and the Vizzuality team for putting on EcoHackNYC. It was great to see so many highly motivated scientists, designers, data analysts, and hardware hackers come together to hack on some of the most compelling new technologies being used to address environmental problems. Among the highlights were csabuilder, a tool to crowdsource interest in Community Supported Agriculture, and new balloons for The Public Laboratory’s balloon mapping kit. One of the most impressive projects was Species Sphere, an interactive visualization of the international trade in wildlife built with CartoDB and d3.js put together in only eight hours. Check out Vizzuality’s blog for a full recap.

New 'Pay as You Grow' Pricing Plans

MapBox — 2012-04-24T00:00:00-07:00

We just launched new ‘pay as you grow’ pricing plans, allowing you to pay for what you use without needing to upgrade to the next plan tier if you have overages. We are now measuring all usage by map views to make our pricing plans easier to understand. Additionally, we just published the progressive discounts we offer high-volume subscribers.

Granular pricing

All MapBox accounts now have automatic overage pricing of $1 per 1,000 additional map views, allowing you to pay for the extra map views you use without having to upgrade your account to the next level when you go over your limit. For example, if you have a $5 Basic plan and see a spike in traffic, there is no need to jump all the way to a $50 Standard plan - you can just pay for the overages incurred. At at certain point, it will become more cost-effective to transition to a larger plan if you see consistent overages, but you can decide that at any time.

Increased Plus and Premium account limits

We’ve raised the map views for our Plus and Premium accounts by 100,000 and 200,000 views respectively, while offering them at the same prices as before. Plus accounts now provide 250,000 map views per month for $149 and Premium accounts now provide 1,000,000 views per month for $499. More details are available on the MapBox pricing page.

High-volume discounts

We’ve published our Premium-level cost structure. Now you can see at a glance the discounts we give for high-volume subscribers, ranging up to 30% off, or use the new calculator to figure out exactly what this means for you.

We want to be transparent, because MapBox is incredibly cost competitive. Our CPM - or cost per thousand map views - is several times lower than Google’s. And while we don’t include all the features and services of the Google Maps API, like imagery and geocoding, we offer beautiful design, custom styles, and the flexibility of open data, which is ideal for those looking for a custom world base map upon which to build a location feature for a website or mobile app.

All current subscribers have the option to opt-in to take advantage of the new plans. Just visit your account page and select ‘change your plan’. If you’d prefer to keep your current bandwidth-based plan, no problem - you’re automatically grandfathered in.

MapBox is free to get started. Try out making a custom map now. And let us know if you have any questions or would like to talk about a custom plan to meet your needs.

Surveying Brasilia with Walking Papers

MapBox — 2012-04-23T00:00:00-07:00

Last week in Brasilia Eric and I joined efforts with Luiz Carlos from the Track Source community, Fabricio and Lima from Transparency Hackers, and John from the Sunlight Foundation to survey several neighborhoods for street names and points of interest to add to OpenStreetMap. We were in town for the Open Government Partnership meetings and wanted to showcase the power of open geo data and take advantage of being in town to catch up with local community members and do useful work. Here’s a quick first report before we start entering the collected data into OpenStreetMap.

Getting started for surveying the Cruzeiro neighborhood of Brasilia. Photo: John Wonderlich.

Prep work

We prepped our surveys in Washington, DC by doing extensive satellite tracing to create good ground coverage. Before we got on the plane, we selected a series of neighborhoods by street name sparseness and accessibility and printed walking papers for them. Walking papers is a great tool from Michal Migurski offering essentially a quick way to print OpenStreetMap onto a series of papers, linking them back to OpenStreetMap with a QR code.

Underestimating the dimensions of Brasilia (it’s at least twice as large as a first look at the map would suggest), we had initially printed our walking papers too small but this was quickly fixed by printing new ones at the hotel before we got started. Next time, we’ll take a much closer look at satellite imagery before getting started in a city we don’t know.

On the ground

We had one team in a car surveying the neighborhoods, and another team walking. The car had the clear advantage in Brasilia, where traffic density is comparatively low and street blocks are large. Working with two to three people per team, we collected block name data and points of interest, entering it on our walking papers.

Aside from the walking papers, we used GPS trackers and cameras to capture points of interest and, from time to time, our assumed location on the walking paper. By synchronizing the camera’s time to the GPS tracker’s time, the photos can be geo coded and used later for reference. This saved us time jotting down long names of points of interest and gives us a way to double check the accuracy of our survey now.

Saving time by taking a picture of what otherwise needs to be jotted down. Screenshot from JOSM after geocoding.

Double checking by taking a picture of the assumed location on the map. Screenshot from JOSM after geocoding.

A special challenge was posed by the fact that Brasilia does not use street names, but rather block names. Brasilienses are quick to admit that the system is messy and indeed it was hard to wrap our minds around it. Even with the help of local Luiz Carlos, we had trouble, as the block system slightly varies in different parts of the city. In Taguatinga for instance, commercial blocks occupy a full city block, while residential blocks occupy two block halves around a street.

Taguatinga residential block sign. Q stands for Quadra (block), S for Sul (South). C is the alphabetic section letter, 9, 10, 11 are different block numbers found on this street.

Data entry

We’re using JOSM for data entry. In JOSM we loaded up our GPS tracks and synchronized them with the pictures we had taken. Once the pictures are synchronized, it’s straightforward to enter data in OpenStreetMap using the information in pictures and the notes on our papers… if it wasn’t for Brasilia’s block system.

Synchronizing the GPS tracker time with camera time by simply taking a picture of the GPS time. Screenshot shows the JOSM dialog comparing the actual photo time with the time read from the GPS tracker.

Before we enter actual block name data, we need to better resolve how to capture them. Turning block names into street names (like Google and others do) forces awkward compromises where streets are shown between two differently named blocks. At the same time, OpenStreetMap does not seem to provide an appropriate tag for blocks. Existing block name data in Brasilia is entered inconsistently. I invite you to join our discussion on the mailing list to hammer out a solution.

If this has gotten you interested in surveying your own neighborhood or the next place you’re traveling to, LearnOSM - Using Walking Papers is a great guide to get started.

Open Mapping Coming to a Campus Near You

MapBox — 2012-04-23T00:00:00-07:00

We just wrapped up a mapping sprint adding building footprints to college campuses in OpenStreetMap, which we prioritized since that’s where students check in using social apps that are switching to MapBox Streets. To find colleges that needed more data added to OpenStreetMap we started by looking at schools where students spend more time on their phones than on their studies: party schools.

We researched Playboy’s Top Ten Party Schools and found a handful of campuses that lacked building footprints and pathways in OpenStreetMap. Then we combed through all U.S. colleges with more than 25,000 students from U.S. News and World Report’s National University Ranking and emerged with a list of roughly 20 universities that needed additional features. Here are our results.

University of Oregon

Before editing, the University of Oregon had many footpaths on OpenStreetMap but zero buildings. After tracing all buildings, quick communications with the university got us the greenlight to use the building names from their website.

Louisiana State University

With almost 29,000 students, Louisiana State University had almost no buildings on the map. You can see how that’s changed after our tracing efforts.

University of Central Florida

The University of Central Florida is the second largest campus in the nation and one of the fastest growing schools in the world, with more than 50,000 students.

Here’s a close up of UCF’s athletic facilities.

Adding your campus on OpenStreetMap

While adding building and footpath traces to OpenStreetMap, we reached out to these colleges to get permission to add building names to OSM based on their online maps. This worked remarkably well, and we received some great feedback and learned exciting things about ongoing campus mapping projects in the process. University of Oregon, for instance, is beginning a program where geography students will update and maintain the University’s buildings, paths, and roads in OpenStreetMap. Some schools like the University of Delaware and the University of Maryland are already using OpenStreetMap as the base layer for their online campus maps. The fact that the majority of U.S. universities already had comprehensive coverage in OpenStreetMap shows how easy it is for schools and students to keep their data up to date. As OSM becomes increasingly easy to edit and customize, we hope to see many other universities using and maintaining their campuses here.

If your campus is missing on OpenStreetMap, you can easily become a contributor and add your most frequented campus buildings to the map. All you have to do is sign up and follow our directions to begin editing. Once you have JOSM downloaded and running, trace any necessary footpaths and use the buildings tool to trace university buildings. To do this go to JOSM’s Preferences, select the plug-ins tab, and add the “buildings” plug-in.

If you know the building names, you can add them. If you don’t you can always try emailing your university to get permission to use their campus map as a naming reference. After you have the buildings and names added, upload changes, and now your campus is on the map!

Using Open Source Tools to Map Health Data

MapBox — 2012-04-21T00:00:00-07:00

Today I’m at the Global Health and Innovation Conference at Yale University to talk about using open source tools to map public health data. The annual conference brings together students and professionals to discuss advances in technology and tools for global health and international development. I’m excited to hear what others are working on and to share examples on how MapBox’s tools can be used to map global health data.

Specifically, I will walk through how to make this map.

Hexagon gridded density map of Kenya’s health facilities

How to make the map

A map like this can be published using open source tools that are freely available: QuantumGIS and TileMill. I’ll talk about how the greater availability and openness of organizations and governments make it easier to access key datasets that can then be used to gain a better understanding topics and regions. For this map, I simply downloaded the master health facility list made available by the Kenya Open Data project. I can browse and manipulate the data using QuantumGIS. Here I’ve used the grid function to create an arbitrary grid and then counted the number of facilities within each grid polygon. Using a grid for density allows for the quick aggregation of data along more understandable geographies than an administrative area.

I then designed a custom map in just a few clicks and lines of code. TileMill allows me to create custom classes for defining a scale and determine the best visualization of the data to create a fully interactive map. I can then export the map to publish it on the web or use on a mobile device. Using MapBox Hosting, I can publish to the web and leverage the API and embed functionality to post the map here.

I’ll be presenting in Session L31: Policy & Research Tools at 1:15 pm. Here are the slides for my idea pitch:

Integrating MapBox with ESRI and more in Wax

MapBox — 2012-04-19T00:00:00-07:00

MapBox Wax now supports ESRI’s Javascript API 2.8, allowing you to easily use tiles from your MapBox account in your ESRI projects and use super-fast interactive overlays.

Open Source Integration

(map)

You can download Wax v6.0.0-beta5 and see demos & docs of the interaction control and connector now.

Wax already supported many APIs: Leaflet, Modest Maps, OpenLayers, and Google Maps API v3. In most cases, the very basic integration code is minimal - Leaflet’s connector, for instance, is only 11 lines long.

Containing the variation between APIs makes it possible to have more consistent code. Here are the APIs for all of the available toolkits:

// Modest Maps
map.addLayer(new wax.mm.connector(tilejson))
// Leaflet
map.addLayer(new wax.leaf.connector(tilejson))
// Google Maps
map.mapTypes.set('mapbox', new wax.g.connector(tilejson));
// OpenLayers
map.addLayer(wax.ol.connector(tilejson));
// ESRI
map.addLayer(new wax.esri.connector(tilejson));

If you don’t want to use Wax in your project, just check out the source code of the connector and use it as a guide - it’s open source.

In order to make this spread maintainable, Wax v6 has a high level of abstraction that allows the new ESRI interaction control to be 50 lines of concise code - containing application logic in the shared library.

Using the ESRI Javascript API

The ESRI API is most similar to OpenLayers in that it devotes much of its complexity to dealing with projections and coordinate systems, and treats much of the web component as an implementation detail. This has some negative ramifications in that Wax needs to rely on a private API in order to provide interactivity - the API doesn’t document access to tile elements.

ESRI’s Javascript API is actually available for download by those who have ESRI accounts. Despite taking advantage of a great deal of open source code in the Dojo toolkit, ESRI’s code is explicitly copywritten and labeled as TRADE SECRETS: ESRI PROPRIETARY AND CONFIDENTIAL. This is unfortunate in an ecosystem of open and understandable code.

Being able to mix tools is what enables people to create powerful stories through maps. Wax is one of the many tools that can be built from the MapBox API and the open standards we’ve been pushing like TileJSON and UTFGrid. Recently NPMap from the National Park Service and slashmaps from Seabourne Consulting have demonstrated other ways to use the toolkit and integrate maps into websites.

Integrating with ESRI and Google Maps open up new possibilities for MapBox users, and we hope this bridges web and GIS worlds, getting the best of both.

Introducing the MapBox iOS SDK

MapBox — 2012-04-17T00:00:00-07:00

We’re proud to introduce our MapBox iOS SDK, a powerful open source tool that lets developers on Apple’s iPhone and iPad build custom, high performance maps with tight integration into MapBox Hosting, offline maps capabilities, embedded interactivity, and gorgeous customizability by leveraging our design studio TileMill.

We’re kicking off this beta release with a suite of resources that we hope to enhance going forward.

SDK codebase

Open source on GitHub and based on the popular Route-Me mapping library, our SDK is everything you need to add custom maps to your app. Pan, pinch, and zoom and watch your map instantly snap into place. We will continue to develop enhancements to the codebase and merge in improvements and bug fixes from the open source community. You can follow the issue queue to track our progress and open feature requests of your own.

Custom styled maps

Get direct integration with MapBox Streets, providing worldwide block-level coverage based on high quality open data from OpenStreetMap. MapBox Streets comes in a variety of gorgeous preset color schemes or you can customize the features of your map to fit the look and feel of your app. This is all built right into the control panel of your MapBox.com user account, letting you change the colors of the base map in seconds. It’s a beautiful alternative to Google Maps.

Overview documentation

We’ve created general documentation on the installation of the tools, major parts of the SDK, and the use of the main classes. Read this if you want a solid technical overview. We are also looking into Xcode-native API documentation available in an Atom feed. We really like what projects like BlocksKit have done with this. Follow the enhancement request on this for future updates.

Example iOS app

We’ve built a simple open source app showing use of online, offline, and embedded interactivity map layers in a straightforward iPhone example. We’ll continue to update this app as our API and features evolve.

Mobile apps guide

We have a planning resource for developers thinking about mobile apps - pointers to technology and examples and important considerations for thinking about maps in a mobile context.

Support resources

On our support site, we’ve created a public forum for discussing use of the SDK in your own apps. Like the rest of our support site, we hope to grow the discussion archive here and continually make it more useful to app developers.

Go make an app!

While this is a beta release, we are planning to evolve the SDK rapidly in the coming months. We’re sure that the year of the open map still has a lot of surprises in store and we want to remain on the forefront of the most dynamic mobile platform as they continue. Go give our SDK or example app a spin and see how you easily you can start using custom maps created with open source tools in your own development.

How to Map Where You've Mapped in OpenStreetMap

MapBox — 2012-04-16T00:00:00-07:00

Over the last month we’ve been tracing satellite imagery in OpenStreetMap, most recently the Sahel region of West Africa and Brasilia. A major focus throughout the process has been on how best to analyze and fix OpenStreetMap data to make it the best world map available, including ways to track your own contributions. Here’s a quick step-by-step tutorial on how to visualize the areas of the world that you have mapped on OpenStreetMap, like we have in the example below. If you want to get involved in improving OpenStreetMap, check out our public issue tracker.

Our team’s contributions to Burkina Faso

Tools to start visualizing

Before getting started, you’ll want to be familiar with OpenStreetMap’s open source desktop editor JOSM and the process of tracing satellite imagery. Then to begin download the region you want to visualize and save the layer as an .osm file.

Next you need to convert this OSM file to GeoJSON format for styling in TileMill, which will require installing Python 2.7 if you don’t have it. To do this Tom wrote a python script that outputs each node as a geo-referenced point along with the user associated and a timestamp marking when it was created.

from xml.etree.ElementTree import ElementTree
from datetime import datetime
import time
from sys import argv
import json
tree = ElementTree()

if (len(argv) < 3):
    print "specify an input & output filename, and year. input is osm, output is geojson"
    exit()

tree.parse(argv[1])

geojson = { "type": "FeatureCollection", "features": [] }

nodeidx = {}

print 'mapping nodes'

for n in tree.iterfind('node'):
    if (n.attrib.has_key('user')):
        pt = {
            "type": "Feature",
            "geometry": {
                "type": 'Point',
                "coordinates": [float(n.attrib['lon']), float(n.attrib['lat'])]
            },
            "properties": {
                "user": n.attrib['user'],
                "version": n.attrib['version'],
                "timestamp": time.mktime(datetime.strptime(n.attrib['timestamp'], '%Y-%m-%dT%H:%M:%SZ').utctimetuple())
            }
        }
        geojson['features'].append(pt)

json.dump(geojson, open(argv[2], 'w'))

Copy and paste this code into a text editor and save the file as converts-times.py to the same directory housing your OSM file. Then navigate to this directory in terminal and run the following command:

python converts-times.py ouagadougou.osm ougadougou-time.json

This will output the oouagadougou-time.json file in the same directory. Now you are ready to import into TileMill. In TileMill create a new project and navigate to the layer list in the toolbar to add a layer.

In the Datasource browser navigate to your .json file, give it an ID, and save the layer.

Styling in TileMill

Now comes the fun part - styling your visualization. For our purposes, we wanted to gauge the progress being made in the regions we were mapping while also providing the context to allow viewers to locate where they are in the world. With that in mind, we went with a dark gray marker fill for the layer as a whole, while styling each individual user differently with bright colors to make their contributions pop.

Style.mss

Map { background-color:#000; }

#countries {
  line-color:#fff;
  line-join:round;
  line-width:0.5;
  }

@blue: #00aacc;
@magenta: #ff0094;
@green: #0F0;
@red: #fc0303;
@yellow: #FF0;
@grey: #2d2c2c;

#ouagadougou {
  marker-width:1;
  marker-line-opacity:0;
  marker-allow-overlap:true;
  marker-fill:@grey;
  [timestamp >= 1330581600] [version = '1'] {
  [user = 'Chad Lawlis'] { marker-fill: @magenta;} 
  [user = 'Rub21'] { marker-fill: @blue;}
  [user = 'Dan Wood'] {marker-fill: @yellow;}
  [user = 'mayar'] {marker-fill: @green;}
  }
}

We also wanted to limit points for each user to only nodes created (excluding nodes altered or tagged) to provide the most honest representation of OSM growth, while limiting individual contributions by date added. We went with all nodes created on or after March 1, 2012, using the unix timestamp 1330581600, which you can convert with any time stamp translator online.

With a bit of html/css in the legend, you are ready to export the visualization. Navigate to Export in the main toolbar and select PNG.

Hold shift while dragging over the area you wish to export, enter the file size and name, and select export.

Once the file has uploaded, hit save. You now have a visualization of where you have mapped in OpenStreetMap.

Remember to include OSM attribution where appropriate, and be sure to check out our public issue tracker for OpenStreetMap contributions if you are looking to get involved.

Letter-Spacing in a Multilingual World

MapBox — 2012-04-13T00:00:00-07:00

On Wednesday at the FOSS4G North America conference here in Washington, DC I spoke about some of the technical and design details that went into the cartography of MapBox Streets. One of the subjects I touched on was typography and methods we are using to make it work well in many different languages at once.

The main typographic tool I discussed was letter-spacing because of the various challenges we ran into with it. This is a design element that is useful to indicate that a label covers a relatively large area, and something we also apply to street names to keep the letters from bunching up around curves.

The Mapnik renderer supports letter-spacing and can be adjusted with Carto in TileMill. This is the simplest and most straight forward way to define letter-spacing:

text-character-spacing: 2;

In the past this would only work for labels positioned on points, not labels positioned along lines (as road names often are). For a long time we took advantage of Mapnik’s replace() feature to fake letter-spacing in these situations by replacing every character in a label with itself plus a space (or multiple spaces). This is no longer necessary with bleeding edge versions of Mapnik.

There is still a problem though. Our map of the world contains text in many different languages and writing systems, including scripts like Arabic and Bengali. The characters in these scripts are meant to flow together and remain attached, not spread apart at all. Applying letter-spacing effectively breaks such text.

To get around this we come back to the same text replacement hack we were using before, but with a more specific expression to match.

text-name:"[name].replace('([^؀-ۿঀ-৿])','\1 ')";

Here we are replacing most characters with themselves plus a space, however the regular expression does not match characters falling within the Unicode blocks for Arabic and Bengali characters. (Note that as of TileMill 0.9.0 the replace feature does not support Unicode text in the Windows or Mac versions, so this is a Linux-only example for now. The next update should address this issue.)

Doing these replacements at the character level is important because labels are not necessarily restricted to a single language.

Something important to note in the above code snippet is that the replacement space is a non-breaking space. This is necessary for labels placed on points where there will potentially be a line break. Using a normal space in that situation could cause the label to break in the middle of a word.

Another thing about spaces - there are lots of different kinds. Wikipedia lists about two dozen in Unicode. What’s cool about this is that you can use some of them to alleviate the lack of control you have with the replacement hack versus traditional letter-spacing methods. Only want to space out your text a little bit? Try a thin space (U+2009) or a hair space (U+200A). There’s also a narrow no-break space (U+202F), useful for slight spacing in multi-line situations.

Ideally we can figure out ways to improve the flexibility of typography tools in Mapnik. Hermann Kraus has proposed a Google Summer of Code project to address issues relating to non-Latin text, which will be an important aspect. But for the short and medium term we’ll continue to look for hacks like this to make sure our maps look great in all languages.

I’ll blog about several other ideas from my Cartography with TileMill, PostGIS, and OpenStreetMap session, and in the meantime here are my slides.

OpenStreetMap Open Government Partnership Mapping Party in Brasilia

MapBox — 2012-04-12T00:00:00-07:00

OpenStreetMap is a critical open data platform for governments embracing open data and is shaping up to be a big focus at the upcoming OGP High-Level Conference in Brasilia next week. Eric and I are participating in Tuesday’s OGP Innovation Village, where we will present our open source mapping tools and MapBox Streets, which is powered by OpenStreetMap, and the panel “Open Data As a Means, Not an End: Getting to Impact” on Wednesday morning. To kick off the OGP meetings we will spend Sunday afternoon mapping Brasilia in OpenStreetMap. We would love it if you joined us.

A lot is already moving. Brasilia has great coverage on OpenStreetMap and it is rapidly improving. Ruben, Chad, and Dan from our team have spent the last two weeks tracing many of Brasilia’s surrounding cities. Here is a look at the progress made by all OpenStreetMap contributors since the beginning of the month.

For our mapping on Sunday on the ground, we are excited to add local knowledge. We plan to prepare walking papers, grab a camera and a GPS, get in a car, and collect street name and point of interest data. We are still figuring out which areas to focus on but we’ll likely pick Taguatinga and around as this is most accessible to us. Let us know of specific areas you think would benefit from better local geo information.

If you would like to join us, we will meet on Sunday (that’s April 15) in the lobby of the Comfort Inn, Setor Hoteleiro-Projeção I, Taguatinga at 11:00 am. You don’t need anything particular to participate, and if you have a camera and a GPS (maybe just your smart phone) bring it along.

If you’re interested, please send me an email to alex at mapbox dot com by Saturday or find us on Twitter at @ericg and @lxbarth. Looking forward to it!

Expanding Mapnik's Cartographic Capabilities

MapBox — 2012-04-12T00:00:00-07:00

We are well into the the year of the open map, and people are waking up to what this means for them in all areas of data and design. One size simply doesn’t fit all where maps are concerned and we are actively developing new features in core Mapnik to give more power to designs.

Up until now Mapnik has concentrated mainly on vector processing - you configure your data sources, apply styles and filters, and get a beautiful map at the end of the process. In the case of tiled maps, these are small raster images which are assembled into the final map. This is where Mapnik would usually stop, after all there lots of tools out there to do post-processing. But bearing in mind the new innovative ways people want to use maps, things would get interesting if you could embed image processing/filtering in that chain. Both vector and image processing should play together, complimenting each other.

Consider the recent beautiful ‘watercolor’ maps from Stamen, or even our own process at MapBox of assembling (compositing) tiles from multiple sources. You can apply post-processing on a rendered map image, perhaps splitting a map into multiple maps, then processing using third-party tools and finally combining them. But this requires a programmatic approach. Granted, writing a Python or JavaScript to accomplish this is reasonably easy and fun, but having this implemented internally opens new, interesting doors.

What about making it possible to control both vector and raster processing from within the UI? This would give the power back to cartographers and designers and give them access to the custom cartography they’re looking for. If this is the goal, here are some first steps to get there. The features I’m working on fall into two categories: vector and raster processing or ‘vertex converters’ and ‘image filters’.

Converters

In the vector domain, we have ‘converters’ whose concept is influenced by AGG design. A ‘converter’ takes input (vertex source) and applies some kind of useful transformation on each vertex. Most importantly, they can be chained. For example, ‘vertex converter’ pipe-line for closed paths (PolygonSymbolizer) looks like:


[original source] -> [projection transform] -> [viewport transform]

To improve rendering speed at high zoom levels a ‘clip’ converter can be introduced:


[original source] -> [clip_poly] -> [projection transform] -> [viewport transform]

LineSymbolizer uses two more converters (conv_stroke and conv_dash(optional)):


[original source] -> [clip_poly] -> [projection transform] -> [viewport transform] -> if(dash):[dash conv] -> [stroke conv]

There is also a ‘smooth’ converter which automatically approximates straight segments with Bezier curves, and can be plugged into this chain:


[original source] -> [clip_poly] -> [projection transform] -> [viewport transform] -> [ smooth val:0.5] -> if(dash)[dash conv] -> [stroke conv]

The ability to ‘lazily’ chain conversions is a powerful concept. More converters can be added - for example “svg-styled transform” to offset/rotate/scale geometries. There are also use cases when clipping is undesirable and needs to be switched off (e.g curvy ocean text).

From the user’s point of view, we want to be able to mix and match converters easily, but there is a problem here. While the use case is clear, the ‘lazy’/compiled nature of these pipe-lines makes implementing them cumbersome. All possible permutations result in lots of lines of repetitive/redundant code. I think I’ve solved this with some help from meta-programming Boost.MPL. I abstracted the ‘converter’ logic into something more managable and easy to use:

// c++   
// declare all possible converters to choose from for this particular case (LineSymbolizer)
typedef boost::mpl::vector<clip_line_tag,transform_tag, smooth_tag, dash_tag, stroke_tag> conv_types;

// create 'converter' object supporting types declared above
vertex_converter<box2d<double>,rasterizer,line_symbolizer, proj_transform, CoordTransform,conv_types> 
converter(ext,*ras_ptr,sym,t_,prj_trans);

// enable individual converters as required
if (sym.clip())  converter.set<clip_line_tag>(); // optional clip
converter.set<transform_tag>(); // always apply coordinate transform
if (sym.smooth() > 0.0) converter.set<smooth_tag>(); // optional smooth converter
if (has_dash()) converter.set<dash_tag>(); // optional dash       
converter.set<stroke_tag>(); //always apply stroke converter

for (unsigned i=0;i<feature->num_geometries();++i)
{
  converter.apply(geom); // apply chained conversions and feed into rasterizer
}

The above might look scary but it is by far more readable and maintainable than what we have currently.

Let’s see how this functionality to turn on/off converters can be used. One such example is applying ‘smoothing’ to water polygons:

Original style from demo/python

  <PolygonSymbolizer fill="rgb(153,204,255)"/>

Smooth water polygons

  <PolygonSymbolizer fill="rgb(153,204,255)" smooth="0.7"/>

In the future I’m sure we can come up with some interesting converters to add to the pipelines.

Image filters and compositing

This is another area I’m working on and where I think Mapnik can lead the way. I’m experimenting with different ways of applying compositing and filters. As it currently stands, compositing can be applied at <Symbolizer> and/or <Style> level e.g:

 
  <Style name="my-style" comp-op="color_dodge"/>
  <PolygonSymbolizer comp-op="src_atop"/>

All compositing modes defined in SVG 1.2 are supported.

One change to be aware of is that a ‘pre-multiplied’ pixel format is used throughout the rendering process now - it is important for correct compositing. At the end, the final image is ‘demultiplied’ to play nicely with PNG format which expects plain (not-multiplied by alpha) R,G,B values.

This week I also started work to implement image filters in the Mapnik core. The idea is to provide a framework for adding and chaining image filters at <Style> level. Similar to converters in many ways. So far, I implemented 3x3 2D convolution kernels as in Gimp using Boost.GIL library. The syntax to expose this has not been finalised, but I envisage something along these lines:

<Style name="my-style" image-filter="grey|blur:rx=4,ry=4|emboss|..."/>

Here are some maps I made using and abusing some of the above features:

Compositing/Blur/Smoothing

Embose

Sobel

Blurred text halos

Watercolor textures/neon roads

These new features are all works in progress and can be pulled from origin/compositing branch.

Transcript of Rendering the World Presentation

MapBox — 2012-04-11T00:00:00-07:00

This is a transcript of my Rendering the World session for anyone who was not able to attend the FOSS4G conference. This transcript is based on my memory and is not a recording so there may be differences from the actual presentation.

My name is Young Hahn. You can follow me on twitter or on GitHub. Today I’m going to be presenting not just my work but the work of my colleagues: AJ Ashton, Dave Cole, Konstantin Kaefer, Artem Pavlenko, Dane Springmeyer, Tom MacWright, and Will White. If you’re going to follow anyone, please follow these people.

We work at MapBox here in Washington, DC. MapBox is a mapping company that focuses on open source and open data solutions for web and mobile mapping. Our best known product is probably the open source map design studio TileMill.

MapBox Streets

We recently launched a new product called MapBox Streets. MapBox Streets is a base map with global coverage - we’re in New York, Paris, Tokyo, and everywhere in between. It includes deep zoom levels so you can see individual city blocks, tunnels and subway stations.

Great, we’ve got a cool map. So let’s render the world! Not so fast. It turns out the world is a really big place. Big can mean a lot of different things, even in the context of rendering the world. For the purposes of this talk, big is going to mean two things. One, taking a long time to render. And two, taking a lot of space to store.

Suppose you were to start up TileMill and ask it to render the world down to zoom level 15, 16, maybe 17. And say you put aside all kinds of other performance factors – not worry about PostGIS tuning, or optimizing styles, or even Mapnik performance. Say we wanted to render a very simple or blank map down to zoom level 17. TileMill would show you a screen that looks something like this: 443 days remaining.

And if you were to do a little googling around the web for how much space it would take to store all these tiles you would probably run across this OSM wiki page. It would take some 54 terabytes of space to store OSM tiles up to z17 or 18 and OSM’s tile images are really small.

If you’re new to tile-based mapping or even if you’re familiar you might think: WTF. It’s pretty hard to wrap your head around how or why these numbers are so big.

Tile-based mapping

Let’s do a quick review of tile-based mapping. By tile based mapping I mean maps that load progressively through 256x256px image tiles. Usually these images are square, but they can really be any size and dimension. These tiles are rendered at multiple zoom levels. So using the ubiquitous web mercator projection, zoom level 0 has 1 tile and contains the entire world. At the next zoom level, we double the amount of detail so our 1x1 grid becomes a 2x2 grid and contains 4 tiles total. The next zoom level is a 4x4 grid and contains 16 tiles. The next 8x8 and 64 tiles. We can calculate the number of tiles at a given zoom level using this equation: 4^z is the number of tiles at a given zoom level z.

Then suppose we have a super fast map renderer and really tiny tiles. Say we can render 1000 tiles/sec and each of our tiles is about 400 bytes. These numbers are pretty arbitrary but you’ll see in a second that tweaking these numbers is not going to make much of a difference.

At zoom level 15 there are 4 to 15 tiles, or 1.1 billion tiles. That would be 12 days of rendering time and take 644 GB of storage. z16 will be four times that amount, so about 4.3 billion tiles, 50 days and 2.6 terabytes of storage. And z17 will be 17.2 billion tiles, 200 days, 10.3 terabytes of storage.

With these numbers you can see that even if we worked really hard to optimize PostGIS or tune our renderer – say we could double our speed and get to 2000 tiles/sec – we’d be looking at a render time around 100 days. If I kicked this off now it would probably finish in time for the next FOSS4G.

Live rendering

Common sense says: don’t even try this. It’s not reasonable. Instead there’s some accepted wisdom to how you serve a global baselayer. Users are only going to request a small fraction of these tiles at any given time. So you could render each tile only when it’s requested by a user.

This scenario looks like this: on the right is a big beefy render server and on the left a computer or mobile device that wants your map. When the device requests the map the server renders it and sends it over.

Of course in reality there may be dozens or hundreds of devices that are viewing your map at any given time and your beefy render server is going to start looking a lot less beefy. So the solution is quite straightforward - you just add a lot more render servers, right?

Not so fast. It turns out the internet is actually a really big place. You’re going to need to scale to not hundreds but thousands of requests per second. Your maps are going to start to get slow and your render servers may actually go down.

Render servers are slow, costly and stressful. We have a small team at MapBox and any time we’re spending babysitting servers is time we could be spending on our open source projects or improving the experience for our users.

On the other hand, serving pre-rendered tiles is fast, cheap and reliable. We don’t have to worry about spikes in traffic or whether our servers can keep up – we know all of the tiles are there and are cheap and fast to serve.

Rendering the world

These are the reasons why we set out to try to solve this very hard problem. So how do you render the world?

We had two tools in our belt when we started work on this problem that I didn’t know would play an important role, but they did. Just keep these in the back of your mind as we go through the problem.

Tool #1: MBTiles is a tile storage format we’ve been using for a while. It’s an sqlite-based format that can store an image once and reference it multiple times for different tile coordinates.

Tool #2: One of the features of MapBox Hosting is fast image compositing. We take multiple pre-rendered tiles and blend them together into a single image and we’ve found that this can be quite fast and efficient.

Tile redundancy

So back to our problem. What’s actually on these 17.2 billion tiles at z17? If you were to pluck a tile out of the pile at random you would sometimes get this - some roads, buildings, parks, lakes. But mostly you’ll get this: a patch of blue somewhere in the middle of the Pacific. In fact, mostly is about 60% of the time. You may remember from gradeschool that the world’s surface is some 70-80% ocean but in the web mercator projection that percentage is more like 60. Antarctica and Greenland are really huge.

This solid blue tile is an example of redundancy. Once I’ve seen this blue tile once, I actually know what 60% of the map looks like - it’s exactly the same. It’s a lot like this example from information theory: “if u cn rd ths msg u cn gt a gd jb w hi pa” or “If you can read this message you can get a good job with high pay.” It turns out some 50% of the letters in English are redundant. You can get your message across without these letters.

How much of the world is actually redundant? This next slide has a clue where we’ve split the world up into 4 different layers. For now let’s focus on the bottom two, the land and water layers. If we take just these two layers our map looks like this. Now we know that of all these tiles, about 60% of them are solid blue areas of water. Pretty good.

But wait, there’s another type of redundant tile on this map. If you were to pan to the middle of Kansas you’re going to see a lot of this tile here: the solid land tile. This tile is also redundant. Once you’ve seen this sandy color you know what all the other solid areas of land look like.

The only type of tile on this map that contains information is a tile containing coastline. Tiles that show where water and land meet, that show a distinction. Under this analysis then, some 38% of the world is solid land leaving just 2% of the tiles containing coastline information.

That’s great but a map with just water and land is pretty boring. What about roads, parks, buildings… human stuff? It turns out that humans really haven’t touched much of the earth at all in this respect – about 1%.

Thinking of tiles in terms of information and redundancy changes our equation from the beginning: 4^z. Consider a map with just a single point. At z0 it would take 1 tile to capture this point. But at z1, if we throw out the redundant tiles, it still takes just 1 tile to represent this point. And at z2, z3, z4 it still takes just 1 tile to represent the point. Instead of quadrupling at each next zoom level, the number of tiles it takes to represent a point stays constant.

If you play this thought experiment out you can see that for points the equation is going to be a lot more like 1^z, and for lines somewhere between 2 to 3^z. For polygons interestingly you only need to represent the perimeter in tiles like in our analysis of the land and water layer where only the coastline contained actual information.

Getting to 2%

Theoretically then we only need to store some 2% of those 17.2 billion tiles. But to do this we’ll need a storage format that actually represents redundancy. MBTiles does this very well. It can store an image once and then reference it many times for different zxy coordinates using an id. So we only need to store the blue ocean tile once for billions of different coordinates.

So 10 terabytes times 2% is 200GB. Pretty good.

What about render time though? Remember those 400 days it would take to render the world? We still have to render a tile before we know that it’s a solid blue ocean tile and can be thrown out or one that contains actual information. Can you know a tile is redundant without rendering, querying or even iterating? If we take Mapnik and PostGIS out of the equation, even just iterating over 17.2 billion zxy coordinates in node.js, python, whatever language you choose, takes a long time.

First, let’s look at how our latest release TileMill 0.9.0 renders an export. It does a scanline. If you’ve ever pre-seeded or rendered tiles you’ll be familiar with this scheme: you start at the first zoom level and do one row, one tile at a time. Once you’ve finished all the tiles at that zoom level you move on to the next, and so on.

In TileMill master we’ve been working on a new rendering scheme. The pyramid scheme takes a tile and looks at its child tiles on the next zoom level. And then it takes each of those and looks at their children. And so on, recursively. If it finds a tile that is redundant at z5 it assumes that all of its child tiles are redundant as well at z6, z7, z8 all the way down to z17. With this method we can skip millions of tiles per second.

It works. Mostly. There are some cases where this scheme will skip tiles too aggressively and I worked hard with our cartographer AJ so that there’s usually an indication at lower zoom levels of features coming up on higher zoom levels.

So 2% of 200 days and we’re down to 4 days of rendering. Not bad. With tiles pre-rendered all our servers need to do is composite them together and serve them. We’ve found that it’s fast, affordable and reliable.

Of course, the truth is a bit more complicated. We also had to tune PostGIS, optimize our OSM import, materialize imposm tables, find a high i/o cloud service, parallelize our renders - we do 32 of them at a time - and optimize our sqlite storage. If you’re curious about these things, please ask us. Thank you.

Geo Meetup in Guadalajara on April 18th

MapBox — 2012-04-09T00:00:00-07:00

Next week on April 18 @hackergarage Guadalajara will host its first Geo Meetup that focuses on OpenStreetMap. After having done serious tracing work for Guadalajara, Mexico, it’s particularly exciting to have an opportunity to work with the local community. While remote satellite tracing allows us to quickly put streets and building footprints in vast areas of a city on a map, local knowledge is needed to add crucial elements like street names.

This is an example where Guadalajara local jackbravo has provided location names on traced street data largely provided by remote tracing user juey (main traffic arteries were provided by minoxfilm).

I look forward to join the meetup via Skype and further explore ways to grow the Guadalajara OpenStreetMap community and coordinate our efforts. In addition to getting a chance to learn about how we can support local mappers, I’ll be giving a short presentation on the work we have done so far in Guadalajara and other cities, with a focus on the import of main traffic arteries we are planning for the state of Jalisco.

FOSS4G: Presenting on the Future of Web Mapping, Cartography, and MapBox Streets

MapBox — 2012-04-08T00:00:00-07:00

This week we’ll be at the FOSS4G conference here in Washington, DC discussing the current state and what the future should look like for open source geospatial technology. The conference schedule looks awesome with deep dives on technologies like PostGIS, PostgreSQL, and Quantam GIS, case studies from folks like the FCC, NOAA, and the World Bank, and design talks on topics like making watercolor maps with OpenStreetMap.

We’re participating in four sessions, described below, on the future of open source web mapping, the tech behind our street-level map of the world, and cartography tricks. We also have an exhibit table where we’ll be running demos of new custom styles for MapBox Streets and the map design tool TileMill. Stop by our sessions or table to chat, or message us on Twitter at Tom, AJ, Eric, DJ, Bonnie, and Young.

Keynote: Gaps and Voids in Open Source Geo Technology: Tom will join four others on this panel to talk about what’s missing in open source geo technology and where the community should focus its efforts on. Design will weigh heavy in this conversation. Tuesday 3:00 pm, Room 103 AB
New Maps: Beyond the Google Maps Paradigm: Big strides have been made recently in map interactivity and user experience. Tom will talk about the technology and APIs that are changing this, how they’re being used to tell stories with maps, and what you can already do to display crazy maps on your devices, whether they’re handheld or giant monitors. Tuesday 1:00 pm, Room 103 AB
Rendering the World: Rendering a street map of the world used to be unthinkable, but not any more. Young will talk about how we made MapBox Streets with our open source technology stack leveraging Node.js, Mapnik, MBTiles, and TileMill.
Cartography with TileMill, PostGIS, and OpenStreetMap: AJ will share his best tricks for designing beautiful, unique maps using TileMill. He’ll touch on import decisions, stylesheet techniques, spatial queries, and data massaging – and how to do it all with free, open-source software.

We’re looking forward to catching up with folks, learning what they’re working on, and sharing our latest thoughts on open source geo technology.

Where Conference Keynote Launching Custom Styles for MapBox Streets

MapBox — 2012-04-06T00:00:00-07:00

Our keynote launching custom styles for MapBox Streets at the O’Reilly Where Conference this week is now online! You can see Eric walk though both the new custom styles and see how this fits into our larger platform. Eric’s talk ends by focusing on the larger conversation around what Alex Howard is calling the ”year of the open map” and how this is about more than just code - it’s about a larger open movement and the community of companies and developers pushing the open stack forward. This ultimately makes technology easier and open data move actionable, and that is going to be what radically changes the larger marketplace.

Eric’s keynote on Designing Fast and Beautiful Maps on Tuesday, April 3rd at the O’Reilly Where Conference.

A special thank you to Brady Forrest and Laurel Ruma for an awesome conference!

Mapping for Nonprofits at the NTC

MapBox — 2012-04-04T00:00:00-07:00

Today at the Nonprofit Technology Network’s annual conference, Eric will present on how nonprofits can use online maps to tell the stories behind their work and their data, along with Sean Larkin and Lev Tsypin from ThinkShout. It’s an interesting time for mapping in the nonprofit space, with the barrier to entry being lowered with more open source and user friendly tools being released, making mapping possible for nonprofits of any budget size.

This session will focus in part on showing what’s possible, showing examples of nonprofits using maps in their advocacy work, to report survey findings, and to track and collaborate around operations. It will provide an overall of existing open source interactive mapping technologies and how they can be used. Then Eric will give a hands on demo on how TileMill can be used to design custom maps, walking through how to go from a spreadsheet of data to custom map, and then how to take it online and embed it in a website or application. With free accounts from MapBox, this is easier than ever.

If you’re at the NTC, check out Geek Out With Online Maps As Storytelling Devices today at 10:30 am in the Franciscan B room. For background on the open source mapping landscape, read Sean’s article, and for a jump start with TileMill, check out the crash course.

GeoDC Meetup: Presenting MapBox Streets New Custom Styles

MapBox — 2012-04-04T00:00:00-07:00

At tonight’s GeoDC meetup, we will demo the new custom styles we released yesterday for our MapBox Streets world base map. This new feature allows you to choose from 12 different map styles and change the colors of land, water, areas, and streets on your map via a color switcher. It brings an new level of design flexibility to our world maps, and we’re already seeing some fun examples of it in action like this neon map of bike path’s in Portland.

There are two other interesting lightning talks on the schedule:

William Hurley from Forum One will talk about the County Health Rankings map that was just launched, specifically touching on how it’s rendering dynamic tiles with Mapnik and Tilelive.
Kevin Webb of OpenPlans will talk about OpenTripPlanner, how it works, and its transit analysis extensions.

The meetup starts at 7:00 pm, with presentations kicking off at 7:30 pm, at the upstairs bar at Stetson’s at the intersection of 16th and U Streets NW.

The GeoDC meetups draw a vibrant crowd of people interested in and working with maps, data collection, GIS, and other related technologies. The group meets once a month on the first Wednesday of the month. Join the meetup group and follow @geo_dc on Twitter for updates.

Custom Styles for MapBox Streets

MapBox — 2012-04-03T00:00:00-07:00

We just launched custom styles for our MapBox Streets world base map at the O’Reilly Where Conference, as part of Eric’s keynote on Designing Fast and Beautiful Maps. Now whether you need a clean canvas for data visualization, a film noir atmosphere for setting the mood, or map labels in the language of your audience, MapBox Streets is up to the job. With our scalable cloud infrastructure, we are proud to offer custom styling to all of our users at no extra charge.

Integrated styling

MapBox Streets is integrated right into our map builder and gives you control over how land, water, areas, and streets look. You can adjust the color scheme of your map, toggle between localized and English labels, and toggle any of these components on or off. To help you get started quickly we’ve created a dozen presets including Nightvision, monochromatic Light, and Graphite that work well with data overlays, and some more artistic schemes like St Clair and Lacquer. For the hackers out there, we’ve even done a cartographic take on the popular vim color scheme Zenburn.

The Lacquer custom style preset. View it full screen.

In addition to matching your look and feel, using MapBox Streets with custom styles is a great way to supplement data visualizations context while maintaining a clean canvas.

Powered by open source and open data

MapBox Hosting is powered by dozens of open source Node.js modules. Custom styles for MapBox Streets are made possible by Konstantin’s work on node-blend for fast image compositing and node-tint for efficient PNG color manipulation.

The data for our latest MapBox Streets render comes from OpenStreetMap’s March 21st database snapshot. MapBox Streets includes recent data contributions from the OSM community and the results of our own efforts to identify and address areas with low coverage.

Get started fast

Creating a custom styled map is easy, fast, and free. Sign up for MapBox Hosting to create a new map and customize MapBox Streets. From there you can embed it in your website or request your custom map tiles directly using our API. We’re looking forward to seeing some awesome maps!

Update from #Whereconf

MapBox — 2012-04-02T00:00:00-07:00

Tom, Eric and I are wrapping up a full day of presentations and conversations with attendees at Where Conference in San Francisco. This afternoon, Tom and I had a chance to present on our tools and process for open source mapping. We had a great response: people here are really excited to see what’s possible with open source geospatial tools and open datasets.

You can check out our slides now, and a video should be posted shortly. Tom summed up the main takeaway as this:

open source libraries > their © equivalents

To start there are incredibly detailed and easy-to-work-with datasets like OpenStreetMap that make it possible to design completely custom base maps, such as what we’ve done with MapBox Streets.

There are also great open source tools for processing both large and small datasets, like Quantum GIS, which we use to reproject, analyze, and join geographic data to tabular spreadsheets.

Check out our slides for more links and thoughts on building an open mapping stack. If you have questions or want more detailed information, reach out to us on Twitter or check out our help docs.

Also, tune in to the live stream tomorrow at 9:25am PT (12:25pm ET) for Eric’s keynote presentation and a special announcement.

Tracing OpenStreetMap data in the Sahel

MapBox — 2012-04-02T00:00:00-07:00

We just finished a three day sprint of OpenStreetMap tracing, adding critical features to the main capitals in the Sahel region of West Africa. This is in response to a new food security alert issued last week from Famine Early Warning System Network (FEWS NET) for the Sahel. Currently five countries are in danger of famine due to drought and food security conditions - Mauritania, Mali, Burkina Faso, Niger, and Chad.

OpenStreetMap is a critical resource in disaster response, particularly in this part of the world where current map data is often lacking. In this sprint, we focused our efforts on adding data to OpenStreetMap for capitals in the five at-risk countries.

Hitting capitals first

We’ve added road data to the capitals using satellite imagery. We are working to receive higher-resolution imagery for critical areas outside of the capitals, and are planning another tracing sprint once that is available. Here are some of the results of the work we’ve completed so far.

Ouagadougou, Burkina Faso

Within the capital of Burkina Faso, we’ve increased detail in the roads and paths in the outskirts of the city.

You can see many areas around the city have informal road tracks.

Bamako, Mali
Areas that weren’t on the map before are now on the map.

Facilitating data sharing between organizations

We’re also working to boost the transportation networks and place data within OpenStreetMap in the Sahel region to facilitate data sharing between organizations. All of this tracing work is being done in conjunction with other data coordination activities in the region. Having a common operating picture is critical for the international community to respond to early warnings.

We’re just kicking this effort off. The mapping team - Dan Wood, Chad Lawlis, Maya Richman, and Ruben Lopez Mendoza - have done incredible work during this first sprint. Join us in our public github repo to contribute and give feedback. We’re tracking our progress and listing out the priority areas of where we’ll be tracing in the wiki page.

MapBox Streets and OpenStreetMap License Change Schedule

MapBox — 2012-03-29T00:00:00-07:00

The service schedule for the OpenStreetMap license change was announced yesterday. It will put the OpenStreetMap database into read only-mode between April 1st and April 4th, and then start slowly processing all non-clean objects, meaning objects contributed by the few users who have not agreed to the license change by April 1st.

We will pull a copy of the last available OpenStreetMap database export on March 31st, meaning that all changes to the OpenStreetMap database made up until then will go into our next render of MapBox Streets, which is scheduled for April 14th. After that, we are aiming for another render of MapBox streets by mid May. By then, we expect the impact of the license change on the data to be largely mitigated by remapping efforts.

For background, read about the license change on the OpenStreetMap Foundation blog.

Open Mapping Discussion on TWiT.tv

MapBox — 2012-03-22T00:00:00-07:00

Yesterday Brady Forrest from O’Reilly Media led a great conversation on This Week in Google on TWiT.tv on open mapping and the growing discussion around Google Maps versus OpenStreetMap. Of course Google’s new pricing came up as one of the driving factors behind companies like StreetEasy and foursquare switching to alternatives, but much of the conversation was also around the benefits of using open data - instead of proprietary - and the flexibility that allows.

One of the things that’s happening with OpenStreetMap is that people will start to look around and go, oh wait a minute, I’m going from free with Google Maps to suddenly I have to pay in some cases hundreds of thousands of dollars for it. Is there another option? And when they see OpenStreetMap suddenly they’re like oh, but I can have a little more control.

We were psyched for out tools to get a generous shout out as not only alternatives to Google Maps, but also open source tools that help people make better, custom maps.

But the people who are really going to capitalize on the switch to OSM are Development Seed, which is a company working out of DC run by this guy Eric Gundersen, and that’s who StreetEasy and foursquare both used. I was chatting with him the other day, and it’s really, in the past year, the data has gotten better and the open stack has gotten better. Leaflet is a JavaScript control for maps, MapBox is Development Seed’s product that they serve the maps from, and then TileMill which lets you make your own tiles.

You can watch the whole segment here, starting at the 48 minute mark.

We’re excited to be at the Where2.0 conference next month to talk more with folks about open geo data, how we’re helping improve OpenStreetMap data, and how to use our tools like TileMill, MapBox, and MapBox Streets. Eric and Tom will give a keynote presentation on designing fast and beautiful maps, and a workshop on making web maps with open source tools. We’ll post more details here on the conference in the coming weeks.

Experiments in the 3rd Dimension

MapBox — 2012-03-22T00:00:00-07:00

The good folks at Sanborn were kind enough to send us a sample extract of processed 3D building data from their 3D Cities archive, so I set out to see what we could do with it using TileMill and the building symbolizer in Mapnik.

Mapping 3D buildings

Out of the box, the Sanborn 3D data is ready to map. To make it fast in TileMill, I reprojected the data to Spherical Mercator and added a Mapnik shape index. Then seeing the buildings pop up was just a matter of adding the shapefile as a layer in TileMill and setting the building-height: 100; style. The Sanborn data contains a bld_h_m value for each polygon, which is the shape’s building height in meters. To accurately extrude shapes according to this value, we need to do a slight patch to TileMill so we can set building-height: "[bld_h_m]";, which uses the data value instead of a fixed number.

Finally, to get the polygons to render in the right order, I imported the data into PostGIS and sorted it with ORDER BY ST_YMax(ST_Envelope(the_geom)) DESC.

Other (ab)uses for 3d in TileMill

In the past, we’ve hacked around with using the building symbolizer to show data, particularly like major earthquakes:

If you’re looking to add some basic depth to your map features, you can get creative with the building symbolizer. It supports three basic styles for the color and height of extruded shapes. There’s a lot you can do with these styles, particularly by stacking lots of semi-transparent features.

Here, I’ve made a 3D population density map by taken randomly distributed points within census tract polygons and summarized them on a diamond grid. Looking close up, you can see this is just a grid with lots of extruded shapes.

TileMill’s not intended to be a substitute for a full 3D rendering application, but it’s quite powerful for building data and some quirky visualizations. Check it out and let us know (@mapbox) what you make.

Analyzing, Fixing, and Improving OpenStreetMap

MapBox — 2012-03-21T12:00:00-07:00

We’re looking at new ways to find out how and where OpenStreetMap needs improvement. The collaborative mapping project that provides the street data in our world map MapBox Streets is remarkably complete, as well as open and free, but it is continually unfinished as roads are built and borders changed.

Foursquare’s adoption of MapBox Streets and OpenStreetMap and Apple’s likely usage have directed many more eyes at the project - all users who are likely to spot missing roads and features, and be able to give incredibly valuable feedback. This feedback has been one of the guiding elements of our tracing efforts over the past few weeks, but we’re working on ways to get global, fine-grained priorities set.

One of the tools we’ve been using is simply an overlay of social checkins on the map. Often city centers are well mapped, but completeness falls off quickly from the center and there’s still a lot of population and activity in the outskirts.

Of course social network activity isn’t an absolute indicator of where general populations are or what areas important to map, but it’s a useful heuristic - and one that can be derived on a global level.

But it’s not just important where people are - we need to know where the map doesn’t cover. For this we’ve been working on finding the street density, measured by the file size of tile pyramids, of MapBox Streets. I talked about that analysis at NodeDC on Monday, a presentation that’s available online.

This is just one angle of finding and improving data gaps and flaws in OpenStreetMap - the community has created many other methods, and novel ideas like fixing routing between cities have been used to drive improvements. The fact that OSM’s data is available in its entirety and freely licensed enables all of these approaches.

When we find areas that need improvement, we start tracing, finding and supporting local contributors, and working with organizations to find freely licenseable data to add to the map.

Design and Data Updates for MapBox Streets Down to Zoom Level 17

MapBox — 2012-03-19T12:00:00-07:00

We’ve rolled out new updates for MapBox Streets, our colorful OpenStreetMap-based world map. The underlying data has been updated to include many of the amazing OSM contributions that have happened in recent weeks, and the design has received substantial refinements to improve readability and display more map features. All of this is also now available down to zoom level 17 across the entire world.

Here are some of the design changes that we just rolled out.

Labels

One of the more noticeable changes is the we are now using abbreviations for many common suffixes and prefixes in English road labels. The abbreviation logic attempts to work intelligently so that, for example, “West Main Street” becomes “W Main St” but “North Road” becomes “North Rd” and not “N Rd”. More short roads are now being labeled as a result of the abbreviations. (Let us know if you find situations where things aren’t working correctly.)

Street label clarity has been improved by enlarging the text, darkening the shade, and decreasing the letter spacing. We are including more important road names at lower zoom levels and showing more labels overall.

Other labels have also been adjusted. Place names are better distinguished from street names, and priorities for school labels have been adjusted to fix many urban areas where they appeared too densely.

Roads and tunnels

Previously tunnel visibility was poor, but they are now properly distinguished from other roads. Various road widths have been adjusted to try to better highlight primary routes, and one way arrows have been made a bit smaller and more subtle.

More map features

A number of features that were not previously included are now visible on the map. Most noticeably a bug that caused many wooded areas to not appear has been fixed. You can also now see beaches, sand, cliffs, piers, and docks.

We’ve added rail stations (including subways) and added icons for airports. We’ve also added labels to larger buildings at the highest zoom levels.

Color adjustments

The colors of many features have been tweaked to improve the overall aesthetic. Forests and buildings have been toned down to not stand out as much. Elementary and high school grounds are no longer as prominent as college and university campuses.

Get started with MapBox Streets

MapBox Streets can be used within MapBox Hosting plans, including a free option. Our documentation site has detailed instructions and examples to help you get started.

Demos and Workshop at TUgis Conference

MapBox — 2012-03-19T11:00:00-07:00

For the next two days, DJ and I are at the TUgis conference, Towson University’s annual GIS conference. We’re looking forward to connecting with folks coming at mapping from the GIS side, learning more about what they need, and teaching them about what’s possible now in web mapping. I’m particularly excited for today’s keynote from Barney Krucoff, Maryland’s GIO officer, who gave a great talk about what the state is doing to release its GIS data at this month’s Geo DC meetup.

DJ and I will be running TileMill and MapBox Streets demos all day from the MapBox exhibit booth. Tomorrow, we’ll lead two sessions:

Designing Fast and Beautiful Maps with MapBox and TileMill: In this session we’ll walk through how to use TileMill to design maps, taking participants from a spreadsheet to a custom designed map, and then sharing it online with our cloud map hosting service and getting it live on a website via embeddable widgets and the MapBox API. Tuesday, 10:45 am

An Open Source Web Mapping Tool Box: This 90 minute workshop starts with a brief walk through of open source mapping tools and examples of what’s possible in the world of web mapping. It will be followed by two one hour simultaneous workshops to suit different mapping needs. Participants will choose between a hands on workshop with GIS analysis and cartography using open sources tools, or a walk through of the fundamentals of building a microsite with your maps and some light web languages. Tuesday, 3:30 pm

If you want to talk about open source maps, MapBox, or anything else, come find us at the MapBox exhibit table. We’ll be posting updates from TUgis via Twitter as well at @ian_villeda and @djindc.

Presenting on Big Data Processing and Visualization at NodeDC Tonight

MapBox — 2012-03-19T00:00:00-07:00

The March NodeDC meetup is tonight at 7:00 pm at Stetson’s Bar and Grill. There will be two presentations, both on visualizing large data sets using node.js.

Tom will talk about big data processing and visualization with node.js, and specifically on solving the problem of analyzing road density data with redis, d3, sqlite, and polymaps. As part of our improvements to OpenStreetMap, we’ve been doing intense analysis of the data that we have and what we’re missing - this talk will review the problems of missing data and fundamentals of processing like binning and prototyping. Check out his slides for a preview.

Then Kyle Hill will talk about visualizing Capital Bikeshare data as a scoreboard for the fastest point-to-point trips in the systsm using the Persevere API.

The NodeDC meetup is open to anyone interested in node.js and is a great opportunity to hear what folks are doing with node. The group meets once a month, usually on the third Monday of the month. Watch nodedc.github.com, join the meetup group, or follow @nodedc on Twitter for updates.

Tracing Satellite Imagery in OpenStreetMap

MapBox — 2012-03-17T00:00:00-07:00

For the past two weeks we have been tracing satellite imagery into OpenStreetMap to support some of our subscribers using MapBox Streets. OpenStreetMap’s real power is that if something isn’t on the map, you can just add it. So this is more than just open data for us, this is about co-creation and collaborating with a larger community.

Around the sprint table at the MapBox office in Washington DC

The fact that the geography around us is constantly changing is why we are using OpenStreetMap to power our key maps. Not only does OpenStreetMap have amazing data today - every day the data improves as the its network of contributors grows. Right now, our team is focusing on completing a few key populous cities, making sure that suburbs and popular features like buildings, rivers, and parks provide meaningful and attractive context for users. With a team of six people heads down we can very quickly light up a city and meet other local contributors working on the ground.

A look at new edits over the last week to Guadalajara, Mexico, a huge city with the metropolitan area of over 4 million people.

The Tools to Start Tracing

To add all of these features, we’re using JOSM an open source desktop editor for OpenStreetMap. Here is a quick step by step of how you can start tracing imagery in 5 minutes using the same tools and processes we are.

Before you start tracing, be sure to read the OpenStreetMap editing standards and conventions, then download JOSM. It’s helpful to have a mouse on hand (versus a trackpad), as you’ll want to do a lot of precise clicking. Once you’ve set up and run JOSM, click the ‘download data’ button at the top left.

You will see a map similar to openstreetmap.org. Find and select the area you are interested in and click download. To pan around the map, press and hold CTRL. To select an area, just use the left mouse button.

To edit, add Bing Imagery under the Imagery dropdown menu.

You’ll then want to toggle between JOSM’s three modes using these key bindings: draw (A), select (S), and delete (D). Use draw to add nodes, ways and areas. When you’re done adding a way, double click on the last node. To create an area simply close a series of nodes by attaching the last node the the first. For more advanced editing, check out the online guide and useful plugins.

Once you’ve created your features, tag them appropriately with a preset (see ‘Presets’ menu) or directly via the ‘Properties’ window (see ‘Windows’ menu). Good resources for learning how to tag elements in OpenStreetMap include TagInfo and the Map Features page of the OSM wiki. Explore the various list-serves and engage others in the OSM community.

Now you’re ready to map the world.

Designing your Map

If you are looking to create your own custom designed map of OpenStreetMap data, it is easy to do so. Once your data is in OpenStreetMap, download TileMill, the open source design studio then read ”Create a custom map of your city in 30 minutes with TileMill and OpenStreetMap”. You will be able to make maps that look a lot like MapBox Streets and host them on your own MapBox account, with even free plans. If you have questions post them to support.mapbox.com.

We're going to the Toronto OSM Hack Weekend

MapBox — 2012-03-16T00:00:00-07:00

This weekend Tristen and I are attending the OpenStreetMap hacking event in Toronto, organized by Richard Weait and hosted by Ryerson University. Over a dozen people have RSVPed so far and we’re excited to get together.

OSM hack weekends are events that bring together community members to tackle programming, design, and documentation tasks for the OpenStreetMap project. The focus is on improving the tools we use to access, edit, and display the map data.

This afternoon starting at 2:00 pm there will be an introductory presentation for Ryerson students, faculty, and anyone who is curious to learn more about OpenStreetMap followed by a roundtable Q&A session. Richard Weait will lead the presentation, outlining various aspects of the project, and questions will be fielded by some of the OpenStreetMappers who are already in town for the hack weekend (including myself and Tristen).

The hacking will happen on Saturday and Sunday. Tristen and I will likely focus on graphics and web interface work, and I may tackle some cartography and Mapnik related tasks as well. Some of this work was already started at the recent OSM hack weekend in DC, and a number of the same attendees will be in Toronto as well so we will be able to keep the momentum and continue conversations here.

You can follow me and Tristen on Twitter for updates throughout the event. And if you’re in the Toronto area and are interested in OpenStreetMap you can join us! Full details are on the OSM wiki.