MapBox

Distributing Maps on a Stick

Flash drives can be very fast for common operations, but for the task of transferring a large number of files at once, flash storage is extremely slow - transferring Maps on a Stick to a USB drive with the Mac OS X file would take more than 17 hours. A better (although more detailed) solution is to create an image of the USB drive, including the filesystem, and essentially 'burn' this virtual volume to the USB drive. This way, the filesystem is controlled and constructed on a fast disk and then the raw information of the files and filesystem is transferred directly to the storage device.

Requirements

  • This process will require familiarity with some low-level POSIX utilities, especially dd. This is not a guide you can run through by copying commands - it will require, at the very least, careful editing of the sample commands.
  • This is written for Apple Macintosh computers. POSIX equivalents to Mac helper applications like hdiutil certainly exist and a guide for doing this with Linux, etc., is certainly possible, but not provided here.

Create an Image from Folder

sudo hdiutil create -verbose -fs MS-DOS -fsargs "-F 32 -c 4" -volname "MAPSONSTICK" -srcfolder your_map_application your_map_application.dmg

This command creates an Apple Disk Image from a folder on your computer. Notably, the image will have a FAT32 filesystem - the -F 32 argument specifies that this will be the case, rather than FAT16, which is limited to a 2 gigabyte partition. The -c 4 option specifies that the clusters on the disk - which determine the minimum space used for a single file - are limited to 4K, which is conveniently a similar size to the size of a typical tile.

Attach the Image as a Device

hdiutil attach -nomount your_map_application.dmg

This command will typically return two paths, like

/dev/disk1          FDisk_partition_scheme        
/dev/disk1s1        DOS_FAT_32

Use the latter path (/dev/disk1s1) for future steps.

Attach USB Drive

Put the USB flash device in your computer's USB port, and then run df in the console and note the output. The leftmost column shows what the device of the USB flash drive is, and the rightmost will show the mount point. Remember the left column (which will be something like /dev/disk1s2) and then run

umount /Volumes/USBDrive

To unmount the drive

Image the drive

Make absolutely sure that the if and of parameters of the following command are accurate; the dd utility is able to write over essential data if you accidentally put the device of another drive in the of parameter.

sudo dd if=/dev/FROM_DISKIMAGE of=/dev/TO_USBDRIVE bs=8m

The last parameter, bs, indicates the blocksize of the transfer; the size of each chunk of data that is transferred. This value is tuned for setups we've encountered, but other values may lead to a faster process.

tip: on Mac computers, disks are exposed both as raw devices, under rdisk3, and buffered devices, under disk3, etc. Writing images to the raw disk, beginning with "r", is often much faster than writing to the standard buffered disk.

Checking progress

The dd operation that is now running can take a while, and it doesn't provide any status information. However, it can.

Open a new terminal and run

~$ ps aux | grep "dd"
root      1698   0.1  0.0  2434768    252 s000  U+   12:18PM   0:01.67 dd if=/dev/disk2 of=/dev/disk1s2 bs=32k

Note the process id: the first number in the row of your dd process. In the above output, it is 1698. Use this id in the following command.

~$ sudo kill -SIGINFO PROCESS_ID

Going back to the terminal which is running dd will show a count of how many bytes have been transferred. To get a figure in megabytes, you can use the units utility.

~$ units "70000 bytes" "megabytes"
* 0.066757202
/ 14.979657

So, 70000 bytes is 0.06 megabytes.

When dd completes, it will output a summary of how much information was transferred and in what timeframe.