Doing "Bulk Imports" to OpenStreetMap

Uploading existing datasets of any significant size ("bulk imports") to OpenStreetMap involves serious consideration, above that of ordinary updates and edits.

I have done bulk imports to OSM before for UD/Newark Data.  I used shp2osm (http://wiki.openstreetmap.org/wiki/Shp2osm) to convert from shapefile to osm format and then used JOSM (http://wiki.openstreetmap.org/wiki/JOSM) to compare to osm data, edit where needed, and upload the data/changes.

Here is a comprehensive page on doing bulk imports: http://wiki.openstreetmap.org/wiki/Imports

OSM gives priority to smaller edits, as part of its grassroots philosophy.  Partially because of this, it's important to tread lightly when doing bulk updates.  That's why I like JOSM. I can see the existing data and how my updates will change that (if so) or potentially create duplicates.  The imports guidelines link (http://wiki.openstreetmap.org/wiki/Import/Guidelines) is especially helpful in understanding norms and rules for updates.

ArcGIS Editor for OpenStreetMap

I recently had a chance to test out ArcGIS Editor for OpenStreetMap (1.1 Beta3), an open source toolbox for interacting with OpenStreetMap (OSM). OSM, a popular open-source data repository, gained extra traction in the US after the Haitian Earthquake. This growth came from NGOs, and from some ArcGIS shops with open source leanings, who advocated on behalf of an extension to OSM from ArcGIS. I've been interacting heavily with OSM in building and maintaining the University of Delaware Map (UD Map). The data schema for UD Map is based on the OpenStreetMap schema, and interfaces with OpenStreetMap for data updates and publishing. As the project demanded it, and as a test of the new tools, I used the toolkit to publish some pending campus base layers. Alas, this trial ended disappointingly, leaving only manual editing of the vector layer. Append, which could have been used to join the vector input to the OSM polygon layer, did not work as expected: The OSM changeset was not updated, and therefore could not be uploaded to OSM for processing. The tools still look very useful for doing manual editing of OSM data and adding OSM data as fully editable vector layers in a map ... and of course, as code to be expanded upon ... this story is to be continued :-)

Duplicate OSM Features in UD Campus Map

I noticed a couple places where we have two polygons on top of each other (particularly for parking polygons) in the UD Campus Map. I just wanted to note, if you're trying to modify an existing polygon on the map, make sure that you're changing the existing shape, rather than adding a new one (otherwise we're loosing attributes on the exisiting shape).

Here are two screenshots to demonstrate what I'm talking about (note that only the second polygon retains attribute information/properties as listed in the right-hand sidebar), though it appears the first polygon was intended to replace the second.

I've fixed this problem by adding nodes to the second polygon (the original one that has attributes/properties) and dragging them to match the footprint of the first (new, intended to replace the first), and deleting that one.

Mapnik ... success!

Here's how I imported, rendered, and served OSM data through mapnik, locally:

1. Do all downloads/installs ... python (2.5 worked for me), postgres/postgis (8.3 worked), osm2pgsql, mapnik, rendering/mapnik utilities for OSM
2. Follow info below on osm2pgsql
3. Make sure mapnik bindings are registered with Python, run the demo under demo\python to make sure mapnik works, here are pretty good instructions from the OSM wiki
4. Follow this readme to run generate_xml.py and generate_tiles.py. Use my tips below if you get stuck ... also make sure you keep your coordinates straight ... these scripts won't throw an error, they'll just crash
5. Once you've generated tiles stick them in a web accessible location and use this tutorial to get them working with OpenLayers ... note the "bespoke" section, which shows how to add your custom tiles

... now to customize the rendering I only need to modify the map.xml file created by generate_xml.py ... I could do this manually or might look into the qgis plugin ... also cascadenik looks cool

OSM2PGSQL

After getting mapnik installed and running I realized:

1. generate_tiles/generate_xml (and the rest of the render utilities) are really necessaryfor rendering OSM through mapnik. Therefore, you must download both mapnik through the mapnik site but also the mapnik OSM rendering tools which you can get through SVN at http://svn.openstreetmap.org/applications/rendering/mapnik/
2. you must download world coastlines to get the osm/mapnik utilities to run as detailed here (if you're getting an error about 'world_boundaries', that's why)
3. you must use OSM2PGSQL to import your osm data into postgres ... you can do this through osmosis, but it won't work

Helpful hints with OSM2PGSQL

1. you must run the included .sql in order to create the proper spatial reference information for the projection mapnik likes
2. you must create the database with a postgis template
3. you might need to export through JSOM and encode as UTM8 ... I did this as a safeguard this time and it worked ... if you're having trouble, try this

Related errors:
failed: ERROR: AddGeometryColumns() - invalid SRID
CONTEXT: SQL statement "SELECT AddGeometryColumn('','', $1 , $2 , $3 , $4 , $5 )"
PL/pgSQL function "addgeometrycolumn" line 4 at SQL statement

relation "planet_osm_polygon" does not exist

Loading OSM with OSMOSIS

Now that I've decided to locally render OSM data for the campus, I need to get the OSM data for the maximum extent of campus areas into a local database (both Mapnik and I prefer Postgres). I initially tried doing this by using OSM2PGSQL, but that ultimately failed ... not completely sure why.

Here are the steps I took to load this data into Postgres using OSMOSIS:

Required installations: Postgres with Postgis, OSMOSIS ... tested environment: osmosis 0.32, postgres 8.3, Windows XP

1. Download desired extent from OSM using JOSM ... save the downloaded data as *.OSM
2. Convert this .osm file to UTM8 encoding (e.g. using notepad++)
3. Login to postgres, create a new database based on postgis template
4. Run pgsql_simple_schema_0.6.sql, which is located in the osmosis install location in the 'script' directory
5. Open a command line and run: PATH_TO_OSMOSIS_BATCH_FILE --read-xml file="PATH_TO_OSM_FILE" --write-pgsql host="HOST_NAME" database="DB_NAME" user="USER_NAME" password="PASSWORD", for example: C:\qgis_projects\render\osmosis-latest-bin\osmosis-0.32\bin\osmosis.bat --read-xml file="C:\qgis_projects\render\newark_josm_utm8.osm" --write-pgsql host="localhost" database="gis" user="postgres" password="mypassword"

Plugging away with OSM

Update: Success! Was able to convert to OSM with Export_to_OSM.py, all was needed was to change some paths in the code. Uploads via JOSM using the Open, and Merge commands (I recommend also adding some tag so you can query these out later if you need to do anything with them), then Upload. Watch as I add more layers on www.openstreetmap.org!

Old Post: It now seems I'll be able to use OpenStreetMaps either as the repository and rendering system for the full campus map, or as a selective baselayer for national data (for directions). The answer will depend, in a large part, on whether I am able to get campus layers into OSM.

I have worked with multiple OSM editors: JOSM, markaarter, and the QGIS OSM plugin and none will successfully take a shapefile, allow import into OSM format, and then upload to the OSM servers. I have tried conversion to KML and GPX, since these are formats accepted by one or the other of these editors ... still no dice.

There are scripts and programs which were specifically created to convert SHP (and all kinds of other formats) to OSM. I've checked out some of these programs osm2shp, shp-to-osm.jar even gml2osm ... problem is that they assume you want to map all the shapefile attributes ... seems like it could be a lot of extra work to figure these out for every layer I want to upload.

Basemap, prority renderers/services based on extent?

A new idea: use custom service/renderer within a given extent and cloud service outside of that extent ... tag campus layers accordingly, so they're picked up by custom renderer

Directions, directions

So I am returning in the new year with a major task at hand: finish work on UD campus maps. As I recall, the major loose end from where I left off was the matter of the basemap -- which resurrected old considerations about maintaining all layers on a cloud site (i.e. openstreetmaps) and send all operations to be dealt with on a local spatial database. During this diversion from the previous course I need to also keep in mind: the basemap issue only comes up because I want to offer national directions to campus in the same interface as the rest of the campus map ... otherwise I could just use local layers without need for larger extent basemap layers.

The OSM scenario is very appealing for a number of reasons:
-cloud hosting would free up local resources here
-the OSM platform is well designed according to collaborative logic, that would allow updates by data stewards on campus, if we were to move in that direction
-OSM is open source, it's free to us, and is very interoperable with other open software ... that would help in efforts to support open source on campus
-possibly better performance on the cloud
-use of existing renderers allows us to leverage existing cartographic/labeling rules

Disadvantages
-layers could be edited by a third party with poor editing skills, inaccurate data, or malicious intent
-limited control over appearance of the map
-strange issues with spatial referencing ... OSM and existing UD layers do not agree ... these are also off with navteq streets, so problem is independent of OSM ... should note that the issue is probably not significant without sidewalks, since these tend to be more closely located to streets than buildings to streets
-still unresolved data sensitivity issues (we need to contact all data "owners")
-tendency to neglect local resources ... no redundancy

These disadvantages could mostly be overcome, but would probably require quite a bit of work, with some of the advantages becoming mooted
-OSM data could be stored locally, taking advantage of some tag which signifies campus data ... that is what UMd does. However, that is probably impractical for the national dataset (UMd only has part of the DC region), and would moot any advantages to using the cloud ... would however help with redundancy
-A custom renderer could be used to achieve more control over appearance ... this may be unwieldy to setup ... could try to tweak tags, etc. in OSM data as a work around ... could Mapnik default rendering be adequate?
-spatial referencing issues are independent of OSM

Cloud basemaps, so elusive

I wanted to use openstreetmaps as the base data layer for the new UD Campus Maps, but am running into the same labeling issues as when I use a google maps base layer ... labels are prerendered to appear on streets, cities, etc. of course not taking into account placement of "UD" layers (e.g. buildings). When the two are displayed together there is a very unsightly overlapping effect.


So the way around this is to use a custom renderer, which is actually quite a bit of work ... the custom renderer requires a large amount of space, not to mention an initial span of CPU time. It also requires that I create custom style specs, which address all sorts of issues, including a plethora of OSM tags, as well as all the details about base layer appearence ... all to be done manually of course.

... not to mention, the existing baselayers seem to be disagree with cloud services ever so slightly

CloudMade offers a very cool service for rendering custom basemaps from OSM data, all through an efficient GUI, and they even host it for you. Trouble is their terms do not seem to permit "non-personal" use, :.(

Sooo, I can:

1. set up a custom render, which *might* require a lot of space/time, and *might* require a lot of work in symbolization, and will consume local resources instead of leveraging the cloud
   
2. *or* use ArcGIS server to tilecache a national basemap ... having much the same disadvantages as above, except that I have a better idea of how much time/resources it will take
   
3. *or* render on top of satellite data (which doesn't really look that good, and not very informative for directions), plus requiring me to create custom label and symbol scaling and symbology for Newark basemap details (tending to be less attractive than those already available in the cloud, and causing disuniformity with surrounding areas)
   
4. *or* I can mask the extent of all "UD layers" and use this as the default extent for the map, putting in some scale sensitivity which causes UD detailed layers to disappear when viewed at a larger "regional" scale and instead showing some extent rectangle ... then showing these layers again when at a closer zoom (having some of the same drawbacks of disuniformity/attractiveness)

... as of today the last option seems the best ... I sure hope it bares fruit, we need to get this project rolling!