Running GDAL and arcpy together (ArcGIS 10.1, GDAL 1.9.2, python 2.7.4)

There are many use cases for running GDAL and arcpy in the same python instance.  GDAL is the most popular open source geopspatial geoprocessing toolset and arcpy is Esri's python wrapper for the entire ArcGIS geoprocessing toolset.

However, doing so can get quite tricky.  GDAL's python bindings are not easily configured for Windows (and ArcGIS/ArcObjects is Windows-only).  The OSGeo4W project can be used to do so, but the bundled Python does not play well with arcpy.  arcpy itself depends on numpy and requires multiple path inclusions.

Here are steps I took to import gdal and arcpy.  
I tested by successfully running arcpy.RasterToNetCDF_md()  (though I haven't tested gdal yet).

Step 1: Install standalone python to C:\Python274 (download from http://www.python.org/ftp/python/2.7.4/python-2.7.4.msi)
Step 2: Add numpy (version bundled with ArcGIS) and Desktop10.1.pth to C:\Python274\Lib\site-packages (download from http://www.udel.edu/001745)
Step 3: Install GDAL binaries and python bindings (download installer from http://www.udel.edu/001742).
Step 4: Run import commands "import arcpy" and "from osgeo import gdal"

Some considerations:
arcpy is 32-bit (since it wraps arcobjects), so 32-bit python (etc.) are necessary
all versions of wrappers must be compatible with the same Python interpreter

Debugging maptiler on Mac

Running maptiler on Mac is no easy task, as I've found out.

While there are a few stumbling blocks when it comes to dependencies, one that I found particularly perplexing is the bundling of a two Python executables in the .app package.  The maptiler executable is written in such a way as to not only to run the rest of the package through one or other of the Python executables, but to choose based on 32 or 64 bit architecture.  That's independent of the version of Mac OS being run, or the "current" Python.  Most dependency installers will install relative to the "current" Python, and its path.  Therefore, on the 32-bit MacBook Pro I was testing on, I was being kicked on to the bundled Python 2.5 executable, and missing all my dependencies.  This caused various dependency errors, including "ImportError: No module named osgeo"

Here is are the important diffs in maptiler.app that caused it to work: in ../MapTiler.app/Contents/MacOS/maptiler 6,9c6,9 < #if sys.maxint > 2**32: < executable = os.path.join(execdir, "PythonSnow") < #else: < # executable = os.path.join(execdir, "Python") --- > if sys.maxint > 2**32: > executable = os.path.join(execdir, "PythonSnow") > else: > executable = os.path.join(execdir, "Python") and in ../MapTiler.app/Contents/Resources/maptiler/pp/__init__.py 141,147d140 < # fix PYTHONEXECUTABLE dict error < < if "PYTHONEXECUTABLE" in os.environ: < executable = os.environ["PYTHONEXECUTABLE"] < else: < executable = sys.executable < 156c149 < command = "\"" + executable + "\" -u \"" \ --- > command = "\"" + os.environ["PYTHONEXECUTABLE"] + "\" -u \"" \

Relevant link at Google Code: http://code.google.com/p/maptiler/issues/detail?id=42

My Places Overlays on Maptiler Google Map

Maptiler is a good option for creating image overlays (such as historical maps) on Google Maps.  There are some snags though -- the code is still using gmaps API v2, which will give some indication for the attention being given to the project right now.  

 The problem I ran into was adding overlays (such as a KML or GeoRSS feed) on TOP of the image overlay, and correct behavior with the opacity slider built into the maptiler output.

Here are the simple edits to fix this issue.

1. Added a My Places KML overlay by copying the KML URL from My Places and these two lines (I put those at 185, just below the construction of the new map object)

 geoXml = new GGeoXml("https://maps.google.com/maps/ms?authuser=0&vps=1&ie=UTF8&msa=0&output=kml&msid=204904299571298064529.0004d01c986fdc0fe9f19");  
 map.addOverlay(geoXml);  

2. Changed zPriority of maptiler overlay so that it sits below other overlays (I edited the line at ~224 which constructs the "overlay" object

     overlay = new GTileLayerOverlay( tilelayer,{zPriority:-1000});  
   

3. Finally, I fix the opacity slider by changing the line that removes all overlays to just remove the maptiler overlay, and reinitilize it at zPriority -1000 around line 62
   

    this.map.removeOverlay(this.overlay);  
    this.map.addOverlay(this.overlay, { zPriority: -1000 });  
   

 

That's it!

Hit the geocoding wall?

Google's Geocoding API limits results to 2,500/day.  So what are your options when you hit a geocoding wall with a public/free API?

1. Using ArcGIS (or alternatively Gisgraphy, which I have not yet experimented wtih http://www.gisgraphy.com/) you can geocode against the best data you can get your hands on.  Esri Streetmaps Data is great, if you have access (as we do at UD).  Or you could download public data, like the Census TIGER data, geonames, or an openstreetmap extract for the area you're interested in, and geocode on these. There is some high quality dataset from cities and states.  In the case of Delaware,  which licenses TeleAtlas (now TomTom) data, I don't think non-state users are permitted to use it.  I have not seen a way to do this in open source GIS software (e.g., QGIS), which is why ArcGIS or Gisgraphy would be required
2. Mapquest/Nomatim (http://developer.mapquest.com/web/products/open/nominatim) and  Cloudmade  have an unlimited geocoding APIs on OpenStreetMap data.  Sometimes that can be slow or inaccurate though.
   
3. Mapquest (non-Nomatim) and Yahoo also have a 5000 result/day limit ... a 2x improvement on Google

NOAA/NCDC CLIMAPS #2 Climate Atlas of the US

I had previously written about a strange projection in the NOAA/CLIMAPS data, and am now following that up with a strange projection in the Climate Atlas of the US data.  This data comes to the end user with no spatial reference or documentation on disk which describes the spatial reference.  However, I found a helpful page on the climate atlas website which describes the reference: http://www.ncdc.noaa.gov/oa/about/cdrom/climatls2/faq.html#SPHEROID.  This reference, however, is not a common one.  A similar projection is available in ArcGIS though, and by modifying the reference latitude (i.e., latitude of origin), we can apply the new custom projection to these files by "define projection".  Here is a link to the custom projection file.

Web Maps for Digital Humanties

As I'm currently working with two digital humanities groups on web projects, I have been thinking about easy/free ways to get humanities, particularly historical, resources on the web.  The heavier weight self-hosted option we're using is Omeka, and the lighter weight-cloud hosted option is ArcGIS Online.  Omeka provides the kind of exhibits, metadata, and other functionality often crucial to scholarly digital humanities publications, while ArcGIS is more suited as a secondary product or as a class project -- with a significantly fewer adoption costs.  On Omeka, we're using the Neatline and Neatline Maps plugins, as well as Geoserver.

Here's a good workflow for Omeka/Neatline Maps/Geoserver:

1. Download or scan a (historical) map image
2. You will need to use GeoTiff format to work with Neatline Maps. I used ImageMagick to convert your format to TIFF.  We'll worry about georeferencing (the Geo in GeoTIFF) in the next part.  You'll need to use the command line IM tool -- "convert" is a built in Windows system executable, so you'll need to specify the path when you execute from CL, if the IM binary folder is not on your system path.
3. Now that you have your TIFF, georeference with ArcGIS Explorer Desktop and the Georeferencing Add-In.  Explorer is freely available.  You'll find links to download Explorer and the Georeferencing Add-In, as well as instructions for doing the Georeferencing, here: http://blogs.esri.com/esri/arcgis/2010/12/17/georeferencing-add-in-for-explorer-desktop/
4.  After you've gereferenced and saved your TIFF, import it into your Omeka/Neatline Maps site (Add an Item > Files > "Add New Files" and browse to your TIFF ... "Add Item"
5. Your image should now be added to Geoserver and is provided back to NL Maps as a WMS.  Though NL Maps is designed to add the reference to the WMS, we identified a bug that prevented this from occurring.  At this time you will need to add the reference directly through the Edit Item dialog in the Web Map Service tab.  You will need to populate the WMS Address and Layers fields in this tab, in this format:
1. WMS Address: http://HOSTNAME:PORT/geoserver/NAMESPACE/wms
2. Layers: LAYERNAME or LAYERNAME1, LAYERNAME2
6. Next you must create a NL Exhibit through the Neatline tab at the top of the Omeka admin interface.  Once you have created the exhibit, you must Edit Query to include the map image.  I found that the easiest way to get the item you want int the query is to Narrow by Specific Fields using the Title field, with the value of the map image title.
7. Now, when you edit the exhibit, you should see the map image displayed at the proper location.  First, I panned and zoomed to the location of my image.  I immediately changed the opacity of the image via Map Settings > Shape Opacity within the neatline exhibit editing interface.  Next, I fixed the starting point of the map with Layout Editor > Fix starting viewpoint position, within the same inteface.  I saved these changes subsequent to making them, and then refreshed the interface via the browser to see the changes.  
8. Finally I could view my NL Maps Exhibit through  http://SITEURL/neatline-exhibits/show/NLEXHIBITNAME

ArcGIS Online Terms of Service, Notes

With the plethora of cloud services being offered these days for hosting and analyzing geospatial data, it is important to know what we're agreeing to.  Unfortunately "the fine print" is still relatively dense.    With ArcGIS.com (ArcGIS Online) opening up their "subscription" service level to their higher ed site license clients, and more of our clients interested in hosting their research, I decided it was time to take another look at the fine print.

Here are some of the main points I took from the License Agreement (E204 06/24/2012) ... you'll want to focus on Addendum 3 (E300-3) which covers Online Services.  These notes should not be taken as legal advice.  I would recommend you check out the license agreement yourself at http://www.esri.com/legal/software-license

Good Stuff

1. Intellectual property -- it looks like Esri doesn't take ownership of any content.  It remains property of the licensee, the university in this case, and Esri only accesses for purposes of providing the service.
2. Common sense: If you share your data or grant others access to edit it, they might use it or might alter your service.  You may want to adopt your own use restrictions, published alongside your data.
3. After the license is terminated (e.g., if the university decided to stop our relationship with Esri, which would be a big deal) the licensee is granted a period of 30 days to retrieve their content -- after which Esri has no further obligations.

Bad Not Good Stuff

1. The licensee (the university) has a limited amount of service credits.  Many of the functions of the subscription (e.g., tile caching and hosting) cost "service credits".  Once we approach 75% of credits consumed we will be notified.  Once our credits are used our subscription services will be suspended until we renew our credits.  For that reason the administrator (me) will be extra careful about who I grant access to use functions that consume credits.
2. If you are using an evaluation subscription you must make sure to export your content before the evaluation ends, or it will be permanently lost.