NOAA/NCDC CLIMAPS #2 Climate Atlas of the US

I had previously written about a strange projection in the NOAA/CLIMAPS data, and am now following that up with a strange projection in the Climate Atlas of the US data.  This data comes to the end user with no spatial reference or documentation on disk which describes the spatial reference.  However, I found a helpful page on the climate atlas website which describes the reference: http://www.ncdc.noaa.gov/oa/about/cdrom/climatls2/faq.html#SPHEROID.  This reference, however, is not a common one.  A similar projection is available in ArcGIS though, and by modifying the reference latitude (i.e., latitude of origin), we can apply the new custom projection to these files by "define projection".  Here is a link to the custom projection file.

Web Maps for Digital Humanties

As I'm currently working with two digital humanities groups on web projects, I have been thinking about easy/free ways to get humanities, particularly historical, resources on the web.  The heavier weight self-hosted option we're using is Omeka, and the lighter weight-cloud hosted option is ArcGIS Online.  Omeka provides the kind of exhibits, metadata, and other functionality often crucial to scholarly digital humanities publications, while ArcGIS is more suited as a secondary product or as a class project -- with a significantly fewer adoption costs.  On Omeka, we're using the Neatline and Neatline Maps plugins, as well as Geoserver.

Here's a good workflow for Omeka/Neatline Maps/Geoserver:

1. Download or scan a (historical) map image
2. You will need to use GeoTiff format to work with Neatline Maps. I used ImageMagick to convert your format to TIFF.  We'll worry about georeferencing (the Geo in GeoTIFF) in the next part.  You'll need to use the command line IM tool -- "convert" is a built in Windows system executable, so you'll need to specify the path when you execute from CL, if the IM binary folder is not on your system path.
3. Now that you have your TIFF, georeference with ArcGIS Explorer Desktop and the Georeferencing Add-In.  Explorer is freely available.  You'll find links to download Explorer and the Georeferencing Add-In, as well as instructions for doing the Georeferencing, here: http://blogs.esri.com/esri/arcgis/2010/12/17/georeferencing-add-in-for-explorer-desktop/
4.  After you've gereferenced and saved your TIFF, import it into your Omeka/Neatline Maps site (Add an Item > Files > "Add New Files" and browse to your TIFF ... "Add Item"
5. Your image should now be added to Geoserver and is provided back to NL Maps as a WMS.  Though NL Maps is designed to add the reference to the WMS, we identified a bug that prevented this from occurring.  At this time you will need to add the reference directly through the Edit Item dialog in the Web Map Service tab.  You will need to populate the WMS Address and Layers fields in this tab, in this format:
1. WMS Address: http://HOSTNAME:PORT/geoserver/NAMESPACE/wms
2. Layers: LAYERNAME or LAYERNAME1, LAYERNAME2
6. Next you must create a NL Exhibit through the Neatline tab at the top of the Omeka admin interface.  Once you have created the exhibit, you must Edit Query to include the map image.  I found that the easiest way to get the item you want int the query is to Narrow by Specific Fields using the Title field, with the value of the map image title.
7. Now, when you edit the exhibit, you should see the map image displayed at the proper location.  First, I panned and zoomed to the location of my image.  I immediately changed the opacity of the image via Map Settings > Shape Opacity within the neatline exhibit editing interface.  Next, I fixed the starting point of the map with Layout Editor > Fix starting viewpoint position, within the same inteface.  I saved these changes subsequent to making them, and then refreshed the interface via the browser to see the changes.  
8. Finally I could view my NL Maps Exhibit through  http://SITEURL/neatline-exhibits/show/NLEXHIBITNAME

ArcGIS Online Terms of Service, Notes

With the plethora of cloud services being offered these days for hosting and analyzing geospatial data, it is important to know what we're agreeing to.  Unfortunately "the fine print" is still relatively dense.    With ArcGIS.com (ArcGIS Online) opening up their "subscription" service level to their higher ed site license clients, and more of our clients interested in hosting their research, I decided it was time to take another look at the fine print.

Here are some of the main points I took from the License Agreement (E204 06/24/2012) ... you'll want to focus on Addendum 3 (E300-3) which covers Online Services.  These notes should not be taken as legal advice.  I would recommend you check out the license agreement yourself at http://www.esri.com/legal/software-license

Good Stuff

1. Intellectual property -- it looks like Esri doesn't take ownership of any content.  It remains property of the licensee, the university in this case, and Esri only accesses for purposes of providing the service.
2. Common sense: If you share your data or grant others access to edit it, they might use it or might alter your service.  You may want to adopt your own use restrictions, published alongside your data.
3. After the license is terminated (e.g., if the university decided to stop our relationship with Esri, which would be a big deal) the licensee is granted a period of 30 days to retrieve their content -- after which Esri has no further obligations.

Bad Not Good Stuff

1. The licensee (the university) has a limited amount of service credits.  Many of the functions of the subscription (e.g., tile caching and hosting) cost "service credits".  Once we approach 75% of credits consumed we will be notified.  Once our credits are used our subscription services will be suspended until we renew our credits.  For that reason the administrator (me) will be extra careful about who I grant access to use functions that consume credits.
2. If you are using an evaluation subscription you must make sure to export your content before the evaluation ends, or it will be permanently lost.

pgAgent Stumbling Blocks

In addition to doing scheduled backups from Postgres, we've also needed to load nightly dumps sent via scp to our database server's filesystem.  I finally got pgAgent working, but it wasn't without much consternation.  If you need to do this sort of thing on Windows, I'd recommend using Task Scheduler and a batch file if pgdump will work (though it likely won't if the file is coming from Oracle), or Task Scheduler with pgsql if it won't.  If you need to use pgAgent, here are some stumbling blocks that I found:

good support doc here: http://www.pgadmin.org/docs/1.8/pgagent-install.html

• add pgpass.conf to C:\Users\postgres\AppData\Roaming\postgresql
• should be localhost:5432:*:postgres:PASSWORD 
• (note, this is not under the default user, it's the "postgres" user on your system) 
• may also need the following in C:\Program Files (x86)\PostgreSQL\9.0\data\pg_hba.conf (but don't think so) 
• local all all trust
• host all all XXX.XXX.XXX.XXX/32 trust 
• reload configuration *running as admin* 
• install pgAgent via application/stackbuilder (*run as admin*) 
• make your steps ignore error, for some reason they error even if they complete successfully and stall subsequent steps

NOAA/NCDC CLIMAPS

National Climatic Data Center's Climate Maps of the United States database offers polygon data for many popular climate variables.  We would like to use this data for preliminary visualizations on bridge decay research -- the primary benefit to this data being its simplicity (it's already averaged by day/year over a 40 year period), national continuity, and file format (shapefile).

However we encountered two issues with the data.

Issue #1: it's based on a funky datum "GCS_NAD_1927_CGQ77", which addresses some issues in Quebec with NAD 1927.

We were able to side-step this issue by just re-defining the projection as something more common ... in our case we just went right to WGS 1984, since we wanted to view in Google Earth, and since a rough visualization was our immediate purpose.  I suppose NAD 1927 would have been fine, since we weren't doing anything with Quebec.  BTW, there are no readily available transformations of the  GCS_NAD_1927_CGQ77 datum.

Issue #2: the data is not continuous or even numerical

Values are expressed as "0 - 5 in", so we have no way of redefining what values are grouped in what colors, and since it's polygon data we also lose the ability to redefine spatial grouping based on new intervals.  That's not a big problem for this first rough visualization, but we will need to find more fine-grained data for subsequent analysis.  We'll probably end up taking point data from other NCDC sources and doing our own interpolation and temporal reduction.

ArcReader 9/ArcGIS 10 Conflict

I've seen a few folks who've had trouble installing ArcGIS 10 because they've been unable to uninstall ArcReader (usually version 9.x).

The best way I've found to do this is to download the ArcReader installer from here.  Run the installer and try to uninstall.

If you're unable to do that (sometimes you'll need to force quit and restart) you may need to run an install, but uncheck the ArcReader feature.  This gets rid of some hanging installer memory.

Then try to run a repair (restart if necessary) and start the downloaded ArcReader installer ... select repair when prompted.

After you've repaired you should be able to uninstall correctly.

If that doesn't work, you're stuck deleting all ArcReader files and ArcReader (and ArcGIS/Esri) references in the registry using regedit.

Delaware GIS 2012

The Delaware 2012 GIS conference was held yesterday.  It was a great opportunity to meet and learn from our colleagues in Delaware and the region.  I presented on Geospatial Data in Cloud and High Performance Environments, which you can access here: http://udel.academia.edu/BenjaminMearns/Talks/81729/Geospatial_Data_in_Cloud_and_High_Performance_Computing_Environments