... geocoding continued

As I had mentioned in previous email Esri has retired it's public-facing geocoding and routing services. These services are now only available behind their paywall (through an ArcGIS Online subscription). Esri gives UD a fixed number of credits on our subscription that we can use towards their various services.  The cost for use of their Geocoder service is relatively expensive, as far as their services go.  For example 300,000 would take nearly our whole allocation of service credits.  However, for lesser numbers of geocode (for one-off needs) we can allow you to join our subscirption through an ArcGIS Online account.  Contact us for more information.

For all other geocoding needs, I recommend the following desktop solution:

Download Disc 2: StreetMap North America (Detailed Road Network, Geocoding, Routing) from the UD GIS Data page.

You can then choose one of the .loc files in streetmap_na\data when it asks for an address locator in these instructions: http://resources.arcgis.com/en/help/main/10.2/index.html#/in_ArcMap/002500000026000000/

Running GDAL and arcpy together (ArcGIS 10.1, GDAL 1.9.2, python 2.7.4)

There are many use cases for running GDAL and arcpy in the same python instance.  GDAL is the most popular open source geopspatial geoprocessing toolset and arcpy is Esri's python wrapper for the entire ArcGIS geoprocessing toolset.

However, doing so can get quite tricky.  GDAL's python bindings are not easily configured for Windows (and ArcGIS/ArcObjects is Windows-only).  The OSGeo4W project can be used to do so, but the bundled Python does not play well with arcpy.  arcpy itself depends on numpy and requires multiple path inclusions.

Here are steps I took to import gdal and arcpy.  
I tested by successfully running arcpy.RasterToNetCDF_md()  (though I haven't tested gdal yet).

Step 1: Install standalone python to C:\Python274 (download from http://www.python.org/ftp/python/2.7.4/python-2.7.4.msi)
Step 2: Add numpy (version bundled with ArcGIS) and Desktop10.1.pth to C:\Python274\Lib\site-packages (download from http://www.udel.edu/001745)
Step 3: Install GDAL binaries and python bindings (download installer from http://www.udel.edu/001742).
Step 4: Run import commands "import arcpy" and "from osgeo import gdal"

Some considerations:
arcpy is 32-bit (since it wraps arcobjects), so 32-bit python (etc.) are necessary
all versions of wrappers must be compatible with the same Python interpreter

ENVI/IDL and ArcGIS/Python integration

I occasionally answer questions on ENVI/IDL and ArcGIS/Python integration.

ENVI functions are available as "tools" through the ArcGIS GUI toolbox:

"With ENVI 4.8, ENVI’s scientifically proven image analysis capabilities will be available as discrete tools in the ArcGIS toolbox and accessible directly from ArcGIS desktop and server environments. Upon installation of ENVI products, a selection of powerful image processing and analysis tools will be available within ArcGIS."

A list of those tools is here (in the context of licensing, but just ignore that): http://www.ittvis.com/ProductsServices/ENVI/ToolsLicensing.aspx 




IDL can also be accessed for extension of ENVI/IDL functionality:

"developing customized ENVI tools can be done using simple IDL scripting and wrappers that facilitate the integration of ENVI tools with the ArcGIS platform."

Here's a whitepaper: http://www.ittvis.com/portals/0/whitepapers/ENVI_Tools_ArcGIS_Server_WP_Tutorial.pdf

Operationalizing SDE Subsetting/Layer Registration

Q: How to get max latitude for each given longitude in a time period subset of points as a GIS layer?

The particular dataset we were working with already had X and Y coordinate fields and was in shapefile format.  After attempting to use ArcGIS tools, including those that exposed the limited SQL that ArcGIS offers without SDE, we found that SDE was necessary.  We had already set up a SDE Postgis/Postgres (PG) database, so we could easily import the shapefile. 

First we wanted to create a subset of max latitudes in PG, through the preferred PG interface (ours was pgAdmin III).  Where max is the max subset and data is the original data:

SELECT * INTO sde.max FROM sde.data Where y in (SELECT max(y) FROM sde.data GROUP BY x); ALTER TABLE sde.max OWNER TO sde; GRANT ALL ON TABLE sde.max TO sde;

.  Notice the use of the 'sde.' schema prefix and the permissions queries, which we found necessary (see stumbling blocks below).  We operationalized this by semi-colon delimiting and changing table names where necessary.  This could be made much more efficient by getting a list of tables with the name format given to the tables that we wished to subset, but we were more interested in getting a result than more completely operationalizing at this time.

Next we needed to register these new tables as SDE layers, so that ArcGIS could "see" them.  After this step, the tables behave as normal ArcGIS feature classes.  Before this set, we could not even get properties of the tables, let alone interact with them (an error message would be displayed).  We ran the following command to do this through Windows cmd (SDE bin must be on the Path):

sdelayer -o register -l max,shape -e npc -C objectid,sde -i sde:postgresql:localhost -D sde -s localhost -u sde -p sde -t st_geometry

  .  We operationalized this by repeating this in a .bat file and changing tables names.  Again, this could be made more efficient by using an sde command to get a table list and looping through all tables that had names which matched the max subset table name format.

Stumbling Blocks

1. The most difficult problem we ran into was the "DBMS table not found (-37)" error, which apparently is quite common for a variety of reasons with SDE.  This ultimate came down to needing to make sure the table was stored under the sde database, in the sde schema, and that the table was owned by sde.  I had assumed that making the owner of the database sde would cause new tables to be sde owned by default, but this was not the case.
2. You may need to update your SDE/ArcGIS Desktop to 10 SP3.  This was mentioned in some forums with the above error and with other errors around not being able to see tables in an SDE DB.  To do the update, you need to download/install SP3 on desktop/sde and then run "update database" from the database properties under ArcCatalog.  However, you must create a direct connection to do the update ... update does not work under the normal ArcCatalog Spatial DB connection.  
3. General SDE recommendations:  My rule of thumb is"anything you can do in arcgis/catalog, do there ... all else do in pgadmin".  Also remember to refresh after all steps if there's some result you're trying to see.  pgAdmin and ArcCatalog both need refreshes before showing updates.

ArcGIS on a High Performance cluster: Part 1, Linux

Now that we have our new community cluster running at UD  it's time to learn how I can optimize GIS software for that environment.

The first hurdle is operating system

ArcInfo Workstation used to be a great way to run Esri/ArcGIS geoprocessing tasks on *nix boxes.  However, it seems the last version of ArcInfo Workstation that ran on Linux was 9.1.  I'd contacted Esri about obtaining a copy of 9.1, but apparently it is out of production and they do not have any copies of the software that they'd be able to send to me.

So basically the workstation/desktop route is out not available at this time.  But there is more than one way to skin a cat: ArcGIS Server 10 (AGS) runs on Linux.

I've recently seen documentation which suggests geoprocessing models can be leveraged by publishing them as geoprocessing services through ArcGIS Server.  I have heard at the Esri conference, and in some documentation that success has even been reported in distributed tasks, such as building caches, through the SOM/SOC architecture that is available out of the box in AGS.  Could this architecture be extended to distribute geoprocessing tasks?

Taking a different tack, AGS exposes the geoprocessing object through a Python wrappers.  That means that we should be able to programatically run our software on Linux through Python.  Python also has wrappers or libraries for multithreading and MPI (distributed), so the implications for taking advantage of our cluster are especially exciting. 

Note: There are some differences to be expected, such as with file path conventions and name lengths. 

Next: Multithreading

http://support.esri.com/en/knowledgebase/techarticles/detail/31903

Intersect ArcGIS Bug

There is a bug in ArcGIS with the intersect tool (I noticed it when intesecting lines as polylines) that causes the result to be an empty table. To fix this I exported all tables being intersected to a new location (and the output to point there as well). This resulted in the expected output.

Flipped Polar Coordinate System

Problem: NIC/NSIDC Antarctica data was appearing flipped 180 dg, without coordinate system or projection defined. When other polar data, thought to be in the same coordinate system/projection was placed with this data, the other data appears to be flipped 180 in the right orientation (though the data in question is still flipped).

Solution: Modify the projection of the data in question to have a central meridian of 180 instead of 0 (with "define projection" using a custom projection based the other data). THE DATA VIEW PROJECTION MUST ALSO BE SET TO THE PROJECTION OF THE "GOOD" DATA (e.g. 0 Meridian, Polar Stereographic)

ArcGIS Editor for OpenStreetMap

I recently had a chance to test out ArcGIS Editor for OpenStreetMap (1.1 Beta3), an open source toolbox for interacting with OpenStreetMap (OSM). OSM, a popular open-source data repository, gained extra traction in the US after the Haitian Earthquake. This growth came from NGOs, and from some ArcGIS shops with open source leanings, who advocated on behalf of an extension to OSM from ArcGIS. I've been interacting heavily with OSM in building and maintaining the University of Delaware Map (UD Map). The data schema for UD Map is based on the OpenStreetMap schema, and interfaces with OpenStreetMap for data updates and publishing. As the project demanded it, and as a test of the new tools, I used the toolkit to publish some pending campus base layers. Alas, this trial ended disappointingly, leaving only manual editing of the vector layer. Append, which could have been used to join the vector input to the OSM polygon layer, did not work as expected: The OSM changeset was not updated, and therefore could not be uploaded to OSM for processing. The tools still look very useful for doing manual editing of OSM data and adding OSM data as fully editable vector layers in a map ... and of course, as code to be expanded upon ... this story is to be continued :-)

Random Feature Selection

I met with a gentleman today who needed to select random streets in Wilmington in order to do a field survey of urban forestry there. Here are the steps we took to do so:

-Use Excel and the RANDBETWEEN function to generate random numbers in a range = total number of features (in this case, streets in Wilmington). Drag this equation down a column = number of features in sample. Once this has been done, save/export (I usually like to use DBF for ArcGIS)

RANDBETWEEN(bottom,top)

Bottom is the smallest integer RANDBETWEEN will return.

Top is the largest integer RANDBETWEEN will return.

- with data added to ArcGIS, use "Frequency" tool to check that no duplicate values were generated

- do a regular one to one join, dropping out all values that don't match. The result will be a random set of features -- in this case, street segments.

Animal Movement and Home Range Options for ArcGIS

The Animal Movement Extension was written in Avenue, and development stopped many years ago. Therefore, Animal Movement Extension is not available for ArcGIS 9.x or 10.

There are a couple of options if you'd like to move forward with ArcGIS

• A lot of people using Animal Movement extension migrated to Hawths Tools, to be able to use ArcGIS. Hawths tools offered many of the functions in Animal Movement, though Hawths tools is also now a discontinued project, and will not work with newer versions of ArcGIS (9.2 was the last version it worked with).
• Hawths tools has been been merged into the Geospatial Modeling Environment, which works with ArcGIS 10. You can download this and find more info at: http://www.spatialecology.com/gme/
• Home Range Tools for ArcGIS works on ArcGIS 9.x, and is available at http://nhsbig.inhs.uiuc.edu/wes/home_range.html
• There is a lot of work going on with Agent Based modeling right now. I met with an agent based modeling working group at the ESRI conference. They've developed an extension for ArcGIS based on repast called Agent Analyst: http://www.institute.redlands.edu/agentanalyst/ . I didn't hear about any scenarios involving wildlife movement, buy there is a book coming out, and I would assume those scenarios might be discussed. I'll keep my ear to the ground on this one
• There are also contributed and native tools and scripts that could be used to perform many of these functions if strung together in the right order

Other Resources

• http://snr.unl.edu/powell/teaching/nres896/density_HR/nres896_links.htm
• http://warnercnr.colostate.edu/class_info/fw663/

Table to table bug (ArcGIS)?

Is there a bug with the "table to table" tool in ArcGIS when using it to export a DBF from an Excel file (XLS, XLSX)? I don't know. BUT, I would avoid using any Excel files in ArcGIS. This would be one of many poorly documented bugs ... support for Excel in ArcGIS is a weak link, and you really don't want to be basing multistep analysis off some poorly imported data (trust me!). So you have an XLSX file that is more than the limit of rows for XLS ... how do you get it to DBF (since Excel 2007 doesn't export DBF). If you have MS Access, I would recommend going that route. I tried this out with a file that I was getting super incorrect output on, and it worked. Seems like a nice, clean, consistent solution.

unexpected rows in tables

Occasionally, the number of rows in a table* is not really the number of rows in the table. Sometimes this peculiarity only makes itself known when you export a table and only see a subset of the total rows in the original attribute table ... where all the other rows go to?

One reason that this happens (perhaps the most common), is that when you use the "keep only matches" option when doing a join, the non-matched rows are kinda there, and kinda not. To fix this you would need to go back and use the "keep non-matched" option.


*as counted by the "Records (1 out of x)" at the bottom of an attribute table window

license server down or not responding -96,7

was getting an error like "license server down or not responding -96,7" when trying to get ArcGIS to run on a Vista machine. Seems like the problem was that since the host name could not be set (I could not browse to the license server and therefore it could not be set through the ArcGIS Administrator dialog) the host's license file wasn't validating (although the machine I was setting up could ping the host). I got around this by setting all registry entries = "Not_Set" (actually were "@Not_Set") to the name/port of my license server

Classify Dialog does not open

Problem: Classify dialog does not open when hitting button "Classify ..." under Spatial Analyst "Reclassify" tool

Solution: Was trying to do this on an image (JPG). Must convert to a native ArcGIS grid to get the Classify dialog with histogram.