The Atlas of Visual Complexity

Very cool dynamic cartograms for visualizing economic complexity: atlas.media.mit.edu

Surface Climate Data

I am consulting on a research project involving bridge decay. For this project we needed to locate a number of surface atmosphere chemical data (some sulfure oxides and chloride) as well as data on precipitation, snowfall, and min/max temperature.

Here are some sources we found:

• Cl, SO4, precipitation etc.: National Atmospheric Deposition Program, http://nadp.sws.uiuc.edu/ ... the National Trends Network (NTN) data was the best
• SO2: http://www.epa.gov/airtrends/sulfur.html ... a KML is even available from this link
• Snowfall, Min/max tempurature (precip is also available): Daily by state, http://cdiac.ornl.gov/ftp/ushcn_daily/, Monthly, http://cdiac.ornl.gov/ftp/ushcn_v2_monthly/

NetCDF

NetCDF is a format well suited for environmental research given the inherent multi-dimensionality of the environment and environmental research topics. 

I have not worked much with NetCDF data, and recently ran into some trouble importing gridded bathymetric data sets (GEBCO) into ArcGIS.

When importing into ArcGIS I attempted to use Multidimension Tools > Make NetCDF Raster Layer.  This tutorial from Esri describes this procedure.  Unlike in the tutorial, default values were not populated for tool parameters, and as I was unable to ascertain these values, I could not run the tool.

My next option was to learn more about the data set from GDAL's gdalinfo command.  After recompiling with the netcdf libraries, I was able to run gdalinfo on the data set, but it returned a rather sparse result that did not illuminate any of the parameters that I needed.  This contrasts with the result show in the GDAL documentation for netcdf.

I then attempted to load NetCDF directly into ENVI and IMAGINE -- common geospatial image proecessing programs -- I could not load it successfully into either of these.

I was somewhat more successful with IDL.  IDL is able to read NetCDF and report on its structure. However, the commands for interacting with the results objects are unwieldy at best.  This is an option that requires more investigation.

The GMT use NetCDF as the native package format, and GEBCO and others seem to base their NetCDF format off of GMT's specifications.  The grdinfo command under GMT gives some information about the data set, but still nothing that can be used in ArcGIS.  This also requires more investigation, as there are many commands for working with NetCDF.

Finally the data set was able to be opened by GEBCO's specialized display software (which is produced by the data provider).

I will be adding more to this post as I learn more about NetCDF and perhaps the peculiarities of this particular data set ... why use open standards if you need a data provider created tool to open the data??

Importing Data Dump into Postgres

I needed to take four steps to import an Oracle binary dump into PostgreSQL

1. Open Oracle dump with something like notepad++ or grap and do a find/replace for all the binary characters (these can be highlighted in notepad++) and replace them with something like a pipe character ("|")
2. If you know the special characters that are not compatible with UTF-8, then do find/replace on these too now ... otherwise you can do this after running the copy command (step 5), since that will give an error for each non UTF-8-able line/character
3. Open the piped file in Excel and replace with tabs.  Tabs are the standard delimiter in Postgres and by saving out of Excel, you get a uniform number of columns for each record, thereby avoiding "missing data for column" when executing the "copy" command
4. Create a table in Postgres with the number of columns seen in Excel, using generic data type such as text
5. execute the copy command ... no delimiters param need be given, since it's tabs

   COPY data FROM 'C:/temp/data.txt'

ArcGIS Weighted Overlay Error #999999

Client was getting error 999999 (generic) error when running weighted overlay analysis.  Fixed this by changing Windows environment variable for temporary folders to c:\Windows\Temp from the users temporary directory, which had a complicated file name

Caching (GWC) Imagery

Issues I've run into while caching imagery on Geoserver/GWC

• Out of Memory Errors (java.lang.OutOfMemoryError: PermGen space): Solved this (as suggested here) by adding -Xms48m -Xmx256M -XX:MaxPermSize=128m to Geoserver's startup.bat/startup.sh on the line that starts the JVM (for me it was line 120)
• also found a post suggesting adding UseCompressedOops fixed some issues with caching imagery ... from

"%RUN_JAVA%" -DGEOSERVER_DATA_DIR="%GEOSERVER_DATA_DIR%" -Djava.awt.headless=true -DSTOP.PORT=8079 -DSTOP.KEY=geoserver -jar start.jar

to

"%RUN_JAVA%" -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSClassUnloadingEnabled -Xms48m -Xmx512M -XX:MaxPermSize=256m -XX:+UseCompressedOops -DGEOSERVER_DATA_DIR="%GEOSERVER_DATA_DIR%" -Djava.awt.headless=true -DSTOP.PORT=8079 -DSTOP.KEY=geoserver -jar start.jar

• ERROR [seed.MTSeeder] - Problem communicating with GeoServer : was getting this error because a DB parameter had recently changed (DB name)
• Error ... Could not list layers for this store, an error occurred retrieving them ... then something about null argument ... this was also fixed with these memory options
• I also found it necessary to scale down my imagery using the outsize parameter  with gdal translate

gdal_translate -of GTiff -co "TILED=YES" -outsize 50% 50% input-file.tif output-file.tif

• I think I may also have been having trouble not adding a world file (possibly prj) which I then created with ArcGIS (not sure how to do that with GDAL, since that saves the geographic info directly to the TIFF)

JQuery Each Iterator

Is your each iterator not working on an Array? Try constructing the array as an object instead. Jquery v1.3.2