Ben Mearns

ArcGIS on a High Performance cluster: Part 1, Linux

2012-02-03T06:41:00.000-08:00

Now that we have our new community cluster running at UD it's time to learn how I can optimize GIS software for that environment.

The first hurdle is operating system

ArcInfo Workstation used to be a great way to run Esri/ArcGIS geoprocessing tasks on *nix boxes. However, it seems the last version of ArcInfo Workstation that ran on Linux was 9.1. I'd contacted Esri about obtaining a copy of 9.1, but apparently it is out of production and they do not have any copies of the software that they'd be able to send to me.

So basically the workstation/desktop route is out not available at this time. But there is more than one way to skin a cat: ArcGIS Server 10 (AGS) runs on Linux.

I've recently seen documentation which suggests geoprocessing models can be leveraged by publishing them as geoprocessing services through ArcGIS Server. I have heard at the Esri conference, and in some documentation that success has even been reported in distributed tasks, such as building caches, through the SOM/SOC architecture that is available out of the box in AGS. Could this architecture be extended to distribute geoprocessing tasks?

Taking a different tack, AGS exposes the geoprocessing object through a Python wrappers. That means that we should be able to programatically run our software on Linux through Python. Python also has wrappers or libraries for multithreading and MPI (distributed), so the implications for taking advantage of our cluster are especially exciting.

Note: There are some differences to be expected, such as with file path conventions and name lengths.

Next: Multithreading

http://support.esri.com/en/knowledgebase/techarticles/detail/31903

The Atlas of Visual Complexity

2012-01-25T08:18:00.000-08:00

Very cool dynamic cartograms for visualizing economic complexity: atlas.media.mit.edu

Surface Climate Data

2012-01-19T06:34:00.000-08:00

I am consulting on a research project involving bridge decay. For this project we needed to locate a number of surface atmosphere chemical data (some sulfure oxides and chloride) as well as data on precipitation, snowfall, and min/max temperature.

Here are some sources we found:

Cl, SO4, precipitation etc.: National Atmospheric Deposition Program, http://nadp.sws.uiuc.edu/ ... the National Trends Network (NTN) data was the best
SO2: http://www.epa.gov/airtrends/sulfur.html ... a KML is even available from this link
Snowfall, Min/max tempurature (precip is also available): Daily by state, http://cdiac.ornl.gov/ftp/ushcn_daily/, Monthly, http://cdiac.ornl.gov/ftp/ushcn_v2_monthly/

NetCDF

2012-01-17T09:03:00.000-08:00

NetCDF is a format well suited for environmental research given the inherent multi-dimensionality of the environment and environmental research topics.

I have not worked much with NetCDF data, and recently ran into some trouble importing gridded bathymetric data sets (GEBCO) into ArcGIS.

When importing into ArcGIS I attempted to use Multidimension Tools > Make NetCDF Raster Layer. This tutorial from Esri describes this procedure. Unlike in the tutorial, default values were not populated for tool parameters, and as I was unable to ascertain these values, I could not run the tool.

My next option was to learn more about the data set from GDAL's gdalinfo command. After recompiling with the netcdf libraries, I was able to run gdalinfo on the data set, but it returned a rather sparse result that did not illuminate any of the parameters that I needed. This contrasts with the result show in the GDAL documentation for netcdf.

I then attempted to load NetCDF directly into ENVI and IMAGINE -- common geospatial image proecessing programs -- I could not load it successfully into either of these.

I was somewhat more successful with IDL. IDL is able to read NetCDF and report on its structure. However, the commands for interacting with the results objects are unwieldy at best. This is an option that requires more investigation.

The GMT use NetCDF as the native package format, and GEBCO and others seem to base their NetCDF format off of GMT's specifications. The grdinfo command under GMT gives some information about the data set, but still nothing that can be used in ArcGIS. This also requires more investigation, as there are many commands for working with NetCDF.

Finally the data set was able to be opened by GEBCO's specialized display software (which is produced by the data provider).

I will be adding more to this post as I learn more about NetCDF and perhaps the peculiarities of this particular data set ... why use open standards if you need a data provider created tool to open the data??

Importing Data Dump into Postgres

2011-12-20T06:43:00.000-08:00

I needed to take four steps to import an Oracle binary dump into PostgreSQL

Open Oracle dump with something like notepad++ or grap and do a find/replace for all the binary characters (these can be highlighted in notepad++) and replace them with something like a pipe character ("|")
If you know the special characters that are not compatible with UTF-8, then do find/replace on these too now ... otherwise you can do this after running the copy command (step 5), since that will give an error for each non UTF-8-able line/character
Open the piped file in Excel and replace with tabs. Tabs are the standard delimiter in Postgres and by saving out of Excel, you get a uniform number of columns for each record, thereby avoiding "missing data for column" when executing the "copy" command
Create a table in Postgres with the number of columns seen in Excel, using generic data type such as text
execute the copy command ... no delimiters param need be given, since it's tabs
```
COPY data FROM 'C:/temp/data.txt'
```

ArcGIS Weighted Overlay Error #999999

2011-11-15T13:17:00.001-08:00

Client was getting error 999999 (generic) error when running weighted overlay analysis. Fixed this by changing Windows environment variable for temporary folders to c:\Windows\Temp from the users temporary directory, which had a complicated file name

Caching (GWC) Imagery

2011-10-10T07:27:00.000-07:00

Issues I've run into while caching imagery on Geoserver/GWC

Out of Memory Errors (java.lang.OutOfMemoryError: PermGen space): Solved this (as suggested here) by adding -Xms48m -Xmx256M -XX:MaxPermSize=128m to Geoserver's startup.bat/startup.sh on the line that starts the JVM (for me it was line 120)
also found a post suggesting adding UseCompressedOops fixed some issues with caching imagery ... from

"%RUN_JAVA%" -DGEOSERVER_DATA_DIR="%GEOSERVER_DATA_DIR%" -Djava.awt.headless=true -DSTOP.PORT=8079 -DSTOP.KEY=geoserver -jar start.jar

"%RUN_JAVA%" -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSClassUnloadingEnabled -Xms48m -Xmx512M -XX:MaxPermSize=256m -XX:+UseCompressedOops -DGEOSERVER_DATA_DIR="%GEOSERVER_DATA_DIR%" -Djava.awt.headless=true -DSTOP.PORT=8079 -DSTOP.KEY=geoserver -jar start.jar

ERROR [seed.MTSeeder] - Problem communicating with GeoServer : was getting this error because a DB parameter had recently changed (DB name)
Error ... Could not list layers for this store, an error occurred retrieving them ... then something about null argument ... this was also fixed with these memory options
I also found it necessary to scale down my imagery using the outsize parameter with gdal translate

gdal_translate -of GTiff -co "TILED=YES" -outsize 50% 50% input-file.tif output-file.tif

I think I may also have been having trouble not adding a world file (possibly prj) which I then created with ArcGIS (not sure how to do that with GDAL, since that saves the geographic info directly to the TIFF)

JQuery Each Iterator

2011-10-05T08:17:00.000-07:00

Is your each iterator not working on an Array? Try constructing the array as an object instead. Jquery v1.3.2

Setting LM_LICENSE_FILE Environment Variable for ENVI/IDL 4.8 on Pre-Lion MacOS

2011-09-22T09:56:00.000-07:00

To set an environment variable for ENVI/IDL on a pre-Lion MacOS System, I would recommend using the environment.plist file. This persists through whatever shells you have installed as well as in X Windows (and therefore when you double-click the ENVI executable from the MacOS GUI). I did not find any correct documentation for doing this from ITT or NASA.

1. Install the software
3. Define the environment variable in the file ~/.MacOSX/environment.plist
----- Open the Terminal
----- You will probably have to create the directory ... mkdir ~/.MacOSX)
----- ... and the file vi ~/.MacOSX/environment.plist

Paste the following into vi in Insert mode (press i) and save (:x) ... if you have more than one environment variable, make sure to add another key/string pair ... make sure you subsistute PORT and HOST below


<!--?xml version="1.0" encoding="UTF-8"?--><!--?xml version="1.0" encoding="UTF-8"?-->
<plist version="1.0">
<dict>
   <key>LM_LICENSE_FILE</key>
   <string>PORT@HOST</string>
</dict>
</plist><plist version="1.0"><dict></dict>
</plist>

4. Close terminal

You should now be able to use ENVI/IDL.

Openlayers Geoserver WMS Caching with GWC

2011-09-01T13:00:00.000-07:00

Now that Geowebcache is integrated with Geoserver and WMS (among other OGC service types), map tile-caching is supposed to be really simple. I found this not to be the case. Among other challenges, document is lacking or blatantly wrong. Here are a couple key concepts that I learned the hard way:

Use the demo from the Geowebcache section (http://localhost:8080/geoserver/gwc/demo) NOT the regular OL "Layer Preview" which does not use the correct gridset parameters (nice demo, eh?)

Issues I was having with labeling and tiles are mostly solved by creating a database view from the centroids of polygons to be labeled and label these instead, and include in a layer group.

Don't use the "Recycle Tiles" option ... especially if you don't have native JAI. This causes a memory leak and subsequently causes the server to crawl/crash after X requests. Also you probably shouldn't tweak max render time/memory unless you have to (despite documentation saying that this helps performance ... would this be helpful at another time? I don't know).

Despite documentation saying to use the wms endpoint and that no special endpoint is needed ... a special endpoint is needed ... it's of this format http://host:8080/geoserver/gwc/service/wms

The gridset scheme is static and opaque, unless a custom config file is created. The documentation gives the impression that the integration of WMS/GWC creates arbitrary gridsets given wms grid request characteristics. One could could create their own geowebcache.xml (saved to GEOSERVER/webapps/WEB-INF/geowebcache.xml ... though there is some confusion about that). See more about geowebcache, the gridset schema, and geowebcache.xml here

Don't pass anything on the wms request that you don't have to! Forums and documentation say that you need to pass things like tilesOrigin, which will cause the request to fail most of the time.

GWC saves tiles to your data directory under the gwc folder. You should check here to make sure tiling is working.

Lookbehind/Lookahead Assertion with Npp Regex

2011-08-24T12:55:00.000-07:00

Today I was looking for a way to get the last character of a pattern so that I could replace it. In this example, I wanted to find the last pipe in a record like 8|34346|. Normally with regex, one could do this using a lookbehind assertion. To do this with notepad++ you must utilize the implicit search buffer, by using a \1 in the replace field (for example, to access the first set in the buffer).

So by searching on: 1|10007|someemail@someplace.com|lastname,first name|X

"Find What:" ([0-9]*[|][0-9]*)[|]
"Replace With:" \1%

I could get: 1|10007%someemail@someplace.com|lastname,first name|X

BIG THANKS TO DEAN NAIRN!

Lay Interpretation of Cancer Incidence

2011-08-03T06:42:00.000-07:00

Cancer is a big deal, and the potential environmental factors make mapping incidence very appealing. This has made maps a handy device for newspapers and other "lay" interpretations, which typically don't employee rigorous statistical tests for significance. Victor Perez, a researcher at UD wants to investigate the ways cancer incidence is discussed in these lay publications. As a discursive preliminary step in this study, Dr. Perez asked me to help him prepare a map that would show what a random incidence of cancer cases might look like. To do this I used the "generate random points" tool using data on census tracts (Total Population from Census 2000) and given average Delaware cancer incidence.

SDE Errors

2011-07-27T10:49:00.000-07:00

"ERROR: ESRI: error getting spatial references for srid = "

... uses public.sde_spatial_references instead of sde.sde_coordinate_systems, so cannot use standard srid ... only have a selection of srid's of loaded features ... therefore use srid defined in public.sde_spatial_references

"DBMS table not found xxx State ID = 9"

... was getting this error after the SDE had become corrupted b/c client was using PG Admin instead of ArcGIS Desktop to do schema changing operations (deleting tables). Clearly these actions should be done with Desktop in the future (Catalog). To fix this I deleted the SDE and reconfigured with the SDE Post-Installer. This may also be fixable using

sdetable -o unregister

or some switch combo on sdegdbrepair

ArcSDE/Postgres Error 000210, Could not Copy Features

2011-06-28T14:20:00.000-07:00

It seems like there is an out of the box error in ArcSDE which causes geodatabase operations like Import Feature to fail out of the box (giving a 000210 error, Could not Copy Feature). In my case, it was because my db user name "postgres", which is usually a default, was not included as a schema in the deafault database. This might be solved by calling the user "sde", but in my case I solved it by simply creating a new schema in the default database beside the existing sde schema.

postgres max integer in an alphanumeric column

2011-06-14T12:41:00.001-07:00

select max(to_number(substring(COLUMNNAME from '([0-9].*)'),'99999')) from TABLENAME

removes letters from a mixed column ... can be used for generating a new id

Intersect ArcGIS Bug

2011-05-10T16:49:00.000-07:00

There is a bug in ArcGIS with the intersect tool (I noticed it when intesecting lines as polylines) that causes the result to be an empty table. To fix this I exported all tables being intersected to a new location (and the output to point there as well). This resulted in the expected output.

Flipped Polar Coordinate System

2011-04-28T08:11:00.000-07:00

Problem: NIC/NSIDC Antarctica data was appearing flipped 180 dg, without coordinate system or projection defined. When other polar data, thought to be in the same coordinate system/projection was placed with this data, the other data appears to be flipped 180 in the right orientation (though the data in question is still flipped).

Solution: Modify the projection of the data in question to have a central meridian of 180 instead of 0 (with "define projection" using a custom projection based the other data). THE DATA VIEW PROJECTION MUST ALSO BE SET TO THE PROJECTION OF THE "GOOD" DATA (e.g. 0 Meridian, Polar Stereographic)

On Simple Structure and Factor Analysis

2011-03-26T09:36:00.001-07:00

"I had become increasingly unhappy with L. L. Thurstone's principle of so-called simple structure and his widely accepted idea that variables are best explained if based on a minimum number of "factors", on only one "factor" if possible. As far as I can see, varieties in nature are nowhere put on single poles, phenomena generally arise from relations of underlying building blocks, from interaction of components and complex combinations. On all levels of inanimate and animate organization wholes emerge from interweaving parts. I could not understand why factor analysts were doggedly attempting to construe a world of utterly independent "dimensions" and associated constructs in splendid isolation." (David M. Glover)

notepad ++ special characters as replace string

2011-03-07T08:09:00.000-08:00

While copy over a newline special character will show up in the "find" field of the "find & replace" dialog if one of these is present in a highlight, these cannot be copied/pasted into the "replace" field notepad ++ allows an extended set of special characters to be used both in the find AND replace fields ... so for example if you replace ";" with ";\r\n", with the "extended" option checked on, you will get a new line after every ";"

ArcGIS Editor for OpenStreetMap

2011-02-14T18:48:00.000-08:00

I recently had a chance to test out ArcGIS Editor for OpenStreetMap (1.1 Beta3), an open source toolbox for interacting with OpenStreetMap (OSM). OSM, a popular open-source data repository, gained extra traction in the US after the Haitian Earthquake. This growth came from NGOs, and from some ArcGIS shops with open source leanings, who advocated on behalf of an extension to OSM from ArcGIS. I've been interacting heavily with OSM in building and maintaining the University of Delaware Map (UD Map). The data schema for UD Map is based on the OpenStreetMap schema, and interfaces with OpenStreetMap for data updates and publishing. As the project demanded it, and as a test of the new tools, I used the toolkit to publish some pending campus base layers. Alas, this trial ended disappointingly, leaving only manual editing of the vector layer. Append, which could have been used to join the vector input to the OSM polygon layer, did not work as expected: The OSM changeset was not updated, and therefore could not be uploaded to OSM for processing. The tools still look very useful for doing manual editing of OSM data and adding OSM data as fully editable vector layers in a map ... and of course, as code to be expanded upon ... this story is to be continued :-)

jquery accordion breaks out of containers on ie7 (ie6?)

2011-02-03T13:19:00.000-08:00

fixed this issue by giving css position:relative on parent container further up the tree (in this case was the tabs parent)

Google Spreadsheet Data API

2011-02-03T12:20:00.000-08:00

I gave a presentation to the U. of Delaware WebDev group and IT-CS&S (RDMS) on the topic of using data from Google Docs Spreadsheets for consumption in a scripted workflow. I'm using this in the UD Map for automatically incorporate rendering decisions from OCM.

The gist is that you open an https stream with headers for
authorizing, capture an authorization token on the response, open
another stream returning an xml of the table, and parse that. In
addition, you could form the url for the second stream with parameters
that query the column of interest, for example.

Here is the presentation:

https://docs.google.com/present/view?id=dx7rb3j_104dxjkq8c6

Here is some sample code (excuse references to my database/tables):



function do_post_request($url, $data, $optional_headers = null)
{
 echo "Begin do_post_request \n";
 $params = array('http' => array(
    'method' => 'POST',
    'content' => $data
  ));
 if ($optional_headers !== null) {
$params['http']['header'] = $optional_headers;
 }
 $ctx = stream_context_create($params);
 $fp = @fopen($url, 'rb', false, $ctx);
 if (!$fp) {
throw new Exception("Problem with $url, $php_errormsg");
 }
 $response = @stream_get_contents($fp);
 if ($response === false) {
throw new Exception("Problem reading data from $url, $php_errormsg");
 }
 return $response;
}

function doFromGoogle(){
echo "Begin doFromGoogle \n";
define("GURL", "https://www.google.com/accounts/ClientLogin");
define("SPREADSHEETREFKEY", "***YOURKEYHERE***");

//creates post to google auth, creates $auth string from google
$dataArr = array();
$dataArr['accountType'] = 'GOOGLE';
$dataArr['Email'] = '***YOUREMAILHERE***';
$dataArr['Passwd'] = '***YOURPASSWORDHERE***';
$dataArr['service'] = 'wise'; //name for spreadsheet data service
$dataArr['source'] = '***YOURSORUCENAMEHERE***'; //this may be optional

$dataStr = http_build_query($dataArr);

$opt_headers = 'application/x-www-form-urlencoded';

$responseStr = do_post_request(GURL, $dataStr, $opt_headers);
$responseArr = explode("\n",$responseStr); //breaking apart request to get auth token
$auth = explode("=", $responseArr[2]);
$auth = $auth[1];

$authStr = 'Authorization: GoogleLogin auth=' . $auth;
$url = 'https://spreadsheets.google.com/feeds/list/' . SPREADSHEETREFKEY . '/1/private/full';


$opts = array(
 'http'=>array(
'method'=>"GET",
'header'=>"$authStr"
 )
);

$context = stream_context_create($opts);

// ini_set('Authorization', "$authStr");

$handle = fopen($url, "r", false, $context);
$contents = stream_get_contents($handle);
fclose($handle);

$xml = new SimpleXMLElement($contents);

$fullquery = null;


$count = 0;
foreach ($xml->entry as $entry){
$count = $count + 1;
$id = $entry->children('http://schemas.google.com/spreadsheets/2006/extended')->osmid;
$render = $entry->children('http://schemas.google.com/spreadsheets/2006/extended')->dorender;
$label = $entry->children('http://schemas.google.com/spreadsheets/2006/extended')->dolabel;
$labelname = $entry->children('http://schemas.google.com/spreadsheets/2006/extended')->labelname;
$maxzoom = $entry->children('http://schemas.google.com/spreadsheets/2006/extended')->maxzoom;
$minzoom = $entry->children('http://schemas.google.com/spreadsheets/2006/extended')->minzoom;
$labelname = addslashes ($labelname);

//$color = $entry->children('http://schemas.google.com/spreadsheets/2006/extended')->color;

//print "$id, $count \n $render \n $label \n $labelname";
print "$id  $maxzoom $minzoom\n";

$query = null;

if($render <> ''){
 $query .= "UPDATE master SET dorender = '" . $render . "' where osm_id = $id and dorender <> '" . $render ."';\n";
}
if($label <> ''){
 $query .= "UPDATE master SET dolabel = '" . $label . "'where osm_id = $id and dolabel <> '" . $label ."';\n";
}
if($labelname <> ''){
 $query .= "UPDATE master SET labelname = '" . $labelname . "'where osm_id = $id and labelname <> '" . $labelname ."';\n\n";
}
if($maxzoom <> ''){
 $query .= "UPDATE master SET maxzoom = '" . $maxzoom . "'where osm_id = $id;\n\n";
}
if($minzoom <> ''){
 $query .= "UPDATE master SET minzoom = '" . $minzoom . "'where osm_id = $id;\n\n";
}

$fullquery .= $query;
}

//print($query);
$result = pg_query($fullquery) or die('Query failed: ' . pg_last_error());
}

Replacing Tomcat .jsp pages with Apache httpd .php pages

2011-01-13T13:59:00.000-08:00

It turned out that this task was much easier than I'd expected. Here are the steps I took:

Edit the httpd.conf file (default path: C:\Apache2.2\conf\httpd.conf). Remove tomcat references to the services (JkMount). Add Redirect 301 lines to .php proxy

Create php proxy which is referenced by Redirect 301. Arguments are passed by the redirect, which can be handled by this proxy. Here is the code I used for a single argument, but could easily be extended to other arguments. This also saves arguments that are not handled to a log file:

$out = array();
$log = '';

foreach($_GET as $key => $value){
   if (strpos($key, 'max')){};
   switch ($key) {
       case bldgcode:
           $out['id'] = $value;
           break;
       default:
           $log .= 'key is ' . $key . ' and value is ' . $value . "\n";
           break;
   }

}

$fp = fopen('log', 'a');
fwrite($fp, $log);
fclose($fp);

$url = 'http://maps.rdms.udel.edu/map/index.php?' . http_build_query($out);

header("Location: $url");

WHERE NOT IN and NULL in Postgresql

2010-12-23T13:00:00.000-08:00

NULL values trip up the WHERE NOT IN condition in Postgres. To fix this behavior, use a conditional (not null) in the subquery.

source: http://www.depesz.com/index.php/2008/08/13/nulls-vs-not-in/

Duplicate OSM Features in UD Campus Map

2010-11-10T09:04:00.000-08:00

I noticed a couple places where we have two polygons on top of each other (particularly for parking polygons) in the UD Campus Map. I just wanted to note, if you're trying to modify an existing polygon on the map, make sure that you're changing the existing shape, rather than adding a new one (otherwise we're loosing attributes on the exisiting shape).

Here are two screenshots to demonstrate what I'm talking about (note that only the second polygon retains attribute information/properties as listed in the right-hand sidebar), though it appears the first polygon was intended to replace the second.

I've fixed this problem by adding nodes to the second polygon (the original one that has attributes/properties) and dragging them to match the footprint of the first (new, intended to replace the first), and deleting that one.

Zoom to extent of new vector layer in OL

2010-11-09T08:35:00.000-08:00

Zooming to the maximum extent of all features in a vector layer does not work unless all features have been loaded. Therefore it is necessary to register an event handler, as demonstrated below ('layer' is any object of class OpenLayers.Layer.Vector):

layer.events.register('featuresadded', map,function(){this.zoomToExtent(layer.getDataExtent())});

Random Feature Selection

2010-10-08T11:10:00.000-07:00

I met with a gentleman today who needed to select random streets in Wilmington in order to do a field survey of urban forestry there. Here are the steps we took to do so:

-Use Excel and the RANDBETWEEN function to generate random numbers in a range = total number of features (in this case, streets in Wilmington). Drag this equation down a column = number of features in sample. Once this has been done, save/export (I usually like to use DBF for ArcGIS)

RANDBETWEEN(bottom,top)
Bottom is the smallest integer RANDBETWEEN will return.
Top is the largest integer RANDBETWEEN will return.

- with data added to ArcGIS, use "Frequency" tool to check that no duplicate values were generated

- do a regular one to one join, dropping out all values that don't match. The result will be a random set of features -- in this case, street segments.

View KML Source from MyMaps

2010-10-07T07:09:00.000-07:00

Google MyMaps allows you to export KML by using the "Export to Google Earth" feature. If you copy/paste the URL that this button links to you get a KML, but it is a network reference to the actual KML. If you copy/paste the URL referenced in this KML in your browser, you'll get a warning "{errorText:"Unable to contact server."}". To get past this warning, you must decode the encoded URL reference. That means getting rid of all the "amp;" strings.

For example, change:

http://maps.google.com/maps/ms?ie=UTF8&hl=en&vps=1&jsv=250a&oe=UTF8 &msa=0 &msid=113097578162956796020.00048fc123b72558b50ff &output=kml

http://maps.google.com/maps/ms?ie=UTF8&;hl=en&vps=1&jsv=250a&oe=UTF8&msa=0&msid=113097578162956796020.00048fc123b72558b50ff&output=kml

and you'll get the actual kml. You do not need to do this within any script or program, since those tend to encode URLs anyway. However, it is very useful for debugging issues with KML.

Geoserver Labeling, Font Changes

2010-10-01T07:37:00.000-07:00

Wierd Geoserver Behavior (bug): font changed from Arial to some kind of serif font for no apparent reason. I fixed this by clearing all caches and restarting the Geoserver service. I'm still not sure what caused the issue.

Animal Movement and Home Range Options for ArcGIS

2010-10-01T06:09:00.000-07:00

The Animal Movement Extension was written in Avenue, and development stopped many years ago. Therefore, Animal Movement Extension is not available for ArcGIS 9.x or 10.

There are a couple of options if you'd like to move forward with ArcGIS

A lot of people using Animal Movement extension migrated to Hawths Tools, to be able to use ArcGIS. Hawths tools offered many of the functions in Animal Movement, though Hawths tools is also now a discontinued project, and will not work with newer versions of ArcGIS (9.2 was the last version it worked with).
Hawths tools has been been merged into the Geospatial Modeling Environment, which works with ArcGIS 10. You can download this and find more info at: http://www.spatialecology.com/gme/
Home Range Tools for ArcGIS works on ArcGIS 9.x, and is available at http://nhsbig.inhs.uiuc.edu/wes/home_range.html
There is a lot of work going on with Agent Based modeling right now. I met with an agent based modeling working group at the ESRI conference. They've developed an extension for ArcGIS based on repast called Agent Analyst: http://www.institute.redlands.edu/agentanalyst/ . I didn't hear about any scenarios involving wildlife movement, buy there is a book coming out, and I would assume those scenarios might be discussed. I'll keep my ear to the ground on this one
There are also contributed and native tools and scripts that could be used to perform many of these functions if strung together in the right order

Other Resources

Geoserver SLD Polygon Labels Repeat

2010-09-29T07:17:00.001-07:00

Fix: had specified larger tile size for earlier debugging. Switch tilesize back to 256 to fix label repeat on polygons.

FIX! Geoserver not displaying labels

2010-09-28T12:54:00.000-07:00

I've been having an extremely frustrating problem over the last month ... in one word: labeling. My latest labeling challenge, was that Geoserver refused to show labels for many buildings. I tried all kinds of things ... isolating the buildings layer, checking my query order, changing conflicts/overlap options ... pretty much anything I could think of to get labels to display.

Finally, I figured it out! There is an obscured parameter in older ('stable') versions of Geoserver, which was finally exposed through SLD vendorOption in the latest release of Geoserver/Geotools. This parameter MIN_GOODNESS_FIT is used when labeling polygons in particular. Based on MIN_GOODNESS_FIT geotools determines if From geoserver docs:

Geoserver will remove labels if they are a particularly bad fit for the geometry they are labeling.

The label is sampled approximately at every letter. The distance from these points to the polygon is determined and each sample votes based on how close it is to the polygon

so labels are removed if letters are beyond a certain threshold from the polygon. That meant that small buildings (especially ones with longish names) would never be labeled.

However, in Geoserver 2.1 the SLD VendorOption "goodnessOfFit" is exposed

Syntax follows this example:

0.3

the default value is 0.5, so anything lower will allow more lables to be placed.

For more, http://docs.geoserver.org/latest/en/user/styling/sld-reference/labeling.html

Issue discussion:

Geoserver WebTileCache

2010-09-16T12:23:00.001-07:00

I was ultimately not able to deal with cutoff labels in Mapnik (or labels not being placed because of tile/overlap issues), so I went with Geoserver for the UD campus map project. The WebTileCache automatically tilecaches all services ... only challenge is to use SLD for styling.

Mapnik label cut off strategies

2010-08-27T08:30:00.000-07:00

I've been spending a lot of time dealing with mapnik labels/shields being cut off. Here are some strategies:

change the "buffer_size" property of the map object in the mapnik xml ... you can also direclty change this in generate_tiles.py (which comes with osm mapnik package). I don't think the mapnik xml was being read in my case. When I changed the line self.m.buffer_size to = 0, my cutoff problems disappear
you'll need to do some tweaking with the order layers are defined within mapnik/cascadenik
also tweaking of text-allow-overlap and text-avoid-edges (these are cascadenik properties, there are similarly named ones for mapnik alone)
you may need to merge segments of the same road

To sum up: my strategy was first getting all my ordering straightened out with how layers are defined/ordered within mapnik/cascadenik. I then turned most layers to text-avoid-edges and text-allow-overlap to true. I set the buffer value to 0 in the .py file that I was using to render my tiles (in the osm package). Then moving around stuff that overlapped badly and will possibly need to merge segments of the same road.

ImportError: No module named mapnik

2010-08-10T05:22:00.000-07:00

I'm still not sure what exactly causes this error: "ImportError: No module named mapnik" .. clearly Python is not finding the mapnik bindings, but why?  I think it might be related to a new install of ArcGIS, which installs a new version of Python.  I am also trying to figure out how exactly to fix this.  Recently I relaunched "cmd" and that worked.

python zipfile.py MemoryError

2010-07-29T05:50:00.001-07:00

Was getting the following error on zipfile.py (actually while attempting to run a cascadenik script):

newdata = self.dc.decompress(newdata)

MemoryError

I fixed this error by changing the compression algorithm that my .zip was being created with to "STORE" (i.e. no compression).

python cgi with xxamp/apache on windows

2010-04-21T11:23:00.000-07:00

I recently figured out that a web service class I was dealing with (WFS in OpenLayers) would only accept requests through a proxy script (despite the requests were sent to my local machine!). OpenLayers has a .cgi proxy script which runs through Python, and here's how I got it to work:

1. Download proxy.cgi to the cgi-bin folder within xampp (or wherever it is), and install Python. I used version 2.5
2. Follow these instructions to setup Apache for running python
3. Finally, make sure you change the "shebang" line at the top of proxy.cgi from #!/usr/bin/env python to #!c:/python25/python (if python.exe is installed at C:\Python25\python.exe)

Tilecaching MXD/ArcGIS Server to file/folder-based only (e.g. google, mapnik)

2010-03-15T05:58:00.001-07:00

So it seems like there's no automated way to go from mxd to file/folder based (rather than web service) tile cache. The best I could do was use maptiler/gdal2tiles (maptiler is a very nice gui for gdal2tiles) to cache raster layers. This will fuse multiple raster layers, but doesn't handle vectors. If you want to pass vectors through here you can always convert to raster with some other tool but you'll end up loosing any scale dependencies. Please note, you'll likely have to incorporate a part of their auto-generated code for google maps, since the file naming doesn't completely conform to x/y/z google scheme naming. I've included my modified tile provider class for the gmap_overlays submodule (part of the gmap addons submod collection ... lives in PATH_TO_DRUPAL\sites\all\modules\gmap_addons\gmap_overlays\js), which handles output from maptiler/gdal2tiles.

note: exposing as wms and using tilecache might be a good idea, but this non-automated solution that is OK for me right now

AGS error: #000569: Failed to generate cache. All server contexts failed to cache map.

2010-03-11T13:06:00.000-08:00

I was getting the following error when trying to cache a map service in ArcGIS Server/ArcCatalog: 000569: Failed to generate cache. All server contexts failed to cache map.

I ultimately resolved this issue by upgrading my server software (ArcGIS Desktop) to the same version as my client (in my case 9.3.1 service pack 1). Caching only within a feature, which in my case was a three county polygon that was much larger than the area I wanted to cache, greatly improved performance ... I'm not sure if it even would have worked without that.

Note: if postinstallation hangs while upgrading server software check out this post: http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=32766

Drupal SWFTools Error

2010-03-04T11:02:00.001-08:00

Was getting the following error: "You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialize correctly." on Chrome and I.E. only. Figured out that by changing the embed method under Embedding settings (Home>Administer>Site configuration>SWF Tools>Embedding settings) back to "Direct embedding - do not use JavaScript replacement" from " SWFObject 2 - JavaScript" I no longer get this error

Tile basemap through Views/GMaps module (Drupal)

2010-03-04T06:48:00.001-08:00

To add a tiled basemap to a gmap created with views (and the gmaps) module:

1. First make sure that you have a gmap set up through views. This is a good tutorial: http://groups.drupal.org/node/19614

2. Next download the GMap Addons module, if you haven't already done so

3. Create a map tile structure using something like tilecache, mapcruncher, or ArcGIS (you can generate a map with the google maps scheme, just not sure where to get an xml that would adequately describe the scheme to ArcGIS)

4. Finally add a tile overlay macro within the style (GMap) configuration for this view. Use the following form:

[gmap |overlay=tile:http://host/tilesdirectory/{Z}/{X}/{Y}.png]

... notice you don't need to substitute anything for Z, X, or Y ... these correspond to folder numbering as generated with tilecache (or other tiling program) using the google maps scheme.

If you save the view you should now be able to see your basemap in a page generated by that view.

Migrating ArcIMS to a new machine

2010-02-04T11:36:00.001-08:00

1. Install Apache (2.2.x, to c:\Apache2.2\, in my case) and Tomcat (6.0.x, to c:\Tomcat6.0\, in my case) make sure each is able to launch independently. You might need to install Java JDK and set environment variables JAVA_HOME and JRE_HOME to the JDK directory
2. Download mod_jk (1.2.28 httpd-2.2.3, in my case). Install by dragging mod_jk-xxx.so to the Apache modules folder and renaming mod_jk.so. Add this line to httpd.conf: LoadModule jk_module modules/mod_jk.so
3. Add a workers.properties file to the apache conf directory. This file sets options for your ajp (probably ajp13). Make sure it points to your JDK, Tomcat home directory, etc.
4. Install ArcIMS with the 'custom' option during post-install. Make sure that you select the JDK you already installed.
5. Copy over old aimsacl.xml from webapps\servlet\WEB-INF\classes. Give * permissions to all your apps for testing, you can change later if necessary
6. Copy over old old apps from the Tomcat webapps directory. Make sure WEB-INF\classes\Authenticate.properties is pointing to the new aimsacl.xml
7. Use AIMS administrator to set up map service, using similiar settings as old service. Note: on my install 'Output', 'Website', and 'Axl' are all under the Apache htdocs directory
8. Add all applicable webapps directories to httpd.conf as in the following (substituting 'example' for your own app):
JkMount /example ajp13
JkMount /example/* ajp13
9. Change any connection properties in your webapps to match new machine
10. Restart Apache, Tomcat, enjoy.

json_encode pg_fetch_array

2010-01-25T06:08:00.000-08:00

For some reason you need to run the result of pg_fetch_array through a loop and store in another array to be able to encode as JSON ... here's an example:

// Connecting, selecting database
$dbconn = pg_connect("host=HOSTNAME dbname=DBNAME user=DBUSER password=PASSWORD")
or die('Could not connect: ' . pg_last_error());

// Performing SQL query
$query = "SELECT * FROM mytable";
$result = pg_query($query) or die('Query failed: ' . pg_last_error());

$resultArray = array();
while ($row = pg_fetch_array($result, null, PGSQL_ASSOC)) {
$resultArray[] = $row;
}

echo json_encode($resultArray);

// Free resultset
pg_free_result($result);

// Closing connection
pg_close($dbconn);

Mapnik ... success!

2010-01-22T07:06:00.000-08:00

Here's how I imported, rendered, and served OSM data through mapnik, locally:

1. Do all downloads/installs ... python (2.5 worked for me), postgres/postgis (8.3 worked), osm2pgsql, mapnik, rendering/mapnik utilities for OSM
2. Follow info below on osm2pgsql
3. Make sure mapnik bindings are registered with Python, run the demo under demo\python to make sure mapnik works, here are pretty good instructions from the OSM wiki
4. Follow this readme to run generate_xml.py and generate_tiles.py. Use my tips below if you get stuck ... also make sure you keep your coordinates straight ... these scripts won't throw an error, they'll just crash
5. Once you've generated tiles stick them in a web accessible location and use this tutorial to get them working with OpenLayers ... note the "bespoke" section, which shows how to add your custom tiles

... now to customize the rendering I only need to modify the map.xml file created by generate_xml.py ... I could do this manually or might look into the qgis plugin ... also cascadenik looks cool

OSM2PGSQL

2010-01-21T08:32:00.000-08:00

After getting mapnik installed and running I realized:

1. generate_tiles/generate_xml (and the rest of the render utilities) are really necessaryfor rendering OSM through mapnik. Therefore, you must download both mapnik through the mapnik site but also the mapnik OSM rendering tools which you can get through SVN at http://svn.openstreetmap.org/applications/rendering/mapnik/
2. you must download world coastlines to get the osm/mapnik utilities to run as detailed here (if you're getting an error about 'world_boundaries', that's why)
3. you must use OSM2PGSQL to import your osm data into postgres ... you can do this through osmosis, but it won't work

Helpful hints with OSM2PGSQL

1. you must run the included .sql in order to create the proper spatial reference information for the projection mapnik likes
2. you must create the database with a postgis template
3. you might need to export through JSOM and encode as UTM8 ... I did this as a safeguard this time and it worked ... if you're having trouble, try this

Related errors:
failed: ERROR: AddGeometryColumns() - invalid SRID
CONTEXT: SQL statement "SELECT AddGeometryColumn('','', $1 , $2 , $3 , $4 , $5 )"
PL/pgSQL function "addgeometrycolumn" line 4 at SQL statement

relation "planet_osm_polygon" does not exist

Loading OSM with OSMOSIS

2010-01-20T05:59:00.000-08:00

Now that I've decided to locally render OSM data for the campus, I need to get the OSM data for the maximum extent of campus areas into a local database (both Mapnik and I prefer Postgres). I initially tried doing this by using OSM2PGSQL, but that ultimately failed ... not completely sure why.

Here are the steps I took to load this data into Postgres using OSMOSIS:

Required installations: Postgres with Postgis, OSMOSIS ... tested environment: osmosis 0.32, postgres 8.3, Windows XP

1. Download desired extent from OSM using JOSM ... save the downloaded data as *.OSM
2. Convert this .osm file to UTM8 encoding (e.g. using notepad++)
3. Login to postgres, create a new database based on postgis template
4. Run pgsql_simple_schema_0.6.sql, which is located in the osmosis install location in the 'script' directory
5. Open a command line and run: PATH_TO_OSMOSIS_BATCH_FILE --read-xml file="PATH_TO_OSM_FILE" --write-pgsql host="HOST_NAME" database="DB_NAME" user="USER_NAME" password="PASSWORD", for example: C:\qgis_projects\render\osmosis-latest-bin\osmosis-0.32\bin\osmosis.bat --read-xml file="C:\qgis_projects\render\newark_josm_utm8.osm" --write-pgsql host="localhost" database="gis" user="postgres" password="mypassword"

Plugging away with OSM

2010-01-12T13:12:00.000-08:00

Update: Success! Was able to convert to OSM with Export_to_OSM.py, all was needed was to change some paths in the code. Uploads via JOSM using the Open, and Merge commands (I recommend also adding some tag so you can query these out later if you need to do anything with them), then Upload. Watch as I add more layers on www.openstreetmap.org!

Old Post: It now seems I'll be able to use OpenStreetMaps either as the repository and rendering system for the full campus map, or as a selective baselayer for national data (for directions). The answer will depend, in a large part, on whether I am able to get campus layers into OSM.

I have worked with multiple OSM editors: JOSM, markaarter, and the QGIS OSM plugin and none will successfully take a shapefile, allow import into OSM format, and then upload to the OSM servers. I have tried conversion to KML and GPX, since these are formats accepted by one or the other of these editors ... still no dice.

There are scripts and programs which were specifically created to convert SHP (and all kinds of other formats) to OSM. I've checked out some of these programs osm2shp, shp-to-osm.jar even gml2osm ... problem is that they assume you want to map all the shapefile attributes ... seems like it could be a lot of extra work to figure these out for every layer I want to upload.

Installing QGIS with osgeo4w, express crashes

2010-01-04T12:17:00.001-08:00

Installing QGIS with a standalone installer is definitely the way to go for someone with little time, no need for other osgeo4w libraries or features in newer ('unstable') releases.

However, as I need other osgeo4w packages as well as features in the 'unstable' release (unstable is somewhat of a misnomer, as I understand, really is just shorthand for saying it has features that aren't fully supported as in the 1.x version), I wanted to install with the osgeo4w package. For some reason osgeo4w crashes when I try to use the express install option, though I needed to use the advanced install anyway in order to get unstable qgis.

Steps for this are as follows

download osgeo4w from http://trac.osgeo.org/osgeo4w/
open installer, select advanced install
select the following packages to install: qgis (unstable if you want, which was 1.4 at the time I wrote this), qt4-libs, libpq, zlib, geos, gdal16, ogdi, expat, proj (you will need to rename, read this), xerces-c, hdf4, hdf5, libjpeg,libgeotiff,libpng,libtiff, netcdf,libjpeg12, sqllite3, tcltk, zlib1 (install zlib and make a duplicate renamed to zlib1.dll), curl, iconv, sip, pyqt4,

Basemap, prority renderers/services based on extent?

2010-01-04T11:11:00.001-08:00

A new idea: use custom service/renderer within a given extent and cloud service outside of that extent ... tag campus layers accordingly, so they're picked up by custom renderer

Directions, directions

2010-01-04T05:50:00.000-08:00

So I am returning in the new year with a major task at hand: finish work on UD campus maps. As I recall, the major loose end from where I left off was the matter of the basemap -- which resurrected old considerations about maintaining all layers on a cloud site (i.e. openstreetmaps) and send all operations to be dealt with on a local spatial database. During this diversion from the previous course I need to also keep in mind: the basemap issue only comes up because I want to offer national directions to campus in the same interface as the rest of the campus map ... otherwise I could just use local layers without need for larger extent basemap layers.

The OSM scenario is very appealing for a number of reasons:
-cloud hosting would free up local resources here
-the OSM platform is well designed according to collaborative logic, that would allow updates by data stewards on campus, if we were to move in that direction
-OSM is open source, it's free to us, and is very interoperable with other open software ... that would help in efforts to support open source on campus
-possibly better performance on the cloud
-use of existing renderers allows us to leverage existing cartographic/labeling rules

Disadvantages
-layers could be edited by a third party with poor editing skills, inaccurate data, or malicious intent
-limited control over appearance of the map
-strange issues with spatial referencing ... OSM and existing UD layers do not agree ... these are also off with navteq streets, so problem is independent of OSM ... should note that the issue is probably not significant without sidewalks, since these tend to be more closely located to streets than buildings to streets
-still unresolved data sensitivity issues (we need to contact all data "owners")
-tendency to neglect local resources ... no redundancy

These disadvantages could mostly be overcome, but would probably require quite a bit of work, with some of the advantages becoming mooted
-OSM data could be stored locally, taking advantage of some tag which signifies campus data ... that is what UMd does. However, that is probably impractical for the national dataset (UMd only has part of the DC region), and would moot any advantages to using the cloud ... would however help with redundancy
-A custom renderer could be used to achieve more control over appearance ... this may be unwieldy to setup ... could try to tweak tags, etc. in OSM data as a work around ... could Mapnik default rendering be adequate?
-spatial referencing issues are independent of OSM

Table to table bug (ArcGIS)?

2009-12-21T15:06:00.000-08:00

Is there a bug with the "table to table" tool in ArcGIS when using it to export a DBF from an Excel file (XLS, XLSX)? I don't know. BUT, I would avoid using any Excel files in ArcGIS. This would be one of many poorly documented bugs ... support for Excel in ArcGIS is a weak link, and you really don't want to be basing multistep analysis off some poorly imported data (trust me!). So you have an XLSX file that is more than the limit of rows for XLS ... how do you get it to DBF (since Excel 2007 doesn't export DBF). If you have MS Access, I would recommend going that route. I tried this out with a file that I was getting super incorrect output on, and it worked. Seems like a nice, clean, consistent solution.

Cloud basemaps, so elusive

2009-12-21T14:28:00.000-08:00

I wanted to use openstreetmaps as the base data layer for the new UD Campus Maps, but am running into the same labeling issues as when I use a google maps base layer ... labels are prerendered to appear on streets, cities, etc. of course not taking into account placement of "UD" layers (e.g. buildings). When the two are displayed together there is a very unsightly overlapping effect.

So the way around this is to use a custom renderer, which is actually quite a bit of work ... the custom renderer requires a large amount of space, not to mention an initial span of CPU time. It also requires that I create custom style specs, which address all sorts of issues, including a plethora of OSM tags, as well as all the details about base layer appearence ... all to be done manually of course.

... not to mention, the existing baselayers seem to be disagree with cloud services ever so slightly

CloudMade offers a very cool service for rendering custom basemaps from OSM data, all through an efficient GUI, and they even host it for you. Trouble is their terms do not seem to permit "non-personal" use, :.(

Sooo, I can:

set up a custom render, which *might* require a lot of space/time, and *might* require a lot of work in symbolization, and will consume local resources instead of leveraging the cloud
*or* use ArcGIS server to tilecache a national basemap ... having much the same disadvantages as above, except that I have a better idea of how much time/resources it will take
*or* render on top of satellite data (which doesn't really look that good, and not very informative for directions), plus requiring me to create custom label and symbol scaling and symbology for Newark basemap details (tending to be less attractive than those already available in the cloud, and causing disuniformity with surrounding areas)
*or* I can mask the extent of all "UD layers" and use this as the default extent for the map, putting in some scale sensitivity which causes UD detailed layers to disappear when viewed at a larger "regional" scale and instead showing some extent rectangle ... then showing these layers again when at a closer zoom (having some of the same drawbacks of disuniformity/attractiveness)

... as of today the last option seems the best ... I sure hope it bares fruit, we need to get this project rolling!

unexpected rows in tables

2009-12-16T05:40:00.000-08:00

Occasionally, the number of rows in a table* is not really the number of rows in the table. Sometimes this peculiarity only makes itself known when you export a table and only see a subset of the total rows in the original attribute table ... where all the other rows go to?

One reason that this happens (perhaps the most common), is that when you use the "keep only matches" option when doing a join, the non-matched rows are kinda there, and kinda not. To fix this you would need to go back and use the "keep non-matched" option.

*as counted by the "Records (1 out of x)" at the bottom of an attribute table window

license server down or not responding -96,7

2009-10-01T07:05:00.000-07:00

was getting an error like "license server down or not responding -96,7" when trying to get ArcGIS to run on a Vista machine. Seems like the problem was that since the host name could not be set (I could not browse to the license server and therefore it could not be set through the ArcGIS Administrator dialog) the host's license file wasn't validating (although the machine I was setting up could ping the host). I got around this by setting all registry entries = "Not_Set" (actually were "@Not_Set") to the name/port of my license server

area cropped when skinning flv to swf

2009-08-27T13:23:00.000-07:00

try using the noscale option and enlarging the dimensions exported ... you can use negative margins on the html output to make it look decent

notepad++ insert carriage return

2009-08-17T11:58:00.000-07:00

In notepad++ you can easily find a carriage return by copying a section of texxt with a carriage return in it and doing ctrl+f or ctrl+h to replace, but you cannot copy that block and paste into the replace box. I found the answer for this here: http://bytes.com/topic/net/answers/767637-find-replace-insert-return-character

In the replace area, type

\1\r\n\2 -- \1 is the first group, \r\n is a carriage return and a new
line, \2 is the second group.

connecting to sde

2009-07-17T10:16:00.000-07:00

Connecting to my new SDE was exceedingly difficult. I'm not sure if it's because it was built on postgres, but ... geez!

Here are a few items you should be aware of

contrary to ESRI's documentation, you need to do a very important piece of configuration on the client, namely add a reference to the service in the services file which is found (for most Windows users) at C:\WINDOWS\system32\drivers\etc\services. At the bottom add the following line (replacing servicename with the name of your sde service): service name serviceportnumber/tcp #ArcSDE for RDBMS
make sure your server is listed in the pg_hba.conf file (in your postgres data folder)
this entry must also be in the same file on the server, and also in C:\Program Files\ArcGIS\ArcSDE\pgexe\etc\services.sde, but that's usually less of a problem since the post-install takes care of it
if you add your port number in the server field in the "connection properties" from your client (in my case arccatalog), it will return an error
make sure your desktop software is the same major version as the sde (example: 9.3 to 9.3)

references:

http://forums.esri.com/Thread.asp?c=2&f=1717&t=142803

hint: was getting the following error: "Failed to connect to the
specified server. Entry for SDE instance not found in service file" ... note 1 above, particularly on the client side, fixed that

setting percent property as3

2009-07-15T07:56:00.000-07:00

when setting percent height property in AS3 was getting the following erro"1084: Syntax error: expecting identifier before semicolon" ... was because I need to use percentHeight, percentWidth, etc. for percentage properties ... I believe this is similar to HTML DOM

Importing ESRI GRID to ENVI

2009-06-23T05:59:00.000-07:00

I'm not sure if ENVI documentation is lousy or just not searchable, but I could not find any good information on how to import ESRI GRID to ENVI ... and the ENVI UI is so unintuitive (hint to ITTVIS, why not just streamline the whole "external file" thing into the same "open image" command used to import geotiffs and the like?) as to make a simple task need explaining!

THIS FUNCTIONALITY IS ONLY AVAILABLE IN THE WIN 32 VERSION!

1. File > Open External File > IP Formats > ESRI GRID
2. Navigate to the hdr.adf within the folder with the same name as the vector you are trying to open and open this

"select * into" in mysql

2009-06-22T05:32:00.000-07:00

"select * into" is not allowed in mysql. Here is an example to get around this limitation:

CREATE TABLE newtable SELECT * FROM oldtable

originally found here

find/replace text mysql

2009-06-08T12:36:00.000-07:00

update TABLE_NAME set FIELD_NAME = replace(FIELD_NAME, ‘find this string’, ‘replace found string with this string’);

from: http://www.mydigitallife.info/2007/04/23/how-to-find-and-replace-text-in-mysql-database-using-sql/

Classify Dialog does not open

2009-04-20T11:07:00.000-07:00

Problem: Classify dialog does not open when hitting button "Classify ..." under Spatial Analyst "Reclassify" tool

Solution: Was trying to do this on an image (JPG). Must convert to a native ArcGIS grid to get the Classify dialog with histogram.

EveryDWG CAD Command Line

2009-03-17T07:25:00.001-07:00

EveryDWG is a utility provided by the Open Design Alliance, a consortium of firms mostly organized around DWG and other CAD standards, for doing conversion from all versions of DWG to DXF (DXF is the more interchangeable format). I was having a hard time getting this utility to work from the command line -- whenever I'd run the command I'd getting a window giving me information about proper syntax. The only trouble is THE SYNTAX USED IN THE WINDOW IS NOT CORRECT! The window says "Command line format is: Quoted input folder Quoted Output folder", but if you actually put quotes on these the command doesn't run and you're back in the same place! Here is an example of the proper syntax:

everydwg C:\TEMP C:\TEMP\OUTPUT ACAD2000 DXF 0 1

make on windows (GNU), dmake

2009-03-13T12:36:00.000-07:00

finally found a suitable (and open source!) make executable for windows 64x, dmake. I posted it on a serve I control, so now it'll always be available (it's GNU)

Blocks not displaying in their regions?

2009-03-12T06:35:00.000-07:00

Blocks not displaying in their regions? Delete the block and then create it again and add again to the appropriate region. This worked for me.

copy mysql tables

2009-03-11T11:04:00.000-07:00

mysql> create database db2; \! /usr/local/mysql/bin/mysqldump -u root –password=pass db1 | /usr/local/mysql/bin/mysql -u root –password=pass db2

originally seen on: http://mywheel.net/blog/index.php/2006/01/05/mysql-duplicate-database-quick-tip/

wildcard regex in notepad

2009-02-11T10:08:00.000-08:00

The general wildcard regex for notepad (simliar to % in sql) is (.*).

So to match both of these lines:

</option><option value="COM">COMPAQ ALPHA UNIX
</option><option value="DGV">DATA GENERAL AOS/VS

The regex/search value looks like:

</option><option value="(.*)">

Just don't forget to toggle search mode to "regular expression"

all changes cause syntax error, python

2009-02-02T06:49:00.001-08:00

Problem: getting a syntax error whenever making any change to a py (python) file

Solution: stop editing py file in regular text editor, needed to edit within development environment, such as pywin

load csv to mysql from local file

2009-02-02T06:45:00.000-08:00

LOAD DATA LOCAL INFILE '[path to your csv]'
INTO TABLE [table to import into, must be created before running with proper columns]
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
([field 1], [field 2], [field 3], [...]);

link to original article

removing carriage return/new line character in mysql

2009-02-02T06:43:00.001-08:00

on importing a csv to mysql I saw these little black square characters, which were carriage return symbols of course. To remove these little artifacts run the following query:

update [table_name] set [field_name] = replace([field_name],'\r','');

terminal server has exceeded max number of allowed connections

2009-02-02T06:41:00.000-08:00

if you get the error just run the following command to override the error:

mstsc -v:0.0.0.0 /f -console

(substitute ip for 0.0.0.0)

link to original article

distinct Drupal user list with views

2008-12-10T08:07:00.001-08:00

having trouble displaying a DISTINCT user list with views? This problem often arises because you need to choose the 'user' type view when you are first creating the new view.

Drupal Views Templates and CCK $fields Array

2008-12-09T06:52:00.001-08:00

Having difficulty accessing the $fields array under your Drupal Views template? Are you sure you have named your template correctly (for instance, if you have named it with 'field' instead of 'fields' the template will apply to individual fields, for which the $fields array is not exposed)

How to migrate Drupal to xampp for Development

2008-12-04T07:04:00.000-08:00

Occasionally it is important to work on a local development machine. For this task I often use the xampp package, which mirrors a standard lamp stack.

To migrate an existing remote drupal site do the following:

install xampp at c:\xampp
modify c:\xampp\apache\conf\httpd.conf

the line 'Listen 80' to 'Listen 88'
If you had Clean URLs on your remote site (i.e. localhost/drupal/admin instead of localhost/drupal/?q=admin)

#LoadModule rewrite_module modules/mod_rewrite.so' to ' LoadModule rewrite_module modules/mod_rewrite.so' (take out the #)

AllowOverride None to AllowOverride All

start apache and mysql from the xampp interface (you may need to change mysql configuration)

copy over the following folders to a new 'drupal_remote_site' folder on your local machine

Themes
Modules
Files

if you don't already have the Backup and Migrate module on your remote machine, install it
backup your database on the remote machine, copy this into the local folder 'drupal_remote_site'
install the same version of drupal on your local machine which you had on your remote -- note: this downloads as a tarball which must be expanded and copied into within c:\xampp\htdocs\drupal (rename the copied folder as drupal from drupal-6.5 for instance)
copy in your themes, modules, and files folder into this new drupal folder
go to http://localhost:88/drupal/install.php, this will install necessary tables,etc. into the db
now enable the Backup and Migrate module on the local machine
use the 'Restore/Import DB' tab in drupal to import your db
voila, all should be back to normal

The $fields array in Drupal (CCK)

2008-12-03T10:50:00.001-08:00

the $fields array is very important for accessing field content when doing custom theming. Unfortunately field names are not well documented, nor are they intuitive. For example, the field name for a node:link field is view_node. In order to track down names of my fields I usually use print_r ... the trick here is that you must look for 'stdClass Object', almost always the index giving these objects as values are the names of the fields used within the $fields array.

For example:
[field_image_fid] => stdClass Object ( [content] =>

can be accessed in your tpl.php by:
$fields['field_image_fid']->content

2008-11-25T07:28:00.001-08:00

having trouble with no "active" class being generated for menu's or primary/secondary links (

) in Drupal 6.x? It has been documented that 6.x has issues with generating an active class ... at 6.5 the active

is classed as "active-trail", so if you are bringing in a theme that worked at 5.x this might be the issue.

For me it was a naming issue within my primary links configuration (the same would be true for a menu). You must use a Drupal recognizable path such as "" for your homepage or "data" for a page at "http://www.yoursite.com/data". To avoid this, make sure to select this menu when creating new content.

2008-11-10T11:13:00.001-08:00

with the php domdocument class was getting the following error:

Warning: main() [function.main]: unterminated entity reference 64-bit

this error was occurring when trying to set the name of an anchor element ... the same error popped up when setting both within the constructor and through the nodeValue method. I haven't tested the solution via the constructor, but by wrapping the value with the built-in htmlspecialchars(), I solved this error

i.e.:
$anchornode = $doc->createElement('a');
$anchornode->nodeValue = htmlspecialchars("$somevalue");

create dom html with php, getelementbyid doesn't work

2008-11-10T10:33:00.000-08:00

I leverage the domdocument php class to create elegant, dynamic html documents via the dom tree. However, I was finding that the "getelementbyid" method was not returning elements as expected. The reason for this is that I had not defined a DTD, which is necessary in this case because we only specify that we are saving as HTML at the end of the process, so the domdocument object is only understood as generic XML, which has no predefined "id." To create a DTD use this code (here using 4.01 transitional html):

$doc = new DOMDocument();
$doc->loadHTML('<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">;');

2008-11-10T07:29:00.000-08:00

there are so many options with pear's db library for php, I sometimes forget the optimal code for getting a sql result as an array that I can easily use within php. The code below fully demonstrates connecting to and getting an associative array with an 'order by' field as the array key. This code is for a postgres rdms, but could easily be modified for mysql, sqlserver, etc. Parameters are in caps surrounded by '%' in the form %PARAMETER%.

require_once 'DB.php';

$dsn = "pgsql://%LOGIN%:%PASSWORD%@%HOST%/%DBNAME%";
$dbh =& DB::connect ($dsn);
if (PEAR::isError($dbh)) {
die($dbh->getMessage());
}

$dbh->setFetchMode(DB_FETCHMODE_ASSOC);
$res =& $dbh->getAssoc('%SQL STATEMENT, USE 'ORDER BY' TO DEFINE ARRAY KEY%');

if (PEAR::isError($res)) {
die($res->getMessage());
}

2008-09-30T08:04:00.000-07:00

When defining projection from ArcCatalog (shapefile properties) I get an error message "Failed to Alter the Spatial Reference". If I try to do this from the "Define Projection" geoprocessing tool (ArcToolbox) I get this error insted: "Cannot create a projection file for filepath Failed to execute (DefineProjection).

I solved this problem by accesing the shapefile through a mapped drive (a.k.a. "Z:\") within ArcCatalog, rather than its \\server\drive$ pathname.

2008-09-19T08:12:00.000-07:00

There are multiple caching levels to be mindful of with a typical ArcGIS Server environment. This factor often makes debugging difficult. Below I have made an expanding list of issues related to cahcing as well as the various cache levels to clear when debugging.

Issues related to caching: query does not return correct field as defined in 'outFields' query property

Cache Levels

AGS web service cache, this can be cleared by restarting the AGS web service from ArcCatalog or by web admin interface
Tile cache, cleared by 'deleting cache', and then reconfiguring tile cache within service properties of the web service
Web browser cache, cleared by clearing cache of web browser
REST API cache, cleared by going to http://yourserveraddress/ArcGIS/rest/admin/cache and clearing cache manually
KML cache, if using a Google hosted KML with your application, to have this load dynamically you need to add a random number to the end of your kml address ... google it!

2008-09-16T21:26:00.000-07:00

To do a non-boolean conditional insert in SQL Server (the "where exists" clause is best for boolean), use the "where in" clause along with an inner join, in the following form:

INSERT INTO [Destination Table]([Destination Field1],[Destination Field2],[...])
SELECT [Origin Field1],[Origin Field2],[...]
FROM [Origin Table]
WHERE ([Test Field A, could be from origin] Not In (select [Test Field B] from
[Test Table, could be destination]));

Gmaps: GEvent and "a is not defined"

2008-08-18T09:28:00.000-07:00

I was getting the (generic and fairly common) "a is not defined" (on line 287 of main.js I believe) javascript error with gmaps when trying to add a GEvent click event listener. I intially thought it was an issue with my some map layers with ArcGIS Javascript Extension for Google Maps, but I resolved this issue by sticking the even handler in the initialize function. For some reason gmaps must not have been available at the scope of the even handler outside this function ... the gmap object was being instantiated inside this function, yet the variable was first declared outside the function in the first lines of the application ... strange.

Flash Map Workflow, described

2008-08-14T09:26:00.001-07:00

Thematic groups created in ArcMap and enhanced with Adobe Illustrator (AI), were saved as separate AI files (by region) and further organized as AI “layers.” Note that a layer should be created for every group of shapes which would be always dealt with in the same way in an animation. For example, if a group of lines would appear and disappear together and never separately, they should be organized into a layer. These layers later correspond to symbols in the Flash application. All layers were then selected and copied/pasted into Flash at once. Flash has the ability to implicitly import AI layers as Flash layers copied and pasted into it. Note that the “paste in place” command should be used in order to guarantee uniform positioning across layers copied/pasted from different AI files -- the dimensions of the Flash canvas must be the same as the AI document for this to work.

The Flash Actionscript program then deals with all user interaction and data display. Symbols are displayed/not displayed, fade in/out, or are magnified based on program functions referencing the symbols indicated by other user interface symbols (buttons, etc.) which are invoked by user interaction.

New Tool: Map Extent Calculator

2008-08-14T09:21:00.000-07:00

I just created a small tool for calculating decimal degree map extents :

you can access it here

add arcgis service remotely from arccatalog

2008-08-11T06:09:00.000-07:00

when adding a service to a remote arcgis server you cannot browse to the actual location of the mxd or geodatabase on the remote server. windows file sharing addresses do not work. you must use the address on the local machine exactly as if you were accessing the mxd from the local machine.

2008-08-05T13:41:00.000-07:00

this is a nice pear db query which returns an associative array

$data = $sqlconnectionobject ->getAssoc('sql statement here',true, array(), DB_FETCHMODE_ASSOC, true);

if (PEAR::isError($data)) {
die($data->getMessage());
}

Setting up a Tiling/Caching Google Maps/ArcGIS Server Solution

2008-08-01T06:01:00.000-07:00

Setting up a Setting up a Tiling/Caching Google Maps/ArcGIS Server Solution has gotten much easier with the advent of ArcGIS Server 9.3 and its REST-ful Javascript API. Specifically, the ArcGIS JavaScript Extension for Google Maps makes creating this kind of mashup almost a breeze ;)

added 8/20/08: this link at an ESRI site really helped a lot

1. reproject shapefiles into world mercator web projection (only available in 9.3)

2. add new service from mxd containing that/those shapefiles

3. enable tiling in the "service properties" for your web service. You should use the google/virtual earth tiling scheme

4. paste the service url into the example code for the extension

ArcGIS Server, not able to "input layers" for cache, or preview map in ArcCatalog

2008-07-23T11:08:00.001-07:00

In trying to make an ArcGIS server (9.3) caching service I was initially not seeing any option for the required "input layers" parameter in the cache manager (through service properties).

Afterwards I noticed that I was unable to preview the map service in ArcCatalog (though coordinates were visible at the bottom of that frame).

This issue was probably fixed by changing properties to include all arcgis server related accounts (used "everyone", just to make sure it was a permissions issue) as having read permissions for the individual shapefiles participating in the mxd I was trying to publish.

It may have also been fixed by not initially specifying a cache directory when creating the service through ArcCatalog, but probably not.

Arcmap geocoding error

2008-07-02T18:43:00.000-07:00

Was getting the following error when trying to geocode, though I suspect it might come up whenever a feature class cannot be created which would have an invalid data table (dbf) associated with it:

Create Feature Class

There was an error trying to process this table.

Operation failed

Solution: Remove any unnecessary fields from the table

Workflow: Access to Arcmap and Back

2008-07-01T22:33:00.000-07:00

export as .xls (with no formatting option)
open w/ excel 2003 or earlier. press ctrl + A twice, do column autofit
convert all double number fields to number w/ adequate number of sig digits (6 is good for decimal degrees)
export as dbf
import dbf to arcmap
when creating shp via geoprocessing/etc name 6 chars/less and no special chars
close arcmap
import into access

postgres to mysql workflow

2008-06-25T10:04:00.000-07:00

postgres to mysql workflow

1. Export database from postgres to plain text sql. You should do this with the pgAdmin III tool by right clicking on the db you want to export and choosing "Backup ..." this will take you to the following screen, where you should select options as displayed below (notice the "Plain" and "Insert" options are checked):

2. Once you have exported your plaintext sql you must create tables in mysql into which you can insert your data. Do this by copying the CREATE statement appearing in the "SQL Pane" (after you have clicked/selected a table) in pgAdmin III into the MySQL query line and execute this. Do this for every table you want to import

3. Now run insert statements ... if you get errors, make sure the table structure of the new table is the same as your old table (and insert statement)

Import/Copy CSV/data to Postgres

2008-06-24T06:10:00.000-07:00

I recently was having trouble importing CSV to postgres. The trouble was with empty trailing columns (the first row had an empty trailing column). I found that a tab delimited file (export from excel, using export as ms-dos text) allowed me to avoid warnings about empty columns.

was getting this error in postgres while trying to run the "COPY" command from sql:

WARNING: nonstandard use of escape in a string literal

This error was followed by a description of where the error was occuring, which was showing the error in the wrong place. To fix this issue, make sure that all slashes (such as in file paths) are "/" rather than "\"

Here is an example of a proper COPY command:

COPY count_crime TO 'c:/temp/temp.csv' USING DELIMITERS ','

2008-06-12T11:52:00.000-07:00

was getting the following error after installing php 5.2.6 on IIS with IIS CGI configuration:

PHP Warning: PHP Startup: Unable to load dynamic library 'C:\Program Files\PHP\ext\php_pgsql.dll' - The specified module could not be found. in Unknown on line 0

I resolved this by replacing the .dll in this location with my .dll from a previous install (php 5.2.5) ... you can probably use the .dll from any version

2008-06-04T08:51:00.000-07:00

are you getting field value with with the leading zeros excluded, but are joining to a field with the zeros included? this is a common issue with census tract numbers, for instance.

you can use the vbscript below in arcmap or other vba data programs to generate a six character value with leading zeros, where the value is less than six characters

*remember new field MUST BE TEXT format, else will always loose leading zeros!

parameters:
short_text_field: shortened field which longer field will be based on
need to change the loop condition to the longer number of characters involved (here is 6)

dim txtShort
txtShort = CStr([short_text_field])
Do Until len(txtShort) = 6
txtShort = "0" & txtShort
Loop

2008-05-20T13:32:00.001-07:00

I was getting the following error on my web browser:

"The directory name is invalid."

I assumed this was a PHP error, and indeed is probably related in some way to PHP, but realized that this was coming up instead of a 404 error ... I'm not sure why.

I just typed the correct URL and the page worked.

2008-04-29T08:21:00.000-07:00

having problems with your pear install on a drive other than c (d?) on iis? you must run the pear install .bat from the cmd prompt referencing the pear install .bat on that drive. otherwise, pear is configured to run on the c drive, and therefore doesn't work. this tip from the cml dba, karl dailey.

2008-04-23T07:01:00.000-07:00

a common task I come across is the need to create a table of "distinct" records based upon "uniqueness" of values in a given field.

SELECT * INTO new_distinct_table FROM [old_non_distinct_table] WHERE [some_unique_field] In (SELECT Max([some_unique_field
]) FROM [old_non_distinct_table
] GROUP BY [non_unique_field_of_interest]);

2008-04-23T06:09:00.001-07:00

I had earlier shown how to do an update query based on a field from another table in MS Access. Here is the syntax for postgres:

UPDATE to_table SET to_column = from_column from from_table where to_table.primary_key= from_table.foreign_key;

handy notes for this query:

append in postgres is || (double pipe)
cast in postgres is like column_to_cast::type_to_cast

2008-04-20T11:39:00.000-07:00

using xxamp for a lamp (linux,mysql, php) environment to do development on php? perhaps you are already running an iis server (or even the others through iis) ... in that case the default ports for webhost and mysql db will already be in use. to get apache to listen on other ports you will have to change the "listen" and "server" parameters in respective httpd.conf. For mysql .cnf the parameters are "port" at 20 and 27

you should do this at least for:

http: in C:\xampp\apache\conf\httpd.conf
ssl: in C:\xampp\apache\conf\extra\httpd-ssl.conf
mysql: C:\xampp\mysql\bin\my.cnf

you may also want to change port values for mysql in:

C:\xampp\mysql\bin\mysqld-nt.exe
C:\xampp\apache\bin\php.ini (line 795)

add the following lines to administer your server with phpmyadmin (at line 20 in C:\xampp\phpMyAdmin\config.inc.php), with whatever your new port number is in the section noted below

$cfg['Servers'][$i]['host'] = 'localhost';
$cfg['Servers'][$i]['port'] = 'whatever your new port number is';