Applying Patches on ArcGIS Server for Linux (as root)

For various reasons, I had found I needed to be root to install ArcGIS Server on Centos.

I ran into my first problem with that configuration on patching, when the script halts with the following message "You must install this [patch] as ArcGIS Server Owner, do not install it as ROOT user."  I was able to successfully bypass this by commenting out this if block from the patch script (applypatch).

... so far, so good.

ArcGIS Server on Centos, continued

When our cluster was upgraded, I somehow lost the ability to run ArcGIS Server. No problem, I said, I'll just rebuild the machine. This is where I was again reminded that ArcGIS Server installation is very different from other Linux software.

The biggest issue is that ArcGIS installers will only run when logged in root.  Yeah, not running a bash as root, or sudo'ing as root, or su'ing to root ... you actually have to log in as root.

Herein lies the difficulty: the rest of the installer assumes a GUI (e.g., gnome or kde).  You can't login to gnome or kde as root.

If you're doing best practice X tunneling with SSH, then you're using private/public keys.  So make sure that you create a .ssh profile with keys and copy over the authorized_keys directory from your normal user.  I think this is where my immediate stumbling block was: when I first did this I wasn't thinking as much about security, just seeing if it would work.  I didn't use keys.  When I rebuilt the system, I did ... and of course I disabled plaintext passwords.  I didn't even consider that I'd want to login as root through X/SSH, but I'd guarantee that's how I did it the first time.  This was also an issue with X, specifically, because X settings (e.g., $DISPLAY) would only be set for the logged in connection (my normal login) and not root, if not connected directly with root.

[root@host]# ssh-keygen

[root@host]# cd .ssh/authorized_keys .

Finally, of course, everything ArcGIS runs through Wine on Linux (the Windows compatibility layer).  It turned out there was a profile directory that was owned by another use

# chown -R root /root/arcgis/server/framework/runtime/.wine

Also two other things I found out during this process:

1. set is a great command for debugging.  I put set -x at the top of authorizeSoftware and this gave me all kinds of great diagnostic info, including THE LOCATION OF THE LOGS!!
2. the software authorization log was located under: /root/arcgis/server/.Setup/softwareauthorization.log

Thanks, Dean Nairn!

Undocumented KML Output Conversion, GDAL

As noted here: http://gis.stackexchange.com/questions/18280/batch-convert-geotiffs-to-kmz-files, an undocumented method for conversion to KML exists in GDAL's gdal_translate as format KMLSUPEROVERLAY.  To do this with an existing image, such as a geotiff, your command will be the following format (on Windows):

C:\OSGeo4W64>gdal_translate -of KMLSUPEROVERLAY   FORMAT=JPEG

with an example path, this looks like:

C:\OSGeo4W64>gdal_translate -of KMLSUPEROVERLAY D:\downloads\tile177_12orthos\177.tif D:\downloads\tile177_12orthos\177.kml -co FORMAT=JPEG

This will output a tiled images with a KML to tie them all together.

Disk I/O Error on sqlite/spatialite

On running sqlite CREATE and INSERT queries, I was getting some mysterious errors, like "Error: disk I/O error", despite having the proper permissions on the .sqlite file and adequate space in the directory.  I then broke down the compound query to a simple query and was getting "GEOS error: IllegalArgumentException: Points of LinearRing do not form a closed linestring", though I found that doing Hex(PolygonFromText(...)) did return a hex string.  My WKT was, however, poorly formed.  WKT requires that the final coordinate pair and the beginning coordinate pair are the same, as in  PolyFromText('POLYGON((-75.8 38.4, -75.0 38.4, -75.0 39.85, -75.8 39.85))',4236).  After I fixed that, my more complex queries were able to run successfully.

github push 403 and Permission denied

Initially I was getting

error: The requested URL returned error: 403 Forbidden while accessing https://github.com/..

on

$ git push origin master

I read a forum post that recommended

$ vi .git/config

and changing the url line from https://github.com/.. to ssh://github.com/..

and then generating public/private keys and adding this to github.  After doing so I was getting

Permission denied (publickey).
fatal: The remote end hung up unexpectedly

eventually I solve this by changing that url line in .git/config to

url = git@github.com:LOGIN/REPONAME.git

... I also made sure to run

git config --global user.name "LOGIN"
git config --global user.email EMAIL

Quick GIS tip of the day, 6/10/14

To keep symbology of a shapefile or geodatabase feature class, save a layer as a "layer file" (.lyr).  If you keep this .lyr file in the same relative path location (in the same directory, for instance), it should continue to reference the data and you can just add the .lyr instead of the data itself.  If the data and lyr become seperated you can apply the lyr to the data through the symbology tab.  I've included some images to illustrate.

RWD Summit 2014

Here are some notes and thoughts I gathered during the 3-day Responsive Design Summit, which concluded today.  Link to the live document.

• Design principals
• Atomic design
• Stop thinking about websites as collections of pages.  They are collections of elements that, through a browser-based, collaborative, design process, that are mixed together into larger and larger collections.  This new paradigm is helpful when doing responsive design, since “a page” is different across different screen sizes.  You can start at the level of meaningful/reused DOM elements, disregarding content and style in conceptions, but adding the style and sometimes content (images?), when creating style tiles, style guides, and pattern libraries (public-facing example)
• Pattern lab is a tool created for doing atomic design.  It emphasizes browser-based design/decisions, going to the browser first instead of wireframing etc., producing the style tile/guide/and pattern libraries, producing clear annotation from code comments, and allowing naaive (non-programmer) changes within a template-like structure through JSON, using mustache
• Design/decide on the browser.  Avoid other deliverables and get the discussion immediately to the browser
• Test with content (not lorem ipsum)
• Wireframes can be bad … if you need to use wireframe, make it a part of your iterative, browser-based design process
•  Your own collection of “templates”, “starter kits”, etc. also help with this
• Test and develop locally
• Many elements of the workflow below are tuned to this (e.g., scaffolder, package/dependency manager, task runner, versioning, etc.)
• Vagrant can be useful for reproducing lower level environments (OS, etc.) although it’s not clear if this should be used over the scaffolder, etc. … or if these complementary
• Supports browser-based design
• Workflow supports collaborative coding
• Performance
• Test your site performance
• http://www.webpagetest.org/
•  
• Wireless network experience will not be improving anytime soon.  This is due to the effect of latency (not bandwidth).  International visitors will also be much more sensitive to site site.  Some things to help
• Use Compression (whitespace)
• Good syntax
• Keep presentation in CSS (which is cached for the entire site)
• Small as possible
• Responsive
• Can no longer use a limited set of breakpoints, use em’s and use additional self-set breakpoints … breakpoints relative to font size?
• These other principals support responsive design … for example: browser based!  Overlap in Designer/UX/Dev process and roles.
• Frameworks: the two frameworks below responsive user interface elements.  Both come as standard web browser syntax files (html, javascript, css)
• Foundation: seems to provide a richer set of elements, which can be selected (or not) via the web interface when first downloading the code to add to the project.
• Bootstrap:
• Workflow (video which was mentioned: https://www.youtube.com/watch?v=vsTrAfJFLXI)
• Scaffolding: yeoman
• Produces the initial code for particular kinds of projects
• Integrates a Package manager (bower) and a task runner: grunt (good “why grunt”).  Many “generators” provided to scaffold various projects.  Installs and runs through node package manager/nodejs
• Know your text editor: Sublime Text 2 is popular
• Versioning: git (github has a friendly client app and is on the cloud)
• Preprocessors (see below)
• Deployment: beanstalk (versioning plus deployment)
• Preprocessors
• Notes
• it is important to consider how you will run these.  Many run through Ruby or Node by default, but have other options (Python, PHP, PHP, etc.).  Ruby is installed by default on Mac and is trivial to install on Windows.  Node seems trivial to install on either platform, as well.
• A simper way to run these preprocessors, as well as some other workflow tools, would be through the standalones: codekit (Mac) or prepros (Windows)
• Cloud-based tool here, with many different preprocessors http://codepen.io/pen/
• HTML
• HAML: runs on ruby.  Sparse syntax which generates complete, well formed, xhtml
• Mustache: use for html includes, fed through JSON.  Preprocessor available on many platforms.
• CSS:
• SASS: CSS preprocessor syntax adding mixins (css snippets), media queries, indents, variables, functions,
• Many github repos offer useful mixins.  For example: Team-Saas, but these sometimes rely on ruby
• Team Sass provides a few prominent projects/tools … rely on ruby … not supported directly on codekit or prepros
• Breakpoint: for responsive, multiple screensizes
• Toolkit … responsive stuff?
• Singularity … layouts!
• SCSS: A more CSS-like version of the SASS syntax, does not require indents … actually this is now the default in SASS at version 3
• Compass: a tool which provides many common mixins, some of which provide cross browser support.  Run for free on ruby or install as a standalone compass.app .. scout is a free standalone version.
•  Javascript
• Coffeescript

UD TA speaks about using Story Maps in the classroom

We recently up put out a new LearnIT Express Webinar A Brief Introduction To Story Maps, with Jim Casey, Teaching Assistant and PhD Student, English.  In this video, Jim tells us how he guided his introductory undergraduate English class through projects displayed on the ArcGIS Online cloud-based platform.  This example demonstrates how (nearly) any type of location-based information can be visualized through GIS software.

Follow this link for an example of a student project.

image: Elana Borinsky, Sophomore, B&E

... geocoding continued

As I had mentioned in previous email Esri has retired it's public-facing geocoding and routing services. These services are now only available behind their paywall (through an ArcGIS Online subscription). Esri gives UD a fixed number of credits on our subscription that we can use towards their various services.  The cost for use of their Geocoder service is relatively expensive, as far as their services go.  For example 300,000 would take nearly our whole allocation of service credits.  However, for lesser numbers of geocode (for one-off needs) we can allow you to join our subscirption through an ArcGIS Online account.  Contact us for more information.

For all other geocoding needs, I recommend the following desktop solution:

Download Disc 2: StreetMap North America (Detailed Road Network, Geocoding, Routing) from the UD GIS Data page.

You can then choose one of the .loc files in streetmap_na\data when it asks for an address locator in these instructions: http://resources.arcgis.com/en/help/main/10.2/index.html#/in_ArcMap/002500000026000000/

Plotting and Projecting Coordinates

So you have some data that includes latitude and longitude, you add it to the map ... and it doesn't line up.  You've got some spatial reference/projection problem.  Figuring out what coordinate/projection system your data came from is not always straightforward, and often involves looking more deeply into the source of your data (for metadata).  One rule of thumb: if your data shows up around the horn of Africa in the Atlantic Ocean, you're probably trying to display unprojected coordinates in a view (or with other data) that's in a projected system.  Most of the time that you have unprojected coordinates the data will be in WGS 1984 or NAD 1983 (HARN or not).  You can often figure out which one by trial and error.

In any case here are all steps for going between unprojected coordinates and a projected system.  These are instructions for ArcMap, but similar would apply for QGIS.

We'll plot the coordinates, export the data, and project it.

1.  Add the tabular data (xls or csv usually) to ArcMap.  Right click on the layer and choose "Display X Y data", choosing latitude, longitude and WGS

2.  From the created plot layer, export to feature class (shapefile), and add this to the map

3.  Project this shapefile into the projection of the GIS data

ArcGIS Online for Geocoding

As I noted in my last post, UD has an ArcGIS Online subscription. As part of that service we have credits to be used (frugally) towards geocoding. Just let me know you are interested, and I'll sign you up. You'll receive an email with details about logging in and completing the sign up. Instructions for pointing to the service from ArcGIS, with a link to instructions on geocoding a table of addresses, is here: http://resources.arcgis.com/en/help/main/10.1/index.html#//00250000004v000000

Geocodes cost UD 1 service credit per 25 addresses. If you would like to geocode more than 100 addresses, I may refer you to other options for geocoding.

UD's ArcGIS Online Subscription

You'll be glad to hear that the ArcGIS Online Subscription is included as part of our campus site license.  We've been granting access to the subscription for no extra charge.  If you are paying your annual ArcGIS charge, you're already paying for a portion of our campus site license.

However, we have a limited number of service credits (12,500 remaining at the moment).  The administration tools for the subscription are pretty limited, so we ask that you use credits carefully.  Here's a link to see how credits are used: http://www.esri.com/software/arcgis/arcgisonline/credits.  In many cases there are desktop-based alternatives for the most "expensive" operations (e.g., tile caching, geocoding, analysis).  I would be happy to suggest alternatives if you think something is looking expensive.  We may also end up charging for the subscription at some point, but I don't anticipate that we'll do that in the coming year.

Once I've invited you to the subscription, you'll see an email notification letting you know how to connect.  If you already have an Esri account, you can sign in with that.  Otherwise you'll be prompted to create an account.  If you already have a "public" ArcGIS Online account, you may want to keep it separate from the subscription.