Position: GIS/Metadata Librarian (Columbia University)

GIS/Metadata Librarian (Columbia University)

This is an intriguing position. As gleaned from the job posting, the Lehman Social Sciences Library has two professional GIS positions: the GIS/Map Librarian and the currently available position, GIS/Metadata Librarian.

This position will focus on the cataloging/metadata maintenance and distribution of geospatial data. That is a wonderful setup that I am envious of as I think of the fantastic spatial catalog we can build if we had someone devoted entirely to it's development and maintenance. It is possible... This prompts me to draft a request for a new professional position or perhaps a professional.

Here are some snippets:

• "Work with colleagues from a variety of Library, IT, academic departments and University institutes in the design and creation of a searchable and easily accessible spatial data catalog that allows for remote access to spatial data collection."


• "Work with the GIS/Map Librarian and the Head of EDS to provide reference services, outreach and instruction in EDS. This includes: providing one-on-one consultations, assisting in instruction sessions, and training student assistants working in the unit."


• Requirements: "MLS or equivalent combination of education and experience. Demonstrated proficiency with: GIS software, spatial data, FGDC and ISO metadata standards, and database design."

There is no mention of salary, but these perks are extremely enticing:

We offer excellent benefits including 100% Columbia tuition exemption for self and family and assistance with University housing. Columbia will also pay 50% tuition for your dependent child who is a candidate for an undergraduate degree at another accredited college or university.

H5N1 Avian Flu GIS & Migratory Bird Flyways

As is to be expected, one of the most popular GIS requests that I have received lately (especially last Spring semester) are data resources concerning the H5N1 Wild Bird flu virus. There are some great resources out there, the best that I am aware of I will list below.

Earlier this summer, a student researching the epidemiology of the virus needed a shapefile of the common worldwide migratory bird flyways. I was unable to locate such a shapefile. If someone knows of such an available file (free or not) out there, please leave a comment here to let me know about it. However, we were able to locate several static maps (images) of migratory bird flyways. I advised the student on how to georeference and digitize such a static map to create a feature class. Unfortunately I have not heard back from the student since then so I do not know if they were successful.

Just last week, the Introduction to GIS course that I am teaching came to the section on georeferencing and digitizing. Instead of using a historical (scanned) map of Texas for the class exercise as I normally assign, they worked on creating a feature class of the migratory bird flyways shown on the H5N1 outbreaks in 2005 and major flyways of migratory birds map. This map was created by the UN: Food and Agriculture Organization.

Here is the best digitized migratory bird path of the lot. (Yes, the student gave permission for her files to be uploaded.)

• Personal geodatabase
• Shapefile
• Google Earth KMZ (created using Arc2Earth)

Additional key H5N1 avian flu resources (that I am aware of):

• Google Earth Maps of Avian Flu Spread
• Declan Butler, reporter for Nature, has created a super-fantastic Google Earth resource. These freely available KML files contain both human and poultry cases of avian flu. More information can be found on Declan's blog.


• Public ArcIMS Services
• Global Resource Information Database (GRID-Nairobi)
• Feature classes containing avian influenza cases in humans by influenza type by country.
• ArcIMS Server: http://gridnairobi.unep.org
  Service: java_make
  Layer: Avian influenza outbreaks
• Interactive Online


• Austin Peay State University: GIS Center
• Feature classes containing cases and deaths from avian flu in both birds and humans by country
• ArcIMS Server: http://gisweb.apsu.edu
  Service: birdflu
• Interactive Online


• Access Raw Data Tables
• WHO: Confirmed Human Cases of Avian Influenza A (H5N1)


• OIE: World Organization for Animal Health
• Provides tables (and a PDF graph) containing avian influenza in animals (type 5).


• WHO: Global Health Atlas
• Great resource for worldwide, country-level, disease data. Provides data tables and even an interactive mapping interface. The data tables are the most useful to me as there is even a download to Excel option.
  

I know there must be all sorts of other great resources out there, but the sources I listed above has suited all of my needs thus far.

Geocoding to Parcel Boundary

A growing request by students on campus lately is the accurate geocoding of addresses. My opinion is that with the high accessibility of aerial imagery these days, students want geocoded addresses to be placed exactly on the house or business. I have been recommending these students to geocode their addresses based on parcel boundaries as opposed to using a street file. I want to take this opportunity to organize my thoughts about the steps needed to geocode directly to the parcel boundaries, specifically using ArcGIS desktop software.

Of course, you will get much better precision geocoding an address to the parcel boundary. For example, if I look at the Google Map image of that old rundown house I lived in while a student in Buffalo, you will see the marker get close but not quite exactly right on. The house is actually the next to last house near the dead-end. StreetMap USA will not geocode the address with high enough precision, nor will Yahoo! Maps, and of course TIGER/Line geocoding (poor-man's geocoding) will be way off.

The only solution for a high level of accuracy is to geocode addresses based on the parcel boundaries, provided the parcel boundary feature class is accurate as well.

So, here's the steps.

1. Of course first you must obtain the parcel polygon containing the address information for each parcel.
2. Ensure that the full street address is broken into two fields, one for the street number and one for the street name. This is essential. If the street number and name occupy the same field, they must be separated. There are a number of ways to do this. A quick regular expressions script in Python will do the trick. Or, using Excel, you can extract numbers from alphanumeric strings or split the field by using convert text to columns.
3. This is the important step. Now we're ready to create our geocoding service. In ArcToolbox, launch Geocoding Tools/Create Address Locator. The style that we want is 'US One Address'. You can see in the Field Map that the street number and the street name must be separated. Fill out the form and let 'r rip.
4. Now that we've got the geocoding service set up properly, we can geocode as we normally would, but using the service we just created. All matched addresses will be placed exactly in the parcel centroid.

Now, be warned that the unmatched percentage might be higher than you anticipated. This is because the 'US One Address' geocoding style will either find a polygon containing that address or it will not. There is no estimating or approximating. Conversely, standard street geocoding finds approximate address information between nodes. For example, 450 Happy Gilmore Street will be placed exactly inbetween the 400 node and the 500 node of Happy Gilmore Street. What if there is no physical address at 450 HG Street? The point will still match there. What if 450 HG Street is the first house on the 400 block of HG Street? The point will still be placed exactly in the center between the nodes.

My Powerful Geospatial Suite of Free GIS

These are the freely available applications and services that make up my own personal free GIS. Individually, many freely available applications do not of themselves constitute a full geographic information system, but when these are all pulled together within one suite of tools...Well, it is remarkable what someone can do without spending a cent. (And without needing to spend an enormous amount of time developing your own applications out of open source components or needing to learn, or install, complex applications, such as GRASS GIS.)

To all of you wonderful and generous developers out there who have spent your own resources to create these applications/services, I give you a sincere and heart-felt thank you for making it so easy and wonderful for the rest of us. Thank you.

For a more comprehensive freeware software list, see FreeGIS.org.

For a more comprehensive list of both commercial and non-commercial applications, see the VerySpatial: Geotechnologies Roundup.

This post is an update of my previous listing of freeware and open source GIS options.

GEOCODING

• BatchGeocode.com
• By far the very best free geocoder out there. In fact this is the only free geocoder that I am aware of that can handle batch jobs. Fantastic.
• Uses the Yahoo! Maps Geocoding API, so the returned lat/long are exactly where Yahoo! Maps would have placed them. That's a world of an improvement over TIGER/Line-based geocoders.
• There is also a single address lookup tool.
• Total online web 2.0 application. No software to download.
• Geocoding limit is 7,000 addresses per computer per day (limit imposed by Yahoo!).
• ** The BatchGeocode.com blog announced that Google Maps has recently released an geocoding API and development is underway to use this new functionality, which will include Europe and Japan.

• ajmGeocode PE
• This was my favorite free geocoder before discovering BatchGeocode.com.
• Requires free registration.
• Geocoding limit is 50 addresses per day.
• Seems pointless to even mention this now that BatchGeocode.com is here.

GIS PROCESSING

• SAGA GIS :: A System for an Automated Geographical Analysis
• This is not the most intuitive application out there, but once you get familiar with the interface it is incredibly powerful. Here are some highlights:
• Create a point shapefile from geocoded addresses (table containing XY)
• Shapefile utilities, including geometry conversion, merge, table join.
• Convert E00 to shapefile
• Georeferencing
• Powerful analysis tools for both grids and shapefiles, including kriging and calculus functions.
• Interpolate to raster.
• Multiple regression
• Terrain analysis
• SAGA is completely open source.
• Provides a manual (PDF)

• fGIS: Forestry GIS -- (Download Site)
• As reported here, fGIS is no longer freely available as of November 7, 2005. Big time bummer. However, this page states: “You can continue to use and freely share versions of fGIS released before November 7, 2005.” Older versiona of fGIS can be downloaded from ForestPal.com.
• More user-friendly interface than SAGA.
• There are some great utilities available in fGIS that are hard to find elsewhere.
• Powerful shapefile editing capabilities.
• Shapefile utilities: clip, merge, table join, select, intersectct
• 3D view
• Nice map layout features
• Georeferencing tools
• Create a point shapefile from geocoded addresses (table containing XY)

• QGIS: Quantum GIS
• This is the most versatile application in this list as it runs on Windows, Mac, Linux, and Unix. Just the other day, someone asked me about an affordable GIS app for their Mac and I sent them to QGIS.
• Another thing that makes QGIS very unique is its native ability to work with PostGIS, a PostgreSQL geodatabase.
• Can read numerous vector file formats, including shapefiles and MapInfo formats.
• API supported (which is nice for a free application)
• Provides a 76 page manual (PDF)

• DIVA-GIS
• This excellent user-friendly application contains numerous GIS utilities and spatial analysis functions. Here are some highlights:
• Interpolation, geometry conversion, distance, autocorrelation, regression, select by polygon, table to point.
• DIVA-GIS also offers numerous specialized models and tools for analyzing biodiversity data.
• Provides a 76 page manual (PDF)

• Christine GIS Viewer 1.2
• I am fond of this little freeware application because of its small size (0.97 MB), very user-friendly interface (similar in ways to ArcView 3), and its simplicity. Many users of GIS simply want to be able to download some TIGER/Line files and join them to some Census data to create static maps. This is the software I always recommend for such purposes.
• Standard simple functions such as table join, selection, query attributes, graduated color symbology.
• The processes are scriptable but the UI is not.

SPATIAL STATISTICS

• GeoDa: Spatial Analysis Software
• Definitely one of my favorites, I have written about GeoDa before.
• This easy-to-use application is fantastic for exploring, visualizing, and analyzing spatial datasets. Some highlights include:
• Centroids and thiessen polygons
• Smoothing
• ESDA (Exploratory Spatial Data Analysis)
• Regression
• Spatial statistics, including spatial autocorrelation
• One of the powerful features of GeoDa is the linking and brushing between all tables, maps, and graphs. For example, select the outliers on a box-plot graph and these features are also highlighted on all other charts, tables, and maps.
• Shapefile and attribute analysis only. Use SAGA (above) for raster analysis.
• In the bulleted list, I pointed out mainly the spatial analysis tools in which GeoDa specializes. However, I do want to point out that this application does indeed have numerous standard GIS functionality, such as table joins, attribute calculations, table to points, etc. If GeoDa were to add just a few small standard GIS functions to its repertoire, it could very well serve as an independent GIS application.
• One other thing that must be pointed out is the most excellent documentation. In particular, the 244 page Exploring Spatial Datwith GeoDaDa: A Workbook (PDF) serves as an entire course on spatial analysis.

• STARS: Space-Time Analysis of Regional Systems
• Functionality similar to GeoDa, but much more flexible and dynamic as it is completely open source (written in Python).
• Python developed, open source, cross-platform. The sample application distributed with the code is simply an example of the applications that can be developed using STARS.
• Anyone interested in open source Python GIS should take a look at PySAL.

LINUX GIS TOOLS FOR THE LINUX-INTIMIDATED

There are some fantastic open source resources out there that people swear work so wonderful on the Linux platform. However, if you are entirely dependent on Windows or merely dabble in Linux (like me), compiling and installing the applications can be more difficult than actually using them. Even if you can get your hands on a good binary file, it is likely that you will not have all of the required dependencies. OK, you can see my Linux frustration here. Anyway, here are a couple of tools that make it easy for absolutely anyone to install the applications and so the only remaining obstacle is using them‚…

• Sourcepole: GIS - Knoppix
• Bootable Linux CD with pre-installed GIS applications. This means that you go to any PC that can boot from CD, and you‚’re ready to run. Easy as pie.
• The website has a complete list of the applications, but here are some highlights:
• GRASS
• UMN MapServer
• PostGIS
• MySQL
• GPS tools

• HostGIS - Linux
• Now, I have not actually installed this or played around with it yet, but perhaps if I have a free weekend sometime I'll give it a shot on my home computer.
• HostGIS Linux is a Slackware Linux/GNU distribution containing all of the applications you will need to run a completely open source GIS server.
• Wow, wow, wow, wow!!!
• Installations include Apache, Postgres, MySQL, MapServer, MapLab, sample datasets, and more.
• Here is the documentation.

REMOTE SENSING

• ScanMagic
• This is the best all-in-one free remote sensing application that I have used.
• ScanEx offers two different freeware applications, ScanMagic Lite and ScanMagic LL. These two offer similar features, but not exactly the same. Download both to get the full potential from their offerings.
• Between the two applications, functionality includes:
• Over 60 raster format support
• 10 vector format support
• Image processing functions
• Map projections
• Export/printing

VIRTUAL GLOBES

• Google Earth
• Do I even need to write something here? GE is a wonderful and exciting application that is impacting the geospatial community in profound ways.
• There are some fantastic free tools out there for converting shapefiles into GE’s KML format. A good one is Shape2Earth. Of course, MapInfo supports exporting to KML and there are numerous 3^rd party ArcGIS extensions as well.

• NASA World Wind
• With the recent discussions surrounding the legality of using the free version of Google Earth at work, NASA World Wind might prove to be a more legal option.

• ArcGIS Explorer
• I have not even downloaded this application yet, which is still in Beta. However, the James Fee GIS Blog has been doing an excellent job describing the application.
• This will be the only virtual globe that I am aware of that can stream ArcIMS data, which is fantastic.

DIGITIZING / VECTORIZING

• WinTopo Raster to Vector Converter
• Includes powerful tools for heads-up and one-touch digitizing. Highlights include:
• Extract centers and edges
• Polyline smoothing and reduction
• Support multiple GIS and CAD formats

OTHER MISCELLANEOUS FREEWARE APPLICATIONS/SERVICES (that I use)

• CarbonTools
• An ‚“Open-Geospatial .NET development toolkit‚”
• In my opinion, the power of the .NET framework is the powerful IDE. Those C# and VB GIS developers should take a look at CarbonTools.

• GMapEZ: Google Maps EZ
• GMapEZ provides public JavaScripts that developers can reference within their HTML code to quickly and easily generate customized Google Maps.
• I used this service in the Friday Night Hangout (IE-Only) application as well as ArcMap2GMap.

Sort , Filter, & Copy Tables in Your Browser

TableTools is a new (to me) Firefox browser extension that is really, really cool.

Tables on the web become interactive with this extension. Right-click menu includes sorting (yes, actually sort previously static tables), filtering (yes, just like Excel's auto-filter), and copying as tab-delimited or HTML.

A great use of this extension is to use it with BatchGeocode.com. As many folks are aware, BatchGeocode.com is a free batch address geocoder that uses the public Yahoo! Maps Geocoding API to take a pasted tab-delimited table and spit out an identical table with lat/long coords (as well as a Yahoo! Maps mashup and Google Earth KML file of the first 100 points). Using the TableTools extension, a table of addresses on the web can be copied directly to tab-delimited format, and then pasted directly into BatchGeocode.com without ever opening Excel. Now THAT is a beautiful thing, eh?

2006 Statistical Abstract + Zoho Sheet Excel Viewer

All of the tables in the 125th edition (2006) of the Statistical Abstract of the United States are available in Excel format. This is a huge leap forward from PDF, which is how StatAb has traditionally been published online.

Via: The librarian is: BLOGGING | Journalism Library, Columbia University

I really appreciate that all tables have static URLs. This means that I can save and share these resources using the Zoho Sheet Online Excel Viewer. I have previously blogged about Zoho Sheet, a powerful (& free) Web 2.0 spreadsheet application. The Online Excel Viewer allows users to view and interact with existing Microsoft Excel spreadsheets entirely over the web.

Here are a few examples of using Zoho Sheet to view these static StatAb Excel files:

• Public High School Graduates by State: 1980 to 2004
• Electoral Vote Cast for President, by Major Political Party--States: 1964 to 2004
• Crude Oil Imports Into the U.S. by Country of Origin: 1980 to 2004
• Christian Church Adherents and Jewish Population, 2003--States
• U.S. and Worldwide Commercial Space Industry Revenue by Type: 2000 to 2003

There's all sorts of fantastic tables included in the StatAb, and the ability to respond to student requests with an online spreadsheet is more effective than sending an Excel file as an attachment.

And, YES, it is very nice to be blogging again.