Area Weighted Join vs. Standard Join

Students from an advanced suitability analysis course this summer needed to create a report that specified the percent of the join layer that intersected the target layer. For example, they needed to calculate the percent of each use from a landuse shapefile that intersect each zip code in Texas. I wrote a quick script for the class that generated the report they needed, but the implications are astounding to me.

Here is what I mean by astounding. As a test, I calculated the median household income and the total population within a 1-mile radius around each dance club in Arlington, TX comparing the following methods: (1) Total population using ArcMap's standard spatial join tool, (2) Total population using an area-weighted summation, (3) Average median household income using standard spatial join, (4) Average median household income using an area-weighted average.

The following table displays the results.



Field A displays the dance club's name. Fields B, C, and D above display the difference between using the standard ArcMap spatial join tool and a weighted-average spatial join tool when calculating average household income. Fields E, F, and G display the differences when calculating the total population.

The differences in both cases are quite high. I am thinking of myself and all of the students who I have seen naively rely on the standard spatial join tool for these types of calculations. Wow...

Why is there such a difference?

An area-weighted spatial join between two polygons comes in two flavors, depending on whether ti is calculating an average or a sum.

If it is calculating an average, the formula is [area-percent] * [value] + [area-percent] * [value]... The most important consideration is the percentages of the join features that are within each target feature. For example, in a particular zip code, there might be 3 block groups. Let's further suppose that block group 1 comprises 50%, block group 2 comprises 35%, and block group 3 comprises 15%.

If it is calculating a sum, the most important consideration is the percent of the join feature that actually intersects the target feature. The formula is ( [% area intersects target feature] * [value] + [% area intersects target feature] * [value] ) / number of intersecting join features. This is why you will see a much larger error when using the standard spatial join tool for summations than for averages. If 2% of a block group intersects a zip code, the standard tool will include the entire population of the block group instead of only 2%.

Is This a Perfect Solution?

No. This assumes a perfectly even distribution within each join feature. It is, however, a huge improvement.

Where Can I Get the Script?

Download it here. Extract the compressed archive and you will see three Python scripts and an ArcGIS toolbox. Open ArcMap or ArcCatalog, ensure ArcToolbox is visible, and add the Spatial Join Tools.tbx (single-click).

Caveat: These scripts are first drafts and have not been tested on any systems other than the ArcINFO Desktop 9.1 & 9.2 systems here at UT Arlington. There is no documentation. Also, the scripts run on the slow side. Eventually these will be optimized, but at the current time they are presented as is.

Description of the three tools:

1. Average Area Weighted: Use this tool to calculate an area-weighted average spatial join between two polygons.
2. Sum Area Weighted Join: Use this tool to calculate an area-weighted summation spatial join between two polygons.
3. Percent Area Report: Use this tool to generate a report that specifies the percent of the join layer that intersected the target layer.

GIS to Select Foreclosure Residential Properties

This summer I taught a graduate real estate course exploring the ways GIS can be used select foreclosure listings for possible investment. As far as student satisfaction is concerned, this was the most successful real estate course I taught. The course was very tight and practical, and hopefully everyone will go out and make some big bucks with the skills they learned...well, when the market eventually turns around, eh?

The 9-week course was divided into two sections. First, we went over the fundamentals of performing a comparative market analysis (CMA) using foreclosure listings and multiple listing service (MLS) listings in Tarrant County, TX. Second, we learned how to automate the process using ArcMap's ModelBuilder.

What data did we use?

We used exclusively the HUD foreclosure listings in Texas available for free from the Southwest Alliance of Asset Managers. This is a fantastic resource as the listings can be batch downloaded in Excel format. Each week, we were able to download a fresh batch of new foreclosure listings for Tarrant County. As an aside, HUD foreclosure listings are freely available from most states. To locate the agency that manages these listings for any state, see the HUD Homes website.

For MLS, we used data from the North Texas Real Estate Information Systems, Inc. (NTREIS). Unfortunately, acquiring batch MLS data is expensive and is only available to realtors. In May I downloaded app. 17,000 MLS listings of all statuses, including both sold and active properties. We used this data for the entire course.

How Can a CMA Help to Locate Foreclosure Properties for Investment?

In its most straight-forward sense, a CMA compares the selling price of properties (from MLS) in the same neighborhood of an active foreclosure listing. The lower the price of the foreclosure as compared to the average selling price the better the deal...on the surface, anyway. There are so many factors to take into consideration, but first let's consider this straight-forward CMA.

To accomplish this bare bones CMA, you first need to geocode both the foreclosure and the MLS listings. You then need to generate a buffer that will designate the neighborhood surrounding the foreclosure property. Then, spatially join the sold MLS points (containing the selling price) to the buffer, making sure to average the selling price field. Then subtract the foreclosure list price from the average selling price, and you are all set.

This is similar to the CMA reports I have seen from many commercial vendors.

Hey, This Oversimplified CMA Is Not Very Useful!

Yes, you are correct. Now, let's take a look at the many, many wrinkles that make this analysis so much fun.

• In our CMA, we want to compare the price per square foot. The cost per square foot often decreases as the size of the property increases.
• This causes a large problem as many foreclosure listings, including the HUD data we used, do not report the square foot of the property. Yes, this data is available in the MLS, but the MLS only includes those properties that have been recently active. The answer is to acquire this data from the local appraisal district. In our case, the Tarrant Appriasal District (TAD). We acquired the complete primary real estate account data from TAD, which includes the square foot (living area), but unfortunately there is no reliable field in common with the foreclosure data so we could not make a tabular relationship. So, we acquired the complete parcel shapefile, tabular joined that to the primary real estate account, and then geocoded the foreclosure listings directly to the parcel boundary. We were receiving accuracy results in the upper 90 percentile. Then, we spatially join the parcel shapefile to the geocoded foreclosure shapefile. Whew...we finally obtained the square foot of each foreclosure and calculated the price per square foot.
• The MLS data already included the price per square foot.
• A standard spatial join between the foreclosure buffer and the MLS sales is not appropriate as not all properties are comparable. Differences between the year built, number of bathrooms, etc. can decrease the accuracy of a CMA. For example, consider a foreclosure buffer with 6 recently sold properties, and 2 of the properties were built in 2004 while the other 4 were built in the 1950s. If the foreclosure property was built in the 1950s, it might not be wise to include the two newer properties in the CMA as the prices of those two might be substantially higher than the other four.
• This caused a major wrinkle, as I know of no way to exclude such properties from a spatial join using ArcGIS built-in tools. In other words, I know of no way to perform a spatial join filtered by a query based on the values of each feature of the target layer. When planning this class I knew this would be the major stumbling point, so I created a Python script that did just this. After I clean it up a bit this Fall semester, I will post it here and to ArcScripts.
• Demographics, especially crime rates and potentially employment outlook, can play a major role in an investor's decision to invest in a property. After the CMA, it is then necessary to filter, rank, or weight the results by these demographic attributes.
• We used block group data provided by Applied Geographic Solutions (AGS), which the library has been purchasing the last few years. We are shifting to accessing the data via SimplyMap, but this will be a whole other post.
• Defining a neighborhood by a circular buffer is not the most reliable way to define a neighborhood. A better method is to include only those properties in the same subdivision.
• While spotty subdivision data is included in MLS, it is not included at all in most foreclosure listings. The solution again was to turn to TAD. If you geocode both the foreclosure listings and MLS data using a parcel shapefile as the reference, you can spatally join the parcel shapefile back to the two geocoded point layers to obtain the subdivision. Then, you need to perform another query-based spatial join (see above), which to the best of my knowledge is not included within the standard ArcGIS tools. The script that I wrote allows for this as well.
• Yeah, there are other issues, but this is enough for now.
  

How is This Process Automated Without Any Coding?

Using the ArcGIS ModelBuilder, this be automated up to a point, but not as smoothly as I hoped. The Python script I wrote can be incorporated as a geoprocessing object into a Model, so that went fairly smoothly. Two major difficulties arose. First, there are limitations that make it difficult to set parameters for temporary layers that are contingent on other parameters. Second, I just find the ModelBuilder flunky and inconsistent in general. While everyone did a fantastic job on the first section, there were varying degrees of success automating the process. I recommended to everyone interested to take the Python Scripting for ArcGIS course I am teaching Spring 2008 semester.

Lot of Work For One Class..What Now?

Yes, initial development on these special courses can be quite exhaustive. I expect to be teaching this course once per year for the foreseeable future and I plan to hold a library workshop this Fall semester showing everyone how GIS can help investors flip houses. If this topic can not get folks into thye library for a workshop, I do not know what will. (Actually, the four workshops held in Spring 2007 semester averaged over 20 attendees each.) This workshop will be part of our GIS Day activities, but this deserves its own post as well.

Position: Map and Data Services Librarian (University of Illinois at Chicago)

Map and Data Services Librarian and Assistant Professor (University of Illinois at Chicago)

"The University of Illinois at Chicago Library seeks a dynamic and energetic librarian to provide reference, research consultation/user education services for cartographic material and social science data sets, as well as implementation of GIS applications within the library and the university community."

There sure seems to be a need for good GIS folks in Illinois, eh? Last week, I posted the newly available position at the University of Illinois Library at Urbana-Champaign, and now there is a similar opening in Chicago.

As this position's title suggests, the posting describes this position as a mixture between GIS, traditional cartographic materials, and data/numeric services. However, there does seem to be a strong focus on the social sciences.

Here are some snippets:

• Works closely with primary users in a wide variety of academic areas including urban planning, history, earth sciences, public administration, sociology, political science, architecture, social work, education, public health.
• Participates in collection development decisions for cartographic materials and social science data (maps, atlases, remote sensing images, geospatial data).
• Participates in planning, design, and maintenance of web pages that include specific information about cartographic resources and GIS and social science data.
• Creates local indexes (property listings, remote sensing products, aerial photography, etc.).
• Works with other library units to assure consistent policies for the cataloging of maps, remote sensing imagery, aerial photographs, and data sets. Creates and reviews metadata for digital geospatial data and data sets, as needed.

Salary begins at $40,000, with faculty status.

Firefox Mapping Extensions in a Single .XPI Package

Lots and lots of fantastic mapping extensions for the Firefox browser. Which ones, you ask? Well, the 16 extensions listed below.

There are also a couple of extensions that allow users to back up all the extensions installed in their browser and package them into one .xpi extension file. The extensions are Firefox Extension Backup Extension and its partner Compact Library Extension Organizer.

So, I used FEBE and CLEO to backup and package all of the extensions in the following list into one convenient .xpi extension file: FFmapping.xpi. Install this file and you will have just about every Firefox mapping extension that I am aware of...Well, those that are compatible with Firefox 2.0 anyways. Now, the server might not be configured properly for direct extension installations, so you might need to download the extension first (right-click/save as), and then File/Open and browse to the file. This will get it installed.

Why use Firefox over IE? Well, here is one reason.

Google/Yahoo!/Live Local et al Map Extensions

• Full Map
• See more of the actual map on Google Maps. Rotate through 3 modes.
  
• Map+
• View a Yahoo! map of a selected address without having to open a new window or tab.
• Map This
• This extension will let you get a Google map for any address on a web page.
• All Your Maps Are Belong To Us
• Translates URLs for other mapping sites to Google Maps.
• Firefox Toolbar for LookLOCAL Maps
• The LookLOCAL Firefox Toolbar is a convenient extension to the Firefox browser that enables you to map a location, get directions, or search for products and services from any web page you are on without first navigating to an online mapping site...
• MapIt!
• Highlight an address and get a map and/or driving directions using your favorite online mapping site.
• GDirections
• Finds directions on Google and Yahoo Maps based on your selected text and one of various home addresses.
• Freeway Driving Directions
• Uses your favorite driving directions web site -- Expedia, MapQuest, or Yahoo! Maps -- to display the driving directions in a new tab when you highlight an address on any web page.
  

Region Specific

• Streetmap
• Simple UK streetmap search from context menu.
  
• BuscaDirs
• Gets a map for a selected address in Argentina, as well as for some cities in other Spanish-speaking countries that use the same address format.
• Locate Address in Israel
• Enables you to locate Address written in hebrew using mapa.co.il map database.
  

Geotags (Websites & Images)

• Shazou
• The product called Shazou (pronounced Shazoo it is Japanese for mapping) enables the user with one-click to map and geo-locate any website they are currently viewing.
• Nearby
• Shows you Flickr photos, provides GeoURL links and Degree Confluences nearby the website your viewing.
• GeoURL
• Opens useful sites for pages geographically marked with ICBM or geo.position META tags.
• Photo Map
• Display user contributed photos on a map...

Track Packages

• PackageMapping.com Extension
• Track and Map packages from the right-click menu.
  

American Hospital Directory: Summary Hospital Data Maps

The American Hospital Directory (AHD) now supplements their public hospital data with Google-powered maps. I (and many of our health care researchers) have used AHD for hospital data, but the ability to access and view the hospital data through maps is fantastic and so very useful.

Check out their map of the 596 Texas hospitals in their database.

Hospitals are broken up into 8 categories:

1. Short term acute care
2. Critical access
3. Psychiatric
4. Long term
5. Rehabilitation
6. Children's
7. Other
8. Unknown

Great data is provided for each hospital, including revenue, number of beds, discharges, type of service, inpatient origin, financial statistics, and more. There are a number of subscription services available on their site, but this free data is great.

Here is the press release announcing their new mapping service.

Position: GIS Librarian (University of Illinois Library at Urbana-Champaign)

Position: GIS Librarian (University of Illinois Library at Urbana-Champaign)

"The University of Illinois Library is seeking an energetic and creative person to serve as the lead in the Library's collection and delivery of digital geospatial data and associated geographic information systems (GIS) services. The GIS Librarian will coordinate all aspects of the Library's digital geospatial information program at the intersection of user needs, technology, and data content. Creating a portal to geospatial data, the GIS Librarian will move the profession and services offered from traditional map librarianship towards geoinformatics with an increased interest in modeling geospatial data and in techniques applied to geospatial information systems for data management, retrieval, and analysis."

What a position! I have not posted a position here in a while as my regular blogging habits gave way to the business of this semester, but this position seems so exciting that I can not help myself... The folks over at Urbana-Champaign sure know what it takes to define the role of a GIS Librarian. If anyone out there is unsure about such a role, this job announcement is a fantastic concise summary.

Here are some snippets from the job announcement:

• "This is a full-time faculty position in the Map and Geography Library of the University Library."
• "Provides reference service, research assistance, and instruction in the selection and possible uses of digital geospatial data through in-person and remote one-on-one interactions, workshops/seminars, websites, blogs, and other avenues of widely distributed communication..."
• "Works with teaching faculty to implement GIS modules in courses. Prepares datasets to support course assignments."
  
• "GIS Librarian will craft a digital geospatial data collection development plan and will develop a campus clearinghouse and archive for digital geospatial data."
• "GIS Librarian will describe acquired data using appropriate metadata schema and mount and maintain data on Library or University servers, providing access to the campus community at large."
• "Serves as Library's contact person for campus site-license software such as ESRI and ERDAS
  products."
• "Develops and maintains close liaison relationships with local and state geospatial data producers."

There is no specific mention of a salary here, but considering the requirements and the high research level of the university, the salary should be nice.

GIS-Related Dissertations: Latest Batch

Here's the latest installment of GIS-related dissertations that have caught my eye. Previous lists include 11/05, 01/06, 01/06, 04/06, 07/06, 10/06. (Of course, this is not a comprehensive list...only those I find interesting.)

Now, I will continue to link to the ProQuest Digital Dissertations public database, which provides free abstracts and 24 page previews for many dissertations and theses. However, due to Proquest's migration to the ProQuest Dissertations & Theses (PQDT) database, the public database will provide only brief citations beginning on July 21, 2007. I will link to full-text versions if Google can point them out to me.

• Airfare, competition, and spatial structure: New evidence in the United States airline deregulation, by Gong, Gang, Ph.D., Boston University, 2006, 170 pages.
• "The dynamics of airline deregulation has caused dramatic changes in airfare and competition structure... The spatial distribution of airfare has not been even. Pricing dynamics have resulted in geographic patterns of lower airfare for cities in the west and southwestern United States while higher airfare was found in the South, New England, and Midwest."
• I find this research extremely interesting and I bet a lot of folks would find this intriguing as well. Perhaps if I can pull the data together, this would make an excellent workshop next Fall 2007 semester. See here for Spring 2007 workshops.
  
• Communal ontology for navigation support in urban region: Getting directions from familiar landmarks, by Hong, Ilyoung, PhD, State University of New York at Buffalo, 2007.
• "This dissertation proposes a communal ontology as a type of regional knowledge with a formal structure that can be incorporated with geographical information systems. As part of an effort toward the realization of community wayfinding, this research explores several methodologies. To figure out what the shared geographical places are, the preference and degree of familiarity of different places are measured using the behavioral geographer's methodology. For investigating similar geographical interests within a community, social network analysis is conducted with the help of a person-place matrix and centrality measures are calculated."
• A co-evolutionary cellular automata for the integration of spatial and temporal scales in forest management planning, by Mathey, Anne-Helene, PhD, The University of British Columbia, 2006.
• "This thesis presents a case for more holistic numerical planning tools which can handle spatial objectives and inter-temporal trade-offs. A novel algorithm based on cellular automata (CA) is designed to address forest planning objectives that are both spatial and temporal and subject to global constraints."
• A geographic information system prototype for archived data from intelligent transportation systems: A multidimensional analysis, by Cusack, Maggie, PhD, State University of New York at Albany, 2006.
• "This work suggests a GIS prototype that will exploit existing industry data collection technologies, and apply sound Information Science (IS) principles to a growing transportation industry database problem. The prototype demonstrates a rational approach to applying those principals to the ITS data archiving and retrieval problem, with emphasis on the possibilities for data analysis."
• Archaeological predictive model of southwestern Kansas, by Campbell, Joshua Stewart, M.A., The University of Kansas, 2006, 131 pages.
• "Knowledge on the archaeological condition of southwestern Kansas is anomalously low, therefore a high-resolution archaeological predictive model has been constructed for the High Plains region of southwestern Kansas. Using quantitative data about the environment as independent variables, the model was constructed using a combination of Geographic Information Systems (GIS) and statistical software."
• Association between ozone and emergency department visits: Application of geostatistics and geographic information systems (GIS), by Choi, Mona, Ph.D., University of Maryland, Baltimore, 2006, 123 pages.
• "Using traditional statistics and geostatistics in combination with GIS, the association between ozone concentration and emergency department (ED) admissions for cardiovascular and respiratory conditions were examined at the ZIP code level... Findings suggest that respiratory and cardiovascular ED visits increased even at lower ozone concentration than the EPA's air quality standards."
• A Web-based spatial decision support system for utilizing organic wastes as renewable energy resources in New York State, by Ma, Jianguo, Ph.D., Cornell University, 2006, 129 pages.
• "As the 3rd largest dairy state in the nation and the host for many food waste generators, New York State produces a large amount of organic waste. Recently there has been a renewed interest in farm-based co-digestion, which has created strong needs for research in this field... [A] Web-based spatial decision support system (SDSS) is developed by integrating geographic information systems (GIS), the Internet, and modeling. ArcGIS, Manifold, VB.Net, JavaScript and HTML are used during the design process. This system consists of three modules: (1) Dynamic Mapping and Querying; (2) Food Waste Estimator; and (3) Co-digestion Economic Analysis."
• Bahamian cave and karst geodatabase, and GIS analysis of San Salvador Island, Bahamas, by Walker, Adam Dennis, M.S., Mississippi State University, 2006, 94 pages.
• Full-Text
• "A geodatabase and a data management program have been created to store and manipulate cave and karst feature data from the Bahamas. A geographic information system was used to recognize any spatial patterns in the cave and karst data from San Salvador Island."
• Comprehensive conservation modeling: A spatially explicit individual-based approach using grizzly bears as a case study, by Backus, Vickie Marie, Ph.D., The University of Utah, 2006, 239 pages.
• "This dissertation illustrates how a mechanistic bottom-up approach to constructing a spatially explicit individual-based model (IBM) provides the proper theoretical and operational frameworks for constructing population viability analysis (PVA) models that avoid many of the substantive and theoretical criticism of the conventional demographic models used in PVA. Using Java™, such a model is developed for the grizzly bear population of the Cabinet-Yaak Ecosystem."
• Creation of a system for assessing and communicating the risks associated with terrestrial chemical spills, by Bryant, Derek L., Ph.D., Vanderbilt University, 2006, 85 pages.
  
• Full-Text
• "Adequately preparing for and responding to potential terrestrial (land-based) chemical spills are critical to the protection of human health and ecology. In this research, an environmental risk management system is developed to support analysis and facilitate decision-making for terrestrial chemical spill planning and response... The system leverages geographic information systems (GIS) technology to assess and delineate the immediate threat to human and environmental receptors from terrestrial chemical spills. It characterizes a spilled chemical's ability to immediately impact human health, groundwater, surface water, and soil resources, and incorporates these four receptors into an overall measure of terrestrial chemical risk."

Mapping & Datasets @ ACRL

Attended the Association of College & Research Libraries (ACRL) Annual Conference a couple of weeks ago. There were a few mention-worthy poster sessions and vendors related to mapping and datasets and I will outline these here.

Poster Sessions

• Historic Images, Google Maps, and Your Digital Collections
• Presenters:
• Bartholomew Schmidt, Digital Projects Librarian, Cowles Library, Drake University
• Scott Phillips, Web Developer, Cowles Library, Drake University
• Download handout
• These guys showcased the Google Map interface they "used to increase the visibility, browsability and usability of [their] Historic Des Moines digital collection." Their project is entitled: Map of Historic Des Moines. They assigned a Google marker for a digital collection of app. 115 photographs. The marker info windows contain the photograph title, a thumbnail image, and a link to a details page. The details page contains a zoomable image and metadata.
• Scott told me that they programmed the API themselves, and their poster/handout showed how YourGMap can also be used without any need to bother with javascript at all. Of course, this was before Google created My Maps.

• Who's Counting? Using the US Census to Introduce Information Literacy into a Beginning Statistics Course
• Presenter:
• Ruth H. Hamann, Reference Librarian, Learning Resources Center, Passaic County Community College
• This was a nifty idea where the library teamed up with the statistics instructor to show the students how to access census data from the American Factfinder website. The students, then, performed statistical analysis on actual data they gathered as opposed to sample datasets provided by book vendors.
• I like this idea very much as it is an outreach approach I have not yet tried. I do not believe I have yet had any interactions with undergraduate math students.
  

• Documents Directly to the People: Using Podcasts to Promote Government Documents Collections
• Presenter:
• Newkirk Barnes, Assistant Professor/Government Documents Librarian, Mitchell Memorial Library, Mississippi State University
• Download handout
• "This poster session describes the Mississippi State University (MSU) Libraries’ podcasting activities, and the Government Documents and Microforms Department’s contribution to these efforts."
• I was intrigued by this as I have often toyed around with the idea of creating a geospatial podcast geared toward our researchers here at UT Arlington. I have always been skeptical whether anyone would even listen to us. I am also a bit daunted by the workload. So many of the datasets we use are government-produced, and so it is encouraging to see a successful government documents podcast.
• Newkirk stated that the project is relatively new, but that it shows promise. I plan on following up with her to see how well it goes.
• Here is a link to Mississippi State University Libraries Podcasts.
  

• OneBook Meets Google Maps: Engaging Freshman in English Composition Library Instruction
• Presenters:
• Joshua Been, GIS Librarian, University of Texas at Arlington Library
• Gretchen Trkay, Instruction & Information Literacy Librarian, University of Texas at Arlington Library
• Handouts (yes, we made 3 of 'em):
• OneBook Meets Google Maps: Engaging Freshmen in English Composition Library Instruction
• Mapping the Afghan Experience: GIS, Google Maps, & Library Instruction
• GIS & Google Maps @ Your Library: A Do-It-Yourself Guide to Integrating GIS Data & Google Maps
• Our poster presentation showcased our inclusion of GIS and demographic data into library instruction sessions for 15 freshman composition courses last semester. We developed the following web application, Mapping the Afghan Experience in the US. Click here for more information about this project.

Vendor Booths

• Geographic Research
  
• They have a product entitled SimplyMap which allows online and interactive access to Census data, as well as EASI and AGS data. The interface is an interactive map environment, where users can easily view demographic distributions down to the block group level for anywhere in the US, and then export reports and maps.
  
• The demonstration of this product was very impressive. The potential of this product is quite high and I am looking forward to playing around with it.
  
• We are in the process of setting up a trial. I will post more about SimplyMap after the trial starts up.

• LexisNexis
• Was told by the rep that there is a new product planned to be released this summer called the U.S. Serial Set Map Module. This module will be a separate product from their Serial Set collection, containing high resolution images of maps from the Serial Set in full color.
  
• Currently, Lexis-Nexis' digital Serial Set product allows users to search the map collection and retrieve low-resolution black/white images. Now, my library has not purchased this product, and I only know this because the reps were kind enough to let me play.
• It would be incredible if there can even be an Export to GIS function...
• The rep told me there would be more information and perhaps even a working demo at ALA this June. I will not be there, however, and so will beg one of my colleagues to ask them about the product for me.
  

• Proquest-CSA
• There is a new intriguing product called Illustrata.
• "Researchers can now specifically search information presented in tables, figures, charts, and other illustrations within scholarly articles"
• Independent of any other CSA product a library subscribes to, Illustrata individually indexes and provides access to tables, figures, charts, and even maps. The demo shows that each item returned also contains a brief abstract of the source article and a blurb about the image or table returned.
  

• ESRI
• Spent a few minutes chatting with the ESRI rep about GIS in libraries/education, and about new software applications. Most notably their new (relatively) Image Server. I do not foresee a need for the library as ArcIMS is meeting our image serving needs, but I passed along the materials to various faculty who might be interested.

Census 2010 - Less Local US Data & ACS Insufficient

Census 2010 is right around the corner, folks.

Have been hearing numerous faculty and students anxiously awaiting for fresh new Census data, as the 2000 data has gotten quite stale and the taste of marketing estimates is never as satisfying. However, the joyous party times we experienced during the early 2000's as summary files 3 and 4 were being released are not to be repeated this time around. As I will discuss below, this has huge implications for all users of demographic data and this in turn will have a huge affect on my role as GIS Librarian.

Census 2010: No Long Form

So, the US Census has gone and submitted the subjects for the 2010 Census to Congress. There will be only 6 subjects. The data for these subjects is planned to be released on the Census block level on April 1, 2011. Here are the 6 subjects:

1. gender
2. age
3. race
4. ethnicity
5. relationship (to the head of household)
   
6. whether you own or rent your home (tenure)

What?

How about the long form questions? How about income, education, citizenship, all those housing characteristics? Since 1960, sample data has been collected on the Census long form sent out to 1/6 households containing oodles of data. Data that I and many others have come to rely upon.

There will be no long form questionnaire in 2010. To cut costs, the US Census Bureau is now relying on the American Community Survey (ACS) instead of the decennial census to provide all questions previously on the long form.

This is not good news, especially in the short term. I have a big problem with the ACS replacing the long form data previously released each decennial census since 1960. The implications scare me.

Let's Take a Closer Look at the ACS

The American Community Survey last year began providing access to annual estimates of various demographics for all cities (places), counties, congressional districts, school districts, and states in the US.

Now, the upside to the ACS as compared to long form decennial census data is the timeliness. No longer will we need to wait until next decade to find the median household income for our fair city of Arlington. The ACS will release this data annually.

The downsides, however, are quite steep. First, the ACS is currently releasing data down to the county and city level. It is extremely rare that our students here use census data on any level other than block, block group, tract, or zip code tabulation areas. However, according to Chris Williamson, Ph.D., Senior Planner for the City of Oxnard, California, on the latest Planetizen Podcast (which is efficiently informative), as soon as the ACS has accumulated enough data they will begin releasing data on the tract level. In am uncertain when. Regardless of when, we have all been enjoying long form data down to the block group level.

Second, the margin of error for ACS data can be quite high. When using ACS data, it is imperative to consider the margin for error that is provided for each attribute for each geography. Now, Dr. Williamson (Planetizen Podcast) stated that the accuracy of ACS data will improve over time, but for the short-term this is what we have.

ACS Data can be accessed via Factfinder, FTP download, and as data briefs (URL fixed)

Two Ways This Will Impact GIS At My Campus

1. Possibly stifle the use of GIS
   
1. Over the last two years, GIS activity on campus has blossomed as more and more faculty and students are embracing geospatial technologies to supplement their research. For many new social science, social work, health care, urban planning, and business users, the ease of accessing census data and joining it to TIGER shapefiles is a huge draw. The ease substantially decreases if the new users must consider the margin of error before using ACS data.
2. Losing the ability to allow users to analyze official census data on the block group level (beyond the basic six subjects). As I discussed above, ACS is currently available to the city (place) level, and there are plans to release data on the tract level.
   
2. Increased reliance on unofficial and less accurate estimates
1. Faculty and students will not be willing to give up their block group level data. We (the library) will have to provide it to them. I assume (hope) companies such as Applied Geographic Solutions will continue to create current and 5 year estimates, and of course we will continue to collect this unofficial data. However, if the census is publishing data only down to the city or tract level, and if the margin for error is higher than ever before, the error of these unofficial estimates will increase. At the same time our users' reliance on these unofficial datasets will increase as ACS is insufficient to satisfy their needs. Perhaps the timeliness of the ACS will help to counteract this.

---

Now, I understand the Census Bureau (which means us, the taxpayers) is saving money by shifting from the decennial long form to the American Community Survey, but is it really worth it. Perhaps if I knew the amount of the savings it would make more sense to me. There are a whole lot of bright folks working at the census, and I reckon if they decided this is the best thing to do, then perhaps it is.

Oak Cliff Interactive - Student ArcMap2GMap Project

Student-Created Map: Oak Cliff Interactive

Want to discuss this political science student's work to show an example of the effectiveness of our library's GIS workshop program and our endeavor to create customized tools to increase the access and attractiveness of geospatial resources.

Have been working this semester with Charles Jackson, an enthusiastic political science student to develop an interactive map for an undergraduate public affairs course. Charles, (who gave me permission to discuss his project here), first attended the Google Maps workshop I held earlier this semester. He was so excited about the idea of creating an interactive map that he spent countless hours (at least 80) in the GIS lab learning how to access business location information, various demographic datasets, pull them all into ArcMap, and create an interactive map using our ArcMap2GMap script. I tell you, his enthusiasm was so infectious and always got other students (at nearby computers) and myself excited as well.

Oak Cliff is an old neighborhood south of Dallas, and Charles is working to develop a revitalization plan. The Oak Cliff Interactive map is the community services component. He explained to me numerous times that there has never before been a comprehensive analysis of the services/resources available to Oak Cliff's residents. This interactive map allows users to view various business, non-business, and housing facilities in Oak Cliff. If an address is geocoded, the 10 closest facilities of each visible facility type is listed, along with links to driving directions.

Hopefully, the professor will be impressed enough to begin encouraging future students to integrate geospatial resources within their projects. I will follow up with the professor and see what happens...

Texas Redistricting & Workshop - Texas Redistricting

# Google Map: Redistricting in Texas

# Workshop Materials: CD Image (350mb)

---

Held our library's 3rd and final workshop of the semester, entitled Congressional Redistricting in Texas: The Maps & The Demographics. It was a blast as we had 15 students, 4 staff, and 2 faculty members attending. Everyone seemed to have a good time.

The 2-hour workshop was organized into three sections.

1. Dr. Jose Gutierrez, political science faculty, first gave an outstanding 30-minute introduction to the redistricting process in Texas, with an emphasis on the 2003 Texas redistricting and the 2006 Supreme Court decision that required the Texas legislature to redesign District 23. Jose is a fantastic speaker and it I learned a lot.
2. Spent approximately 30 minutes exploring local voting results and congressional district boundaries using the UT Arlington Library: Texas Redistricting interactive map. Using the same interface design as Mapping the Afghan Experience and the developing Texas Time Machine. The Texas Redistricting Google Map (image above) incorporates voter tabulation district (VTD) election data from the 2000 & 2004 presidential elections, and the 2006 gubernatorial election. There are app. 8,500 VTDs in Texas, so the data allows for local analysis. Not quite as numerous as block groups (app. 14,500 in Texas), but still quite small.
3. Spent the remaining 45 minutes using ArcMap.
1. First, we showed everyone how to use the spatial join function to find the % Hispanic population (from block group Census data) in the infamous District 23 in the 108th Congress (before the 2003 redistricting), the 109th Congress (results of the 2003 redistricting), and the 110th Congress (results of the Supreme Court demand to redraw the Congressional district boundaries. This was quite effective as the percentage dropped over 10% from the 108th to the 109th.


2. Second, we ran through an exercise where participants adjusted the 109th Congressional Districts based on their preferences, whether they be biased or impartial. No programming was necessary for this automation. I was able to create it rather quickly using ArcMap's Model Builder. I implemented an extremely simple algorithm making use of ArcMap's built-in mean center function and then creating a simple voronoi polygon map. Now, our campus has not yet upgraded to ArcGIS 9.2, so I used the euclidian allocation function instead of the voronoi map functions included in 9.2.

So, where did all of this great data come from? Mostly from the Texas Legislative Council's Redistricting FTP Site. Note that Firefox has some issues with this FTP site, so if you use a browser to access FTP, use IE. For more details about data sources, see the workshop handout, which is basically a bibliography of relevant data sources.

Free Historical Census Data: 1970, 1980, 1990

Received a request over the weekend for 1990 Census data on the block level beyond what is currently available through our local holdings. We are purchasing the CensusCD 1990 Blocks from Geolytics, but in the meanwhile this forced me to take some time exploring free historical Census data availability and so here are the results of what I discovered.

U.S. Census Bureau

• Data Files: Census 1980
  
• Data Files: Census 1990
• These downloadable data files are fantastic in that users can freely download the entire 1980 and 1990 dataset, including all the summary tape files (STF). Downside is they are not easy to use. Choice is to install the Extract software or access the DBF files yourself. In the past I imported the DBF file containing the STF 1B for Texas into an Access database and made that database available to our library users, who would then run queries against the data. (Access, of course, will be much improved when we acquire CensusCD 1990 Blocks.
• American Factfinder
• Everyone loves this fantastic resource, but it only offers data from 1990 and 2000. Their 1990 holdings do not contain STF 1B, and so do not go down to the block level.

University of Wisconsin-Madison Center for Demography and Ecology

• 1970 Census Summary Tape Files
• Files for all available states and the District of Columbia for the 1970 4th Count A and C population and housing files may be downloaded.
• Current Population Surveys (various, from 1963 - 1990)
• 1980 Public Use Microdata Sample (PUMS)
  

UC Berkeley Social Science and Government Data Library: Files to FTP

• Been having trouble accessing their public archive, which is too bad as it has been a fantastic resource. Seems as if their FTP server ftp://sunsite2.berkeley.edu/pub dis not permitting public anonymous logon. Their FTP server ftp://sunsite.berkeley.edu/pub is permitting anonymous logon, but does not contain their data. Hopefully their data will be made available to the public again.

Interuniversity Consortium for Political and Social Research (ICPSR)

• This is not a free resource, but it seems silly comprising this list without mentioning the ICPSR. Many (if not most) academic universities should be a member of ICPSR and have access to its data holdings.
• Here are their Decennial Census holdings.
  

Public Transportation Workshop a Success!

Here is an example of a bus route developed by a workshop participant.

20 folks attended to hear Dr. Anjomani discuss student endeavors to plan transportation routes and to then plan their own during this hands-on GIS workshop. For more information about this workshop, see this previous post.

Everyone had a lot of fun, bunch of folks stayed afterwards with questions, and I am already booked to repeat the workshop next week specifically for the City & Regional Planning GIS folks. Everyone generated their cost matrices, used some on-the-fly weighted mean techniques to position the stops, and then created the bus route. Then, of course, the script called ArcMap2GMap to overlay the stops and route via Google Maps.

As an added bonus, the event was covered in The Shorthorn, our student-run newspaper (see image). Click here (pdf) to view the entire issue from Friday, March 2.

As I promised earlier in the week, the workshop materials, including the Python scripts, can be downloaded here. This is actually the zipped contents of the CDs that each participant received. Unzip the contents and launch the Public_Transportation_in_Arlington_Plan_the_Route.exe file (which would have auto-run if you had the CD). ArcGIS 9.1 is required. It might work with 9.2, but I have not tested it.

ACRL Poster Presentation: ArcIMS, Google Maps, & Information Literacy

As I wrote a couple of days ago, Gretchen Trkay, the Information Literacy librarian and I needed to get the handouts in this week for our ACRL Poster Presentation: OneBook Meets Google Maps: Engaging Freshman in English Composition Library Instruction.

As you can see from the 4 handouts below, our presentation focuses on how we integrated ArcIMS 9.1 services with Google Maps to develop a web application for freshman English Composition courses. Here is the web application: Mapping the Afghan Population of the US.

Well, we submitted our four handouts this afternoon, and it feels good to get those documents complete and shipped out.

Here are the 4 handouts:

• OneBook Meets Google Maps: Engaging Freshmen in English Composition Library Instruction
• Mapping the Afghan Experience: GIS, Google Maps, & Library Instruction
• GIS & Google Maps @ Your Library: A Do-It-Yourself Guide to Integrating GIS Data & Google Maps
• Handout Given to Students: ENGL 1301 Kite Runner Library Activity
  

I am really looking forward to presenting the results of our collaboration in Baltimore later this month. I have done quite a number of conference presentations, but never a poster presentation before. Should be fun.

Free Batch Geocoding: Juice Analytics Excel Geocoding Tool

As Meg Stewart stated in the GIS @ Vassar blog, "Geocoding in ArcGIS is not pleasant. Success rates hit a high of about 70 percent, in my experience." We all feel your pain, especially those of us in education where we are constantly put in the position where we need to explain this geocoding unpleasantness at least once per week.

The GIS @ Vassar post points out the fantastic BatchGeocode.com, which makes up a major component of my My Powerful Geospatial Suite of Free GIS.

However, lately I have been directing students to download and use the Juice Analytics: Excel Geocoding Tool. I first learned of this tool via Ogle Earth last year, but for whatever reason, it is only lately that I have really begun appreciating it.

The Excel Geocoding Tool is a downloadable Excel file with super-easy-to-use Macros built into it, so of course you need to enable Macros to use this tool. There are two worksheets. The first worksheet prompts which geocoding service to use (Yahoo! API or geocoder.us). Yeah...who's going to use geocoder.us when compared to Yahoo! Maps API? The second sheet has fields for address, city, state, and zip. Paste tens, hundreds, or thousands (I have never done more than 2,000) of addresses into this sheet, click Geocode All Rows, and voila. At lightening speeds compared to BatchGeocode.com, the lat/long of each address appear in the first two columns. The 500 addresses per geocode limit in BatchGeocode.com is nowhere to be seen with this tool. Another advantage is this Excel tool provides the geocoding precision level for each address, which is truly, truly wonderful. Not nearly as powerful as ArcMap's geocoding score, but for each geocoded address, this tool specifies whether the supplied lat/long is matched for the specific address, street, zip code, or city. Just like BatchGeocode.com, the Excel Geocoding Tool can even create KML files.

I introduced my Intro to GIS class to the Excel Geocoding Tool last week, and as you can imagine, it was a big hit.

March 2007 Conference Presentations

If anyone wants to drop by and say hello, I will be holding the following two conference presentations during March:

Texas Map Society 2007 Spring Meeting
Location: Nacagdoches, TX
Dates: March 23-24
Presentation : Historical Maps & GIS: Peeling the Cartographic Layers
I will be discussing our library's current endeavors to georeference and digitize historical Texas maps and distribute the maps and vector data via Google Maps and Google Earth.

ACRL 13th National Conference
Location: Baltimore, MD
Dates: March 29 - April 1
Poster Presentation: OneBook Meets Google Maps: Engaging Freshman in English Composition Library Instruction (co-presented with Gretchen Trkay)
Our poster presentation will highlight our inclusion of GIS and demographic data into library instruction sessions for 15 freshman composition courses last semester. We developed the following web application, Mapping the Afghan Experience in the US. Click here for more information about this project.

I will post more details about these presentations as the dates get closer. Like most folks, I tend to wait until the last possible moment. We actually need to submit electronic copies of all handouts for the poster presentation for ACRL this week, so I will post more details within the next two days...

ArcMap2GMap for ArcGIS 9.0, 9.1, and 9.2

As I just posted earlier, I have devised band-aid solutions that will allow all of my scripts to operate in ArcGIS 9.whatever, but that I must create separate scripts for each version.


Just created ArcMap2GMap scripts for ArcGIS 9.2 and 9.0, in addition to 9.1. Click here to download from ArcScripts.

Native Support for Geoprocessor in ArcGIS 9.2...Doh!

OK, so finally re-installed ArcGIS 9.2 after my rash initial installation went awry. Now experiencing first-hand that all of the Python scripts I developed in for ArcGIS 9.0 & 9.1 do not work in ArcGIS 9.2.

Why? As the ESRI documentation states here, "At ArcGIS 9.2, there is native Python support for geoprocessing scripting."

What does this mean in practical terms? Replace the COM connection code at the tippety-top of your scripts.


Replace:
import win32com.client
gp = win32com.client.Dispatch("esriGeoprocessing.GpDispatch.1")

With:
import arcgisscripting
gp = arcgisscripting.create()

Now, how to make a single script that will work with either ArcGIS 9.1 or 9.2? I really do not know. The only method that I am aware of to detect the version number is to pull that info from the registry. For me, that is way, way not worth it. Until a nice solution comes along, I will create different scripts for different versions.

While I am discussing Python scripting differences, there is still an unresolved issue (for me) using the searchcursor to access the geometry object for points between 9.0 and 9.1/9.2. Sent a query a while back to the ArcView-L list, but no one was able to help. Which is a shame because the last time I posted a scripting query (about constructing multipart polygons and inner circles) I received a fantastic answer within a couple of days from Nathan Warmer (ESRI).

Anyway, here is a snippet that highlights the difference:

rows = gp.SearchCursor(inputFC)
row = rows.Next()
# For each row
while row:
feat = row.shape
LUArray = feat.GetPart()
# Following line is required for 9.1 & 9.2
pnt = LUArray
# Following 2 lines must be removed for version 9.1 & 9.2 They are essential for 9.0.
# LUArray.Reset()
# pnt = LUArray.Next()
row = rows.Next()
...

I do see that it is no longer necessary to store point features in object arrays, but I have been unable to devise a solution that would work seamlessly across all ArcGIS versions.

Anyway, just thought I would post this difference here while I had it in my mind.

Plan the Route: Arlington TX Largest City Without Public Transit

"Arlington [Texas] remains the largest metropolitan city in the country without a public transit system for the general population."
City of Arlington, 2001

So, I live, work, and play in the largest city in the United States without public transportation, eh? Now to be honest, I bike everywhere on my Trek 7100, but not everyone is fortunate enough to live so close to life's many amenities and short destinations.

The great up-side to this is that I have worked with oodles of students from our City & Regional Planning GIS program wanting to plan public transit routes through Arlington, TX. Working with these students over the past few years have given me a real appreciation of how ArcGIS's spatial analyst and lately (since 9.1) the network analyst extension can be used. One of the students who I helped to devise a method for implementing a bus routing algorithm into ArcMap actually won an award for his work. Very exciting. Of course, his project was planning a bus route through our fair city of Arlington.

So, I thought I'd try my hand at holding a GIS workshop where participants will use GIS to plan a new bus route route through Arlington. Now, the primary purpose of my workshops is not to demonstrate advanced analysis techniques to GIS students, but to increase awareness and interest in GIS and the library's services and collections. This means that in order to hold such a workshop, I needed to automate everything. A students (or faculty) who beforehand would have difficulty spelling GIS would need to create a bus route within 1 to 1.5 hours.

So I fire up my Pywin32 IDE and just about finished up the Python geoprocessing yesterday. Everything is completely, 100%, automated. Today at noon, a couple of my colleagues here in the library agreed to meet with me to do some quick usability testing.

As soon as the testing is complete, and I put the finishing touches on the user interface, I will post all of the code here.

I divided the analysis into two forms. The forms are simple VBA forms that launch the scripts and pass the parameters. First, workshop participants will develop their cost matrix using the following intuitive criteria: number of commuters, number of single mothers, income, and road class. Second, participants will select 4 categories of bus stops, such as airports, colleges, grocery stores, etc. They can then assign specific stops within these categories or allow the GIS to pick a specific stop for them. A cost raster (which is also an output from the first step) is used to select the stop(s) for the participant. Then, because I could not help myself, the Python code calls ArcMap2GMap and plots the bus stops and the derived route on a Google Map that they can take with themselves. Works like a charm. The only downside is that the second step takes over 3 minutes to process, so I will need to put on all of my charm to keep everyone engaged and entertained while we wait for the process to finish. The results are totally worth it, though as the default browser window opens automatically with Google Maps displaying their results.

Then, last week, Dr. Ardeshir Anjomani, coordinator of the our City & Regional Planning department's GIS program, agreed to give a brief introduction to the topic and to highlight various student public transit projects that have come out of his department. It's going to be fantastic...

If anyone is in the area, here are the workshop details:

Title: Public Transportation in Arlington: Plan the Route
Introduction: Dr. Ardeshir Anjomani, coordinator of SUPA's City & Regional Planning GIS Program, will give a brief introduction and highlight various student endeavors to plan public transit routes.
Time: Thursday, March 1, 2007, 3pm - 5pm
Location: Central Library, Room B20 (basement)
Description: Learn how to use Geographic Information Systems to plan a public transportation (bus) route through Arlington, TX.
Flyer: http://gis.uta.edu/arlingtonTransit.pdf

TIGER Positional Accuracy & Geocoding: 2 Articles

The following two recent articles focusing on positional accuracy and geocoding are very intriguing. Learned quite a bit about the science behind positional accuracy and geocoding. I usually advise our students to use our local parcel boundary feature class for geocoding (as I discuss here), but I'll focus first on these articles.

• A Snake-based Approach for TIGER Road Data Conflation, Song, Wenbo; Haithcoat, Timothy L.; Keller, James M. Cartography and Geographic Information Science, Volume 33, Number 4, October 2006, pp. 287-298(12) [abstract]


• Modeling the probability distribution of positional errors incurred by residential address geocoding, Dale L Zimmerman, Xiangming Fang, Soumya Mazumdar and Gerard Rushton, International Journal of Health Geographics 2007, 6:1 [full-text]

The first article describes a new conflation method for improving the positional accuracy of TIGER files. As described by this article, conflation is the combining of attribute-rich TIGER data with positionally-superior local data. Traditional conflation methods include (1) feature matching, (2) map alignment (rubber-sheeting), and (3) attribute transfers. The article describes a new conflation method, based on snakes. Snakes [background information] are active (dynamic) contour models based on image or 3D data. See Mark Schulze's nifty java tool to learn more about snakes. Traditional conflation are between TIGER and other vector files. This new method extends conflation to interact between TIGER and snakes based on raster orthophotos. This article was a fantastic insight into the creation of the improved vector street files we enjoy here at the university as a result of the hard work of local government analysts (and their contractors).

The second article presents the results of comparing three geocoding methods: (1) Automated Geocodes, (2) E911 Geocodes, (3) Use of an Orthophoto. The automated geocoding was accomplished using TIGER data and ArcGIS 9.1. The E911 geocoding used the local 911 listing. The orthophoto method overlaid parcel boundaries over an orthophoto to enhance the E911 geocode. As expected, the orthophoto method was the most accurate, and was termed the "gold standard". The real beauty of this article is the analysis of the error measurements and their attempts to model the errors.

So, how does this impact how I do my job? Reckon the best is to continue to advise students to geocode based on our local government-produced parcel shapefile or cadastral. If I was not fortunate to work in the GIS-rich DFW Metroplex, and so did not have access to these high quality files? Guess then I would try to work with local faculty and governments to work on conflation techniques to improve the accuracy of TIGER data.

FYI, here is a good resource to scan the recent table of contents of the major GIS-related journals.