To Catch a Thief - Made the Front Page!!

Holey mackerel...the To Catch a Thief contest and workshop has been a huge success. Here is a list of what has happened so far:

• This morning, The Shorthorn (student-run newspaper) featured a nice article right right on the front page!! That has got to drive up student participation, eh??
• A criminology professor (cited in the article) has invited me to come and meet with her victimology and forensics classes to discuss how GIS can be used in their research. This is my first ever breakthrough to form a solid contact with one of our criminology faculty, and this is just the start of something great.
• The first person who Played to Win! (solved the means of transportation contest correctly) is actually an employee at the Canada Border Services Agency.
• The second of four clues (police report and accompanying map) were released today, so you too should Play to Win! If you can figure out the means of transportation used by the criminal(s) to get to the crime scenes, simply email your guess over to gislib@uta.edu.

Python Geoprocessing: Shape2Text

As I discussed here, I was developing a Python script that will convert any shapefile to a text file. This text file will then be manipulated by an external application (lat/long of each vertices will be adjusted) and then the manipulated text file needs to be converted back into a shapefile. I developed the first third of this process, which converts the original shapefile to a text file. Work on the other two is nearing completion as well.

Well, I need to complete this project and have not really touched or thought about it for over a month. Consider this post an effort to remember and to organize my thoughts...and perhaps even organize and tighten my existing code up a bit. Like most new (budding) programmers, I am indebted to the amazing samples and assistance out there for both ESRI geoprocessing scripting and Python programming in general. If someone out there is undertaking a similar task, perhaps this will be of help to them.

There are actually three scripts that need to be developed. First is shape2text, which converts the shapefile to a text file. Second is text2exchange which converts the manipulated text file into a standard exchange formatted text file. Third is exchange2shape, which converts the standard exchange text file to a shapefile.

Python Geoprocessing Script: shape2text

So, let's focus on the first third here: converting from shape2text. Sounds like a simple enough procedure, right? Nope. Here are the problems that I had to overcome:

1. Convert polygon and line shapefiles to a table of X/Y coordinates
2. Provide flexibility for users to select the shapefile attributes they would like to be preserved in the text file (and so also in the resulting manipulated shapefile).
3. Account for the possibility of inner circles (donut holes) and multipart shapes (such as a chain of islands as one feature).
4. The format of the text file needs to be standardized as the corresponding text2exchange and exchange2shape script will need a standard format to work with.
5. The external program that will manipulate the text file by adding an adjusted latitude and longitude fields might also remove some vertices (points). New vertices will never be added, but it is possible that vertices will be removed.
6. The external application requires a fixed set of fields. The number of attributes users select to carry over is not fixed, so this flexibility must be built in such a way that the text file manipulated by the external program always contains the same number of fixed fields. (The game would be too easy without such restricitve rules, eh??)
7. All of these processes need to happen seamlessly, in sequence, from within ArcMap, and I would prefer to program this in Python as opposed to VBA/ArcObjects.

So, how to read through a polygon or polyline shapefile and write to a text file containing XY vertices? By using data cursors to access the easy-to-use geometry object. Documentation on the geometry object is provided by ESRI in the ArcGIS Desktop Help 9.1. Scroll about half way down and you will even find sample code (which I used successfully) to access the XY for each vertices in a polygon and print it to the screen. The provided script even shows you how to detect multipart and inner circle polygons. Recording these anomolies in a text file and then dutifully recreating them, however, is an entirely other matter. This was very difficult, but I was able to complete it with the help of the great folks over at ArcView-L Listserv.

So, how to account for inner circles and multipart shapes? Well, the sample script provided by ESRI (above) shows how to detect them. What I did was to have the script write a new line in the text file containing only the word NEW. These NEW lines are preserved through the text2exchange script. Preview a sample output file of three state polygon features converted to text. The exchange2shape script generates the adjusted shapefile by reading the exchange file line by line. When it hits a line containing the word NEW, it starts a new part if the FID of the next vertices do not change and a new feature if the FID does indeed change. The geometry object will automatically create either an inner circle or multipart shape depending on the orientation (direction) of the vertices in the polygon. This is actually easy as long as the original order the vertices were read by shape2text are preserved. The geoprocessing capabilities do all the work for you then.

So, how to handle the possible removal of vertices by the external application? By creating a new autonumber field, named pid (point id). (See sample output text file) This field can then be used to properly sort the vertices in the same relative order in which they were read. If points or entire features are removed by the external application, the exchange2shape will still draw the adjusted shapes as they should be drawn as long as the relative order remains.

So, how to provide the flexibility for a variable set of fields while creating a text file with a set number of predefined fields for the external application to consume? Create two text files, one of which has those minimum fields required by the external application. Preview a sample of the text file which shape2text creates specifically for the external program. The text2exchange script then recombines the two text files based on the pid number.

So, how to enable users to select the fields they wish to preserve in the adjusted shapefile at the end of the process?
First the script reads in the user-selected field names as a series of comma seperated values .

fields = sys.argv[2]

Second, a function is called to split the values into accessable variables.

def parseKeyFields(fields):
     StandardFields = fields.split(",")
     return StandardFields

Third, as the data cursor reads the original shapefile, it records the index number of those field names that match the StandardFields list (above).

for item in StandardFields:
     if str(field.Name) == item:
          StandardFieldIndex[n]= str(fieldNum+2)
          n = n + 1

This way, the script has a list that is used to determine whether a particular field gets written to the larger text file. No user-selected fields are written to the fixed fields text file which will be manipulated by the external program.

So, how to coordinate all of these activities from one ArcMap button? Here is what must seamlessly happen.

• shape2text
• launch external program
• text2exchange
• exchange2shape

I am using VBA for this by creating a geoprocessing object within the VBA instead of the more traditional method of invoking a geoprocessing tool from a button on a toolbar with parameters. This allows the VBA to call the various scripts and to wait until the script has finished running before going to the next line of code.

Dim gp As Object
Set gp = CreateObject("esriGeoprocessing.GpDispatch.1") gp.shape2text InputForm.ComboBox1.Text,
_InputForm.TextBox1.Text

Works like a charm...

Position: Map Librarian - UCSD

Map Librarian - UCSD

I have written this before, but this time I really mean it when I say that this sounds like a super-exciting position. I especially like the way that it blends print cartographic materials and GIS data/skills. Furthermore, this position is within the Social Sciences & Humanities Library, which is where my strongest mapping interests lie...

And it is in San Diego. So convenient for the ESRI conference each year, and the chance to live with the happy munchkin people of California is always fun. Just don't try to stand in one place for too long or the Earth will move you.

Here is a snippet:

Primary duties include providing reference and research consultation for cartographic materials, participating in outreach and user education programs related to maps and geospatial data, building and developing map and geospatial data collections, developing map collection web pages, and collaborating with the GIS Coordinator to support GIS services and to better integrate GIS skills into instruction and learning campus-wide. This position will also consult across the UCSD Libraries regarding current and future map collection and service needs, and work together with other University of California map librarians on collaborative purchase and cataloging projects.

The salary of $41,376 - $53,676 seems OK. Similar to a position I posted earlier, the cost of living in San Diego can be so astronomical and I am uncertain how well this salary will adjust for that cost of living. Librarianship will always be a labor of love...and I do indeed love it.

GIS4LIB: Supporting Those Who Barely Need GIS

There is a great thread going on over at the GIS4LIB Listserv focusing on what GIS services are offered to support those who barely need GIS.

While this is not the purpose of this thread, the numerous replies from a variety of libraries shine light on this oft-asked question: So what does a GIS Librarian actually do? For example, see the Geographic Information Systems (GIS) Geoblog. While metadata, cataloging, and selection are also a major component of many GIS Librarian's responsibilities, this thread does give some insight into the public services aspect.

OK, let's suppose that a 17 year old freshman comes to the library GIS lab needing to create a map showing median household income in their particular county. Let's also suppose that this student knows nothing about GIS beyond Google Earth. This is something that would take me no more than 15 minutes to do for the student. Or, do I require that the student learn the skills and resources necessary to create the map on his own. Not every library has someone on staff who can help with GIS. Would this student then be referred to a professor?

The answers are varied, depending on the resources of the particular library. As one reply within this thread noted,

Some libraries are more in the teaching mode, offering GIS workshops, one-one instruction, and in depth project assistance. I would guess that these libraries have librarians that are actually on par with GIS Technicians and are extremely savvy, devoting the bulk of their day primarily to GIS and/or maps.

But for those libraries that do not have titled 'GIS' librarians, but librarians with many other subject specialties/liaison duties of which GIS is just one very small part, teaching newbies or doing projects for those that just don't want to take the time to learn for themselves is just not practical or feasible.

We definitely aim for the first scenario above here at the University of Texas at Arlington Libraries.

Here is the original question that prompted this thread and after that I will post my reply (which I just shipped off to the list).

Hello,
I'm curious to hear feedback from the members of this list on how you handle the following situations. We encounter these sort of patrons several times a quarter, so we're interested in finding a better way to meet their needs.

Situation A - A faculty member understands conceptually that GIS can solve many of his needs for a current research project, but he doesn't have the time to learn and operate the software, so he wants to hire someone on campus to do the work for him. Where do you send him?

Situation B - Patron *just* needs to get her study area/sample points on a map for her paper/thesis/etc. She has the lat/lon coordinates of her area (or points). The patron heard that GIS might help her, but is in a hurry and doesn't want to spend extra time learning the GIS software. Do you still direct her to use GIS or are there other, simpler software that might give her the desired results faster and easier?

Situation C - A grad student has a table of addresses he needs to geocode for his professor. He says he doesn't need GIS for anything else. The professor is pressuring him to get it done yesterday because he heard that the GIS software can do it automatically. Do you sit down with him and show him how to geocode in ArcMap? Other ideas?

I look forward to reading your responses. Hopefully we can use this list to help each other provide better services.

Here was my response:

Here is how we would handle the three scenarios.

A.
First, we would discuss the needs of the faculty member to see whether we can handle the request in-house. On a few occasions, this scenario occurred and we found that what the faculty member really needed was within our resources (usually time) to complete. This has helped us to forge extremely strong ties with the faculty. If the project seems interesting, then we always find ways to make time. (Yes, this has gotten me into a lot of trouble, but I learn more from these side projects than in doing anything else. Everything always seems to get done...So far.)

Second, if we cannot handle the request in-house, we would help the faculty member draft a job description and post them around the GIS labs on campus as well as to the university GIS listserv.

B.
If the patron was indeed in a hurry, we would do it for them, but would try to reach for those teachable moments. We would do it for the student, but with them watching and explaining to them the logic of each step of the procedure. We try hard not to do any studentÂ’s work for them without at least them being present and learning at least a tad-bit about GIS. I know that GIS is a lot of fun and I always try to allow students to share my enthusiasm. Often times, there are portions of what they need that can become repetitive enough that we show them once and they are able to complete that portion before asking for our help again. If the patron was a faculty member, then we would be more likely to do it for them. Strong ties with faculty members are essential, while finding those teachable moments with students are essential.

C.
As others have said, the first thing that we would do is to contact the professor so that we can verify exactly what the professor needs (reference interview) so that we can make the professor aware of the many things that GIS and library GIS services can do for them. If the GRA was indeed correct and geocoding is all they need, then we will just run it for them.

Play to Win! - To Catch a Thief

http://library.uta.edu/gis/toCatchAThief

The day has finally arrived!

After too many hours of preparation (as usual...), the To Catch a Thief contest has begun. We have released the first of four clues (map and police report) yesterday, so anyone who can correctly guess the means of transportation used by the burglar(s) can email such guesses to gislib@uta.edu. Only UTA students/staff can actually win the ultimate grand prize (1 free plotter map or poster), but anyone who so wished can...Play to Win!.

As I mentioned before ( Jan. 20 | Oct. 14 ), this was inspired by the TV show Numb3rs.

There will be 3 additional police reports/maps released before the hands-on workshop on March 2. The workshop will focus on two spatial analysis tasks. First, we will attempt to locate other targets with a high probability of being struck by the serial burglar(s). Second, we will attempt to track down potential neighborhoods where the burglar(s) might live.

Is the scenario fake?? Of course, but the data sources are actual.

Am I well versed in spatial criminology?? No way...just having some fun. The analysis will mostly depend on multivariate regression and suitability analysis. Who knows...maybe we'll even throw in a touch of an actual algorithm used in the field...

Should a criminology professor or student attend the workshop and scoff at the methods I use, then I will consider the exercise a success. This is actually my hope. I am not aware of any of our criminology faculty or students using GIS or spatial analysis, and would love the opportunity to collaborate with them. Got my fingers crossed.

Why did I have to spend many hours developing this contest with the clues?? Well, I had to ensure that the results would be statistically significant. I had to whip out some of my old stats textbooks to calculate the best way to select targets (points) that will yield a significant correlation. Yes, the analysis will work better than it could ever work in real life. ;) That's the fun of it, eh?