Evaluation

Measuring the Project, not the Data

Objectives of lecture:

1. Celebrate GIS Day
2. Prepare for Review (Friday)
   • Materials: List of terms, Glossary of book, Worksheet on Measurement frameworks, Worksheets on measurement and reference systems (.doc), Original form of Objectives,
   • reminder of grading policies
3. Technical Evaluation (a form of review of the whole quarter so far)
   Measuring Data Quality
   Allocation of Resources
   Computer, Financial, Personnel
   Limits to Benefit/Cost
4. Land Inventory as an example of evaluation choices

Technical Evaluation

Evaluation compares results to intended goals.

Often, what is measured is the component of utmost interest. (Since the choices to control reduce variability, this makes sense.)

Data Quality concerns evaluate the technical result of the system from measurement through representation, operations and all transformations.

It is always possible to do better on data quality... Projects must be done with incomplete data, inadequate data.

BIG QUESTION: When would the flaws and inadequacies in the data get SO bad that you would have to tell them the project is not feasible?

Allocation of Resources

A manager of a GIS will be judged on other criteria than simply the data quality.

• Computer Resources
  • How much storage?
  • How much processing time?
  • How quickly delivered?
• Financial Resources
  • How much money?
  • What returns on investment?
  • What 'benefits'?
• Human Resources
  • Who can do all this?
  • What training do they need?

Duplication, a major sign of misallocation?

Finland: Joint Use report, has material on allocation, compatibility, and inter-operability

Interoperability, a continuing hot-topic

• As a research issue:
  • UCGIS research priority (Interoperability white paper 1996)
  • NCGIA event (1997)
  • Not explicitly in the 2002 list, however.
• As an industry movement
  • Open GIS has its own Interoperability Program
  • Demos of interoperability

Approaches to Land Inventory

Choices about measurement for geographic phenomena relate to resolution and accuracy, basic concerns of data quality. They also raise questions of the choices involved in implementing a design.

A continuum of approaches:

Parametric extreme

Each attribute measured separately. Measurement framework chosen for that attribute separately. Multipurpose in the sense that the measurement be used each time that particular parameter is needed. The "layer cake" diagram shows this conceptually, but real layer cakes are NOT as extreme, because each map may have a lot of attributes attached.

Integrated Survey

One map as combined result of all relevant experts. Linked to a purpose that must be known before the survey starts. Origins in Australian 'land systems' regionalizations; adopted by UN Food and Agricultural Organization (1976) for rural development work. [an example]

Intermediary possibilities

Almost all projects fall somewhere along this continuum. Indirect measurement (assigning continuous attributes to categories on the map) still quite common. Soil maps and forest inventories often fall closer to the 'integrated' end. The tighter the list of attributes the more 'parametric' it is.

The tradeoffs involve:

• Time to conduct survey
• Training of staff
• Processing time to convert survey into results
  Indirect measurement & choropleth maps are easy compared to overlay
  This factor changed dramatically with GIS...
• Accuracy, resolution, etc. of result (for various purposes...)