Forging Forward:Using Evaluation as a Stepping Stone

Joe Matthews

SLA – San Diego Fall SeminarOctober 30, 2015

Evaluation is the process of …• Identifying and collecting data

• Focusing on specific services or activities

• Establishing criteria to gauge success

• Determining the degree to which stated goals and objectives are achieved

Types of Evaluation Activities

• Program or service planning

• Program or service monitoring

• Impact assessment (outcomes)

• Economic efficiency

Evaluation QuestionsHow much? How many? How economical? How prompt?

Magnitude

% of change

% of overall change

Costs

Magnitude

Change

Resources used

Units processed

Cycle times

Turnaround time

Anticipatory

Library Controls

Evaluation QuestionsHow valuable? How reliable? How accurate? How well?

Effort expended

Cost-benefit obtained

Dependability

Access

Accuracy

Currency

Completeness

Comprehensiveness

Accuracy

Expertise

Performance meeting expectations

Library & Customer Decide

Evaluation QuestionsHow courteous? How responsive? How satisfied?Welcoming

Attentive

Anticipatory

Helpful

Empathetic

Expectations met

Materials obtained

Personal interaction

Ease of use

Equipment used

Facilities

Willingness to return

Customers Decide

Evidence-Based Library and Information Practice

1. Formulate a clearly defined question2. Find the best evidence to answer the

question3. Critically appraise the evidence4. Combine the appraisal of the evidence with

professional judgment to make a decision5. Evaluate the effectiveness of the results6. Disseminate the results

Library Metrics

Resources Capability Outputs Outcomes Impacts

Input Measures

Process Measures

Output Measures

Outcome Measures

ImpactMeasures

Library Metrics

• National Center for Education Statistics Academic Library Survey

• Integrated Postsecondary Educational Data System (IPEDS)

• ACRL Academic Library Trends and Statistics survey

• ACRLMetrics• IMLS Public Libraries Survey• PLAMetrics

Internal Evaluation• Facilities• Technologies• Library Personnel• Collections• Services• Other

Input, Process, and Output Measures

External EvaluationLibrary’s Perspective

• Demographics and use• Collection strengths• Library personnel• Library deployed technologies• Facilities• Services provided• Unique programs and services

External Evaluation

Customer Perspective – Outcomes

• ACRL – Megan Oakleaf’s Value report• ACRL – Assessment in Action• The LibValue Project – Carol Tenopir

• PLA – Project Outcome

Measuring Satisfaction

Customer Satisfaction

In-person interview

Focus groupinterview

Exit interviewComment cards

Surveys

Complaint/Compliment analysis

Mystery shopping

Satisfaction Surveys

• Gaps Model of Service Quality – LibQUAL– Affect of service– Information control– Library as place

• Counting Opinions – LibSat• DIY

Customer Satisfaction

• Customer comments

• Net Promoter Score (NPS)

• Opportunity index

• Resolution rate

Value of the Library

Value of theLibrary

Personal &Organizational

Benefits

Economic Benefits

Direct

Direct

Indirect

Nonuse

Indirect

Value of the Special Library

• Cognitive results• Affective results• Meeting expectations• Task accomplishments• Time aspects• Money aspects

Value of the Academic LibraryStudent Faculty Institution

Recruitment/EnrollmentRetention & GraduationSuccessAchievementLearningExperience, Attitude & Perception of Quality

Research ProductivityGrantsTeaching

Reputation & Prestige

Value of the Public Library

GenericLearning

Outcomes

Knowledge & Understanding

Attitudes & Values

Activity, Behavior & ProgressionEnjoyment, Inspiration

& Creativity

Skills

Communicating the Results• Library staff members• Friends groups• Prospective donors• Board of Trustees• Community governing

boards

• Institutional stakeholders• Library administrative

agencies• State legislatures• Manager/Partner• Customers

– Faculty– Graduate students– Undergraduate students– Others

1. Accountability is part of librarianship

2. Austerity increases the need for evaluation

3. Librarians are engaged in organizational change management

4. Evaluation is not brain surgery

5. Meaningful evaluation focuses on customers and demonstrates the value of the library to them

6. The hardest thing is to start – “Just do it”

7. Play with the various concepts

8. Make evaluation a part of your regular work

9. Share the results

10. Implement a results reporting system

Joe AT JoeMatthews.Org