Forging Forward:Using Evaluation as a Stepping Stone
Joe Matthews
SLA – San Diego Fall SeminarOctober 30, 2015
Evaluation is the process of …• Identifying and collecting data
• Focusing on specific services or activities
• Establishing criteria to gauge success
• Determining the degree to which stated goals and objectives are achieved
Types of Evaluation Activities
• Program or service planning
• Program or service monitoring
• Impact assessment (outcomes)
• Economic efficiency
Evaluation QuestionsHow much? How many? How economical? How prompt?
Magnitude
% of change
% of overall change
Costs
Magnitude
Change
Resources used
Units processed
Cycle times
Turnaround time
Anticipatory
Library Controls
Evaluation QuestionsHow valuable? How reliable? How accurate? How well?
Effort expended
Cost-benefit obtained
Dependability
Access
Accuracy
Currency
Completeness
Comprehensiveness
Accuracy
Expertise
Performance meeting expectations
Library & Customer Decide
Evaluation QuestionsHow courteous? How responsive? How satisfied?Welcoming
Attentive
Anticipatory
Helpful
Empathetic
Expectations met
Materials obtained
Personal interaction
Ease of use
Equipment used
Facilities
Willingness to return
Customers Decide
Evidence-Based Library and Information Practice
1. Formulate a clearly defined question2. Find the best evidence to answer the
question3. Critically appraise the evidence4. Combine the appraisal of the evidence with
professional judgment to make a decision5. Evaluate the effectiveness of the results6. Disseminate the results
Library Metrics
Resources Capability Outputs Outcomes Impacts
Input Measures
Process Measures
Output Measures
Outcome Measures
ImpactMeasures
Library Metrics
• National Center for Education Statistics Academic Library Survey
• Integrated Postsecondary Educational Data System (IPEDS)
• ACRL Academic Library Trends and Statistics survey
• ACRLMetrics• IMLS Public Libraries Survey• PLAMetrics
Internal Evaluation• Facilities• Technologies• Library Personnel• Collections• Services• Other
Input, Process, and Output Measures
External EvaluationLibrary’s Perspective
• Demographics and use• Collection strengths• Library personnel• Library deployed technologies• Facilities• Services provided• Unique programs and services
External Evaluation
Customer Perspective – Outcomes
• ACRL – Megan Oakleaf’s Value report• ACRL – Assessment in Action• The LibValue Project – Carol Tenopir
• PLA – Project Outcome
Measuring Satisfaction
Customer Satisfaction
In-person interview
Focus groupinterview
Exit interviewComment cards
Surveys
Complaint/Compliment analysis
Mystery shopping
Satisfaction Surveys
• Gaps Model of Service Quality – LibQUAL– Affect of service– Information control– Library as place
• Counting Opinions – LibSat• DIY
Customer Satisfaction
• Customer comments
• Net Promoter Score (NPS)
• Opportunity index
• Resolution rate
Value of the Library
Value of theLibrary
Personal &Organizational
Benefits
Economic Benefits
Direct
Direct
Indirect
Nonuse
Indirect
Value of the Special Library
• Cognitive results• Affective results• Meeting expectations• Task accomplishments• Time aspects• Money aspects
Value of the Academic LibraryStudent Faculty Institution
Recruitment/EnrollmentRetention & GraduationSuccessAchievementLearningExperience, Attitude & Perception of Quality
Research ProductivityGrantsTeaching
Reputation & Prestige
Value of the Public Library
GenericLearning
Outcomes
Knowledge & Understanding
Attitudes & Values
Activity, Behavior & ProgressionEnjoyment, Inspiration
& Creativity
Skills
Communicating the Results• Library staff members• Friends groups• Prospective donors• Board of Trustees• Community governing
boards
• Institutional stakeholders• Library administrative
agencies• State legislatures• Manager/Partner• Customers
– Faculty– Graduate students– Undergraduate students– Others
1. Accountability is part of librarianship
2. Austerity increases the need for evaluation
3. Librarians are engaged in organizational change management
4. Evaluation is not brain surgery
5. Meaningful evaluation focuses on customers and demonstrates the value of the library to them
6. The hardest thing is to start – “Just do it”
7. Play with the various concepts
8. Make evaluation a part of your regular work
9. Share the results
10. Implement a results reporting system
Joe AT JoeMatthews.Org