Evidence-Based Practice

Sioo, 8 april 2013Utrecht

Wat is het?

Waar komt het vandaan?

Hoe ziet het er uit in een opleiding?

Evidence-based practice

Evidence based practice:

Wat is het?

Evidence based practice:

Improve information to support decision making

Definition

Uitgangspunt bij evidence-based practice is dat beslissingen gebaseerd dienen te zijn op een combinatie van kritisch denken en de best beschikbare 'evidence'.

Evidence-based practice

Met het begrip 'evidence' wordt niet meer bedoeld dan 'informatie'.

Dit kan informatie zijn afkomstig uit wetenschappelijk onderzoek, maar ook interne bedrijfsinformatie en persoonlijke ervaring geldt als 'evidence’.

Evidence based practice

In principe neemt iedere manager dus beslissingen op basis van 'evidence'.

De meeste managers besteden echter nauwelijks aandacht aan de kwaliteit van de 'evidence' waarop ze hun beslissingen baseren.

Evidence based practice

Evidence-based practice:

kritisch denken

van verschillende informatiebronnen gebruik maken >> best available evidence?

de evidence kritisch tegen het licht houden

denken in termen van waarschijnlijkheid in plaats van 'golden bullets'.

Professional expertise and judgment

Best available internal evidence

Stakeholders’ values and concerns

Best available external evidence

Evidence-based decision

Evidence based practice

Evidence based practice:

Waar komt het vandaan?

“there is a large research-user gap”

“practitioners do not read academic journals”

“the findings of research into what is an effective intervention are not being translated into actual practice”

“academics not practitioners are driving the research agenda”

“the relevance, quality and applicability of research is questionable”

“practice is being driven more by fads and fashions than research”

“many practices are doing more harm than good”

What field is this?

McMaster University Medical School, Canada

Medicine: Founding fathers

David Sackett Gordon Guyatt

How it all started

if you’re hyperventilating breathe into a bag

Problem I: persistent convictions

elderly people who have an irregular heartbeat are much more likely to die of

coronary disease

give them a drug that reduces the number of

irregular beats

Problem I: persistent convictions

How 40,000 cardiologists can be wrong

In the early 1980s newly introduced anti-arrhythmic drugs were found to be

highly successful at suppressing arrhythmias.

Not until a RCT was performed was it realized that, although these drugs suppressed arrhythmias, they actually increased mortality.

By the time the results of this trial were published, at least 100,000 such patients had been taking these drugs.

More than 1 million articles in 40,000 medical journals per year (= 1995; now probably more than 2 million). For a specialist to keep up this means reading 25 articles every day (for a GP more than 100!)

Problem II: too much information

Problem I: persistent convictions

Maslow, A.H. (1943). "A Theory of Human Motivation," Psychological Review 50(4)

Wahba, M. A., & Bridwell, L. G. (1976). Maslow reconsidered: A review of research on the need hierarchy theory. Organizational Behavior and Human Performance, 15(2)

1. Incompetent people benefit more from feedback than highly competent people.

2. Task conflict improves work group performance while relational conflict harms it.

3. Encouraging employees to participate in decision making is more effective for improving organizational performance than setting performance goals.

True or false?

How evidence-based are we?

959 (US) + 626 (Dutch) HR professionals 35 statements, based on an extensive body of

evidence true / false / uncertain

On average: 35% - 57% correct

HR Professionals' beliefs about effective human resource practices: correspondence between research and practice, (Rynes et al, 2002, Sanders et al 2008)

Problem II: too much information

HRM: 1,350 articles in 2010 (ABI/INFORM). For an HR manager to keep up this means reading 3 to 4 articles every day (for a ‘general’ manager more than 50!)

BTW: most of the research is seriously flawed or irrelevant for practice

David Sackett Half of what you learn in medical school will be

shown to be either dead wrong or out-of-date within 5 years of your graduation; the trouble is that nobody can tell you which half.

The most important thing to learn is how to learn on your own: search for the evidence!

(Remember that your teachers are as full of bullshit as your parents)

The 5 steps EBP: pull approach

1. Formulate a focused question (Ask)

2. Search for the best available evidence (Acquire)

3. Critically appraise the evidence (Appraise)

4. Integrate the evidence with your professional expertise and apply (Apply)

5. Monitor the outcome (Assess)

Evidence-Based Practice

1991Medicine

1998Education

1999Social care, public policy

2000Nursing

2000Criminal justice

????Management?

Evidence based practice:Hoe ziet het er uit in de opleiding?

Fase 1: Het ontwikkelen van kritisch en wetenschappelijk denken dat leidt tot een professioneel-kritische attitude met betrekking tot organisatievraagstukken

Fase 2: Het kunnen formuleren van een expliciete vraag en op basis van deze vraag kunnen zoeken in online databases naar uitkomst van relevant wetenschappelijk onderzoek.

Fase 3: Het kritisch kunnen beoordelen van wetenschappelijke en organizational evidence (critical appraisal)

Fase 4: Uitkomst van wetenschappelijk onderzoek kunnen toepassen in de eigen beroepspraktijk (application of science)

Evidence-based practice

Professional expertise and judgment

Best available organizational

evidence

Stakeholders’ values and concerns

Best available scientific evidence

Evidence-based decision

Evidence based practice

Trust me, 20 years of experience

Bounded rationality

Bounded rationality

Het feilbare brein

System 1 Snel, actie Intuitief, associatief shortcuts & biasses

System 2

Langzaam (lui!) Rationeel Nadenken

Bounded rationality

limbische systeem en hersenstam

(systeem 1)

neo cortex(systeem 2)

Systeem 1

Seeing order in randomness Mental corner cutting Misinterpretation of incomplete data Halo effect False consensus effect Group think Self serving bias Sunk cost fallacy Cognitive dissonance reduction

Confirmation bias Authority bias Small numbers fallacy In-group bias Recall bias Anchoring bias Inaccurate covariation detection Distortions due to plausibility

Systeem 1: het feilbare brein

1. Denkfouten

2. Informatiebronnen

3. Mythbusting

4. Assumpties

Fase 1: Kritisch & wetenschappelijk denken

“The first principle is that you must not fool yourself - and you are the easiest person to fool”.

Richard Feynman

Een Type I fout of een vals positief: denken dat er een patroon / verband is terwijl dat er in het echt niet is.

Een Type II fout of een vals negatief: denken dat er geen patroon / verband is terwijl dat er in het echt wel is

Dr. Michael Shermer (Director of the Skeptics Society)

Seeing order in randomness

Een Type I fout of een vals positief: denken dat het geritsel in de bosjes een gevaarlijk roofdier is, terwijl het gewoon de wind is (goedkoop foutje)

Het feilbare brein: patern recognition

Een Type II fout of een vals negatief: denken dat het geritsel in de bosjes gewoon de wind is, terwijl het een gevaarlijk roofdier is (duur foutje)

Het feilbare brein: patern recognition

Een Type I fout of een vals positief: denken dat het geritsel in de bosjes een gevaarlijk roofdier is, terwijl het gewoon de wind is (goedkoop foutje)

DEFAULT

Een Type II fout of een vals negatief: denken dat het geritsel in de bosjes gewoon de wind is, terwijl het een gevaarlijk roofdier is (duur foutje)

Het feilbare brein: patern recognition

Ook ervaren mensen en experts zien patronen en verbanden waar ze niet zijn.

stress & lifestyle peptic ulcer

Het feilbare brein: patern recognition

Peptic ulcer – an infectious disease!

This year's Nobel Prize in Physiology or Medicine goes to Barry Marshall and Robin Warren, who with tenacity and a prepared mind challenged prevailing dogmas. By using technologies generally available (fibre endoscopy, silver staining of histological sections and culture techniques for microaerophilic bacteria), they made an irrefutable case that the bacterium Helicobacter pylori is causing disease. By culturing the bacteria they made them amenable to scientific study.

In 1982, when this bacterium was discovered by Marshall and Warren, stress and lifestyle were considered the major causes of peptic ulcer disease. It is now

firmly established that Helicobacter pylori causes more then 90% of duodenal ulcers. The link between Helicobacter pylori infection and peptic ulcer disease has been established through studies of human volunteers, antibiotic treatment studies and epidemiological studies.

Oct 2005

Errors and Biases of Human Judgment

superstitious rituals

superstitious rituals

more stress = more prone to biases

Errors and Biases of Human Judgment

Beliefs vs Evidence

“What gets us into trouble is not what we don't know,

it's what we know for sure that just isn't so.”

Mark Twain

Ook slimme mensen houden er verkeerde ideeën op

na, niet omdat ze dom of eigenwijs zijn, maar omdat het

de meest logische conclusie is op basis van hun eigen

ervaringen.

Het feilbare brein: patern recognition

(systeem 1 doet altijd mee!)

Fase 1: Het ontwikkelen van kritisch en wetenschappelijk denken dat leidt tot een professioneel-kritische attitude met betrekking tot organisatievraagstukken

Fase 2: Het kunnen formuleren van een expliciete vraag en op basis van deze vraag kunnen zoeken in online databases naar uitkomst van relevant wetenschappelijk onderzoek.

Fase 3: Het kritisch kunnen beoordelen van wetenschappelijke en organizational evidence (critical appraisal)

Fase 4: Uitkomst van wetenschappelijk onderzoek kunnen toepassen in de eigen beroepspraktijk (application of science)

Evidence-based practice

The 5 steps EBP

1. Formulate a focused question (Ask)

2. Search for the best available evidence (Acquire)

3. Critically appraise the evidence (Appraise)

4. Integrate the evidence with your professional expertise and apply (Apply)

5. Monitor the outcome (Assess)

Formulate a focused question

Focused question?

Does team-building work?

What are the costs and benefits of self-steering teams?

What are the success factors for culture change?

Does management development improve the

performance of managers?

Does employee participation prevent resistance to

change?

How do employees feel about 360 degree feedback?

What is a ‘team’? What kind of teams? In what contexts/settings? What counts as ‘team-building’? What does ‘work’ mean? What outcomes are relevant? Over what time periods?

Foreground question?

Does team-building work?

P = Population

I = Intervention or success factor

C = Comparison

O = Outcome

C = Context

Answerable question: PICOC

2. Finding the best available evidence

Searching evidence

What do we search?

Current Information

Overview of a subject

General background

Academic Information

Statistical Information

Theories about a subject

What do we search?

Company information

Peer reviewed journals

Searching for evidence

Databases

ABI/INFORM

Business Source Elite

PsycINFO

Web of Knowledge

ERIC

Google Scholar

Searching for evidence

Searching evidence

How do we search?

Search Strategy

Two types of search strategies

Search strategy

Building blocks methodSnowball method

Search strategy

Hands on instruction

Example: merger

Question

Imagine you are a consultant, your client is the board of directors of a large Canadian health-care organization. The board of directors has plans for a merger with a smaller healthcare organization in a nearby town. However, it’s been said that the organizational culture differs widely between the two organizations. The board of directors asks you if this culture-difference can impede a successful outcome of a merger. Most of them intuitively sense that cultural differences matter, but they want an evidence-based advice.

Answerable question: PICOC

What else would you like to know?

Answerable question: PICOC

P: What kind of Population are we talking about? Middle managers, back-office employees, medical staff, clerical staff?

O: What kind of Outcome are we aiming for? Employee productivity, return on investment, profit margin, competitive position, innovation power, market share, customer satisfaction?

P/C: And how is the assumed cultural difference assessed? Is it the personal view of some managers or is it measured by a validated instrument?

According to the board the objective of the merger is to integrate the back-office of the two organizations (ICT, finance, purchasing, facilities, personnel administration, etc.) in order to create economy of scale. The front offices and primary process of the two organizations will remain separate.

The cultural difference is not objectively assessed (it is the perception of the senior managers of both organizations).

Answerable question: PICOC

P = back office employees in a healthcare organisation

I = merger, integration back office

C = status quo

O = economy of scale

C = different organizational culture, unequal

Answerable question: PICOC

Fase 1: Het ontwikkelen van kritisch en wetenschappelijk denken dat leidt tot een professioneel-kritische attitude met betrekking tot organisatievraagstukken

Fase 2: Het kunnen formuleren van een expliciete vraag en op basis van deze vraag kunnen zoeken in online databases naar uitkomst van relevant wetenschappelijk onderzoek.

Fase 3: Het kritisch kunnen beoordelen van wetenschappelijke en organizational evidence (critical appraisal)

Fase 4: Uitkomst van wetenschappelijk onderzoek kunnen toepassen in de eigen beroepspraktijk (application of science)

Evidence-based practice

Randomized controlled study? Grounded theory approach? Cohort / panel study? Qualitative field research? Longitudinal study? Post-test only study? Survey? Action research? Case study?

What is the best research design?

What is the best design?

quants vs quallies, positivists vs post structuralist, etc

What is the BEST car?

on the research question

Which design for which question?

Research designs

Effect vs Non-effect

Types of questions

Does it work?

Does it work better than ....?

Does it have an effect on ....?

What is the success factor for ....?

What is required to make it work ...?

Will it do more good than harm?

Effect

Types of questions: non-effect

Needs: What do people want or need?

Attitude: What do people think or feel?

Experience: What are peoples’ experiences?

Prevalence: How many / often do people / organizations ...?

Procedure: How can we implement ...?

Process: How does it work?

Explanation: Why does it work?

Economics: How much does it cost?

Internal validity

internal validity = indicates to what extent the results of the research may be biased and is thus a comment on the degree to which alternative explanations for the outcome found are possible.

Internal validity

We are pattern seeking primates:

we are predisposed to see order and causal relations in the world

Causal relations

1. Are the "cause" and the "effect” related?

2. Does the "cause" precede the "effect" in time?

3. Are there no plausible alternative explanations for the observed effect?

measurements, effect size

before and after measurement

randomization, control group

When do we know there is a causal relation?

Causality

Bias & Confounding

Research shows:

Shoe size > quality of handwriting

Smoking youngsters > better lung function

Levels of internal validity

Explanation

Which design for which question?

Different types of research questions require different types of research designs,

but ...

Best research design?

Step 3: Critical appraisal of studies

How to read a research article?

Critical appraisal: quick and dirty

Is the study design appropriate to the stated aims?

Are the measurements likely to be valid and reliable?

Was there a relevant effect size?

Is the outcome (population, type of organization) generalizable to your situation?

Appraisal

Critical appraisal questionnaires

www.cebma.org/ebp-tools

Professional expertise and judgment

Best available organizational

evidence

Stakeholders’ values and concerns

Best available scientific evidence

Evidence-based decision

Aantal te laat betalers neemt toe

CAT: Critically Appraised Topic

CAT: structure

1) Background / context

2) Question (PICOC)

3) Search strategy

4) Results / evidence summary

5) Findings

6) Limitations

7) Recommendation

max 3 pag.

CAT-walk