Proefschrift Fokkema

INNOVATING THE PRACTICE OF

MEDICAL SPECIALTY TRAININGJoanne P. I. Fokkema

INN

OVATIN

G TH

E PRACTICE OF M

EDICA

L SPECIALTY TRA

ININ

G

Joanne P. I. Fokkema

UITNODIGING

Voor het bijwonen van de openbare verdediging van

het proefschrift

INNOVATING THE PRACTICE OF MEDICAL SPECIALTY TRAINING

door Joanne Fokkema

op woensdag 29 oktober 2014om 13:45 uur in de aula van de

Vrije Universiteit,De Boelelaan 1005,

Amsterdam

en voor de hierop aansluitende receptie in

The Basket,De Boelelaan 1111,

Amsterdam

Paranimfen:Michiel Westerman

0642307595Friedolien de Fraiture

[email protected]

Joanne Fokkema

Tweede Oosterparkstraat 249

1092 BM Amsterdam

0627056411

[email protected]

Innovating the practice of medical specialty training

Joanne Fokkema

Copyright © Joanne Fokkema, 2014

The copyright of the published articles has been transferred to the respective journals

or publishers.

Cover design by Iris Muilwijk, YRIS Design: www.yrisdesign.nl

Layout and print by Gildeprint, the Netherlands

ISBN: 978-94-6108-776-8

vrije universiteit


academisch proefschrift

ter verkrijging van de graad Doctor aan

de Vrije Universiteit Amsterdam,

op gezag van de rector magnificus

prof.dr. F.A. van der Duyn Schouten,

in het openbaar te verdedigen

ten overstaan van de promotiecommissie

van de Faculteit der Geneeskunde

op woensdag 29 oktober 2014 om 13.45 uur

in de aula van de universiteit,

De Boelelaan 1105

door

Joanne Petra Ingeborg Fokkemageboren te Groningen

promotoren: prof.dr. F. Scheele

prof.dr. P.J. Dörr (†)

copromotoren: dr. P.W. Teunissen

prof.dr. A.J.J.A. Scherpbier

CONTENTS

1. General introduction 7

2. How lead consultants approach educational change in postgraduate 21

medical education

Medical Education 2012;46:390-398

3. Exploration of perceived effects of innovations in postgraduate medical 41

education


4. Perceived effects of innovations in postgraduate medical education: 63

a Q study focusing on workplace-based assessment

Academic Medicine 2014;89:1259-66

5. Implementing transparency and competition in medical specialty training 87

Under review

6. General discussion 109

7. Summary 129

8. Samenvatting 137

9. Dankwoord 147

10. About the author 153

1General introduction

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 1

8

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

General introduction

9

1Changes in healthcare demands and related changes in thinking about training

healthcare professionals have inspired innovation of medical education. For the

people who participate in the current practice of medical education this means

that they have to deal with a variety of changes. The ones who have started to

use new methods in specialty training include trainees and registered specialists

working in training departments. Some of them are in charge of bringing about the

intended innovations too. The subject of interest in this thesis is how the people who

participate in medical specialty training deal with innovations in this training.

This introduction first sketches roughly the changes in the fields of healthcare and

medical education of the recent past. It continues with an overview of resulting

innovations in specialty training and current insight in their implications for training.

This overview makes clear that there is a need for better understanding how to

approach innovating specialty training. The introduction concludes with an outline

of the studies that were performed in order to add to this understanding.

The term innovation is used frequently in this thesis. There is not one exact definition

of the concept because of the differing terminology in various fields and overlap

with the concept of change.1 In this thesis, the term innovation in medical education

indicates something new or changed to training departments, which requires certain

intended behaviour of the people involved. Moreover, it typically includes new

concrete methods and tangible tools. For example, workplace-based assessment is

considered an innovation in this thesis, because it is was a new concept in specialty

training for which behavioural change of trainees and specialists was needed to enact

it in practice, including observation of performance and registering feedback on

forms. It comprised new methods like organized observation and specific feedback

directly afterwards, and new tools to support that like the mini-clinical evaluation

exercise (mini-CEX) form for structuring and registering feedback. The word change

in this thesis usually indicates matters that are less intentional than innovations or do

not include tangible tools. Terminology regarding medical education and the people

involved is diverse as well. Table 1 presents an overview of widely used terms across

the world.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 1

10

Table 1. Terminology regarding medical education*

Stage in medicaltrajectory

Terminology inthe Netherlands

Terminology in the UK Terminology in the USA

Phase followingundergraduatetraining

Postgraduate medical educationSpecialty training

Postgraduate medical educationSpecialty trainingSpecialist registrar training

Graduate medical educationResidency training

Trainee withinspecialty training

Specialist traineeTrainee

Specialist registrarRegistrar

Resident

Registered medical specialist

Medical specialist Hospital consultant Attending physicianAttending

Person in charge of specialty training

Specialty trainerLead consultant

Dean Program director

* Table adapted with author’s permission from Westerman, M. Mind the gap. 2012

Changes in healthcare and medical education

Undergraduate medical education of the past century was organized in academic

curricula with a firm ground in the basic sciences. Meanwhile, postgraduate medical

education consisted almost exclusively of unstructured on the job learning in an

apprenticeship model. These characteristics fitted the state of science, practice

and societal demands of that time.2 Yet, healthcare has continued to increase

in complexity, related to developments like endless possibilities in diagnostic and

treatment options, and ageing of populations leading a to large share of patients

with multi morbidity. This has been accompanied by sub specialization of patient

care and rising healthcare costs. At the same time, societal demand has risen for

accountability of these costs and of performance of physicians. Also, apart from

having excellent medical or surgical skills, patients expect from their doctors that

they communicate clearly, organize care adequately, and possess other general

qualities like these.3

These developments ask for advancement of medical education. Undergraduate

medical curricula have become more student-centred including problem-based

leaning approaches that are directed at active learning and integrating basic and

clinical sciences.4 The current requirements for postgraduate medical education

include that it needs to prepare physicians in such a way that they are competent to

meet current and future demands, while at the same time delivering good, safe and

accountable care. As a solution to do so, the contemporary approach to specialty

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


11

1training is competency based and outcome oriented.5 This includes the use of sets

of attributes that trainees should develop during training as a basis for designing

training programs.6 It also means that demonstrated performance of trainees is

increasingly considered important for certification rather than time in training only.7;8

Contemporary innovations in specialty training and their implications

The above described developments in medical education have been accompanied

by a variety of innovative applications intended to support safe and competency

based specialty training and assessment of trainees. One example is the structured

practicing of skills of in simulated settings, before trainees are allowed to perform

those in practice, called simulation-based training.9 Also, a variety of new assessment

methods and tools has been developed for both formative and summative

assessment of learners. Well known is the example of workplace-based assessment

(WBA), meaning the assessment of actual performance of trainees at work, of which

there are several variations.10 It can concern single patient encounters of a trainee

observed by a registered specialist or senior trainee, followed by immediate feedback.

Widely used tools for this variation are the mini-clinical evaluation exercise (mini-

CEX)11 and the objective structured assessment of technical skill (OSATS)12. Another

variation of WBA is collecting ratings and comments of various people that a trainee

has worked with in a certain time period, typically called multi-source feedback

(MSF)10. Furthermore, a tangible new tool now used in most training programs is the

portfolio. This is a collection, now usually digital, of evidence of a trainee’s activities

and assessments. Portfolios are intended to keep track of trainee development,

create an overview to define learning goals and to support overall assessment. 13

Medical education research about these innovations up until now has mainly

focused on the tools and methods, and the extent to which desired effects can

be reached by using those.10;14;15 For example, the properties of the mini-CEX have

been scrutinized.14;16 Also, the educational effects of portfolios and various types of

workplace based assessment have been looked into thoroughly.13;17-20 Very generally

speaking, all of these tools have been found to be able to support learning and

assessment in specialty training. However, it has become clear that the effects of

the innovations depend largely on how they are used by people in daily practice

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 1

12

of training.21 Experience with mini-CEXs illustrates that an innovation does not

automatically come into play fully. This tool is often used as a check-box and regarded

an administrative procedure that does not contribute to training, while the extra task

increases workload.22 Good psychometric properties of mini-CEXs are then merely

a theoretical benefit, but not relevant for the contribution to training. It is the way

in which supervisors and trainees handle the required feedback that makes them

contributory to learning or not. Thus, innovations for specialty training not only have

to be apt themselves. It is important even more so that using the innovations catches

on the people who participate in training. This means that innovations have to be

introduced into practice in a way that enables meaningful use. This is where the field

of medical education meets the field of change management.

Different aspects of innovation and change processes have since long received

interest in various fields, including business organization and management, sociology

and economics, and healthcare. In business, the main focus is on how people in

charge of change should lead these processes to do this effectively and bring about

intended effects.23 Sociologic and economic interest has since long been about how

innovations diffuse through groups of people, and when people adopt the use of a

novelty.24 The research of these fields combined is extensive and diverse, and more

detailed discussion of these fields is beyond the scope of this introduction. In the

field possibly most closely related to medical education, being that of healthcare,

implementation science is growing with the aim to understand complex issues like

implementing evidence-based guidelines for optimizing quality of care.25-27

In medical education research, innovation processes have received some attention

as well. They mainly concern undergraduate educational change. Reflections

on curriculum changes (for instance into using team-based learning28) aimed at

uncovering elements important for successful change are most abundant, and

have led to conclusions that most issues known from the above discussed fields are

important for medical education innovation as well. This means that issues related

to the people involved, the innovation, the context and the process are recognized

as important for medical education innovation. For instance, elements like strong

leadership, expertise, buy-in, fit with practice, approaching the process in phases,

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


13

1and ensuring adequate time have been acknowledged as important for successful

curriculum innovation.28-30 Also, there is a vast body of reports that are based on

one of these elements, including the person in charge of curricular change31, or on

specific qualities that are deemed important for change like leadership qualities32;33,

communication skills34 or societal and organizational culture.4

Compared to the interest in innovating undergraduate medical education, processes

of innovating postgraduate medical education have remained underexposed.

However, innovating undergraduate and postgraduate training cannot expected to be

similar, since specialty training innovation involves the dynamic area of the practice

of patient care. The existing reports on postgraduate reforms are mainly thoughtful

reflections or advice.21;35-39 Empiric studies about innovating postgraduate medical

education are still scant5;40-42 and theory based insight into innovation processes has

only just started to form.43-45 Thus, while it is clear that innovating specialty training is

challenging, it is still largely unclear how innovations in this area should be dealt with

by the people in daily practice, in order to contribute to training. Therefore, insight is

needed into how the people who participate in medical specialty training deal with

innovations. This kind of knowledge can potentially support approaching innovation

of specialty training in such a way that it leads to high quality training, does not waste

precious energy of all, and ultimately leads to excellent patient care.

Aim of this thesis

The aim of this thesis is to contribute to the knowledge about innovating specialty

training. It uses knowledge about innovation processes from other fields as a basis,

and focuses on the experiences of the people who participate in medical specialty

training: residency program directors, consultants, and residents. The overall

question of this thesis is: how do people who participate in medical specialty training

deal with innovations in this training?

We approach this question by first looking into three distinct aspects that are

involved with innovation, respectively: approaches of the people in charge of training

for bringing about change, the effects in practice that using an innovation brings

about and the perceptions of the people involved with training regarding these

effects. These elements are combined in a fourth study of an innovation process,

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 1

14

looking into the mutual influence of implementation approach and arising effects in

an innovation process.

Starting off with the people in charge of programmatic changes in actual specialty

training, chapter 2 addresses the question how program directors approach bringing

about change at a training department. Specifically, it concerns the senior consultant

with this responsibility for a department, designated in this chapter by the term

‘lead consultant’. An exploratory qualitative study using semi-structured interviews

is described, for which concepts relating to change management from business and

social psychology were used as a basis. The specific research questions of this study

were: which approaches to changes in specialty training are used by lead consultants?

And what factors influence these approaches?

Chapter 3 then looks into how the people involved with daily practice of specialty

training feel they are affected by an innovation. It describes a study among (lead)

consultants and trainees as users of workplace-based assessment (WBA), which

was studied as a case of an innovation in specialty training. In order to take into

account that their perceptions are not necessarily limited to the strictly education-

related domain, the study design was informed by sociologic theory on diffusion

of innovation. It sought to establish what types of effects of WBA are perceived by

consultants and trainees in the course of using WBA in the clinical workplace.

Chapter 4 looks further into a finding from the previous chapter by focussing on the

distinct perceptions that users of the same innovation can have about the effects

of using it. This was studied using Q methodology, which combines aspects of

qualitative and quantitative approaches for systematic investigation of perceptions

of trainees and consultants across various departments. The research question was:

What perceptions of the effects of using WBA exist among its users?

Combining the elements from the preceding chapters, chapter 5 looks into how the

effects of an innovation and of the approach to its implementation are intertwined.

The innovation in this study is the concept of transparency and competition in

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


15

1specialty training, aimed at stimulating accountability and quality of training.

As a case in which this innovation was introduced, a Dutch national project was

studied. This was done using a theory-driven methodology based on theory from

implementation science. Qualitative analysis of project generated documents and

stakeholder insight was conducted guided by the question: what are the effects of

implementing transparency and competition in specialty training, and how are these

effects influenced by the implementation approach?

Finally, in chapter 6 the main findings from this thesis are discussed, and accordingly

a new conceptualization of innovating medical specialty training is proposed.

Furthermore, reflections on the research in this thesis are provided, including its

strengths and limitations and implications for further research and practice.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 1

16

REFERENCES

(1) Fagerberg J. Innovation: a guide to the literature. In: Fagerberg J, Mowery DC, Nelson RR, eds. The Oxford handbook of innovation. Oxford [etc.]: Oxford University Press; 2005;1-26.

(2) Flexner A. Medical education in the United States and Canada: a report to the Carnegie Foundation for the Advancement of Teaching. no. 44. 1910. New York ; Buffalo. Bulletin (Carnegie Foundation for the Advancement of Teaching) ; no. 4.

(3) Neufeld VR, Maudsley RF, Pickering RJ et al. Educating future physicians for Ontario. Acad Med 1998;73:1133-1148.

(4) Jippes M. Culture matters in medical schools. How values shape a successful curriculum change. 2013.

(5) Scheele F, Teunissen P, Van Luijk S et al. Introducing competency-based postgraduate medical education in the Netherlands. Med Teach 2008;30:248-253.

(6) Frank JR, Snell LS, Cate OT et al. Competency-based medical education: theory to practice. Med Teach 2010;32:638-645.

(7) Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system--rationale and benefits. N Engl J Med 2012;366:1051-1056.

(8) Leung WC. Competency based medical training: review. BMJ 2002;325:693-696. (9) Motola I, Devine LA, Chung HS, Sullivan JE, Issenberg SB. Simulation in healthcare

education: a best evidence practical guide. AMEE Guide No. 82. Med Teach 2013;35:e1511-e1530.

(10) Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ 2010;341:c5064.

(11) Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med 1995;123:795-799.

(12) Martin JA, Regehr G, Reznick R et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84:273-278.

(13) van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE Guide no. 45. Med Teach 2009;31:790-801.

(14) Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007;29:855-871.

(15) Ma L, Brindle M, Ronksley P, Lorenzetti D, Sauve R, Ghali W. Use of Simulation-Based Education to Improve Outcomes of Central Venous Catheterization: A Systematic Review and Meta-Analysis. Acad Med 2011.

(16) Hawkins RE, Margolis MJ, Durning SJ, Norcini JJ. Constructing a validity argument for the mini-Clinical Evaluation Exercise: a review of the research. Acad Med 2010;85:1453-1461.

(17) Driessen E, van Tartwijk J, van der Vleuten C, Wass V. Portfolios in medical education: why do they meet with mixed success? A systematic review. Med Educ 2007;41:1224-1233.

(18) Pelgrim EAM, Kramer AWM, Mokkink HGA, van den Elsen L, Grol RPTM, van der Vleuten CPM. In-training assessment using direct observation of single-patient encounters: a literature review. Adv Health Sci Educ Theory Pract 2011;16:131-142.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


17

1 (19) Ringsted C, Henriksen AH, Skaarup AM, van der Vleuten CPM. Educational impact of

in-training assessment (ITA) in postgraduate medical education: a qualitative study of an ITA programme in actual practice. Med Educ 2004;38:767-777.

(20) Daelmans HEM, Overmeer RM, van der Hem-Stokroos HH, Scherpbier AJJA, Stehouwer CDA, van der Vleuten CPM. In-training assessment: qualitative study of effects on supervision and feedback in an undergraduate clinical rotation. Med Educ 2006;40:51-58.

(21) van der Vleuten C, Verhoeven B. In-training assessment developments in postgraduate education in Europe. ANZ J Surg 2013;83:454-459.

(22) Bindal T, Wall D, Goodyear HM. Trainee doctors’ views on workplace-based assessments: Are they just a tick box exercise? Med Teach 2011;33:919-927.

(23) Kotter JP. Leading change. Boston: Harvard Business School Press, 1996. (24) Rogers EM. Diffusion of innovations. 5 ed. New York: Free Press, 2003. (25) Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations

in service organizations: systematic review and recommendations. Milbank Quarterly 2004;82:581-629.

(26) Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet 2003;362:1225-1230.

(27) Huis A, Schoonhoven L, Grol R et al. Helping hands: a cluster randomised trial to evaluate the effectiveness of two different strategies for promoting hand hygiene in hospital nurses. Implement Sci 2011;6:101.

(28) Thompson BM, Schneider VF, Haidet P, Perkowski LC, Richards BF. Factors influencing implementation of team-based learning in health sciences education. Acad Med 2007;82:S53-S56.

(29) Bland CJP, Starnaman SP, Wersal LM, Moorhead-Rosenberg LP, Zonia SP, Henry RP. Curricular Change in Medical Schools: How to Succeed. Academic Medicine 2000;75:575-594.

(30) Loeser H, O’Sullivan P, Irby DM. Leadership lessons from curricular change at the University of California, San Francisco, School of Medicine. Acad Med 2007;82:324-330.

(31) Rich ECM, Magrane DM, Kirch DGM. Qualities of the Medical School Dean: Insights From the Literature. Academic Medicine 2008;83:483-487.

(32) Souba WW, Day DV. Leadership values in academic medicine. Acad Med 2006;81:20-26. (33) Kaufman A. Leadership and governance. Acad Med 1998;73:S11-S15. (34) Dannefer EF, Johnston MA, Krackov SK. Communication and the process of educational

change. Acad Med 1998;73:S16-S23. (35) Calman KC, Temple JG, Naysmith R, Cairncross RG, Bennett SJ. Reforming higher

specialist training in the United Kingdom--a step along the continuum of medical education. Med Educ 1999;33:28-33.

(36) Dzau VJ, Yoediono Z, Ellaissi WF, Cho AH. Fostering innovation in medicine and health care: what must academic health centers do? Acad Med 2013;88:1424-1429.

(37) Grant JR. Changing postgraduate medical education: a commentary from the United Kingdom. Med J Aust 2007;186:S9-13.

(38) Ludmerer KM, Johns MM. Reforming graduate medical education. JAMA 2005;294:1083-1087.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 1

18

(39) Snelgrove H, Familiari G, Gallo P et al. The challenge of reform: 10 years of curricula change in Italian medical schools. Med Teach 2009;31:1047-1055.

(40) Lillevang G, Bugge L, Beck H, Joost-Rethans J, Ringsted C. Evaluation of a national process of reforming curricula in postgraduate medical education. Med Teach 2009;31:e260-e266.

(41) Malling B, Bonderup T, Mortensen L, Ringsted C, Scherpbier A. Effects of multi-source feedback on developmental plans for leaders of postgraduate medical education. Med Educ 2009;43:159-167.

(42) Wallenburg I, van Exel J, Stolk E, Scheele F, de Bont A, Meurs P. Between Trust and Accountability: Different Perspectives on the Modernization of Postgraduate Medical Training in the Netherlands. Academic Medicine 2010;85:1082-1090.

(43) Varpio L, Bell R, Hollingworth G et al. Is transferring an educational innovation actually a process of transformation? Adv Health Sci Educ Theory Pract 2012;17:357-367.

(44) Ginsburg L, Tregunno D. New approaches to interprofessional education and collaborative practice: lessons from the organizational change literature. J Interprof Care 2005;19 Suppl 1:177-187.

(45) Jippes E. The role of social communication networks in implementing educational innovations in healthcare

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

2How lead consultants approach educational

change in postgraduate medical education

Joanne P.I. Fokkema, Michiel Westerman, Pim W. Teunissen, Nadine van der Lee,

Albert J.J.A. Scherpbier, Cees P.M. van der Vleuten, P. Joep Dörr, Fedde Scheele


R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 2

22

ABSTRACT

ContextConsultants in charge of postgraduate medical education in hospital departments (‘lead consultants’) are responsible for the implementation of educational change. Although difficulties in innovating medical education are described in the literature, little is known about how lead consultants approach educational change.

ObjectivesThis study was conducted to explore lead consultants’ approaches to educational change in specialty training and factors influencing these approaches.

MethodFrom an interpretative constructivist perspective we conducted a qualitative exploratory study using semi-structured interviews with a purposive sample of 16 lead consultants in the Netherlands between August 2010 and February 2011. The study design was based on the research questions and notions from corporate business and social psychology about the role of change managers. The interview transcripts were analysed thematically using template analysis.

ResultsThe lead consultants described change processes with different stages, including cause, development of content, and the execution and evaluation of change, and used individual change strategies consisting of elements, such as ideas, intentions and behaviour. Communication was is necessary to the forming of a strategy and the implementation of change, while the nature of communication was influenced by the strategy in use. Lead consultants differed in their degree of awareness of the strategies they used. Factors influencing approaches to change were: knowledge, ideas and beliefs about change, level of reflection, task interpretation, personal style, and department culture.

ConclusionsMost lead consultants showed limited awareness of their own approaches to change. This can lead them to adopt a rigid approach, whereas the ability to adapt strategies to circumstances is considered important for effective change management. Interventions and research should be aimed at enhancing the awareness of lead consultants of approaches to change in postgraduate medical education.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Change approach of lead consultants

23

2

INTRODUCTION

Clinical departments are required to introduce changes in postgraduate medical

education (PGME), to align specialty training with changing societal demands

and new educational insights.1-5 Although details and terminology vary between

countries, responsibility for specialty training in a hospital department generally

rests with a senior consultant of the department, such as the local site director in the

United States and Canada and the consultant responsible for education in Denmark.

In this paper we use the term ‘lead consultant’ to designate this role.

Lead consultants are responsible for the quality of specialty training in their

department. They are in charge of introducing programmatic changes into the

actual residency training. They can also adjust daily practice of training to their own

insights, within the boundaries of the official residency programme. Further, they

are responsible for the assessment of the residents in accordance with modern

assessment methods. Despite their responsibilities regarding changes in residency

training, the approaches to change used by lead consultants are under-researched.

Change of organisational routines gets abundant attention in domains outside

medicine, such as corporate business and social psychology. Theories about the

complexity of change contain concepts like coalitions and power blocks6 and ambiguity

in organizations.7 Recommendations to achieve change usually involve the role of the

change manager, which is generally considered to be crucial.6-9 Important themes

for this role in change management are leadership style10 and communication.11 A

manager’s or leader’s beliefs and assumptions about change7;12 are regarded to affect

his style. These concepts and recommendations about change may also be relevant

to medical education and to the role of lead consultants as leaders of change.

Many reports about medical education reform have stressed the necessity of

attention for the change process.13-18 Various of these reports reflect on whole

change processes and emphasize important issues, such as attending to different

phases of change.19;20 Other reports focus on factors of the person in charge of

change, such as leadership qualities21;22 and attitudes of deans like commitment

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 2

24

and patience23, and skills like communication.24 However, most studies have focused

on the role of administrators of medical education programmes. Few studies have

specifically examined the role of lead consultants.25 Knowledge of lead consultants’

current approaches to change could identify important issues for their approaches to

effectively reach and sustain changes in PGME.

We used an exploratory qualitative study design informed by concepts of change

management from corporate business and social psychology. Semi structured

individual interviews with lead consultants were conducted and analysed to explore

the research questions: Which approaches to changes in specialty training are used

by lead consultants? What factors influence these approaches?

METHOD

Setting

The study was performed in the Netherlands, where competency-based programmes

were being introduced in postgraduate specialty training at the time. The new

programmes have to comply with general and specialty-related national guidelines.

The latter are developed by the national professional societies of the different

specialties.

Training is organised at the level of hospital departments, which have considerable

autonomy in determining delivery methods and scheduling of training, provided

they comply with the above-mentioned guidelines. Residents undertake several

training posts in a university hospital and affiliated teaching hospitals. All consultants

in a hospital department contribute to training in the workplace. Lead consultants

have final responsibility for delivery, organisation and quality of training in their

departments, and the lead consultant in the university hospital coordinates the

overall programme. Lead consultants are appointed by the Dutch Central College

of Medical Specialties (CCMS). Specific management training is not required for the

position. Length of tenure was the main criterion for appointment until 2011: lead

consultants need to have been a consultant for at least five years, not specifically in

the same department. Usually, they have functioned as assistant lead consultant for

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


25

2

some years prior to their post as lead consultant. Often, they already had a natural

leading role within the team of consultants.

Design

The study was performed from a constructivist point of view with an interpretative

phenomenological epistemology.26;27 Based on the notion that social phenomena are

constructed by communal meaning-making about those phenomena, we aimed to

construct insightful accounts of lead consultants’ approaches to change, rather than

to identify the ‘true’ nature of lead consultants’ approaches to educational change.

Because management of change by lead consultants is an under-researched area,

we conducted an exploratory qualitative study using semi structured interviews. We

interviewed consultants individually, because we expected that this would encourage

more openness of responses than group interviews.

Research team

The daily research team consisted of three junior doctors/PhD students, a resident in

obstetrics-gynaecology with a PhD in medical education, and a professor of medical

education who is a lead consultant gynaecologist. The supervising team consisted

of a professor of medical education, who is a psychologist, a professor of quality

assurance in medical education, who is a medical doctor, and a lead consultant

gynaecologist. The professors and lead consultants all regularly lead or engage in

reform initiatives for medical education in the Netherlands or international. The

main researcher and one other PhD student both participate in national initiatives

for reform in PGME. None of the team members is formally educated as a change

manager.

Participants and procedure

Between August 2010 and February 2011, the main researcher (JF) interviewed lead

consultants with at least one year’s experience in this role. Potential participants were

identified from lists of lead consultants distributed by the national societies of the

different medical specialties. Because we were interested in variety of approaches,

we interviewed a purposive sample of lead consultants from different hospitals and

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 2

26

different specialties, i.e. surgical specialties (obstetrics-gynaecology and surgery),

internal medicine and radiology.

Of 34 lead consultants whom we invited by email, sixteen agreed to participate,

eight did not respond, eight refused to participate due to lack of time (3), lack of

interest (3) or for unspecified reasons (2), and two were excluded from participation

because they were no longer a lead consultant. We made an appointment by email

or telephone with each participant for an interview in their office. Written informed

consent was obtained from all participants, who were assured that the data would

be processed anonymously. The study was approved by the ethical review board of

the Dutch Society of Medical Education (NVMO-ERB).

Interviews

The interview questions were based on the research questions and on concepts

relating to change management from corporate business and social psychology. Since

in those fields the role of the change manager is considered important for effective

change management, concepts such as leadership style, approach fitting context6

and the influence of beliefs about change7 were included in the interview guide to

explore in the approaches to change of lead consultants. We used semi structured

interviews with open-ended questions to accommodate the exploratory goal of the

study.

Pilot interviews with two lead consultants who did not participate in the study

resulted in an additional introductory question about changes experienced during

the past two years. The aim of this question was to prevent generalisation in the

answers. This resulted in the following interview guide:

1. Tell me about any changes or innovations in residency training in which you

were involved during the past two years.

2. How did you approach the change in / the introduction of [example]?

3. To which elements did you pay attention in order to successfully change

[example]?

4. How did you lead the change process?

5. Did this approach differ from the one you used in one of your other

examples? If so, in what way and why?

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


27

2

6. Would you share with us any ideas about possible influences that shaped

your approach?

Analysis

The interviews were transcribed verbatim. For respondent validation, we asked the

participants to comment on a one-page summary of their interview. Twelve out of

sixteen participants responded and agreed on the main content of the summaries.

We analysed the data using template analysis28;29, which involves creation of a

template: a list of codes representing themes, which reflects the (hierarchical)

relationships between the themes as conceived by the researcher. Since template

analysis is a technique rather than a methodology, it can be used in studies based

on different epistemological positions, including constructivism. It also enables

researchers to be explicit about their assumptions about possible themes in the data.

The analysis starts from an ‘initial template’ containing a priori themes based on

the researchers’ assumptions and/or themes derived from the initial coding of part

of the dataset. This template is then modified by iteratively adding, deleting and

reorganising themes as coding continues.

The main researcher (JF) coded the data and created the template. The initial

template combined topics from the interview guide and themes resulting from

the analysis of the first three transcripts. During a discussion of this template by

the whole team, the level of detail was determined. During continued coding and

development of the template, the research team met several times to look for

additional themes and prevent early narrowing of ideas. For the same purpose, a

second researcher (MW) coded the transcript of the seventh interview. Theoretical

saturation was reached after coding of fourteen transcripts. At this point, the

research team discussed possible relationships between codes until agreement was

reached. The two remaining interviews were coded to finalise and confirm the final

template. All coding was done using qualitative data analysis software (MaxQDA,

Marburg, Germany).

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 2

28

RESULTS

All participants were male and they were from four university hospitals and six

teaching hospitals: six lead consultants in a surgical specialty (four obstetrics/

gynaecology, two surgery), five internal medicine consultants and five radiology

consultants. The participants from the three different specialties had been lead

consultant for a mean duration of eight, five and eight years respectively. Their mean

age was 55 (range 50-64), 53 (range 47-59) and 50 (range 42-56) years respectively.

The participants’ accounts varied considerably in the changes they described, their

approaches to change and the depth of their accounts. We found no systematic

differences between specialties, hospitals or age of the lead consultants with respect

to the content of their approaches.

In order to show the variety in consultants’ experiences, we first describe the types

of change they reported. Next we report on their approaches to change, followed by

a discussion of factors that influenced these approaches. The results are illustrated

by examples and quotes from the interviews.

Types of change

The participants mentioned different types of change, ranging from concrete

changes, such as switching from paper to electronic resident portfolios or changes in

on-call schedules, to more general changes like creating a better educational climate

during handovers. Changes also differed in that some were externally imposed, while

others were induced by local and personal initiatives. New national regulations,

such as the nationwide introduction of a standard form to record feedback, were

frequently mentioned as an external cause of change; local and personal initiatives

included a lead consultant’s initiative to offer residents the opportunity to talk to

a psychologist once a year and steps to improve cooperation with colleagues from

another specialty.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


29

2

Approaches to change

Process of change in stages

When talking about their efforts to implement change, consultants talked about

processes of change consisting of different elements. In many accounts, mention was

made of stages of change processes.

The analysis revealed distinct types of stages: the cause of change, formulation of

concrete changes, the actual execution of change and evaluation.

The causes of change that were mentioned can be characterised as official external

causes or causes originating within the local organisation. The former include the

introduction of a standard feedback form imposed by external regulations, while

examples of the latter were residents asking for more teaching moments or the

initiative of a lead consultant who believed that residents learn best from examples.

Concrete changes are formulated in response to the above-mentioned causes, for

example the decision to introduce the new feedback forms, the decision to shorten

handover moments in order to free up time for lectures and more attention during

bedside teaching for demonstrating physical examination.

The execution of change involved the moment when the new forms actually came

into use and the scheduling of a lecture after morning report and of more bedside

teaching by consultants.

Although execution can be the final stage, it may be followed by evaluation and/or

adjustment of the change. For example, when the feedback form turned out to be

very time consuming, the lead consultant advised that only one feedback category

should be addressed at a time.

Strategy and communication

During the analysis we noted that every lead consultant’s approach to change

consisted of a characteristic set of elements, such as views, intentions and behaviour.

We will refer to this as the consultant’s strategy.

A strategy applies to both the content and the execution of the process of change. For

example, based on the notion that change should not be abrupt, one lead consultant

introduced change gradually:

“You have to phase it. And make sure they don’t see it [the whole process]

at once.” (participant 9)

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 2

30

Communication was important regarding strategies. We identified three aspects

of its role: 1) communication during strategy formation, consisting of gathering

information about the issues and the people involved, 2) communication aimed at

enabling the lead consultant to carry out his strategy, 3) the effect of a consultant’s

change strategy on the nature of communication.

Strategies were dynamic in that they could vary depending on the specific change

process and alter in the course of a particular change process.

Awareness

The participants differed in the extent to which they showed awareness of the nature

of the approaches they described, such as planning a strategy and the stages of the

change process. Some consultants talked about different aspects of their approach

without explicitly identifying these as stages or strategy. Nevertheless, irrespective

of their level of awareness, all consultants clearly used a strategy in their reasoning

and acting to achieve change.

“The way it eventually happens depends on whether you can motivate

people, whether you can explain it to the people so that they get it.”

(participant 8)

Their accounts showed a certain order in which actions were performed, although

many did not deliberately divide the change process into different stages.

Different levels of awareness were also evident in the consultants’ descriptions of

strategy development, ranging from conscious advance deliberation to spontaneous

emergence during the process. One consultant, for example, deliberately planned a

strategy based on his belief that people only change when motivated. He therefore

sought ways to create enthusiasm for and engagement with the implementation

of the above-mentioned feedback form, and invited an enthusiastic user to the

department to explain its use and point out its educational benefits.

Influencing factors

We identified several factors that affected the consultants’ approaches to change.

These factors had a direct effect on a consultants’ approach to change and an indirect

effect on other factors. We will describe the main factors and related interactions.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


31

2

Factors originated from within the lead consultant (personal factors): knowledge,

task interpretation, ideas and beliefs about change, levels of reflection and personal

style, and from the environment, departmental culture in particular.

Knowledge

An important influence was the lead consultant’s knowledge about change processes,

strategies, elements of strategies and communication. Such knowledge seemed to

influence consultants’ awareness of the content of their own approaches (stages

and strategies as well as communication). Lead consultants who lacked this type of

knowledge seemed to have a limited repertoire of approaches to change.

“I don’t think people have strategies, I think everyone just acts in their own

way. I’ve never attended a course in communication, so I just communicate

the way I’m used to and the way I think is right. I don’t have any, I don’t

know, I don’t have any deeper thoughts on that.” (participant 10)

Ideas and beliefs about change

The lead consultants held specific views with regard to other people’s perceptions of

change and consequently about the best ways to get people to accept change.

“People like things to be uncomplicated. And everything that comes

unexpectedly is experienced as a threat or as something unpleasant. So you

have to remove everything that is unexpected (...) Everything that’s new is

scary. So you have to make new things look like they are old.” (participant 9)

Views regarding ways to motivate people to accept change led consultants to

stimulate interest or enjoyment in the task (intrinsic motivation) or to apply external

pressure (extrinsic motivation).

Lead consultants also had ideas about change for themselves. Some viewed change

as a burden; others saw it as a positive challenge.

“I solemnly believe that you should always work on what we are actually

doing, what we could possibly change, what could make things more fun,

and a little better [...]” (participant 12)

Strategies were designed in accordance with a lead consultant’s ideas and beliefs

about change, which are influenced by knowledge about other people’s ideas and

approaches to change.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 2

32

Levels of reflection

The lead consultants reflected on results of and situations during change processes

on one of three levels. They attributed failure or success to circumstances and

other people (externally directed reflections) or to their own behaviour and actions

or their own capabilities (internally directed reflections). Lead consultants who

reported internally directed reflections, usually also reported efforts to adjust their

approaches.

“With me the rub is, and I’m very well aware of that: I’m not very good at

working out details. [...] Well, if they don’t do it, I reckon I haven’t made up

a good enough plan” (participant 12)

Participants who showed externally directed reflection on the other hand were less

inclined to adapt their approach.

“Sometimes you’re working on an easy problem and you still haven’t solved

it after six months (...) sometimes unexpected resistance is to blame, or

unexpected complexity, or just disinterest of people.”(participant 9)

Task interpretation

The lead consultants expressed different views regarding their obligations and efforts

with respect to their post. For example, one consultant, who felt he should adhere

to the new guidelines regardless of his own opinion, spent a lot of time incorporating

them into the programme. By contrast, a consultant who saw it as his primary

responsibility to prepare residents for their careers prioritised supervising residents

in the workplace over compliance with new guidelines.

Lead consultants also had ideas about what other people expected of them.

“They look to me when something has to be done regarding the subject of

specialty training” (participant 13).

Both aspects of task interpretation influenced strategies.

Personal style

Every lead consultant mentioned personal habits and characteristics with regard to

thinking and acting in general, which we labelled personal style.

“I think I have a natural tendency to sloppiness” (participant 14)

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


33

2

Personal style directly influenced aspects of the approach to change, such as

communication, or it affected other factors, like the acquisition of knowledge or task

interpretation.

Culture

The consultants also indicated that culture and customs within the department

influenced their approaches. A whole team of consultants being education minded

could directly affect strategies (asking others for solutions first) and communication

(not having to clarify the importance of attention for education) and indirectly

influence other factors, such as task interpretation (considering it normal to delegate

educational tasks to others).

DISCUSSION

We performed an exploratory study about the approaches used by lead consultants

in managing educational change for specialty training in their departments. We

found that lead consultants’ approaches to change consisted of (stages of) change

processes and change strategies, and that individual consultant’s approaches

were mainly characterised by their level of awareness of the strategies they used.

We identified several factors that influenced consultants’ approaches to change:

knowledge about change, ideas and beliefs about change, levels of reflection, task

interpretation, personal style and departmental culture.

Comparison with the existing literature

The different stages we identified in consultants’ change processes are in line with

stage models of change processes, which are well established in the literature on

corporate6;9 and educational30 change. Results of other studies of change processes

in medical education also fit with these models.16;19;20 Although different models vary

in number and content of stages, there is a common pattern that also emerged from

our results: an implementation stage, preceded by preliminary stages and followed

by a closing stage.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 2

34

Our findings about lead consultants’ formation and use of strategies are supported

by Henry Mintzberg’s work on this subject.31;32 His thesis is that the creation of

strategies can not only involve orderly and rational planning in advance, but also

involves intuition and creativity during the process. Furthermore, strategies can

be formed and used unaware; “a pattern need not result from a plan”.32 Likewise,

we found lead consultants’ strategies to range from planned and deliberate to

spontaneous emergent.

Our results with regard to the influence of lead consultants’ ideas and beliefs

about change on the approaches they use are supported by the literature on social

psychology and organisational development6;7, where underlying values and beliefs

about change have been shown to influence people’s strategies and communication.

It is thus considered important for change managers to be aware of their own and

others’ change paradigms, since this can help them to attune their approach to the

local environment.

Adapting strategies to circumstances is considered important for effective change

management in general6;9;10 and in medical education.33 Our results revealed only

one contextual influence: departmental culture. Other factors, such as the persons

involved or the nature or goals of change, were not found to be influential. This

suggests that lead consultants do not fit their approaches to specific circumstances.

This may reflect a degree of rigidity, which may impair the effectiveness of change

management.

Our findings regarding lead consultants’ awareness of the strategies they used also

relate to the ability to adapt approaches to circumstances. This requires awareness

of the effects of one’s actions, and is a key element of learning theories stating

that learning entails detection and correction of error. Kolb’s ‘experiential learning’

theory34, for example, claims that it is necessary to reflect on mistakes in order to

choose new behaviour to experiment with. Similarly, Argyris and Schön’s ‘single-loop

learning’35 states that, when something goes wrong, people will look for another

strategy that will work within the so called ‘governing variables’ of underlying norms,

policies and objectives. Our observation that lead consultants’ awareness of their

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


35

2

actions and strategies as well as reflection on these are prerequisite for detecting

options for adjustment of approaches is in line with both Kolb’s and Argyris and

Schön’s models.

Argyris and Schön’s ‘double-loop learning’ goes even further by also including

the modification of norms, objectives and policies (‘governing variables’): double-

loop learning occurs when correction of errors involves the modification of these

variables. Again consultants will only be able to engage in this type of learning when

they are aware of their beliefs about change, strategies and expectations. They can

adapt these variables to deal with discrepancies between their approach and the

beliefs of other people involved in the change process or the course of change.

By contrast to reports about change in medical education in which ‘leadership’ was

identified as an important element of change management19;36, our results do not

mention leadership as such. This may be due to the broad scope of the concept

of leadership, which comprises several facets of our results. Yukl’s definition of

leadership, “influencing others to understand and agree about what needs to

be done ... and facilitating ... efforts to accomplish shared objectives”37 covers

different elements of our findings about lead consultants’ approaches: strategy,

communication, beliefs about change and task interpretation. Therefore, our results

do not contradict the notion that leadership is important for change, but seem to

address different aspects of the concept of leadership.

Strengths and limitations

To our knowledge, this is the first study to explore lead consultants’ approaches to

change management in specialty training. Our findings are supported by comparable

notions about change management from other fields.

Another strength of this study is its methodological rigour. Since little is known about

lead consultants’ approaches to change, we appropriately conducted an exploratory

study. We built on knowledge from other fields by basing the study design on

concepts from the literature on organisational change. We used template analysis,

because it allowed us to work from a theoretical basis without having to adhere to

themes that turned out to be inadequate.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 2

36

A limitation of this study is that it was conducted in one country. Since organisation

and management of PGME differ between countries, some topics may be less

relevant to other settings. Nevertheless, we believe there are sufficient similarities

between the organisation of PGME programmes to warrant the assumption that the

findings will have some relevance to other settings.

Another limitation is the absence of female lead consultants in our study sample,

reflecting the current underrepresentation of women among lead consultants in the

Netherlands. Thus our results do not show any differences between the approaches

of male and female consultants, which are to be expected based on reported gender

differences in leadership.38

Since our data are limited to interviews with lead consultants, the results are likely to

present a limited picture of change management in PGME. The consultants’ espoused

theory (i.e. explanations of their actions which they would like themselves or others to

believe) may differ from their ‘theory-in-use’, i.e. the reasons that actually determine

their behaviour.35 In order to present a fuller and more accurate picture our study

should be supplemented by observational studies of lead consultants’ approaches to

change and studies of the perceptions of other parties involved.

Suggestions for future research

Although we have gained some insight into lead consultants’ approaches to change,

future research will have to determine which issues cause the most problems or for

which aspects of change processes lead consultants would appreciate support.

Investigations of the effectiveness of different approaches to change would be

helpful in developing specific practical advice for lead consultants. Ethnographic

studies might make a valuable contribution to our understanding of lead consultants’

actual practice and its effects.

Because of the increasing contribution of women in health care and medical

education, it is to be expected that the number of women in lead consultant positions

will increase. Women’s approaches to change management should therefore be

addressed in future research.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


37

2

Implications for practice

It is important for lead consultants to be aware of their approaches to change and of

the need to fit their approaches, including their underlying beliefs, to circumstances.

Furthermore, it seems that lead consultants would benefit from expanding their

repertoire of elements of strategies. Reflection on the effects of elements of

their approaches could promote these goals, as would acquisition of theoretical

knowledge about management and change processes. Initiators and advocates of

change in PGME should pay attention to the implementation and management of

change if they are serious about putting theory into practice.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 2

38

REFERENCES

(1) Curriculum for the foundation years in postgraduate education and training, UK Department of Health, (2007).

(2) Modernising Medical Careers: The Next Steps - The future shape of Foundation, Specialist and General Practice Training Programmes, UK Department of Health, (2004).

(3) Accreditation Council for Graduate Medical Education. Outcome Project 2011 [cited 2011 Mar 28];Available from: URL: http://www.acgme.org/outcome

(4) Segouin C, Jouquan J, Hodges B, Brechat PH, David S, Maillard D, et al. Country report: medical education in France. Med Educ 2007 Mar;41(3):295-301.

(5) Teo A. The current state of medical education in Japan: a system under reform. Med Educ 2007 Mar;41(3):302-8.

(6) Cummings TG, Worley CG. Organization development and change. 9th ed. Mason: South Western; 2009.

(7) De Caluwé L, Vermaak H. Learning to change: a guide for organization change agents. Thousand Oaks: Sage; 2003.

(8) Mintzberg H. Structure in fives: designing effective organizations. 2nd ed. Englewood Cliffs: Prentice Hall; 1993.

(9) Kotter JP. Leading change. Boston: Harvard Business School Press; 1996. (10) Burton R, Obel B. Leadership and management style. Strategic organizational diagnosis

and design: the dynamics of fit. 3rd ed. Boston: Kluwer Academic Publishers; 2004. p. 87-126.

(11) Gosling J, Mintzberg H. The Five Minds of a Manager. Harvard Business Review 2003 Nov;81(11):54-63.

(12) Bolman LG, Deal TE. Reframing organizations: artistry, choice, and leadership. San Francisco, CA: Jossey-Bass; 2003.

(13) Rubin I, Plovnick M, Fry R. Initiating Planned Change in Health Care Systems. Journal of Applied Behavioral Science 1974 Jan 1;10(1):107-24.

(14) Carroll JS, Edmondson AC. Leading organisational learning in health care. Qual Saf Health Care 2002 Mar;11(1):51-6.

(15) Yedidia MJ. Challenges to effective medical school leadership: perspectives of 22 current and former deans. Acad Med 1998 Jun;73(6):631-9.

(16) Hillis DJ. Managing the complexity of change in postgraduate surgical education and training. ANZ Journal of Surgery 2009 Mar;79(3):208-13.

(17) Dowton SB, Stokes ML, Rawstron EJ, Pogson PR, Brown MA. Postgraduate medical education: rethinking and integrating a complex landscape. Med J Aust 2005 Feb 21;182(4):177-80.

(18) Mennin SP, Krackov SK. Reflections on Relevance, Resistance, and Reform in Medical Education. Academic Medicine 1998;73(9):-S64.

(19) Bland CJP, Starnaman SP, Wersal LM, Moorhead-Rosenberg LP, Zonia SP, Henry RP. Curricular Change in Medical Schools: How to Succeed. Academic Medicine 2000 Jun;75(6):575-94.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


39

2

(20) Loeser H, O’Sullivan P, Irby DM. Leadership lessons from curricular change at the University of California, San Francisco, School of Medicine. Acad Med 2007 Apr;82(4):324-30.

(21) Souba WW, Day DV. Leadership values in academic medicine. Acad Med 2006 Jan;81(1):20-6.

(22) Kaufman A. Leadership and governance. Acad Med 1998 Sep;73(9 Suppl):S11-S15. (23) Rich ECM, Magrane DM, Kirch DGM. Qualities of the Medical School Dean: Insights

From the Literature. Academic Medicine 2008 May;83(5):483-7. (24) Dannefer EF, Johnston MA, Krackov SK. Communication and the process of educational

change. Acad Med 1998 Sep;73(9 Suppl):S16-S23. (25) Malling B, Scherpbier AJJA, Ringsted C. What is the role of the consultant responsible

for postgraduate education in the clinical department? Medical Teacher 2007 Jun;29(5):471-7.

(26) Guba EG, Lincoln YS. Paradigmatic controversies, contradictions, and emerging confluences. In: Denzin NK, Lincoln YS, editors. The SAGE Handbook of Qualitative Research.Thousands Oaks: Sage Publications; 2005. p. 191-215.

(27) Bunniss S, Kelly DR. Research paradigms in medical education research. Med Educ 2010 Apr;44(4):358-66.

(28) King N. Using templates in the thematic analysis of text. In: Cassell C, Symon G, editors. Essential guide to qualitative methods in organizational research.London: Sage Publications Inc.; 2004. p. 256-70.

(29) King N. Template analysis 2011 [cited 2011 Apr 21];Available from: URL: http://www2.hud.ac.uk/hhs/research/template_analysis/

(30) Fullan M. The new meaning of educational change. New York, NY: Teachers College Press; 2007.

(31) Mintzberg H. Mintzberg on management: inside our strange world of organizations. New York, NY: Free Press; 1989.

(32) Mintzberg H. Crafting strategy. Harvard Business Review 1987 Jul;65(4):66-75. (33) Lieff SJ, Albert M. The mindsets of medical education leaders: how do they conceive of

their work? Acad Med 2010 Jan;85(1):57-62. (34) Kolb DA. Experiential learning. Englewood Cliffs: Prentice-Hall; 1984. (35) Argyris C, Schön DA. Organizational learning: a theory of action perspective. Reading:

Addison-Wesley; 1978. (36) Bogdewic SP, Baxley EG, Jamison PK. Leadership and organizational skills in academic

medicine. Fam Med 1997 Apr;29(4):262-5. (37) Yukl GA. Leadership in organizations. 6th ed. Upper Saddle River, N.J.: Pearson/Prentice

Hall; 2006. (38) Bolman LG, Deal TE. Leading and Managing: Effects of Context, Culture, and Gender.

Educational Administration Quarterly 1992 Aug 1;28(3):314-29.

3Exploration of perceived effects of innovations

in postgraduate medical education

Joanne P.I. Fokkema, Pim W. Teunissen, Michiel Westerman, Nadine van der Lee,

Cees P.M. van der Vleuten, Albert J.J.A. Scherpbier, P. Joep Dörr, Fedde Scheele


R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 3

42

ABSTRACT

ContextMany studies have examined how educational innovations in postgraduate medical education (PGME) impact on teaching and learning, but little is known about effects in the clinical workplace outside the strictly educational domain. Insights into the full scope of effects may facilitate the implementation and acceptance of innovations, because expectations can be more realistic and difficulties and pitfalls anticipated.

ObjectivesTo explore, using workplace-based assessment (WBA) as a case-study, which different types of effects are perceived by users of innovations in PGME.

MethodsFocusing on WBA as a recent case of innovation in PGME, we conducted semi-structured interviews to explore the perceptions of effects of WBA in a purposive sample of Dutch trainees and (lead) consultants in surgical and non-surgical specialties. Interviews conducted in 2011 with seventeen participants were analysed thematically using template analysis. To support exploration of effects outside the educational domain, the study design was informed by theory on the diffusion of innovations.

ResultsSix domains of effects of WBA were identified: sentiments (affinity with the innovation and emotions), dealing with the innovation, specialty training, teaching and learning, workload and tasks, and patient care. Users’ affinity with WBA partly determined its effects on teaching and learning. Organisational support and the match between the innovation and routine practice were considered important to minimise additional workload and ensure that WBA was used for relevant rather than easily assessable training activities. Dealing with WBA stimulated attention for specialty training and placed specialty training on the agenda of clinical departments.

DiscussionThe outcomes are in line with theoretical notions regarding innovations in general, and may be helpful in the implementation of other innovations in PGME. Given the substantial effects of innovations outside the strictly educational domain, those implementing innovations should consider all potential effects, including those identified in this study.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Effects of innovations

43

3

INTRODUCTION

It is widely recognised that, besides on hard work and tenacity, the success of

innovations depends on early identification of potential pitfalls and opportunities.1;2

This notion has driven some studies of innovations in medical training programmes.3;4

In the domains of business, psychology and sociology, innovation research has focused

on the diffusion and implementation of innovations5;6, but very few, if any, empirical

studies have addressed effects of innovations that emerge only after implementation

but may hold the key to their lasting success.7 In medical education research the

extent to which desired educational effects are achieved has been the subject of

most innovation studies, including those on recent innovations in postgraduate

medical education (PGME), such as simulation-based education, workplace-based

assessment (WBA) and portfolios.8-11 Apart from their intended impact, however,

innovations may affect other areas of day-to-day practice.7 For innovations in the field

of medical education, it is not yet clear what kinds of effects they can bring about.

Therefore, it is yet unclear which areas of day-to-day practice might be affected by an

innovation and should be considered when designing and implementing innovations

in PGME.

The main established theory about diffusion of innovations is based in sociology. Its

founder Rogers states that innovations are diffused through communication between

members of the social system affected by them.2 He proposed five dimensions of

effects of innovations: (i) desirable versus undesirable, (ii) direct versus indirect and

(iii) anticipated versus unanticipated effects; (iv) effects on adopters versus effects

on rejecters of an innovation and (v) effects that increase or decrease equality

between people. Rogers also posited that innovations have a form, function and

social meaning, which may be perceived differently by the developers and adopters

of an innovation. Although it can trigger unanticipated effects, the social meaning of

innovations is particularly prone to being overlooked by developers.7

Although it provides a starting point for notions about effects of innovations, this

theory does not point out the domains of day-to-day practice that might be affected

by an innovation. To study the range of effects in the domain of postgraduate medical

education, we explored how users of one innovation perceived its effects in day-to-

day clinical and educational practice.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 3

44

We studied the case of WBA, which is currently in various stages of implementation

in many PGME programmes worldwide. Numerous studies of its educational

impact12-15 and of instruments like the mini-clinical evaluation exercise (mini-CEX)16

- for assessment of clinical and generic competencies - and objective structured

assessment of technical skills (OSATS)17 -for technical and procedural skills- have been

conducted. The effects of WBA on learning, teaching, supervision, trainees’ clinical

confidence and trainees’ and assessors’ attitudes towards the instrument have been

studied14;18;19, and concerns regarding its appropriate use have prompted further

research.20 Some of these studies also include in their reports unintended effects

of innovations. However, these noted unintended effects remain mostly within

the educational scope (e.g. stimulation of structure in training activities14, inducing

stress15 or improvement of junior residents’ skills after training seniors21). This is not

surprising, since these studies were not conducted with the intention to explore all

kinds of effects of innovations, including those beyond the educational scope.

Addition of insights into effects in all areas of day-to-day practice may paint a more

realistic picture of educational innovations and their effects, which may facilitate

their implementation and adoption and enhance their effectiveness. Therefore, we

addressed the question: What kinds of effects of WBA are perceived by consultants

and trainees in using WBA in the clinical workplace?

METHOD

Setting

The study was performed in the Netherlands, where national guidelines for

competency-based PGME came into effect in 2011. Specialty training is delivered by

hospital departments, of which some have used WBA instruments since before 2011.

All consultants in a department are expected to contribute to training, and trainees

are expected to actively engage in their learning by reflecting, seeking feedback and

documenting their progress, usually in an electronic portfolio. The ‘lead consultant’

in the department has overall responsibility for the programme. The guidelines

require two to four annual progress interviews with each trainee, guided by WBA

data in trainees’ portfolios. Commonly used WBA instruments include the mini-

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


45

3

CEX, OSATS and multi-source feedback (MSF). The ‘training group’, consisting of all

consultants and trainees in a department, are expected to contribute to training,

both individually and as a team.

Design

The research team consisted of medical doctors and educationalists with ample

experience in medical education. Our epistemology was constructivist: we assumed

that knowledge about the phenomenon at hand is constructed in dialogue between

researcher and participant, and therefore diverse interpretations of reality could

arise, depending on the individuals involved.22 Along these lines, we performed this

study with a phenomenological approach, aiming to gain insight into participants’

own experiences and perceptions, and through interpretation of these accounts

identify some commonalities in these perceptions.23 Given the paucity of research

into non-educational effects of educational innovations in PGME, we conducted

an exploratory qualitative study using a design informed by Rogers’ diffusion of

innovations theory. We conducted and analysed individual, face-to-face, semi-

structured interviews with trainees and consultants guided by theoretical concepts

concerning the diffusion, implementation and dimensions of effects of innovations.2;6

Individual interviews as opposed to group interviews were expected to elicit more

details about personal experiences.24


Looking for variety of effects, we purposively sampled trainees and (lead) consultants

from different hospitals, from surgical (obstetrics-gynaecology and surgery) and non-

surgical (internal medicine and paediatrics) specialties and from specialties with

differing degrees of experience with WBA - obstetrics-gynaecology and paediatrics

introduced WBA earlier than surgery and internal medicine.25 To ensure an equal

distribution of different WBA users in the sample, we aimed to include per specialty

at least two trainees and two consultants (but only one lead consultant). To explore

interactions between users at departmental level, we aimed to recruit at least two

participants from each department.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 3

46

Email addresses of trainees and consultants from ten departments (2 internal

medicine, 2 paediatrics, 3 obstetrics-gynaecology, and 3 surgical departments,

ranging in size from 5 trainees and 9 consultants to 80 trainees and 75 consultants)

of six different hospitals were obtained via the departmental secretaries. They all

received an invitation to participate through individual email. Because only one out

of the eleven participants in the first two months was from a surgical department,

we sent a second email to the (trainee) surgeons of one surgical department at

that time. Of the total of 32 potential participants who responded to our invitation,

28 agreed and 4 declined to participate due to time constraints or for unspecified

reasons. Individual appointments were made with each participant for an interview

in their office. The interviews were conducted between September and December

2011.

Ethical considerations

Written informed consent was obtained from all participants, who were assured that

the data would be processed anonymously. The study was approved by the ethical

review board of the Dutch Society of Medical Education (NVMO-ERB; dossier number

81).

Interviews

All interviews were conducted by the principal researcher using an interview guide

based on the research questions and notions regarding the potential consequences

of innovations, such as different dimensions and theories on the development of

consequences (text box 1).2 In keeping with the research approach and the goal

of the study, the interviewer asked open-ended questions regarding the topics in

the interview guide, and also probed emerging issues that seemed of interest, for

which diversion from the proposed order of topics was accepted. The interviews took

30-45 minutes, were audio recorded and transcribed verbatim by an experienced

transcriber.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


47

3

Text box 1. Interview guide.

Aim of the study: to gain insight into which kinds of effects are perceived of new elements in specialty training; not just effects on training, but also other kinds of effects, i.e. on practical work or organisation.Specification: interview not about all innovations in training, but narrowed down to effects of using new methods for supervision and assessments of performance in the workplace, like mini-CEX and OSATS.

1. Tell me about your experiences with WBA methods and instruments that you use regularly.2. In your experience what are the effects of these methods and instruments?Optional exploration of: a. Nature of effects i. Desirable/undesirable ii. Expected/unexpected iii. Direct/indirect (including current situation/future) b. Impact of effects i. On participant, others, team, organisation ii. On adapters and rejecters of [method] iii. On power structures and communication3. (How) do you react on certain effects of these innovations?4. Do certain effects also create new possibilities?5. How do you anticipate on possible future innovations in specialty training?

Analysis

We analysed the data using template analysis.26 This a supporting technique for the

analysis of qualitative data, which has characteristics that make it suitable to use

in a constructivist study approach that is guided by theory. It involves creation of

a template, which is a schema of (coded) themes that are identified as important

in the data and represents the relationships between these themes as recognised

in analysis. It enables researchers to explicate their assumptions (i.e. from existing

theory) about possible themes in the data, without having these assumptions restrict

the process of analysis to these assumptions. Namely, the analysis starts from an

‘initial template’ containing a priori themes which can based on relevant literature,

themes derived from initial coding of part of the dataset and/or on researchers’

own assumptions. This template is then modified by iteratively adding, deleting and

reorganising themes as coding continues.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 3

48

Open coding of the data and construction of the templates was conducted by the

main researcher (JF). The initial template consisted of theoretical topics as used

for the interview guide combined with themes that had resulted from analysis of

the first two interviews. Based on this initial template, the consecutive interviews

were analysed by JF, modifying the template in the process. To prevent premature

narrowing of ideas, identified themes and relations were discussed with the whole

research team at the points of analysis of interviews three, six and eleven. To this

same purpose, the seventh transcript was also analysed by a second researcher (MW),

using open coding and comparing that to the template generated by the principal

researcher. Discussion of discrepancies slightly altered the relations between themes

but yielded no new themes. After fifteen interviews theoretical saturation of the data

was reached, as no new insights were emerging. Inclusion of new participants was

stopped, but two more interviews had already been conducted. A discussion by JF,

MW, NL and FS of the template and the relationships between the categories led

to modification of the template: initial division in effects on individual versus group

level was abandoned. After examining the applicability of the modified template to

all fifteen interviews, JF adjusted the wording of the modified template to better fit

daily practice vocabulary. After JF had applied the final template to the sixteenth and

seventeenth transcripts, the template was agreed on by the research team.

RESULTS

The total of seventeen participants, seven trainees and ten consultants, including four

lead consultants, represented four different specialties, eight different departments

and five hospitals (table 1).

We consecutively present the six different, albeit interrelated, domains of effects of

WBA that resulted from the analysis of the participants’ reports: sentiments, dealing

with the innovation, specialty training, teaching and learning, workload and tasks,

and patient care, illustrated by examples and quotations from the interviews. No

contrasting differences were found between consultants and trainees and between

specialties. In fact, different participants made complementary contributions to the

range of effects in each domain.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


49

3

Table 1. Characteristics of participants.

Consultants Trainees TotalInternal Medicine 2 2 4Paediatrics 3 2 5Obstetrics & Gynaecology 3 1 4Surgery 2 2 4Total 10 (7 men) 7 (5 men) 17Mean age in years (range) 48 (35-61) 36 (30-44)Years of experience in present position (range) 12 years (1-27) 3.5 years (2-6)

Sentiments

Participants expressed sentiments that related to their affinity with WBA.

Affinity

Some participants expressed a clear understanding of the ideas underpinning WBA,

such as direct observation and documentation of focused feedback to promote

learning, and they felt the innovation made sense and was appropriate.

“It makes you notice things at an earlier stage, which enables you to correct

things and also, yes, make you provide a more nuanced training.” (Consultant

4)

As the objectives of WBA and its place in the training programme were consistent

with or complementary to their natural approach to education, they seemed to

incorporate WBA instruments into their work routines and did not feel constrained

by the mandatory use of standard assessment forms. Rather they indicated that they

customised their use of the forms to match different situations in the workplace.

Other participants said they understood the WBA concepts and subscribed to its

goals, but found the instruments quite unfamiliar and not compatible with their

customary approach to feedback and assessment. They mentioned that, although

they experienced WBA as quite demanding, they incorporated it into their teaching

because they considered it worth the effort.

Participants who did not really understand the objectives of WBA said to adhere

to their customary approaches and use the WBA formats as obligatory add-ons.

Considering WBA a formal exercise with little educational value, they experienced

it as a burden.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 3

50

“In my opinion, you learn this profession by doing. It’s a craft, we shouldn’t

complicate things: you need to see a lot and do a lot. Feedback follows

naturally. I can’t squeeze everything into forms.” (Consultant 1)

Affinity with WBA was recognised in analysis not just to be a characteristic of individual

participants. Individual affinity and sentiments in the training groups seemed to be

interrelated. Training groups, as a social system in which the lead consultant played

a prominent role, appeared to develop a shared attitude towards WBA. Individual

and group affinity with WBA impacted on other effects of working with WBA as well.

Emotions

Participants’ emotions on the topic seemed to be mostly related to positive or negative

experiences with WBA and its perceived value. Affinity with WBA was related to

positive emotions, such as satisfaction with effective teaching and learning, pleasure

from a conversation about the specialty and satisfaction with good organisation of

training in the department.

“I like it [OSATS]. I liked it as a trainee, and now as a consultant I still like it. [...]

it’s good to talk through the procedure together beforehand.” (Consultant 5)

Negative emotions related mainly to an experienced imbalance between the burden

of regular mandatory assessments using standardised instruments and the perceived

(low) value of the assessments. Frustration and irritation were expressed mainly, but

not exclusively, by participants who had little affinity with WBA. Most consultants

mentioned emotions like irritation or guilt when trainees asked for assessment

at moments of high time pressure. Trainees reported feeling uneasy about asking

a clearly reluctant consultant to assess them and tense when being observed or

receiving feedback. Some consultants were apprehensive when they had to give

difficult feedback.

Dealing with the innovation

The participants mentioned customisation of WBA to fit their personal preferences,

and their experiences with this innovation shaped their expectations and anticipation

of any future innovations.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


51

3

Shaping the use of WBA

Acknowledging that WBA was an innovation, participants assumed it would take

time and practice to achieve optimal results.

I think that at first people thought: “Oh my, another load!” [..] But not

anymore, I think. Because by now everyone knows that it actually doesn’t

take much time, and that it does add value.” (Trainee 2)

Individually and in group interactions, they deliberated about the acceptance and

practical implementation of WBA and the experiences of other groups. Consultants

indicated that they adapted their usage of WBA to fit conditions in the workplace.

One gynaecologist reported that when she did not do an OSATS immediately after a

laparoscopic procedure, she later watched the video of the procedure together with

the trainee to be able to give concrete feedback. Participants revealed that WBA is

mainly used for what are considered core components of training in their specialty.

Most surgical participants, for example, expressed that relevance of assessing trainee

performance in the outpatient clinic is considered to be low, when a trainee has

already mastered the required technical skills like suturing or physical examination.

“And actually, yes, it is just expected of you that you’re capable of doing that

[outpatient consultations].” (Trainee 6)

Anticipating future innovations

Attitudes to future innovations appeared to be shaped by participants’ experiences

with the current innovation. Participants who felt their group was successfully

managing the use of WBA and understood and valued its contribution to training

voiced no explicit misgivings at the prospect of further educational innovations.

Participants who had experienced significant difficulties with the implementation

of WBA, however, were more likely to express a strong aversion to this prospect.

To most participants, WBA was only one among many innovations in PGME, with

rationales that were not always clear to them. A frequently mentioned barrier to

acceptance of innovations was the perceived lack of scientific evidence to support

their value.

“With this [WBA] as well, I think you should do much more research [...]

instead of changing things without reviewing them.” (Consultant 9)

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 3

52

Specialty training

Consultants and trainees voiced increased interest in matters relating to specialty

training, which seemed to be related to the introduction of WBA.

Specialty training as an area of interest

In analysis, the researchers noticed that the implementation of WBA created

attention for education and training. Trainees and consultants engaged in formal

and informal conversations about ways to accommodate WBA in their work. The

introduction of WBA made specialty training a topic of discussion in departments. As

noted by some participants, this kind of attention promotes recognition of PGME as

an area of interest in its own right.

“I think there is increasing awareness that, if you really want to learn

something during training, that you really have to be in charge. I think that’s

what’s going on.” (Trainee 2)

Shaping specialty training

The growing recognition of PGME as a field of interest stimulated training groups to

discuss education, compare their activities with those of other training groups and

consider matters of consent for content and activities. Individual trainers and trainees

and training groups as a social unit appeared to be customising training activities

to fit within their practice routines. Several participants, for example, pointed to an

emerging shared value in their training group: Good training involves WBA, good

WBA depends on good feedback, good feedback depends on the application of the

Pendleton rules, so if you want good training you should always use the Pendleton

rules for feedback.

Teaching and learning

WBA was regarded by the participants as a stimulus and a hindrance for teaching

and learning. It was said to stimulate the learning of trainees by promoting higher

quality and frequency of feedback, and the WBA instruments, the mini-CEX form

in particular, to stimulate consultants to give competency orientated and specific

feedback. This type of feedback was generally considered to require practice and/

or training.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


53

3

“A standard structure, that’s the essence. So everyone has the mini-CEX’s

structure in mind, that you have to focus on specific competencies. [..]

Maybe another structure would be as good or even better, I don’t know,

but a standard structure that you can look up, on the computer, that’s

important.” (Trainee 1)

Consultants and trainees said that writing down comments when discussing an

observed activity stimulated precision and comprehension of feedback. They also

mentioned that WBA instruments or structure seemed to encourage consultants

to report poor performance, something they might have avoided previously. In

analysing the data, the above effects were recognised to be strongly associated with

an affinity with WBA. Those with less affinity usually expressed to be unable to fit

their comments into the prescribed structure of the instruments and consequently

did not give frequent feedback. Some of them felt that feedback was deteriorating

due to the compulsory use of WBA instruments.

“I think they’re annoying forms to fill out. And sometimes that makes me

think: let’s just skip it this time.” (Consultant 1)

The structure imposed by WBA was considered by most participants to generate

more frequent observation-based feedback and more feedback on inadequate

performance.

“Afterwards, they always tell you what went well and then also what went

badly. Always. It’s never the case anymore that you only hear what you did

well. That absolutely changed in the past two years.” (Trainee 2)

As trainees whose performance was generally adequate or even exceptional received

more feedback in the new system, some participants felt that trainees made more

progress from an earlier stage of training.

“They get feedback more quickly and can correct themselves when they do

something wrong.”

(Consultant 4)

If working properly, electronic recording of WBA was considered to afford a good

overview of trainees’ strengths, weaknesses and areas requiring improvement, but

when programs were slow, crashes frequent or computers not readily available,

electronic recording was experienced by participants as a hindrance to assessment.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 3

54

According to some, recording focused feedback on all competencies helped trainees

and (lead) consultants to pinpoint strengths and weaknesses and to formulate focus

points for training.

A good overview of their performance boosted the self-confidence of some trainees.

Moreover, the availability of solid information from the review of assessments to

guide progress interviews increased the value of these interviews in the eyes of lead

consultants and trainees.

Focus of educational attention

In analysis, the mandatory nature of WBA was recognised to affect participants’

focus on trainees’ activities. Trainees and consultants mentioned to actively look for

opportunities in the workplace to ‘get one done’. This made them realise that WBA was

easier to arrange for some activities than for others and practical considerations rather

than educational relevance tended to direct educational attention. More specifically,

tasks not related to patient encounters, such as handovers and presentations, were

cited as opportunities for WBA, because they involved the simultaneous presence of

trainees and consultants while lacking strong time constraints.

“So routinely, after a nightshift, you get a mini-CEX for your patient handover,

how it went.”

(Trainee 3)

By contrast, it took considerable effort to arrange for observation and discussion of

patient contacts that trainees routinely performed on their own, such as outpatient

consultations.

This logistical challenge discouraged frequent assessment of trainees’ tasks in patient

care.

“I just don’t know how to arrange that, if I’m not there together with a

surgeon already. Very often, that’s just not the case.” (Trainee 6)

In a similar vein, participants from surgical specialties reported that mainly logistical

considerations led to OSATS being conducted more easily and more frequently than

mini-CEXs. This appeared to be strongly influenced by the culture in the training

group: groups that considered surgical skills the core business of their specialty were

less willing to arrange for assessment of other activities, resulting in the mini-CEX

being largely ignored and reduced to a mere check box exercise.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


55

3

Workload and tasks

WBA influenced users’ tasks and responsibilities and their experienced workload.

Workload

All participants regarded WBA as an extra task, but views of the related workload

differed. Those with an affinity with WBA experienced less workload and considered

tasks easier to perform or well worth the extra effort. The same applied for

participants from departments where clinical work was organised to create natural

opportunities and time for assessment.

“There has to be an opportunity during supervision. If there is enough time

or at least set moments for supervision, than you can ask them: ‘Could you

please fill out a mini-CEX?’” (Trainee 5)

Both trainees and supervisors expressed preference to be relieved of the workload

created by their joint responsibility for WBA, each preferring arrangements to be

made by the other party. Some trainees experienced a WBA-related reduction of

workload due to the insight it afforded into their performance and learning goals.

Task allocation

Certain informal practices of allocation of WBA tasks in training groups were

mentioned. Feeling responsible for the success of WBA, lead consultants were

noticed by everyone to do many assessments to set a good example. Trainees sought

assessment mainly from lead consultants or consultants in favour of WBA who

appeared to be least bothered by these requests and gave the best feedback.

“Some consultants [..] are more inclined to sit down at the computer and

take time to discuss it. Well, and the lead consultant himself is also mini-CEX-

minded. For the rest, it differs per consultant.” (Trainee 3)

Balancing care and training

Participants’ struggles to balance patient care and teaching usually turned out

unfavourably for training activities, which participants confessed to skip or shorten.

Participants with a strong commitment to training expressed regret over missed

training opportunities, which again could increase their workload.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 3

56

Patient care

Supporters of WBA felt it helped trainees to provide better care at an earlier stage

of their training. They saw improvement in all competencies and skills. Some noted

that WBA also met current societal demands by its focus on generic competencies,

particularly patient centred communication, the only aspect of patient care that

participants considered likely to benefit from WBA in the long run. Consultants and

trainees had experiences in which observation caused trainees to perform more

awkwardly or more correctly than usual. More experienced trainees said these effects

diminished as they got used to observation. Due to more frequent observation,

patients were confronted more often with the presence of an extra doctor.

“It feels unnatural, you’re not used to it, logistics wise it’s often inconvenient.

And patients, they automatically start to talk to the person with most grey

hair.” (Trainee 4)

Effects on patients were speculated by the participants to range from confusing to

reassuring.

DISCUSSION

We focused on the case of WBA to explore different kinds of effects of innovations in

PGME as perceived by the users of this innovation. Six domains of interrelated effects

were distinguished: sentiments, dealing with the innovation, specialty training,

teaching and learning, workload and tasks, and patient care.

Comparison with the literature

The effects that were found extend beyond the range of the intended ones, in the

case of WBA the facilitation and documentation of learning.16 This finding is in line

with Rogers’ proposal to consider unintended, undesired and unexpected effects of

innovations.7 The interrelatedness of the effect domains underlines the relevance of

considering both intended and unintended consequences of innovations.

Several domains of perceived effects that we identified in this study about a medical

educational innovation are comparable to effects that were recognised as impact of

innovations in health care practice; effects on task allocation, workflow issues, and

sentiments and emotions.27 This consistency in affected domains seems to indicate

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


57

3

that, regardless of the nature of innovation, some aspects of the daily practice of

health care are particularly susceptible to effects of an innovation.

Of particular relevance to the meaningful use of innovation was the effect of users’

affinity with WBA. An explanation for the pervasiveness of this effect may be found

in theory on diffusion of innovations, in particular the notion that every actor has a

certain probability of adopting an innovation.5 Our results suggest that adoption is

affected by user affinity. The broad impact of affinity is also supported by the social

psychological notion that beliefs influence behaviour.28 Diffusion of innovations theory

further states that the probability of adoption can be altered by “communication and

influence”5, which may explain the strong effect on user affinity of lead consultants’

attitudes.

High WBA-related workload was not experienced exclusively by participants with

low affinity. It occurred also when departments failed to incorporate WBA in work

schedules. This finding indicates that adapting departmental organisation could

accommodate the implementation of an innovation. This is in line with occupational

psychology’s recognition of that a supportive environment can motivate and engage

people.29 Workload appears to be an important area to consider in innovations in

specialty training.

Some of the effects, specifically users’ sentiments about the innovation and users

moulding their use of the innovation to fit their personal beliefs, resonate with

effects reported for change management, both in general and in medical education

in particular.1;3;30 This aptly illustrates that the effect domains we found relate to

different aspects of innovations, such as their implementation and the way users

incorporate it in their work routines. To anticipate and deal with effects like negative

emotions, it may be advisable to determine whether these are due to change

management or to inherent characteristics of the innovation.

The reported tendency to focus on assessability rather than on educational and

professional relevance emphasises the importance of a good fit between innovation

and practice, suggesting that mandatory implementation of an innovation may not

be the best way to promote meaningful use in specialty training.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 3

58


Broadening the perspective on consequences of changes in medical education,

we explored a wide range of effects of an innovation in PGME. We appropriately

conducted an exploratory study, with some guidance from existing theory. Template

analysis enabled the researchers to give the analysis a theoretical foundation without

being restricted by it. The resulting effect domains complement the existing theory in

that do not require labelling as being “unintended” or “unexpected”.7

Due to our focus on the users of one particular innovation, the resulting domains

are not necessarily exhaustive and studies of other innovations or from different

perspectives may reveal different effect profiles. As our findings find support in

various theoretical perspectives, however, we expect that our conclusions bear some

relevance to other innovations in PGME and in other settings.

We found no consistent differences between male and female participants. However,

the overrepresentation of male participants in this study may have influenced the

findings, due to between gender differences, for example in perception and coping.31

Future research

Further research should determine whether the domains we identified apply also to

other kinds of innovations and whether different perspectives yield additional effect

domains. It also seems worthwhile to examine which kinds of effects develop under

which conditions to enable optimal tailoring of the implementation of innovations to

specific circumstances.


Educationalists, administrators and clinicians who design and implement innovations

in PGME should be aware that innovations may trigger a variety of effects in the

workplace. Considering the interrelatedness of the different effects, we suggest that

all potential effects deserve careful attention. Through looking at the intended and

unintended effects of an innovation in medical education, this study offers those

involved in current and future changes a framework for recognizing implementation

pitfalls by directing their attention to six domains of effects of innovations.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


59

3

REFERENCES

(1) Kotter JP. Leading change. Boston: Harvard Business School Press; 1996. (2) Rogers EM. Diffusion of innovations. 5 ed. New York: Free Press; 2003. (3) Bland CJP, Starnaman SP, Wersal LM, Moorhead-Rosenberg LP, Zonia SP, Henry RP.

Curricular Change in Medical Schools: How to Succeed. Academic Medicine 2000 Jun;75(6):575-94.

(4) Fokkema J, Westerman M, Teunissen P, Van der Lee N, Scherpbier A, van der Vleuten C, et al. How lead consultants approach educational change in postgraduate medical education. Med Educ 2012 Apr;46(4):390-8.

(5) Wejnert B. Integrating Models of Diffusion of Innovations: A Conceptual Framework. Annual Review of Sociology 2002 Jan 1;28:297-326.

(6) Cummings TG, Worley CG. Organization development and change. 9th ed. Mason: South Western; 2009.

(7) Rogers EM. Consequences of innovations. Diffusion of innovations. 5 ed. New York: Free Press; 2003. p. 436-71.

(8) Ma L, Brindle M, Ronksley P, Lorenzetti D, Sauve R, Ghali W. Use of Simulation-Based Education to Improve Outcomes of Central Venous Catheterization: A Systematic Review and Meta-Analysis. Acad Med 2011 Jul 21.

(9) Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007 Nov;29(9):855-71.


(11) Driessen E, van Tartwijk J, van der Vleuten C, Wass V. Portfolios in medical education: why do they meet with mixed success? A systematic review. Med Educ 2007 Dec;41(12):1224-33.

(12) Pelgrim E, Kramer A, Mokkink H, van den Elsen L, Grol R, van der Vleuten C. In-training assessment using direct observation of single-patient encounters: a literature review. Advances in Health Sciences Education 2011 Mar 1;16(1):131-42.

(13) Hawkins RE, Margolis MJ, Durning SJ, Norcini JJ. Constructing a validity argument for the mini-Clinical Evaluation Exercise: a review of the research. Acad Med 2010 Sep;85(9):1453-61.

(14) Ringsted C, Henriksen AH, Skaarup AM, van der Vleuten CPM. Educational impact of in-training assessment (ITA) in postgraduate medical education: a qualitative study of an ITA programme in actual practice. Med Educ 2004 Jul;38(7):767-77.

(15) Bindal T, Wall D, Goodyear HM. Trainee doctors’ views on workplace-based assessments: Are they just a tick box exercise? Med Teach 2011;33(11):919-27.

(16) Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med 1995 Nov 15;123(10):795-9.

(17) Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997 Feb;84(2):273-8.

(18) Daelmans HEM, Overmeer RM, van der Hem-Stokroos HH, Scherpbier AJJA, Stehouwer CDA, van der Vleuten CPM. In-training assessment: qualitative study of effects on supervision and feedback in an undergraduate clinical rotation. Med Educ 2006 Jan;40(1):51-8.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 3

60

(19) Ringsted C, Pallisgaard J, Ostergaard D, Scherpbier A. The effect of in-training assessment on clinical confidence in postgraduate education. Med Educ 2004 Dec;38(12):1261-9.

(20) Schuwirth LWT, van der Vleuten CPM. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach 2011;33(6):478-85.

(21) Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Unexpected collateral effects of simulation-based medical education. Acad Med 2011 Dec;86(12):1513-7.

(22) Bunniss S, Kelly DR. Research paradigms in medical education research. Med Educ 2010 Apr;44(4):358-66.

(23) Fraenkel JR, Wallen NE. The nature of qualitative research. How to design and evaluate research in education. 7th ed. New York: McGraw-Hill Higher Education; 2008. p. 420-38.

(24) Pope C, van Royen P, Baker R. Qualitative methods in research on healthcare quality. Qual Saf Health Care 2002 Jun;11(2):148-52.

(25) Scheele F, Teunissen P, Van Luijk S, Heineman E, Fluit L, Mulder H, et al. Introducing competency-based postgraduate medical education in the Netherlands. Med Teach 2008;30(3):248-53.

(26) King N. Template analysis. Website 2011 [cited 2011 Apr 21];Available from: URL: http://www2.hud.ac.uk/hhs/research/template_analysis/

(27) Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006 Sep;13(5):547-56.

(28) Ajzen I, Fishbein M. The prediction of behavior from attitudinal and normative variables. Journal of expirimental social psychology 1970;6:466-87.

(29) Demerouti E, Bakker AB, Nachreiner F, Schaufeli WB. The job demands-resources model of burnout. Journal of Applied Psychology 2001 Jun;86(3):499-512.

(30) Lieff SJ, Albert M. The mindsets of medical education leaders: how do they conceive of their work? Acad Med 2010 Jan;85(1):57-62.

(31) Billings AG, Moos RH. The role of coping responses and social resources in attenuating the stress of life events. Journal of Behavioral Medicine 1981 Jun 1;4(2):139-57.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

4Perceived effects of innovations in postgraduate

medical education: a Q study focusing on

workplace-based assessment

Joanne P.I. Fokkema, Fedde Scheele, Michiel Westerman, Job van Exel, Albert J.J.A.

Scherpbier, Cees P.M. van der Vleuten, P. Joep Dörr, and Pim W. Teunissen

Academic Medicine 2014;89:1259-66

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 4

64

ABSTRACT

Purpose

Anticipating users’ perceptions of the effects an innovation will have in daily practice

prior to implementation may lead to a more optimal innovation process. In this

study, the authors aimed to identify the kinds of perceptions that exist concerning

the effects of workplace-based assessment (WBA), an innovation that is widely used

in medical education, among its users.

Method

In 2012, the authors used Q methodology to ascertain the principal user perceptions

of effects of WBA in practice. Participating obstetrics–gynecology residents and

attending physicians (including residency program directors) at six hospitals in the

Netherlands performed individual Q sorts to rank 36 statements concerning WBA

and WBA tools according to their level of agreement. The authors conducted by-

person factor analysis to uncover patterns in the rankings of the statements. They

used the statistical results and participant comments about their sorts to interpret

and describe distinct perceptions.

Results

The analysis of 65 Q sorts (completed by 22 residents and 43 attendings) identified

five distinct user perceptions regarding the effects of WBA in practice, which the

authors labeled enthusiasm, compliance, effort, neutrality, and skepticism. These

perceptions were characterized by differences in views on three main issues: the

intended goals of the innovation, its applicability (ease of applying it to practice), and

its actual impact.

Conclusions

User perceptions of the effects of innovations in medical education can be typified

and should be anticipated. This study’s insights into five principal user perceptions

can support the design and implementation of innovations in medical education.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

User perceptions of innovation effects

65

4

INTRODUCTION

Innovations in postgraduate medical education (PGME) aim to enhance teaching and

learning to optimize residents’ preparation for future practice. However, innovations’

effects on daily practice may reach further than intended.1,2 For example, a recent

exploratory study found that attending physicians and residents experienced effects

of workplace-based assessment (WBA) to varying degrees in six domains, only one of

which was the intended domain of teaching and learning.3

It has become clear that introducing and implementing innovations in medical

education is difficult,4-6 as it is in other fields, such as business.7 Therefore, it is

important to anticipate pitfalls and opportunities.8,9

Given the varying effects of innovations in medical education in daily practice, it

seems insufficient to base approaches to change on the assumption that innovations

will produce intended and beneficial effects. Anticipating how the effects of an

innovation might be experienced during actual use would help customize approaches

for successful design, adoption, and implementation of innovations.10 However,

knowledge is lacking about what can be expected of a medical education innovation

once it is used in actual practice.

To contribute to this required knowledge, we conducted a study focused on WBA.

We selected WBA because it is a widely used innovation that is beyond the stage of

implementation and its educational effects have received abundant attention.11-15 We

investigated the effects of WBA during actual use in residency training as perceived

by the innovation’s users—residents, attending physicians (attendings), and program

directors. Our research question was: What perceptions of the effects of using WBA

exist among its users? We performed this study using Q methodology, which is a

method for the systematic investigation of people’s viewpoints regarding a topic.16

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 4

66

METHOD

Setting

We conducted this study in the Netherlands between October and December

2012. While national guidelines for competency-based residency training came into

effect in 2011,17 some residency programs started to use WBA instruments long

before then. The main responsibility for implementing WBA (and other innovations)

into training practices lies with department-level residency program directors.

Obstetrics–gynecology (ob–gyn) and pediatrics introduced WBA in residency training

in 200518; they were the first specialties to adopt the innovation and therefore their

training programs had years of usage experience by the time of our study. Commonly

used WBA instruments are the mini-clinical evaluation exercise (mini-CEX),19 which

assesses clinical and generic competencies, and the objective structured assessment

of technical skill (OSATS),20 which assesses technical and procedural skills.

In residency training programs, WBA users typically are residents, attendings, and

program directors. All attendings in a department are expected to contribute to the

training of residents, and residents are expected to actively engage in their own

training by seeking feedback, reflecting on their progress, and documenting their

reflections and progress in an electronic portfolio. Together, the attendings and

residents of a department are commonly referred to as the “training group” in the

Netherlands. In each training group, one of the attendings is the program director,

who has overall responsibility for the program and holds two to four annual progress

meetings with each resident. These meetings should be guided by the WBA data in

the resident’s portfolio.

Q methodology

To investigate the users’ perceptions of the effects of WBA, we used Q methodology,

which combines aspects of qualitative and quantitative approaches. This methodology

fits well with our purpose of systematically exploring which principal viewpoints

on the effects of WBA exist among its users and describing the communalities and

distinctions between these views,16 and it has been successfully used in similar

studies.21,22 Stemming from the social sciences, it is an established methodology in

health services and medical education21,23,24 as well as in other fields.25-27

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


67

4

Participants in a Q study typically are asked to create Q sorts by positioning a variety

of statements about the topic on a score sheet, where column placement options

range from most disagree to most agree (scored from –4 to +4 in this study, as

displayed in Figure 1). Participants are encouraged to place the number of statements

that fits in the spaces provided in each column, but are allowed to deviate from

this arrangement. By creating their Q sorts, participants reveal their views on the

subject—in this case, the effects of using WBA in practice. After finishing their Q

sorts, participants are asked to explain their ranking of the statements.

The Q sorts of all participants are then subject to by-person factor analysis to uncover

patterns in the rankings of the statements, under the assumption that correlation

between the Q sorts of certain participants indicates similarity of their viewpoints.

The statistical results and the qualitative data (i.e., participant comments) are used

to interpret and describe the distinct perceptions.

62

agree (scored from –4 to +4 in this study, as displayed in Figure 1). Participants are encouraged to place

the number of statements that fits in the spaces provided in each column, but are allowed to deviate

from this arrangement. By creating their Q sorts, participants reveal their views on the subject—in this

case, the effects of using WBA in practice. After finishing their Q sorts, participants are asked to explain

their ranking of the statements.

Figure 1

Score sheet for the Q sort of 36 statements in a study of user perceptions of workplace-based assessment in daily practice in

residency training programs in the Netherlands, 2012. The values –4 to +4 correspond to the scores given to statements in the

analysis phase, according to their placement by a participant in his or her Q sort. The participant placed one card on each

position on the score sheet, representing his or her level of agreement with the statement (from most disagree to most agree).

In the original sheet presented to participants, the cells were numbered 1 to 9 (from left to right). For the statements used in

the Q sort, see Table 1.

The Q sorts of all participants are then subject to by-person factor analysis to uncover patterns in the

rankings of the statements, under the assumption that correlation between the Q sorts of certain

participants indicates similarity of their viewpoints. The statistical results and the qualitative data (i.e.,

participant comments) are used to interpret and describe the distinct perceptions.

Developing the statement set

Figure 1. Score sheet for the Q sort of 36 statements in a study of user perceptions of workplace-based assessment in daily practice in residency training programs in the Netherlands, 2012.

The values –4 to +4 correspond to the scores given to statements in the analysis phase, according to their placement by a participant in his or her Q sort. The participant placed one card on each position on the score sheet, representing his or her level of agreement with the statement (from most disagree to most agree). In the original sheet presented to participants, the cells were numbered 1 to 9 (from left to right). For the statements used in the Q sort, see Table 1.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 4

68

Developing the statement set

The crucial part of the Q study research instrument is the set of statements to be

used in the Q sort. It is important that the statements are a representative sample of

the subjective communicability around the topic of study—in this case, what users

of WBAs say about the effects of using WBA in PGME—and so allow for expression

of the variety of possible viewpoints. Therefore, we developed our statement set

for this study using the six domains of effects identified in a recent study on this

topic: sentiments, dealing with the innovation, teaching and learning, specialty

training, workload and tasks, and patient care.3 We strove to create a mixture of

statements concerning WBA in general and the mini-CEX and OSATS specifically,

because participants have to be able to express themselves about both general and

specific issues.

The lead researcher (J.F.) formulated statements to represent effects in all six

domains, producing a first set of 72 statements. Subsequently, three of the authors

(M.W., P.W.T., and F.S.) commented on the ambiguity, clarity, and suitability of the

statements. After J.F. made adjustments, the research instrument was pilot-tested

by four research team members (M.W., F.S., N.L., and E.P.) who each conducted a Q

sort with the remaining 59 statements, critically reviewed the full interview materials

(including verbal and written Q sort instructions and the post-sort interview

questionnaire for both written and verbal comments), and commented on the

completeness of the statement set, overlap between statements, and intelligibility

of the statements. Using their comments, J.F. and J.E. further refined wording,

deleted and merged statements, and added one statement. This resulted in a final

set of 36 statements, which was approved by the complete research team (see Table

1). Finally, we randomly numbered the 36 statements and printed them on cards

for participants to use during their Q sorts. We asked the first five participants to

comment on the completeness of the statement set; no further revisions seemed

necessary.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


69

4


Because we aimed to investigate and clarify28 the diversity of viewpoints (rather than

prevalence) we used a purposeful sampling approach in the selection of the study

participants. We invited residents and attendings (including program directors) of

the ob–gyn departments of six hospitals in the Netherlands, and continued inclusion

until a wide variation of viewpoints was seemingly achieved, which is in accordance

with common practice in Q studies.16

We approached the participants through their training groups, because we considered

this to be an effective way to get in contact with them and because the expert

knowledge of two authors (F.S. and J.D.) enabled us to select training groups that

differed in composition and training culture. Invitation and participation preferably

took place during scheduled departmental training group meetings. We requested

that program directors allow J.F. to attend a training group meeting to inform the

residents and attendings who were present about the purpose and procedure of

the study. The residents and attendings who agreed to participate received the

materials to perform the Q sort individually, which they could do immediately after

the meeting or at a later time; in the latter case, they could return the Q sort by mail

or have it collected by J.F. Participants received both written and verbal instructions,

which included tips to facilitate the sorting and an explanation that WBA in this

study referred to the mini-CEX and OSATS. After participants completed the Q sort

by placing the 36 cards on the score sheet, J.F. asked them to comment individually

on why they placed certain statements at the extreme ends of the sort (i.e., most

disagree and most agree). Participants could provide comments in writing or during

brief in-person interviews with J.F. immediately following the Q sort, guided by

the post-sort interview questionnaire mentioned above. Verbal comments were

recorded and transcribed. It took participants on average 20 minutes to complete

the sorting and to provide comments.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 4

70

Analysis

We analyzed the data using dedicated software (PQMethod 2.11, schmolck.userweb.

mwn.de/qmethod). The collected Q sorts were subject to by-person factor analysis

(not by-item, as in conventional factor analysis). The analysis was conducted using

common techniques in Q analysis (centroid factor analysis for extraction of the

factors, followed by varimax rotation).16 First, all factor structures supported by

the data were identified using common criteria of Eigenvalue >1 and a minimum

of two statistically significantly associated Q sorts (P < .05).29 Here, the maximum

number of factors was five. Then, the factor structures were examined by inspecting

all factors in each structure for interpretability and the most comprehensible factor

solution was selected. This was the five-factor solution. For each of these factors, we

generated an idealized Q sort, representing how someone using WBA with exactly

that perception would have ranked the 36 statements in the sorting grid (Figure

1). Finally, statements that had a statistically significantly positioning (P < .05) in a

factor as compared to all other factors (called the distinguishing statements) and

those that were not positioned differently (P < .05) between any pair of factors

(called the consensus statements) were identified. Each factor was then interpreted

and described iteratively by J.F., P.W.T., and J.E. using the idealized Q sort, the

distinguishing statements of that factor, and the comments of participants whose Q

sorts were statistically significantly associated with that factor.


Written informed consent was obtained from all participants, and the data were

processed anonymously. The study was approved by the ethical review board of the

Dutch Society of Medical Education (NVMO-ERB; file number 183).

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


71

4

Tabl

e 1

Com

plet

e Li

st o

f 36

Q S

ort S

tate

men

ts a

nd Id

ealiz

ed Q

Sor

ts fo

r th

e Fi

ve F

acto

rs R

epre

senti

ng P

rinc

ipal

Per

cepti

ons

of E

ffect

s of

Usi

ng

Wor

kpla

ce-B

ased

Ass

essm

ent (

WBA

) in

Prac

tice,

as

Det

erm

ined

Fro

m a

Stu

dy o

f Use

rs o

f WBA

in R

esid

ency

Tra

inin

g in

the

Net

herl

ands

, 201

2a

Fact

or: P

erce

ption

Stat

emen

t1:

Enth

usia

sm2:

Com

plia

nce

3:Eff

ort

4N

eutr

ality

5:Sk

eptic

ism

1.

Hav

ing

a W

BA-f

orm

is c

onve

nien

t for

giv

ing

and

rece

ivin

g fe

edba

ck–1

+20

+2–1

2.

Colle

cting

WBA

s in

a p

ortf

olio

stim

ulat

es c

reati

ng a

list

of f

ocus

poi

nts

for

the

resi

dent

’s e

duca

tion

+4+1

0+3

+1

3.

WBA

s ar

e m

ore

likel

y to

be

done

on

mom

ents

for

whi

ch it

is c

onve

nien

t to

do

them

than

on

mom

ents

whi

ch a

re im

port

ant f

or le

arni

ng th

e jo

b w

ell

–1+1

c–1

–1+4

c

4.

Obs

erva

tion

duri

ng W

BAs

influ

ence

s th

e pe

rfor

man

ce o

f res

iden

tsb

0+1

+10

+15.

D

oing

min

i-CEX

s is

fun

+1–1

–1+2

–16.

W

BAs

prom

ote

atten

tion

to e

duca

tion

+3+2

–1–1

+27.

Co

lleag

ues

disc

uss

the

way

WBA

s sh

ould

be

done

–2–2

–3–2

–48.

Re

side

nts

who

just

sta

rted

thei

r ed

ucati

on p

rofit

mor

e fr

om W

BAs

than

sen

ior

resi

dent

s–1

d–3

+2d

+3d

–2

9.

WBA

s ne

cess

ary

to s

afeg

uard

pati

ent s

afet

y+1

c–3

–2–3

–310

. D

oing

OSA

TS is

diffi

cult

–3–2

–3–2

–1d

11.

WBA

s co

nstit

ute

an e

xtra

wor

k lo

ad

–2c

0+4

d–1

+2d

12.

WBA

s in

crea

se th

e va

lue

of p

rogr

ess

mee

tings

bet

wee

n re

side

nts

and

prog

ram

dire

ctor

s+3

+1+1

+20

13.

Doi

ng O

SATS

is fu

n0d

–1d

+3+3

–2d

14.

WBA

s m

ake

sure

that

sup

ervi

sors

can

not a

void

giv

ing

feed

back

on

poor

pe

rfor

man

ce o

f res

iden

ts a

s w

ell

–1–1

+1+1

–2c

15.

WBA

s cr

eate

resi

stan

ce to

inno

vatio

n in

med

ical

edu

catio

n –2

–3d

–2–2

+1c

16.

Not

eve

rybo

dy u

ses

WBA

s in

an

appr

opri

ate

way

0–1

00

+3c

17.

WBA

s ad

vanc

e th

e ed

ucati

on o

f res

iden

ts

+3c

+10

0–1

18.

WBA

s im

prov

e pa

tient

car

e +1

c–2

–2–3

–419

. Yo

u le

arn

to a

ppre

ciat

e W

BAs

by d

oing

them

–1+2

c–2

+1d

–220

. W

BAs

stim

ulat

e su

perv

isor

s to

giv

e fe

edba

ck+4

+4+2

–10

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 4

72

Fact

or: P

erce

ption

Stat

emen

t1:

Enth

usia

sm2:

Com

plia

nce

3:Eff

ort

4N

eutr

ality

5:Sk

eptic

ism

21.

WBA

s sti

mul

ate

that

resi

dent

s w

ho a

lread

y pe

rfor

m s

uffici

ently

stil

l rec

eive

co

nstr

uctiv

e fe

edba

ck to

impr

ove

furt

herb

+2+1

+1+1

+1

22.

WBA

s di

scou

rage

giv

ing

and

rece

ivin

g fe

edba

ck–4

–4–4

–1d

–323

. D

oing

min

i-CEX

s is

diffi

cult

–3–2

–1–3

+1d

24.

WBA

s sti

mul

ate

the

trai

ning

of g

ener

al c

ompe

tenc

ies

+10

+10

–1c

25.

The

WBA

form

s sti

mul

ate

regi

stra

tion

of im

port

ant f

eedb

ackb

+2+3

+2+1

+126

. W

BAs

stim

ulat

e re

side

nts

to a

sk fo

r fe

edba

ck0

+4c

0–4

c0

27.

WBA

s ta

lly w

ith m

y ow

n id

eas

abou

t wha

t edu

catio

n sh

ould

be

like

00

–10

–128

. W

BAs

prom

ote

the

educ

ation

al c

apac

ities

of s

uper

viso

rs

+2c

00

–10

29.

Doi

ng a

WBA

is e

asie

r w

ith s

ome

peop

le th

an w

ith o

ther

s –1

+3–1

+4+3

30.

WBA

s pr

omot

e fo

rmul

ating

spe

cific

feed

back

+2–1

+30

–3c

31.

Som

e su

perv

isor

s ar

e m

ore

activ

e in

doi

ng W

BAs

than

oth

ers

+1+2

+1+4

+232

. I s

ee li

ttle

use

in d

oing

WBA

s–4

–4–4

–40c

33.

I do

WBA

s be

caus

e do

ing

them

is c

ompu

lsor

y–3

–1–3

–2+2

c

34.

WBA

s in

crea

se th

e nu

mbe

r of

obs

erve

d in

tera

ction

s w

ith p

atien

tsb

00

+2+1

035

. So

me

resi

dent

s in

sist

mor

e on

doi

ng W

BAs

than

oth

ersb

+1+3

+3+2

+336

. O

bser

ving

inte

racti

ons

with

pati

ents

in th

e ou

tpati

ent c

linic

is c

ompl

icat

ed–2

c0d

+4+1

d+4

Abb

revi

ation

s: m

ini-C

EX in

dica

tes

min

i-clin

ical

eva

luati

on e

xerc

ise;

OSA

TS, o

bjec

tive

stru

ctur

ed a

sses

smen

t of t

echn

ical

ski

ll.a Th

e pa

rtici

patin

g us

ers

wer

e re

side

nts,

att

endi

ngs,

and

pro

gram

dire

ctor

s in

dep

artm

ents

of o

bste

tric

s–gy

neco

logy

at 6

hos

pita

ls. T

he s

core

s of

-4 to

+4

corr

espo

nd to

a ra

nkin

g of

the

stat

emen

t fro

m m

ost d

isag

ree

(–4)

to m

ost a

gree

(+4)

; see

Fig

ure

1. T

he s

tate

men

ts w

ere

orig

inal

ly

wri

tten

and

use

d in

the

stud

y in

Dut

ch, a

nd w

ere

tran

slat

ed fo

r th

is p

ublic

ation

. For

the

fact

or lo

adin

gs b

y in

divi

dual

Q s

ort,

see

App

endi

x 1.

b Cons

ensu

s st

atem

ent (

P <

.05)

. c D

istin

guis

hing

sta

tem

ent (

P <

.01)

.d D

istin

guis

hing

sta

tem

ent (

P <

.05)

.

Tabl

e 1

Conti

nued

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


73

4

RESULTS

Of the 74 WBA users invited, 65 individuals (87%) across the ob–gyn residency

programs of 6 institutions (1 academic medical center and 5 general teaching

hospitals) participated. The mean age of participants was 42 years (range: 27–62

years). Among the 65 participants were 22 residents (34%) and 43 attendings (66%);

3 (7%) of the attendings were residency program directors. Nineteen (86%) of the

residents, 21 (49%) of the attendings, and 2 (67%) of the program directors were

female. Two residents, 4 attendings, and 3 program directors who were invited did

not participate due to time constraints.

The 65 Q sorts supported a maximum of five factors, representing five clearly

distinguishable viewpoints about the effects of using WBA in specialty training. Each

factor was defined by 4 to 21 participants, and 19 participants were not (uniquely)

associated with a single factor (see Appendix 1 for the factor loadings matrix by

Q sort). Table 1 presents the idealized Q sorts for the five factors as well as the

distinguishing and consensus statements. The correlation between factors ranged

between 0.27 and 0.62, and the five factors together explained 60% of the total

variance in the Q sorts (Table 2).

Table 2 Characteristics of Five Factors Representing Principal Perceptions of Effects of Using Workplace-Based Assessment in Practice, as Determined From a Study of Users of WBA in Residency Training in the Netherlands, 2012a

Factor: PerceptionCharacteristic 1:

Enthusiasm2:

Compliance3:

Effort4:

Neutrality5:

SkepticismNumber of defining participants 11 21 4 5 5% of variance explained by the factor 15 22 10 7 6

aThe participating users were residents, attendings, and program directors in departments of obstetrics–gynecology at 6 hospitals. Of the 65 participants, 19 were not uniquely associated with a single factor. For the factor loadings by individual Q sort, see Appendix 1.

Below, we describe the five factors, each of which represents a distinct perception

of the effects of using WBA in PGME: enthusiasm, compliance, effort, neutrality, and

skepticism. The perceptions vary with regard to three main issues: the goals the

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 4

74

innovation was intended to achieve, the ease or difficulty of applying the innovation

to practice (the innovation’s applicability), and the innovation’s actual impact. In

each description, the parenthetical numbers refer to the number of the defining

statement (e.g., #32) and its position in the factor’s idealized Q sort (e.g., –4) (see

Table 1). Each description concludes with an overview of the group of corresponding

participants; additional details about individual Q sorts can be found in Appendix 1.

Table 3 provides representative comments of the participants associated with each

perception.

Table 3 Impressions of the Five Principal Perceptions of Effects of Using Workplace-Based Assessment (WBA) in Practice, From a Study of Users of WBA in Residency Training in The Netherlands, 2012a

Perception Representative comments (participant type, Q sort number)Enthusiasm • The usefulness is obvious. It is important for the residents as well as for the

complete chain of care. (A, 12)• The method makes it easier to give feedback, including difficult feedback.

(A, 3)• WBAs stimulate learning, because those are the specific assessments,

when they [the residents] hear what their points of improvement are. (A, 4)

• WBAs stimulate attention to training, because in the ongoing daily practice it is important to consciously choose moments of feedback. (A, 12)

• Feedback is essential in an open and safe work atmosphere, which is necessary for patient safety. (A, 5)

• Well, about the workload… I rather consider it a contribution to why I enjoy my job. (A, 5)

Compliance • Where no feedback was given before, we are now obliged to say something about it [resident performance], which is good. (A, 51)

• I do see the benefit, the structured feedback. (R, 58)• Has little to do with patient safety. (A, 33)• It seems a big time investment, and you fear the criticism, but afterwards I

am usually glad that I received feedback. (R, 65)• WBAs create awareness of the necessity of permanent attention to

feedback. (A, 64)

Effort • It does make sense, and I endorse the system, but you can’t deny that it means an extra workload. Also, it all takes more time. (A, 26)

• WBAs encourage feedback. (A, 9)• It’s a good tool for structuring the training. (A, 56)• The items on the OSATS forms simply do not fit with reality. That does

make it hard to profit from it as much as I would want to. (A, 26)• Mini-CEXs turn out to be hard to organize in practice; planning them helps

up to a point.(A, 43)

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


75

4

Neutrality • I do acknowledge the usefulness of structured feedback. (A, 14)• You can teach the juniors more [than the senior residents]. (A, 13)• Certain people ask for feedback, others less so. I doubt if WBAs stimulate

that, asking for feedback. On the contrary, perhaps, sometimes [they discourage feedback]. (A, 24)

• It’s not difficult, of course, you can just discuss it together. (A, 24)

Skepticism • Compulsion makes me do it, but personally I have reasonable doubts about this method. (A, 16)

• In practice, it gets down to collecting as many WBAs each year as are required for the portfolio. (A, 52)

• It’s very easy to fill out a WBA without being specific about anything: ‘well done’, ‘keep it up’ and ‘no points for improvement’. (R, 47)

• The goal is training, not improvement of CARE. (A, 16)• It is not a pleasure. It’s compulsory for the residents, which also makes it

compulsory for me as a supervisor. (A, 52)

Abbreviations: A indicates attending physician; R, resident.aThe 65 participating users were residents, attendings, and program directors in departments of obstetrics–gynecology at 6 hospitals. Program directors are labeled as attending physicians to preserve their anonymity (n < 5). The comments were originally provided in Dutch and were translated for this publication.

Enthusiasm (factor 1)

In this perception, WBA is viewed as a useful innovation (#32: –4) because it

stimulates multiple aspects of training. First, WBA is considered to contribute to

providing feedback and formulating points of improvement for residents (#2 and

#20: +4; #22: –4). Furthermore, it is valued for its broader effects on training, such

as stimulating learning (#17: +3), increasing the significance of progress meetings

between program directors and residents (#12: +3), and promoting attention to

training (#6: +3). The innovation is also perceived as related to improving patient

care (#18: +1) and safety (#9: +1). WBA is not considered difficult to use (#36: –2; #10

and #23: –3), and obligation is not the main impetus for using WBA (#33: –3). The

innovation, which is considered valuable and easy to use, is not perceived as an extra

workload (#11: –2).

The 11 participants (1 resident, 9 attendings, 1 program director) who contributed to

this perception worked in 4 different hospitals.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 4

76

Compliance (factor 2)

In this perception, WBA is deemed highly useful (#32: –4), because it empowers users

to initiate feedback, including difficult feedback, and to request attention for training

(#20 and #26: +4; #22: –4). Use of the innovation is viewed as directed by practical

opportunity rather than by the educational importance of the moment (#3: +1).

This perception focuses on direct effects of WBA; for instance, giving and registering

feedback is perceived as supported by the structured forms (#1: +2; #25: +3). Broader

educational effects are not experienced--such as formulating learning goals, or

improving progress meetings or educational capabilities of attendings (#2, #12, and

#28: 0)—and no links are perceived with patient care (#18: –2) and safety (#9: –3).

Appreciation for the innovation increases once the user gets used to it (#19: +2),

which is unique to this perception. Creating resistance to further medical education

innovations is not deemed a problem (#15: –3).

This perception was defined by 21 participants (7 attendings, 14 residents) from 4

different hospitals.

Effort (factor 3)

In this perception, the usefulness of WBA is endorsed (#32: –4) and there is a

willingness to use the innovation (#33: –3). The perceived usefulness is mainly

related to direct effects, such as encouragement of exchange (#22: –4; #20: +2),

specificity (#30: +3), and registration (#25: +2) of feedback. Benefits of the innovation

are considered to affect beginning residents more than senior residents (#8: +2).

Despite the endorsement, using the innovation in practice is experienced as an extra

workload (#11: +4). While OSATS is considered fun and not difficult (#13: +3; #10:

–3), WBA is perceived as taking time and the forms are not considered helpful (#1

and #2: 0). Organizing mini-CEXs to observe interactions with patients is considered

particularly hard work (#36: +4). Although there is an evident struggle to integrate

the innovation into practice, the use of WBA is not a topic of discussion among

colleagues (#7: –3).

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


77

4

Four participants (3 attendings, 1 program director) in 4 hospitals adhered to this

perception.

Neutrality (factor 4)

In this perception, the potential value of WBA is endorsed in principle (#32: –4). The

innovation is mainly perceived as supporting the training of inexperienced residents

(#8: +3); it helps target points of improvement (#2: +3) and increases the value of

progress meetings (#12: +2). However, the actual execution and consequences of

WBA are perceived as depending on the individuals involved (#29: +4; #35: +2). WBA

is not thought to stimulate seeking feedback (#26: –4) or providing feedback (#22:

–1), although this view is not related to difficulty using the innovation (#10: –2; #23:

–3).

Five participants (all attendings) in 2 hospitals were associated with this perception.

Skepticism (factor 5)

In this perception, WBA is not viewed as useful (#32: 0). The innovation is considered

to enhance attention for education in general (#6: +2), but not to contribute to

specificity of feedback (#30: –3), the value of progress meetings (#12: 0), or residents’

education (#17: –1). Patient care and safety are not believed to benefit either (#18:

–4; #9: –3).

Obligation drives use of the innovation (#33: +2). Convenience, rather than relevance

for practice, directs utilization of the tools (#3: +4), which are often filled out in a

meaningless way (#16: +3). Accordingly, WBA is perceived as an extra workload (#11:

+2), and appreciation does not increase with use of the innovation (#19: –2). Further,

WBA is viewed as creating resistance to other innovations in PGME (#15: +1).

This perception was defined by 5 participants (3 attendings, 2 residents) in 4 hospitals.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 4

78

DISCUSSION

This study focusing on WBA shows that users of an innovation in residency training

programs have different perceptions of its effects in practice. We uncovered and

described five principal perceptions of the effects of using WBA in PGME: enthusiasm,

compliance, effort, neutrality, and skepticism.

These five perceptions are characterized by the users’ differing views on three main

issues: the goals, the applicability, and the impact of the innovation. Participants’

views of the goals of WBA—experienced as the reasons why the innovation is or is

not useful—ranged from being confined to stimulating feedback to being broader

and including indirect effects, such as influencing patient care and attitudes toward

change in general. The views of WBA’s applicability (i.e., whether applying the

innovation to practice is easy or difficult) varied by perception as did views of its

impact on the quality of residents’ training; the latter ranged from positive impact

to no impact.

Our findings resonate with others’ recent descriptions of differences in the viewpoints

of individuals involved in changes in medical education. For example, Jacobs et al

found that undergraduate medical education teachers have differing conceptions

of learning and teaching.30 In a study on the influences of culture on successful

curriculum change in medical schools, Jippes noted that individuals have different

perceptions of their organization’s readiness for change.31 These kinds of individual

differences should be taken into account in change management.

In our study, all five factors, representing different perceptions, were statistically

significantly associated with participants from more than one training group;

similarly, participants from each training group were associated with various

factors. This finding indicates that the variability in perceptions does not result only

from contextual differences (e.g., implementation strategy, organization of clinical

practice), but also from differences among individuals in the same context.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


79

4

The five different perceptions of the effects of an innovation in this study expand

two aspects of existing sociological theory about innovations. First, theory regarding

innovations in social systems has mainly focused on the diffusion of an innovation

through a system. Rogers’s well-known classification of individuals is based on the

speed with which they adopt new ideas relative to other individuals within the system

and specifies five adopter categories: innovators, early adopters, early majority, late

majority, and laggards.32 Our findings add to this classification of adoption speed

by showing that individuals also differ in what they think they are adopting, as

they perceive different goals, differences inapplicability to practice, and different

impacts. Adopters are not all travelling along the same track at different speeds;

instead, they may be taking different routes and therefore may be reaching different

destinations. Second, diffusion theory makes distinctions among an innovation’s

form, its function, and its social meaning. The theory states that form is indisputable,

but that developers and adopters may perceive the function and the social meaning

very differently.10 Our results indicate that such differences exist not only between

developers and adopters, but also among adopters.

Using Q methodology allowed us to systematically investigate individuals’ perceptions.

The five principal perceptions we identified are based on—and applicable to—use of

an innovation in daily practice, as users’ behaviors and attitudes can be recognized as

belonging to certain perceptions. This may help colleagues understand one another’s

responses to an innovation. It should be noted that individuals are likely to recognize

aspects of several perceptions as their own.

We focused on the use of one innovation, WBA, in residency training and on users of

this innovation in one specialty, ob–gyn. Therefore, the identified typology may be

limited by perceptions typical for this group of people involved with innovations in

residency training. Further research on other innovations and in different contexts is

needed to determine the consistency of the described perceptions.

Also, we focused on perceived effects of the actual use of an innovation as opposed

to perceived effects of implementation; therefore, we cannot assess effects of

differences in different hospitals’ approaches to implementation. Still, while local

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 4

80

influences may have contributed to the viewpoints of various participants, none

of the perceptions we identified was exclusively associated with any one of the

participating hospitals.

Participants were included by purposeful sampling for variation, which strove for

representation of the entire range of viewpoints. The range of the perceptions we

identified indicates that this variation was covered. However, as we were dependent

on individuals’ willingness to participate and some individuals declined participation,

we cannot exclude the possibility of an optimistic or enthusiastic bias in the reported

perceptions.

Our results provide a starting point for customizing approaches to change in medical

education. How or why individuals develop certain perceptions remains largely

unexplored, however. For example, our finding that the compliance perception was

associated with a large share of residents could serve as a basis for exploring whether

a user’s role in the application of an innovation influences that user’s perception

of the innovation’s effects. We did not investigate if perceptions developed over

time. However, knowledge about the development of perceptions could inform

implementation interventions at an early stage of change, when users’ perceptions

may be more pliable.

The primary aim of implementation should be to reach formulated goals, that is,

to bring about the intended effects of an innovation rather than to achieve the

highest possible usage rate of a method or tool. Our findings imply that leaders of

change need to anticipate that different users will have different perceptions of an

innovation. Basing the implementation approach on leaders’ own perceptions is

unlikely to suffice for meaningful adoption of the innovation.33 A starting point for

discussing the innovation can be found in the commonalities between the different

perceptions--in the shared issues of perceived goals, applicability, and impact.

Inviting others to share their perceptions could reveal unintended effects that turn

out to be hidden treasures of the innovation.

Of importance for the implementation of innovations is our finding that pressuring

people to use the innovation resulted only in intended effects for those users who

acknowledged the method’s usefulness and needed empowerment to use it. For

others, this compulsion resulted in opportunistic and meaningless use, which does

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


81

4

not contribute to the intended training effects. This reconfirms that obligation is a

questionable way to achieve full adoption of an innovation.34

Emphasis on effects instead of tools should lead the way not only for implementation,

but also for evaluation. Evaluation should at least demonstrate to what extent the

intended effects of the innovation have been achieved; it would be preferable to

also demonstrate what unintended effects have occurred. With respect to an

implemented method or tool, exploring in what ways it is used may be more relevant

than determining to what extent it is used.

CONCLUSION

Innovations in medical education elicit various effects in practice, and users of

innovations perceive these effects differently. This study provides insight into five

principal user perceptions which can support the design and implementation

of innovations in PGME. For effective change in medical education, it is critical to

pay attention to users’ perceptions of the goals, the applicability, and the impact

of innovations. Paying attention only to the operationalization of an innovation and

pressuring people to adopt it are unlikely to suffice for bringing about the intended

effects of an innovation in medical education.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 4

82

REFERENCES

1. Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Unexpected collateral effects of simulation-based medical education. Acad Med. 2011;86:1513-1517.

2. Sargeant J, Mann K, Sinclair D, van der Vleuten C, Metsemakers J. Challenges in multisource feedback: Intended and unintended outcomes. Medical Education. 2007;41:583-591.

3. Fokkema JPI, Teunissen PW, Westerman M et al. Exploration of perceived effects of innovations in postgraduate medical education. Med Educ. 2013;47:271-281.

4. Mennin SP, Krackov SK. Reflections on relevance, resistance, and reform in medical education. Academic Medicine. 1998;73:S60-S64.

5. Hillis DJ. Managing the complexity of change in postgraduate surgical education and training. ANZ Journal of Surgery. 2009;79:208-213.

6. Lillevang G, Bugge L, Beck H, Joost-Rethans J, Ringsted C. Evaluation of a national process of reforming curricula in postgraduate medical education. Med Teach. 2009;31:e260-e266.

7. Kotter JP. Leading change. Boston, Mass: Harvard Business School Press; 1996. 8. Bland CJP, Starnaman SP, Wersal LM, Moorhead-Rosenberg LP, Zonia SP, Henry

RP. Curricular change in medical schools: How to succeed. Academic Medicine. 2000;75:575-594.

9. Jippes E, Van Luijk S, Pols J, Achterkamp M, Brand P, Van Engelen J. Facilitators and barriers to a nationwide implementation of competency-based postgraduate medical curricula: A qualitative study. Med Teach. 2012;34:e589-e602.

10. Rogers EM. Consequences of innovations. In: Rogers EM. Diffusion of innovations. 5th ed. New York: Free Press; 2003;436-471.

11. Pelgrim E, Kramer A, Mokkink H, van den Elsen L, Grol R, van der Vleuten C. In-training assessment using direct observation of single-patient encounters: A literature review. Advances in Health Sciences Education. 2011;16:131-142.

12. Bindal T, Wall D, Goodyear HM. Trainee doctors’ views on workplace-based assessments: Are they just a tick box exercise? Med Teach. 2011;33:919-927.

13. Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: A systematic review. BMJ. 2010;341:c5064.

14. Daelmans HEM, Overmeer RM, van der Hem-Stokroos HH, Scherpbier AJJA, Stehouwer CDA, van der Vleuten CPM. In-training assessment: Qualitative study of effects on supervision and feedback in an undergraduate clinical rotation. Med Educ. 2006;40:51-58.

15. Ringsted C, Henriksen AH, Skaarup AM, van der Vleuten CPM. Educational impact of in-training assessment (ITA) in postgraduate medical education: A qualitative study of an ITA programme in actual practice. Med Educ. 2004;38:767-777.

16. Watts S, Stenner P. Doing Q Methodological Research: Theory, Method and Interpretation. London: SAGE; 2012.

17. van Reenen R, den Rooyen C, Schelfhout-van Deventer V. Modernisering medische vervolgopleidingen: nieuw kaderbesluit CCMS [framework for the Dutch Specialist Training Program]. [In Dutch.] Kaderbesluit centraal college medische specialismen. Utrecht, the Netherlands: Koninklijke Nederlandse Maatschappij tot bevordering der

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


83

4

Geneeskunst. Updated 2011. http://knmg.artsennet.nl/Opleiding-en-herregistratie/Algemene-informatie/Nieuws/O-R-Nieuwsartikel/Modernisering-medische-vervolgopleidingen-nieuw-kaderbesluit-CCMS.htm. Accessed January 7, 2014.

18. Scheele F, Teunissen P, Van Luijk S et al. Introducing competency-based postgraduate medical education in the Netherlands. Med Teach. 2008;30:248-253.

19. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): A preliminary investigation. Ann Intern Med. 1995;123:795-799.

20. Martin JA, Regehr G, Reznick R et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84:273-278.

21. Wallenburg I, van Exel J, Stolk E, Scheele F, de Bont A, Meurs P. Between trust and accountability: Different perspectives on the modernization of postgraduate medical training in the Netherlands. Academic Medicine. 2010;85:1082-1090.

22. Buljac-Samardzic M, van Wijngaarden JDH, van Wijk KP, van Exel NJA. Perceptions of team workers in youth care of what makes teamwork effective. Health Soc Care Community 2011;19:307-316.

23. Nikolaus S, Bode C, Taal E, van de Laar MAFJ. Four different patterns of fatigue in rheumatoid arthritis patients: Results of a Q-sort study. Rheumatology. 2010;49:2191-2199.

24. Cross RM. Exploring attitudes: The case for Q methodology. Health Education Research. 2005;20:206-213.

25. Steelman TA, Maguire LA. Understanding participant perspectives: Q-methodology in national forest management. J Pol Anal Manage. 1999;18:361-388.

26. Rajé F. Using Q methodology to develop more perceptive insights on transport and social inclusion. Transport Policy. 2007;14:467-477.

27. De Graaf G, van Exel J. Using Q methodology in administrative ethics. Public Integrity 2008;11:63-78.

28. Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: A framework for classifying the purposes of research in medical education. Med Educ. 2008;42:128-133.

29. Watts S, Stenner P. Understanding the analytic process (1): Factor extraction. In: Watts S, Stenner P. Doing Q Methodological Research: Theory, Method and Interpretation. London: SAGE; 2012;91-112.

30. Jacobs JCG, Van Luijk SJ, Van Berkel H, van der Vleuten CPM, Croiset G, Scheele F. Development of an instrument (the COLT) to measure conceptions on learning and teaching of teachers, in student-centred medical education. Med Teach. 2012;34:e483-e491.

31. Jippes M. Culture Matters in Medical Schools. How Values Shape a Successful Curriculum Change. Doctoral dissertation, School of Health Professions Education, Maastricht University, 2012. http://digitalarchive.maastrichtuniversity.nl/fedora/get/guid:363a64da-6ac6-49b0-98f8-ae99fcd1cf27/ASSET1. Accessed April 25, 2014.

32. Rogers EM. Elements of diffusion. In: Rogers EM. Diffusion of Innovations. 5th ed. New York: Free Press; 2003:1-38.

33. Fokkema J, Westerman M, Teunissen P et al. How lead consultants approach educational change in postgraduate medical education. Med Educ. 2012;46:390-398.

34. van der Vleuten C, Verhoeven B. In-training assessment developments in postgraduate education in Europe. ANZ J Surg. 2013;83:454-459.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 4

84

Appendix 1: Individual Q Sort Loadings for Five Factors Representing Principal Perceptions of Effects of Using Workplace-Based Assessment (WBA) in Practice, as Determined From a Study of Users of WBA in Residency Training in the Netherlands, 2012a

Q sort Participantcharacteristic

Factor: Perception

1:Enthusiasm

2:Compliance

3:Effort

4:Neutrality

5:SkepticismRole Hospital

1 R 1 0.30 0.33 0.40 0.11 0.162 A 1 0.14 0.66b 0.16 0.10 –0.093 A 1 0.69b 0.53 0.15 0.16 0.134 A 1 0.83b –0.01 0.22 0.09 –0.045 A 1 0.61b 0.26 0.26 0.27 –0.256 A 2 0.60 0.52 0.23 0.16 –0.117 R 1 0.31 0.53 0.45 0.11 0.078 A 2 0.33 0.56 0.61 0.08 –0.059 A 2 0.19 0.20 0.66b 0.06 0.2910 A 2 0.14 0.39 0.11 0.08 0.71b

11 R 2 0.26 0.58b 0.06 0.02 0.0512 A 2 0.73b 0.36 0.01 –0.15 –0.1313 A 2 0.23 –0.25 0.22 0.66b 0.2014 A 2 0.23 0.29 0.07 0.56b 0.2515 R 2 0.15 0.75b 0.10 0.36 0.0216 A 2 –0.23 0.17 0.11 0.09 0.69b

17 R 2 0.26 0.62b 0.11 0.21 0.2418 R 1 0.44 0.69b 0.19 0.02 0.0519 A 1 0.62b 0.25 0.17 0.45 –0.2420 A 1 0.34 0.50 0.43 0.04 0.2121 R 2 0.29 0.58b 0.32 0.23 0.1122 A 2 0.35 0.36 0.43 0.37 –0.0323 A 2 0.34 0.64b 0.10 0.07 0.3124 A 2 0.34 0.27 0.24 0.59b 0.1925 A 2 0.53b 0.04 0.35 0.26 0.0226 A 1 0.01 0.21 0.54b 0.13 0.0227 A 1 0.68b –0.03 0.15 0.24 0.1028 R 2 0.20 0.54b 0.31 0.12 0.3629 A 1 0.68b 0.35 0.14 –0.07 –0.3530 A 2 0.33 0.54 0.44 –0.02 0.1631 R 2 0.47 0.62 –0.02 0.26 0.3032 R 3 0.23 0.67b 0.24 0.01 0.3933 A 3 0.14 0.57b 0.42 0.22 0.0834 R 3 –0.04 –0.07 0.07 0.04 0.46b

35 A 3 0.20 0.60b 0.11 0.45 0.3336 A 2 0.49 0.42 0.09 0.44 –0.0637 R 1 0.36 0.37 0.48 0.05 –0.2538 R 1 –0.10 0.49b 0.44 0.03 0.03

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


85

4

Q sort Participantcharacteristic

Factor: Perception

1:Enthusiasm

2:Compliance

3:Effort

4:Neutrality

5:SkepticismRole Hospital

39 R 2 0.31 0.55b 0.35 0.01 0.1140 A 4 0.27 0.47 0.57 0.19 0.1441 A 4 0.38 0.37 0.19 0.60b –0.0242 A 4 0.53 0.48 0.02 0.24 0.2743 A 4 0.20 0.09 0.74b 0.36 0.1944 A 4 0.29 0.46 0.21 0.43 –0.0545 A 4 –0.14 0.08 0.13 0.70b –0.0746 A 4 0.35 0.15 0.43 0.41 0.0047 R 5 0.01 0.54 0.03 0.14 0.68b

48 R 5 0.28 0.51b 0.37 0.13 0.0349 R 5 0.02 0.65b 0.46 0.08 0.1150 A 5 0.34 0.69b 0.27 0.14 –0.0951 A 5 0.44 0.62b 0.38 –0.02 –0.0452 A 5 –0.14 0.07 0.07 –0.03 0.53b

53 A 5 0.58b 0.45 0.15 0.17 –0.1354 A 5 0.19 0.23 0.25 –0.31 0.2855 A 6 0.38 0.44 0.33 0.05 –0.3856 A 6 0.22 0.07 0.52b 0.33 0.1457 A 6 0.48 0.43 0.46 0.10 0.1258 R 6 0.00 0.60b 0.24 0.36 0.1759 R 6 0.11 0.48 0.24 0.43 0.1360 A 6 0.70b 0.55 0.18 –0.14 –0.0661 A 6 0.48 0.36 0.39 0.37 –0.1062 R 6 0.46 0.67b 0.06 0.11 –0.1263 R 6 0.57b 0.30 0.24 0.25 0.1964 A 2 0.10 0.56b 0.36 0.00 0.3265 R 5 –0.05 0.69b 0.26 0.07 0.29

aThe 65 participating users were residents (Rs), attending physicians (As), and program directors in departments of obstetrics–gynecology at 6 hospitals. In this table, program directors are labeled as attending physicians to preserve their anonymity (n < 5). The six hospitals are labeled 1 through 6.bDefining Q sort for factor. A Q sort defines a factor if: (i) the Q sort correlates statistically significantly with that factor; the loading of a respondent on a factor should exceed the multiplier for the statistical significance level (P = .05) divided by the square root of the number of statements, in this case:

84

Q sort

Participant characteristic

Factor: Perception 1:

Enthusiasm 2:

Compliance 3:

Effort 4:

Neutrality 5:

Skepticism Role Hospital 64 A 2 0.10 0.56b 0.36 0.00 0.32

65 R 5 –0.05 0.69b 0.26 0.07 0.29 aThe 65 participating users were residents (Rs), attending physicians (As), and program directors in departments of obstetrics–gynecology at 6 hospitals. In this table, program directors are labeled as attending physicians to preserve their anonymity (n < 5). The six hospitals are labeled 1 through 6.

bDefining Q sort for factor. A Q sort defines a factor if: (i) the Q sort correlates statistically significantly with that factor; the loading of a respondent on a factor should exceed the multiplier for the statistical significance level (P = .05) divided by the square root of the number of statements, in this

case: 0.33361.96 = ; and(ii) the factor explains more than half of the common variance; that is, the square of the loading on that factor should

exceed the sum of the squares of the factor loadings on the remaining factors.

; and(ii) the factor explains more than half of the common variance; that is, the square of the loading on that factor should exceed the sum of the squares of the factor loadings on the remaining factors.

5Implementing transparency and competition

in medical specialty training

Joanne P.I. Fokkema, Pim W. Teunissen, Albert J.J.A. Scherpbier,

Cees P.M. van der Vleuten, P. Joep Dörr, Fedde Scheele

Under review

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 5

88

ABSTRACT

PurposeThe feasibility of transparency and competition in medical specialty training to raise accountability and quality of training is unclear. By studying a case in specialty training, the authors aimed to explore effects of introducing transparency and competition and how these are influenced by the implementation approach.

MethodIn 2013, the authors conducted a case study of a Dutch project between 2009 and 2013 that aimed to improve accountability and quality of specialty training by introducing transparency and competition for electives. Proceedings of project meetings were thematically analyzed to identify choices and developments regarding the implementation approach and to assess the effects of various approaches. A theory-driven methodology was used for the analysis, based on theory from implementation science.

ResultsThe authors identified effects and features of the implementation approach in the themes Transparency, Competition, and Obstacles for competition. They found effects and implementation approach to be strongly interdependent. By the end of the project, 79% of participating programs had displayed electives on an openly accessible website. Displaying electives increased feasibility of individualizing training, but did not yet lead to competition for electives. The authors saw a shift from a discourse of training command by program directors, to a discourse where trainees’ command became relevant. Stakeholder involvement revealed various obstacles to resident mobility, including social-structural obstacles (like strictness of regulations), and social-cognitive obstacles (like the value of senior trainees to do clinical work).

ConclusionTransparency about available options in specialty training has the potential to contribute to quality. However, competition may be hampered by various obstacles, which can be revealed by stakeholder involvement. Adapting implementation approach to the obstacles supports development of fundamental changes. With these findings, this study illustrates the dynamic nature of the approach and effects in an innovation process in specialty training.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Implementing transparency and competition

89

5

INTRODUCTION

Postgraduate medical education is well on its way towards a competency-driven and

outcome-based approach. The main features of the newly implemented specialty

training programs are clear learning goals and assessment, mainly based on

performance in actual practice. The reform included, among others, the introduction

of a well-defined training structure, faculty development and a more systematic

approach to assessment of trainees.1

Now, new aspects of specialty training come to the fore that warrant attention

and require innovative solutions. A first matter of attention is that physicians need

to develop specific areas of expertise to cope with the increasing complexity and

specialization of patient care.2 A competency-based approach to training caters to

this requirement by allowing individual trainees to optimally customize training to

their learning needs. As a consequence, these individualized learning paths will no

longer all have the same end point or even duration. Optimal preparation for future

practice could be supported by rounding off specialty training with purposefully

chosen electives.

Secondly, quality of training is still hard to determine. Various new elements in

training are supposed to contribute to training quality, such as workplace-based

assessment tools like mini-clinical evaluation exercises.3 Also, instruments have been

introduced to assess components of clinical training, for instance learning climate4 or

teaching qualities of faculty5. However, determining the overall quality of a training

program, i.e. the outcome of all the elements combined, remains an unresolved

challenge. This means that it is not transparent how training programs measure up

against each other and how programs can learn from each other to stimulate training

quality and to increase accountability of investments.6

In the fields of healthcare and higher education, the current trend is to utilize

transparency about performance to generate accountability and to improve

quality.7-9 The concept of transparency about performance and other relevant data is

well-established in traditionally competitive sectors like the airline industry. There, it

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 5

90

is known to stimulate quality aspects, such as customer satisfaction and productivity,

and to reduce costs.10 Following this line of reasoning, individualization and quality of

medical specialty training might be improved by introducing more transparency and

competition in this field. It is worth investigating what effects would be achieved by

employing such a mechanism in specialty training.

However, it is clear that bringing about an intended change in medical education by

introducing innovative concepts is not straightforward, as has been demonstrated

by the introduction of competency-based education.1;11;12 An innovation in specialty

training is likely to bring about unintended effects in other domains than the intended

ones13;14, and individuals differ in which effects they perceive most clearly.15 There

is relatively little understanding of how effects of innovations in medical education

come about.16 As long as these processes are not understood, it is insufficient to

limit assessment of innovations in specialty training to intended outcomes. Instead,

assessors of innovations need to be perceptive to all effects, and to the underlying

mechanisms of development of the effects.17 The nature of the implementation

approach has been suggested to influence the development of effects.18

In this study, we aimed to gain insight into the effects of transparency and

competition in specialty training, and into aspects of the implementation approach

that are relevant for the achievement of those effects. We studied a recent case

of introducing transparency and competition for final-stage training electives in the

Netherlands. The questions that guided our research were: what are the effects

of implementing transparency and competition in specialty training, and how are

these effects influenced by the implementation approach? We took a theory-driven

approach to qualitatively analyze project-generated documents and stakeholder

insights.19

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


91

5

METHOD

Setting

Specialty training in the Netherlands

Medical specialty training in the Netherlands lasts three to six years, depending on

the specialty. It is fully funded by the Ministry of Health, which also yearly decrees

the number of trainees that can enroll in each specialty training program. There are

national training guidelines for specialty training in general, and for each specialty

separately.20

For most specialties, training is offered by eight clusters consisting of one university

hospital and several affiliated general teaching hospitals. Most of these hospitals

employ training staff who support the program directors of all specialties. Following

national requirements, all specialty training programs had to be organized as

competency-based curricula in 2011 at the latest. The new curricula of most specialty

training programs contain elective parts in the final one or two years, which usually

focus on specialized medical content. Each training department is supposed to have

an own training directive, which includes descriptions of the electives offered in that

department.

Display Project

In an effort to stimulate individualization and quality of residency training, the Dutch

government and several organizations of medical specialists saw a possible solution

in combining these two issues by creating transparency and competition for electives.

The ministry of Health provided a 1.5 million euro grant to test this concept in a

4-year project called the Display Project (2010 until 2013) (www.opleidingsetalage.

nl).

The Display Project’s objective was to have program directors display information

about their electives on an openly accessible national website, including all the

details that they considered relevant for trainees to make an optimal choice. Also,

trainees who had completed an elective should be enabled to add their review to

the website’s information, in order to support other trainees in choosing the most

suitable elective. This process was to provoke professional competition among

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 5

92

training programs as well as among trainees: program directors would be encouraged

to design relevant electives and maintain high quality of training in order to attract

trainees, while trainees could face competition for admission to popular electives.

Eventually, trends in trainees’ choices might even provide indications of the quality

differences between electives offered at different training departments.

The national organizations of nine specialties applied to participate in the project:

Internal Medicine, Surgery, Obstetrics-Gynecology, Pediatrics, Neurology, Radiology,

Anesthesiology, Orthopedic surgery, and Plastic, Reconstructive and Aesthetic

surgery. The project management was entrusted to OMS, the organization that

represents all medical specialties in the Netherlands, which employed staff consisting

of a project manager, a project supervisor, and a secretary. The project was led by

a project team consisting of 11 persons: the initiators (representatives of program

directors and training staff), staff, trainee representatives and educationalists. Also, a

comprehensive project group was formed, consisting of representatives of all parties

involved with specialty training, to ensure contact with all those stakeholders.

Design

This study was conducted from a constructivist viewpoint.21 This means that we

assume that knowledge is constructed by researchers in interaction with data and

that it is not possible to present an absolute truth. We conducted a case study of

the Display Project to clarify what happens when introducing transparency and

competition in specialty training. Case studies are suitable for clarification in general16

and to gain understanding of innovation processes specifically22.

Using a theoretical model about complex innovations in the field of healthcare23, we

took a theory-driven approach19, because this is known to support the study of complex

processes. The model contains four main constructs involved in implementation

processes: 1) Contribution: what agents do to implement an innovation; 2) Potential:

social-cognitive resources available to agents, e.g. agents’ readiness to implement

and adopt the innovation; 3) Capacity: social-structural resources available to agents,

e.g. what means agents can employ to implement the innovation, including their

social networks; 4) Capability: possibilities presented by the innovation, e.g. its

components, whether these can be integrated into practice, and its effects.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


93

5

We employed a thematic analysis of the official documents generated during the

four-year project period, because this provided a way to include possible alterations

in the implementation approach and track effects and developments.24;25

Researchers and analysis

The main researcher was involved with the Display Project as a critical observer,

whose role it was to provide the project team with feedback. In this function, she

spoke with various stakeholders and attended meetings throughout the project.

Two of the other research team members (PJD and FS) were active members of the

project team.

We used template analysis as a supporting technique for thematic analysis of data.26

An initial template was used containing the four constructs of the General Theory of

Implementation. In 2013, the main researcher conducted the analysis by identifying

sections that considered implementation approaches or effects of the project, and

labeled them with a code referring to the content (either a code from the template or,

if necessary, a new code). An iterative process was chosen for thematic data analysis

and modification of the template. PJD and FS did not participate in the first round

of data analysis in order to prevent directing findings to their personal experiences.

Also, the entire research team discussed the initial template, major alterations

and the final template in order to explicate possible preconceptions, prevent early

narrowing of ideas and challenge the thought process.

Documents

All documents produced or collected by the project staff were available to the

research team. Since our interest included the implementation approach, we

selected documents reporting meetings of the project’s agents as primary documents

for analysis. These included minutes of all meetings of the project team and the

project group, project proposals, progress reports, and minutes of meetings with the

ministry of Health and other specially organized meetings with other parties. Of the

staff meetings, which took place weekly, minutes of one meeting per semester were

selected. Other documents could be studied as secondary documents if the content

of primary documents pointed to information that might be relevant to our research,

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 5

94

for example if proceedings referred to published articles about the Display Project or

the survey on stakeholders’ attitudes that was conducted in 2013.


This study was approved by the Ethical Review Board of the Dutch Society for Medical

Education (NVMO-ERB, file number 288)

RESULTS

In analysis, we recognized several kinds of effects, both within and outside the scope

of training quality and individualization. Also, we recognized several distinctive

features of the implementation approach that influenced these effects. Effects and

implementation approach were found to be strongly interdependent: effects were

influenced by the approach, and the approach was adapted depending on developing

effects. We present our findings in themes, illustrated by specific examples from our

data.

Transparency

A national website was created to facilitate display and comparison of electives of

nine participating specialties (www.opleidingsetalage.nl). Several rounds of input

of trainees and program directors resulted in adjustments to the website, until

all the requirements were met. In the course of the project, preexisting electives

were displayed on the website. Also, new electives were developed, sometimes by

collaboration of two departments. Most electives aimed for medical specialization,

while some focused on generic competences, such as supervising skills. By the end

of the project period, the majority (222/271, 79%) of program directors of the nine

participating specialties had at least one elective on the national website.

Approaches and effects

The project team stimulated display of electives by appealing to the program directors’

professional entrepreneurship and to their personal motivation, for instance by

listing advantages of participating in the project, like attracting the best residents.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


95

5

A few program directors created informative displays and started to develop new

electives at an early stage of the project, while the majority did not immediately start

to create electives for the display website. When the response fell short of the project

team’s expectations, documented remarks of the project leader (“We’re running

behind schedule”) indicated a growing concern about meeting the requirements for

continuation of the project. These stipulated that 80% of the training departments

had to have at least one elective displayed on the website by mid-2011.

Aiming to speed up the process, the project team started to put additional pressure

on program directors by referring to these requirements and the strict deadlines.

This provoked meaningless contributions threatening the quality of displays and the

motivation to develop displays. For example, when the project team communicated

that program directors were responsible for meeting the requirements for

continuation of the project, program directors of one specialty all published

identical electives. The project team then had to adapt the formal guidelines for

electives on display by adding that they had to have distinctive features. The project

team increased pressure by monthly mailings with information on which training

departments had electives on display, and announced that funds for developing

electives would be available to all specialties that met the deadline. In response,

one specialty announced that they did not intend to apply for this fund. Eventually,

seven out of nine specialties met the requirements before the deadline. However,

the project group judged that many electives on the display website may not be as

informative for trainees looking for an elective as they had strived for.

Competition

While the display of electives on the national website increased the feasibility of

individualizing training, it did not appear to lead to a competition for particular

electives between residents from different departments within the time frame of

the project. Hence, no indications of a difference in quality between the electives

could be deduced so far from the residents’ preferences.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 5

96


In discussions with program directors and in the survey that was held about their

attitudes it became clear that they felt strained by the requirements of the project,

while this was not compensated immediately by an advantage like the addition of

extra, highly motivated, trainees to their department. For some of the program

directors, this diminished their commitment to the project and their faith in the

proposed mechanism. Since the prospect of competition between residents for the

best electives had been part of the project team’s appeal to participate in the project,

the delay in the emergence of this expected effect was particularly disappointing for

program directors who had made a special effort to create an informative display of

their elective.

Obstacles for competition

Various issues that were regarded as hindrances for the intended competition were

raised during meetings of the comprehensive project group, ranging from social-

structural obstacles like geographical positions and national regulations (Capacity)

to social-cognitive obstacles like competing interests and lack of knowledge and

skills (Potential). While some of these obstacles could not be eliminated within

the premises of the project, others were successfully addressed by initiatives of

the project team, which tried to facilitate individualization of training by a variety

of approaches, whether these were within the original project plan or not. We

selected a number of examples to illustrate the nature of the perceived obstacles

and to demonstrate that dealing with these obstacles may impact the feasibility of

individualizing training and introducing competition in the future.

Social-structural obstacles

A) Some stakeholders felt that trainees preferred to live in certain geographic regions

in the country rather than in others, and that their family lives might prevent them

from choosing an elective further away.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


97

5


The project team had aimed for broad stakeholder involvement to raise ownership

and support among stakeholders, and to prevent objections to the project.

This stakeholder involvement was accommodated by regular meetings with the

comprehensive project group, which comprised representatives of program directors

and trainees of all specialties, hospital boards, training staff and regulatory bodies.

Nevertheless, this particular obstacle to resident mobility as perceived by members

of the project group could not be changed during the project. Since no concession

to this of issue could be made in the design of the project, it kept reoccurring in

discussions as a barrier that could not be resolved. For some stakeholders, this

diminished their belief in the project’s impact and their inclination to employ the

mechanism of transparency and competition.

B) The national regulations concerning modification of trainees’ training schedules

were very strict, which impeded actual individualization of training during the year.


Our analysis revealed that an important part of the efforts that went into the

implementation process was directed at altering existing national regulations, since

they hampered the mobility of trainees. This had been partly anticipated by the

initiators and was soon repeatedly stressed by the project’s stakeholders. Impelled

by the experienced need for change in regulations, the project team persistently

discussed this when they spoke with representatives of the ministry of Health. This

eventually led to modifications of the regulations for the nine specialties involved,

creating more possibilities for trainees and program directors to make adjustments.

Also, the original regulation system was evaluated by the government and changed

into another, somewhat more flexible system for all specialty training programs. This

created an important foundation for the individualization of training, in practical

terms but also in establishing the opportunity to choose an elective as a real and

feasible option for trainees.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 5

98

Social-cognitive obstacles

A) A recurrent topic in discussions at meetings was that senior trainees were valuable

for clinical work, which made it attractive for program directors to retain them at their

department. This could be in conflict with the trainees’ interests, since an application

for an elective at another department had to be authorized by the trainee’s current

program director.


We noticed that the implicit tension between the interests of training and the interests

of clinical practice became an explicit concern, as the project team addressed this

issue in their communication throughout the project. Directed at program directors,

this mainly included a moral appeal that sincere program directors should separate

the interests of clinical practice from the interests of training. For example, a letter

to all program directors reminded them that it was their responsibility “to enable

trainees to make choices that are important for their training”. At the same time,

the project team communicated to trainees that program directors are entitled

to expect a convincing argumentation for their choice of electives. In this way,

the project reinforced the importance of keeping an accurate portfolio of training

developments, which was an important element of the newly implemented specialty

training programs

B) When trainee representatives in the project group were asked what prevented

trainees from enrolling in electives, they revealed two weaknesses of trainees

regarding enrolling in electives. Firstly, it became clear that many trainees seemed

to have little knowledge about the exact regulations and their rights regarding

training possibilities. Secondly, residents had poor negotiation competencies to elicit

permission from program directors.


The project team took on the task of informing trainees about the exact regulations

and their rights regarding training possibilities. This involved various approaches

outside the original scope of the project, ranging from repeated explanations about

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


99

5

the regulations during meetings and an informative section on the website to

workshops and videos about negotiation strategies. As a result, we noticed that the

discourse about the importance of electives started to change. We saw that, in the

course of the project, individualizing training by deliberately choosing electives was

increasingly considered important for trainees to create a personal training profile.

This notion started to spread through the community of trainees. For example, in

discussions about developments in health care and future tenure, the Display website

was brought up as a relevant development for trainees. We even noted that “display

elective” appeared as a novel term in discussions, to indicate a consciously chosen

elective in another hospital. Furthermore, some specialties that previously lacked

dedicated time for electives in their official training programs started to include this

in their curricula. We also observed that the discourse regarding who should be in

control over planning a resident’s training started to change in the course of the

project period. We observed a gradual shift from a discourse in which it was accepted

that program directors were in command of planning training, to a discourse in

which trainees’ views of their own training priorities were regarded as relevant. For

example, the project started off without a website-functionality for trainees’ reviews

of electives, even though this was one of the original objectives of the project; in

the final stage, however, the functionality was added and regarded as an important

element to stimulate training quality.

DISCUSSION

This study investigated the effects of implementing transparency and competition in

medical specialty training in relation to implementation approach. We found effects

both within and beyond the scope of the goals of training quality and individualization.

Not only did the project initiate transparency and facilitate individualization, it also

raised awareness of modern training principles and changed the discourse about

control over resident training. We noted that these effects were closely related to

aspects of the implementation approach, including: inviting program directors to

display electives, appealing to their professional entrepreneurship supplemented with

pressuring participation, involving all stakeholders, and broadening communication

and actions beyond the original scope of the project to address unforeseen obstacles.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 5

100

Comparison to the literature

Implementing transparency involved facilitating display of information about training

electives on an openly accessible national website. After four years, about 80 percent

of participating specialty training departments had an elective displayed. At that

point, a considerable amount of the information on display was not very clear,

which can be regarded as limiting actual transparency and competition. However,

the mechanism that started off in this way has the potential to stimulate quality of

information and thus further increase transparency. For example, when the English

National Health Service introduced transparency, they did not start off with perfect

data: in some cases, less than 50 percent of the information fields were coded

correctly. After introducing transparency, coding accuracy improved greatly.10

Several conditions other than transparency of information came up that were important

for individualizing training, such as flexibility of regulations and trainees’ capability

to enroll in electives. We noticed that the intended approach to implementation was

adjusted to address these conditions, and that this was important for the achievement

of intended effects. This finding of differences between a formal plan and the actual

execution of a plan resembles the acknowledged difference between the “intended”

curriculum and the “enacted” curriculum in medical education.27 Social and practical

circumstances contribute to this difference, and in pursuit of congruence between

the two levels, these circumstances should be anticipated while developing the plan.

Along these lines, general change management advice to adjust one’s approach to

circumstances is fitting to medical education innovation as well.28

We found that the implementation in the present project relied heavily on involving

all stakeholders of specialty training. This generated communication between all

parties throughout the country and facilitated meaningful changes for training, rather

than merely imposing the execution of a new method. Generating communication

between all professionals involved has been suggested before as a fitting approach

for working on quality of medical specialty training.29 It could be seen as creating

room for so-called re-invention, which refers to the modification of an innovation by

its users in the process of adoption. This process helps to match the innovation to

the adopter’s problems. Re-invention fits innovations that are complex and aimed at

solving a wide range of problems.30 It is known to be applied by users with a strong

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


101

5

sense of pride, for whom ownership is important, and medical professionals are

considered to belong to that type of users.30;31 Furthermore, for leading professionals

it has since long been recognized that “the amount of management added to their

self-management” needs conscious balancing, because limiting their possibilities for

self-determination introduces the risk of demotivation.32 In consonance with this,

we observed that program directors who felt pressured to display information were

inclined to take short-cuts with contributions of low quality, and to perceive these

contributions as an extra workload for which they received nothing in return. This

observation is in line with similar experiences during other innovations in specialty

training, like workplace-based assessment.14;33


By looking into the effects of introducing transparency and competition in specialty

training, our study adds to knowledge about conditions and obstacles that may be

relevant for introducing this innovation into this field. We focused on one specific

setting as a case to study. The consideration of this unique case can inform the

consideration of other cases that involve issues of individualization, quality of

specialty training, transparency and competition.34

Our study adds to existing research on innovation in medical education by going

beyond a separate investigation of either the effectiveness of an implementation

approach or the effects of an innovation. Instead, we studied the combination of

these elements to do justice to the complex nature of innovating specialty training.

Theory from implementation science provided a basis to do so.19 Still, our findings

are limited by resulting from a single case of a specific innovation in a specific

context. Other cases of implementing innovation in specialty training are likely to

have aspects in common with our case, either regarding the nature of the innovation

(e.g. multiple diverse stakeholder groups, aiming to increase training quality) or the

nature of the implementation approach (e.g. in a national project). People involved

with other cases of innovation in specialty training could benefit from those findings

that seem relevant to their specific setting.

Our research approach was not suited to assess the quality of the electives as followed

by trainees in clinical practice. The positive effects that we found on fundamentals

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 5

102

like awareness of modern training principles and opportunities for individualization

of training schedules seem important contributions to quality of training. However,

it seems unlikely that this may already have caused a notable increase in training

quality.

Implications for practice and future research

Our findings hold practical implications for both expectations of transparency

in specialty training as well as for implementing innovations in specialty training.

Although we did not study the quality of electives that trainees chose, our findings

suggest that transparency can contribute to quality improvement, but that it takes at

least several years to achieve this effect. This illustrates once more that innovation

in medical education takes time1, as is known about innovating healthcare.35 This

knowledge is important for managing the expectations of anyone who intends to

implement or study transparency or other innovations in specialty training; time

should be taken to assess innovations. For complex innovations such as this one, it

seems sensible to plan pilot periods of more than the four years that were available

for the project in this case study. Another challenge for the implementation of this

innovation was the delay in the emergence of effects. People who put effort into the

innovation, like the program directors in our case, expect something in return for their

efforts. Solutions to prevent disappointment might include managing expectations

and indicating noticeable intermediate effects. Further empirical research is needed

to proceed from the speculative level of these suggestions.

Our findings call for reticence in pushing for progress and deadlines, which seems to

be elicited by quantitative planning and minimum requirements for grant allotment.

Both project leaders and funders of innovations should accept the challenge of setting

goals that seem within easy reach, and instead strive for meaningful change.1;36 The

people involved with specialty training may not be professionals in the domain of

management, but managers should keep in mind that they are professionals within

the medical domain and should be treated as such.

Our results highlight that the best interests of training can be hindered by the interests

of clinical practice. This matter of attention for training quality has been recognized by

others who contemplate improvements in medical education.37Our findings suggest

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


103

5

that the interests of training can be brought to attention by appealing to professional

responsibility of the professionals involved in specialty training. However, such an

appeal can hardly be expected to solve this challenge completely, since patient care

is as least as important to program directors as specialty training. Contributions to

meeting the challenge of balancing the service and education missions of training

departments are needed urgently, because an unfavorable balance might jeopardize

future initiatives to improve training quality.

CONCLUSION

Transparency in specialty training has the potential to contribute to individualization

of training. Other conditions can be found important for resident mobility, for

example regulations and the capacity of trainees to organize their own training.

Detecting and addressing these required conditions is facilitated by an approach

that involves all stakeholders and requires willingness to go beyond implementing

transparency. Meaningful changes to training may be achieved by appealing to

professional attitude, providing freedom and allowing reinvention rather than by

pressuring participation. The effects of implementing transparency and competition

in specialty training emerge over many years. Accordingly, the assessment of this

innovation remains unfinished at this point and should be ongoing.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 5

104

REFERENCES


(2) van der Lee N, Westerman M, Fokkema JPI, van der Vleuten CPM, Scherpbier AJJA, Scheele F. The curriculum for the doctor of the future: messages from the clinician’s perspective. Med Teach 2011;33:555-561.


(4) Boor K, van der Vleuten C, Teunissen P, Scherpbier A, Scheele F. Development and analysis of D-RECT, an instrument measuring residents’ learning climate. Med Teach 2011;33:820-827.

(5) van der Leeuw R, Lombarts K, Heineman MJ, Arah O. Systematic evaluation of the teaching qualities of Obstetrics and Gynecology faculty: reliability and validity of the SETQ tools. PLoS One 2011;6:e19142.

(6) Chen C, Petterson S, Phillips R, Mullan F, Bazemore A, O’Donnell S. Toward Graduate Medical Education (GME) Accountability: Measuring the Outcomes of GME Institutions. Acad Med 2013;88:1267-1280.

(7) Marshall MN, Shekelle PG, Leatherman S, Brook RH. The public release of performance data: what do we expect to gain? A review of the evidence. JAMA 2000;283:1866-1874.

(8) Fung CH, Lim YW, Mattke S, Damberg C, Shekelle PG. Systematic review: the evidence that publishing patient care performance data improves quality of care. Ann Intern Med 2008;148:111-123.

(9) Europa Summaries of EU legislation. The Bologna process: setting up the European higher education area. http://europa.eu/legislation_summaries/education_training_youth/lifelong_learning/c11088_en.htm. 2014. 4-2-2014.

(10) McKinsey’s Health Systems and Services Practice. Transparency - the most powerful driver of health care improvement? 2011. Health International. 18-2-2014.

(11) Ringsted C, Henriksen AH, Skaarup AM, van der Vleuten CPM. Educational impact of in-training assessment (ITA) in postgraduate medical education: a qualitative study of an ITA programme in actual practice. Med Educ 2004;38:767-777.

(12) ten Cate O, Scheele F. Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad Med 2007;82:542-547.


(14) Fokkema JPI, Teunissen PW, Westerman M et al. Exploration of perceived effects of innovations in postgraduate medical education. Med Educ 2013;47:271-281.

(15) Fokkema JPI, Scheele F, Westerman M et al. Profiling perceived effects of innovations in medical education; a Q study on workplace-based assessment. Academic Medicine 2014;89:1259-66.

(16) Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of research in medical education. Med Educ 2008;42:128-133.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34


105

5

(17) Sklar DP. Sharing new ideas and giving them wings: introducing innovation reports. Acad Med 2013;88:1401-1402.

(18) Jippes E, Van Luijk S, Pols J, Achterkamp M, Brand P, Van Engelen J. Facilitators and barriers to a nationwide implementation of competency-based postgraduate medical curricula: A qualitative study. Med Teach 2012;34:e589-602.

(19) Marchal B, Van Belle S, Van Olmen J, Hoerée T, Kegels G. Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation 2013;18:192-212.

(20) Central College of Medical Specialties. Kaderbesluit CCMS. http://knmg artsennet nl/Opleiding-en-herregistratie/Algemene-informatie/Nieuws/O-R-Nieuwsartikel/Modernisering-medische-vervolgopleidingen-nieuw-kaderbesluit-CCMS htm [serial online] 2011; Accessed January 7, 2014.

(21) Bunniss S, Kelly DR. Research paradigms in medical education research. Med Educ 2010;44:358-366.


(23) May C. Towards a general theory of implementation. Implement Sci 2013;8:18. (24) Bowen GA. Document analysis as a qualitative research method. Qualitative research

journal 2009;9:27-40. (25) Kuper A, Whitehead C, Hodges BD. Looking back to move forward: using history,

discourse and text in medical education research: AMEE guide no. 73. Med Teach 2013;35:e849-e860.

(26) King N. Using templates in the thematic analysis of text. In: Cassell C, Symon G, eds. Essential guide to qualitative methods in organizational research. London: Sage Publications Inc.; 2004;256-270.

(27) Billet S. Constituting the workplace curriculum. Journal of curriculum studies 2006;38:31-48.

(28) Burton R, Obel B. The dynamics of the change process. Strategic organizational diagnosis and design: the dynamics of fit. 3rd ed. Boston: Kluwer Academic Publishers; 2004;385-420.

(29) Dowton SB, Stokes ML, Rawstron EJ, Pogson PR, Brown MA. Postgraduate medical education: rethinking and integrating a complex landscape. Med J Aust 2005;182:177-180.

(30) Rogers EM. The innovation-decision process. Diffusion of innovations. 5 ed. New York: Free Press; 2003;168-218.

(31) Freidson E. Professionalism: the third logic. Cambridge: Polity Press, 2001. (32) Weggeman M. Is the professional self-managing or is there really a need for professional

management? European management journal 1989;7:422-430. (33) Bindal T, Wall D, Goodyear HM. Trainee doctors’ views on workplace-based assessments:

Are they just a tick box exercise? Med Teach 2011;33:919-927. (34) Lingard L. Qualitative research in the RIME community: critical reflections and future

directions. Acad Med 2007;82:S129-S130. (35) Grol R. Planning and organizing the change process. Improving patient care: the

implementation of change in health care. Chichester, West Sussex: Wiley Blackwell; 2013;64-74.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 5

106

(36) Fokkema J, Teunissen PW. Assessing the assessment of interventions: we’re not there yet. Med Educ 2013;47:954-956.

(37) Mehta NB, Hull AL, Young JB, Stoller JK. Just imagine: new paradigms for medical education. Acad Med 2013;88:1418-1423.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

6general discussion

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 6

110

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

General discussion

111

6

As sketched in the introductory chapter to this thesis, people participating in the

practice of specialty training are often dealing with innovations in this training. Most

of these innovations are aimed at supporting training that is competency based

and outcome oriented, which follows developments in healthcare and societal

expectations about doctors’ performance. However, it is becoming clear that, in the

practice of specialty training, it is challenging to deal with innovations in such a way

that they contribute to their intended effects. In order to contribute to the desired

changes to specialty training, the medical and medical education communities need

an understanding of how innovation in the practice of specialty training evolves. Such

an understanding has not been developed, mostly because research on innovations

in specialty training has failed to follow up innovation processes to unravel what

actually happens after the initial implementation. The aim of this thesis was to

contribute to the understanding of how innovations in practice evolve to result in

manifold effects. The central question was: how do people involved with the practice

of specialty training deal with innovations in that training? In order to address this

question we looked at a number of distinct aspects: approaches of those people in

charge of training to bring about change, effects of an innovation in practice, and

perceptions of the people involved with training regarding these effects. Furthermore,

we studied links between implementation approach and effects of an innovation.

In this chapter, I provide an overview of our main findings, leading to an answer to

our central research question. This answer points towards the need for a different

conceptualization of innovating specialty training. I propose a conceptualization that

facilitates understanding the manifold effects that innovations in specialty training

can bring about and discuss this conceptualization in light of existing literature.

Furthermore, I reflect on our research approach, and propose suggestions for

practice and future research.

Main findings

As reported in chapter 2, lead consultants approach educational change using

idiosyncratic change strategies. They had individual ideas and beliefs about change

that clearly influenced what they regarded as the best way to manage change. They

differed in their degree of awareness of the strategies they used, and in the way in

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 6

112

which they reflected on their efforts. Differences in knowledge, task interpretation,

and personal style also influenced their approaches, as did culture and customs in

the department.

Chapter 3 illustrated that an innovation can bring about a variety of effects that

extend beyond the range of the intended, expected, and desired effects. Trainees

and consultants experienced effects of workplace-based assessment (WBA) in six

domains of their professional lives: sentiments (affinity with the innovation and

emotions), dealing with the innovation, specialty training, teaching and learning,

workload and tasks, and patient care. Affinity with the innovation varied between

users and appeared to be one of the influences on teaching and learning effects.

Organisational support and the match between the innovation and practice were

considered important to minimise additional workload and to ensure that the

WBA was used in a way that is relevant for training. Dealing with WBA stimulated

attention for specialty training and placed specialty training on the agenda of clinical

departments. We noted that individuals clearly differed from each other with respect

to which types of effects they experienced, and in which amount.

In chapter 4 we looked further into these different user perceptions of effects. In a Q

study to determine perceptions of trainees and consultants regarding the effects of

using WBA, we found five distinct perception-profiles: enthusiasm, compliance, effort,

neutrality, and scepticism. The five perceptions were characterized by differences in

the views on three main issues: the goals the innovation was intended to achieve, its

applicability in practice, and its actual impact. Thus, we found that those involved in

an innovation can vary substantially in their perceptions of effects of that innovation,

even if they work in the same department and have similar characteristics like

amount of experience or function.

To look into the mutual influence of approaches to, responses on, and effects of

an innovation, in chapter 5 we focused on a case of introducing transparency

and competition in specialty training electives. We found intertwined effects and

features of the implementation approach in the themes transparency, competition,

and obstacles for competition. For instance, the project team’s approach leaned

on appealing to professional motivation, but was supplemented by pressure to

participate. This did elicit use of the innovation by parties who had first resisted, but

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

General discussion

113

6

in a way that was not conducive to the goals. Furthermore, the approach involved

stakeholder involvement, which revealed obstacles for implementation, to which the

approach was then adapted. Attending to these issues enabled the development of

effects that went beyond the intentions, but that were relevant for training quality,

for instance growing awareness of modern training principles.

New conceptualization of innovating specialty training

Coming back to our central research question of ‘how do people who participate in

medical specialty training deal with innovations in this training?’, we can conclude

that our answer to this question needs to consist of several components. The

people involved in specialty training deal with innovations in a variety of ways and

consequently innovations lead to a range of effects, which are again perceived in

various ways. People’s approaches to innovations depend on an interplay between

different factors, among the most prominent we uncovered were their affinity

with the innovation, other personal factors like their individual ideas, beliefs and

understanding of the innovation, and social and contextual factors like if it fits with

routine practice, communication about the innovation, and department culture.

The dominant perspective on innovations in medical education that was explained

in the introduction is an insufficient model for understanding the processes and

outcomes of innovations in specialty training that resulted from the research in this

thesis. A linear input - black box - output model (figure 1) is too simplistic to explain the

complexities we unearthed. Even when the black box is supposed to contain all the

processes related to introduction, diffusion and adoption of innovations that explain

our findings, it doesn’t set realistic expectations about how future innovations might

play out. It neglects the complex and iterative nature of the process, with effects

that, as soon as they are developing, become influences and context as well, which

again alters the implementation approaches and the developing effects.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 6

114

innovative application

effects -predefined and measurable

innovative philosophy and concepts implementation

activities

Figure 1. Linear model representing the current dominant perspective on innovating medical specialty training

Based on the findings in this thesis, I propose to abandon the dominant linear

perspective on innovations in specialty training, and suggest a dynamic model that

can account for, and anticipates, the complexities of such innovations. This model

links the conceptual foundation of the innovation, its translation to practice, and

effects in practice. Furthermore, in order to do justice to reality, it accentuates the

notion of translation of a concept to practice, which is formed by the combination of

proposed applications and activities to implement these in practice, see figure 2. In

the following section, I explain this view on the translation of an innovative concept,

and then proceed to describe the rest of the model by placing this translation in

a dynamic relationship with the conceptual foundations of an innovation and its

effects.

implementationactivities

conceptual foundation of innovation

practicetranslation

concrete applications

Figure 2. Dynamic model representing the proposed new perspective on innovating medical specialty training

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

General discussion

115

6

Translation of a concept to practice is made up by the combination of proposed

applications and activities for implementation into practice

I propose to more consciously consider the notion of the translation of the conceptual

foundation of an innovation to practice. This translation is made up by both the

concrete application and implementation activities. As outlined in the introduction to

this thesis, all kinds of concrete applications have been developed to use in practice

of specialty training. These applications are part of the translations that make

concepts operational for practice. For example, the concrete form of the mini-CEX

is supposed to facilitate that feedback is documented, in addition to feedback being

discussed verbally. In this way, it is supposed to translate to practice the principle

of documenting training activities, which had as point of departure accountability

of training. Yet, as the research in this thesis pointed out, the implementation of an

innovation actually contributes to the translation of a concept to practice of specialty

training as much as the concrete applications. Thus, the translation of the conceptual

foundation of an innovation to practice is composed of both the concrete application

and implementation activities. Effects in practice are brought about through this

translation.

The conceptual foundation, its translation and effects of the innovation in practice

are a dynamic system

The translation of the innovation’s concept as constructed by the application and

implementation activities brings about effects in practice. As shown in this thesis,

the effects are not all predefined, affect multiple domains of practice, and are

perceived in idiosyncratic ways. Thus, the process of innovating specialty training

does not stop when initial effects have come about. Since the effects have to do

with how people involved handle the innovation, the nature of applications and

the implementation approach are adjusted in reaction to these initial effects. This

means that the translation is attuned, which leads to altered effects, and so on. For

example, when in practice only minimal feedback is written down for mini-CEXs,

both the implementation approach and the concrete application can be adjusted

in all kinds of ways to address this. Examples of adjustments to implementation

activities are stressing the importance of documentation or offering some kind of

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 6

116

reward for each filled out mini-CEX. Adjustments to the concrete application could be

adding suggestions of topics to standard forms, or abandoning standard forms and

letting people find their own way to document. These adjusted implementations and

applications will construct a different translation, and effects in practice will change,

and so on. Moreover, the dynamics between practice and translation can penetrate

the level of the innovative concept. For example, experience in practice can lead to

the insight that accountability is mainly made up by transparency of expenses for

training and only in a lesser part by insight into an individual’s training progress.

This new concept will lead to other concrete applications, implementation activities,

translation and effects.

Comparison to the literature

In the adjacent domain of implementation science, there is a shift in emphasis in

research about innovations for healthcare, which is in congruence with the above

proposed acknowledgement of complexity of innovation processes. Innovations

studied in this field are for instance evidence-based practice and guidelines46 and

digital recording and prescribing systems47. Numerous studies have been conducted

in the past decades that aimed to point out best strategies or barriers and facilitators

to implement these kinds of innovations. These individual studies have resulted

in valuable insights into factors influencing the implementation of innovations

in healthcare. Greenhalgh and colleagues performed an extensive review of this

literature on diffusion of service innovations and composed a comprehensive

conceptual model in which the separate determinants of innovation processes

are united.25 This model features principal findings from across different research

traditions, ranging from key attributes of the innovation, to adoption by individuals

and the system, contextual features and readiness, and implementation approaches.

It also provides some guidance as to how these components are linked. According to

these and other authors, at this stage of knowledge more studies that concentrate

on separate elements will not suffice to further understanding of the innovation

process.25;48 This has led them to call for acknowledging the complexity of innovation

processes and for placing this complexity at the centre of attention of further

research. Still, these pleas in this field of implementation science refer mainly to

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

General discussion

117

6

understanding the black box’s complexity in a linear view on the process, in which

successful implementation of an application will produce predefined and measurable

outcomes. This is illustrated by Grol’s call for “more research on mechanisms that

determine whether a specific innovation will be successful in a particular health

setting.” (page 125).48

As outlined in the introduction to this thesis, in the field of postgraduate medical

education few studies about innovations have focused on the process of innovating.49

The majority of studies about innovations in this field concern the use of a specific

application (e.g. portfolios, simulation-based education or in-training assessment),

its effects, or both. Accordingly, the research is usually designed with the application

as point of departure. Despite this focus on applications and their effects, the

influence of implementation activities on the application’s impact in practice is

increasingly pointed out for the field of postgraduate medicine. As Van der Vleuten

and Verhoeven stated in their paper on in-training assessment developments, “the

concern is more with the actual implementation of change than with the assessment

technology per se. If we fail in our efforts to implement real change, postgraduate

education may be at risk for bureaucratization and trivialization.”.21 In this way, the

importance of the combination of application and implementation to translate a

concept to practice is penetrating the postgraduate medical education field. The

above proposed new conceptualization of innovation processes corresponds with

the desire for understanding innovation processes as translations of a concept to

practice, instead of as the uptake of a method or tool.

The new conceptualization of issues in specialty training as complex and dynamic

seems not to be confined to the subject of this thesis. For instance, the notion of

transfer of learning, which has been an important concept for education since long50,

has been undergoing a similar development. In the traditional view on transfer, it

was considered a static concept independent of its context that can be measured

as a specific behaviour. In the contemporary view, the dynamics between changing

individuals and changing context are regarded more important, which also means that

measuring transfer is regarded more difficult. For instance, a recent review to identify

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 6

118

gaps in research on transfer of medical communication skills signalled the following:

that the traditional approach and its resulting insights have significantly advanced

research in the field, but appear somehow limited in their possibility to explain the

remaining challenges, and that research should be complemented with perspectives

that take into account real life complexity.51 The background of this similarity of

developments in conceptualization of various elements in medical education seems

to be that researchers in the field of medicine who study educational topics continue

to loosen the ties with the traditionally positivist paradigm of biomedical research,

and are more and more taking up post-positivist viewpoints.52

Reflections on methodology

The research in this thesis has not led to a detailed manual for dealing with

innovations in specialty training. In my opinion, there are two reasons why striving

for composition of such a manual isn’t really appropriate: one relates to the

complexity of innovation processes in specialty training, the other relates to my

constructivist view on research. Firstly, as set out above, I have come to appreciate

innovation in specialty training as a dynamic process with involvement of a variety of

interdependent elements. Consequently, it seems far-fetched to compose a decision

model that captures all possible situations. Yet, the insight gained through the four

studies in this thesis has led to a different conceptualization of innovating specialty

training. This conceptualization can support future research as well as practice, which

I will discuss in the concerning paragraphs below.

The second reason I would not strive for a detailed manual for dealing with innovations

is that I have a constructivist view on research52, as mentioned in the separate

chapters. This means that I consider it impossible to establish a presentation of one

single truth through research, because, even if one single truth exists, knowledge

about it is created between an observer and the observed. Thus, presented

knowledge is influenced by the characteristics of the observer. Furthermore, a reader

of the presented findings again creates his own knowledge in interaction with the

presented findings, and is influenced by his own characteristics. This constructivist

view on research means that findings never present one single truth that counts for

everyone at all times. This does not mean that the findings in this thesis are untrue

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

General discussion

119

6

or irrelevant, but that individuals should use these findings in a way that is relevant

to their own situation. Moreover, the findings can and need to be added to by others

through observations from different points of view.

Following on these reflections, there are some specific strengths and limitations

of the work presented in this thesis that readers should be aware of. This thesis

combines studies about different parts of the process of innovating specialty

training: implementation approach, effects in practice, perceptions of people

involved, and the mutual influence of these elements, in order to answer the central

research question of ‘how do people who participate in medical specialty training

deal with innovations in this training?’. The main strengths of this thesis relate to

the relevance of this research, the approach that we took in studying this topic, and

the methodological rigour of our research. As explained in the introductory chapter,

there currently is a paucity of knowledge about the evolvement of innovations in

specialty training, while many innovations are introduced into practice and many

meet with mixed success.17;21 Thus, at this time, a knowledge base about the subject

is needed, so that suggestions and expectations can be realistic, and the handling

of innovations can be rewarding. The findings in this thesis add to this knowledge.

With respect to the approach that we took, combining studies on distinct aspects

that are involved with innovation enabled to formulate an answer to the central

research question. At the same time, the explorative nature of the research question

and of the separate studies allowed for acknowledging that a new conceptualization

innovating specialty training is needed for further advancement of the topic. The

proposed conceptualization does not reject the value of existing theories. Rather, it

stimulates contemplating how these theories relate to each other and can be applied

to the field of postgraduate medical education. Regarding the methodological

rigour the separate studies were based on existing theory from adjacent fields like

sociology and implementation science.24;53 This answers to the call within the medical

education research community to use theories and conceptual models for the design

and interpretation of research.54 The use of existing theory in our studies supports

the relevance of our findings and the understanding of innovation processes.

Furthermore, we used Q methodology when it was relevant, although it is relatively

new to the field of medical education.55 Finally, three studies have been published (or

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 6

120

accepted for publication) in international peer-reviewed journals so far. Sharing our

findings in this way allows others to apply our findings to practice, and to build on

our research, challenge and add to our findings, and thus strengthen the knowledge

on the subject of innovations in medical education.

Some limitations of the research in this thesis should be considered as well. Among

these are sources of possible bias and the transferability of our findings. Several

factors might have biased how we have come to regard innovations in postgraduate

medical education. Firstly, my collaborators and I have a mainly medical or medical

education background. Although we have explicated this in the research process,

this may have led to underexposing certain other views, for example patients’

perspectives and organizational issues. Secondly, since this thesis focuses on the

people who participate in medical specialty training, we have conducted most

research among trainees and medical specialists. Still, other people - ranging from

administrators to medical students - are involved in the process as well, albeit to

a lesser extent. The findings in this thesis may be biased because people involved

other than trainees and consultants were studied to a lesser extent. For better

inclusion of varying views in research, a shift away from the currently dominating

approaches seems necessary, as will be discussed further in the next paragraph. The

study in chapter 5 is the only one that also covered stakeholders from other groups.

Thirdly, the findings in chapter 2, 3 and 4 were based on individual’s perceptions

and report of those, which might have led to underexposure or overrepresentation

of certain elements, such as sensitive or social-desirable ones.56 Furthermore, we

conducted the studies of chapter 2 and 3 among various medical specialties and

training departments, and of chapter 4 among various departments of one specialty.

Although this approach ensured some variability in participants, the transferability

of the findings to other contexts or specialties is limited. All studies were conducted

in the Netherlands, where the setting of implementation of innovations and culture

in postgraduate medical training differs from the rest of the world. These limitations

to the transferability of findings apply even more to the study described in chapter 5,

since this was a case study.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

General discussion

121

6

Implications for research1

Further research on innovations in medical specialty training might contribute to

determining which investments should be made. The proposed new conceptualization

can support this research for several reasons. Firstly, as explained in the introduction

to this thesis, quite some research on innovations in medical education up to now

has focused on applications.10;16 The findings of this thesis facilitate a research focus

on translation of an innovative concept instead. This implication is not limited to

the field of postgraduate medical education, but may pertain to other fields where

translation of a concept and philosophy are eventually more important than use

of a certain application, like other types of education, business, psychology, or life-

coaching. Related to this shift in focus from applications to translation of concepts,

a new angle to the study of innovation effects is needed. Assessments of medical

education innovations have been mainly directed at their intended effects, or at both

intended and unintended effects within the educational domain.19;57 The dynamic

model proposed here points out that this kind of confined evaluation will not help

us to build a complete picture of the full impact of an innovation. It is insufficient

to keep measuring usage rates of an application or effectiveness on predefined

outcomes. This thesis has contributed to the knowledge about any other effects that

might be expected, but research that uses the new conceptualization is needed to

further the understanding of development of various effects in practice. This should

include research in other settings than those studied in this thesis.

The above implicates that future studies need a methodology that is suited to

study dynamics between educational concept, translation through application and

implementation, and effects. Thus, approaches are needed that are suitable for

studying complex systems. However, researchers in the field of medical education

may not be well equipped yet to conduct complexity research, although some

exceptions exist.58 There are other fields that are more familiar with this kind of

research, for example systems innovation in agriculture through reflexive monitoring

in action.59 Moreover, conducting research in transdisciplinary teams including non-

scientific actors is needed to ensure that the variety of viewpoints involved is taken

into account.59;60 For studying innovation in their field, medical education researchers

1 This section is based on: Assessing the assessment of interventions: we’re not there yet. JPI Fokkema, PW Teunissen. Med Educ 2013;47:954-6

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 6

122

should take up these kinds of approaches. Likewise, both scientific journals and

funders of research need to acknowledge these research approaches as relevant.


The practice of innovating medical specialty training could find support in insights

from this thesis. First of all, the conceptualization of this topic that resulted from

the presented work has implications for the approach to developing innovations

in specialty training. When striving for translation of a concept to practice, not

only applications need to be developed, but designing implementation approach

elements is needed as well. Moreover, the dynamic nature of the process requires

that implementation and application should be developed synchronous and in an

iterative manner instead of sequential. These are relevant insight for designers of

innovations for specialty training. However, current relevant training programmes

for this area, like educational studies or management, do not cover both kinds

of elements and mainly take linear approaches. Thus, the content of these kinds

of training programmes should be updated in order to fit to these needs, and to

introduce reflexive methods to the innovation professionals of the future.

All studies in this thesis have shown to some extent that most people who are involved

with innovations in medical specialty training are learners in this area. This ranges

from experienced leaders who get the responsibility for introducing an innovation to

insecure first year trainees who have to ask for feedback. Additionally, it seemed that

pressuring them to take up an application risks meaningless use of well-intended

tools. These findings imply that those involved deserve dedicated time, practice and

guidance to get to meaningful translation of a concept to practice. This is important

to realize, for example when planning elements like support, time span, costs and

evaluation of an innovation in medical training.

All involved with innovations in specialty training need to be aware of the complexity.

This will prevent them from expecting unambiguity, clear-cut use of applications,

and immediately reaching expected effects. This counts for individuals as well as

for organizations or projects, ranging from local initiatives to large scale projects

like the Dutch example studied in chapter 5. The complexity of the subject should

be respected in order to support meaningful interpretation of aspects like targets,

requirements, conditions and evaluation of effects.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

General discussion

123

6

And finally, the proof is in the pudding

People who struggle with innovations in specialty training and hoped to find in this

thesis the perfect recipe that guarantees success might be disappointed by the

absence of such a recipe. Hopefully, the open approach to this challenging subject,

the findings and the new conceptualization will support people to get cooking

themselves. Dealing with innovations in specialty training means considering what

ingredients one is provided with, and putting together the sometimes unusual

combinations in such a way that the product is worth the effort and suits the moment.

Conceptualizing innovation processes as proposed in the dynamic model can support

people in considering what should work for them, the ingredients, and the moment.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 6

124

REFERENCES


(2) Flexner A. Medical education in the United States and Canada: a report to the Carnegie Foundation for the Advancement of Teaching. no. 44. 1910. New York ; Buffalo. Bulletin (Carnegie Foundation for the Advancement of Teaching) ; no. 4.

(3) Neufeld VR, Maudsley RF, Pickering RJ et al. Educating future physicians for Ontario. Acad Med 1998;73:1133-1148.

(4) Jippes M. Culture matters in medical schools. How values shape a successful curriculum change. [ 2013.

(5) Scheele F, Teunissen P, Van Luijk S et al. Introducing competency-based postgraduate medical education in the Netherlands. Med Teach 2008;30:248-253.

(6) Frank JR, Snell LS, Cate OT et al. Competency-based medical education: theory to practice. Med Teach 2010;32:638-645.

(7) Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system--rationale and benefits. N Engl J Med 2012;366:1051-1056.

(8) Leung WC. Competency based medical training: review. BMJ 2002;325:693-696. (9) Motola I, Devine LA, Chung HS, Sullivan JE, Issenberg SB. Simulation in healthcare

education: a best evidence practical guide. AMEE Guide No. 82. Med Teach 2013;35:e1511-e1530.


(11) Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med 1995;123:795-799.

(12) Martin JA, Regehr G, Reznick R et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84:273-278.

(13) van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE Guide no. 45. Med Teach 2009;31:790-801.



(16) Hawkins RE, Margolis MJ, Durning SJ, Norcini JJ. Constructing a validity argument for the mini-Clinical Evaluation Exercise: a review of the research. Acad Med 2010;85:1453-1461.

(17) Driessen E, van Tartwijk J, van der Vleuten C, Wass V. Portfolios in medical education: why do they meet with mixed success? A systematic review. Med Educ 2007;41:1224-1233.

(18) Pelgrim EAM, Kramer AWM, Mokkink HGA, van den Elsen L, Grol RPTM, van der Vleuten CPM. In-training assessment using direct observation of single-patient encounters: a literature review. Adv Health Sci Educ Theory Pract 2011;16:131-142.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

General discussion

125

6

(19) Ringsted C, Henriksen AH, Skaarup AM, van der Vleuten CPM. Educational impact of in-training assessment (ITA) in postgraduate medical education: a qualitative study of an ITA programme in actual practice. Med Educ 2004;38:767-777.

(20) Daelmans HEM, Overmeer RM, van der Hem-Stokroos HH, Scherpbier AJJA, Stehouwer CDA, van der Vleuten CPM. In-training assessment: qualitative study of effects on supervision and feedback in an undergraduate clinical rotation. Med Educ 2006;40:51-58.


(22) Bindal T, Wall D, Goodyear HM. Trainee doctors’ views on workplace-based assessments: Are they just a tick box exercise? Med Teach 2011;33:919-927.

(23) Kotter JP. Leading change. Boston: Harvard Business School Press, 1996. (24) Rogers EM. Diffusion of innovations. 5 ed. New York: Free Press, 2003. (25) Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations

in service organizations: systematic review and recommendations. Milbank Quarterly 2004;82:581-629.

(26) Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet 2003;362:1225-1230.

(27) Huis A, Schoonhoven L, Grol R et al. Helping hands: a cluster randomised trial to evaluate the effectiveness of two different strategies for promoting hand hygiene in hospital nurses. Implement Sci 2011;6:101.

(28) Thompson BM, Schneider VF, Haidet P, Perkowski LC, Richards BF. Factors influencing implementation of team-based learning in health sciences education. Acad Med 2007;82:S53-S56.

(29) Bland CJP, Starnaman SP, Wersal LM, Moorhead-Rosenberg LP, Zonia SP, Henry RP. Curricular Change in Medical Schools: How to Succeed. Academic Medicine 2000;75:575-594.

(30) Loeser H, O’Sullivan P, Irby DM. Leadership lessons from curricular change at the University of California, San Francisco, School of Medicine. Acad Med 2007;82:324-330.

(31) Rich ECM, Magrane DM, Kirch DGM. Qualities of the Medical School Dean: Insights From the Literature. Academic Medicine 2008;83:483-487.

(32) Souba WW, Day DV. Leadership values in academic medicine. Acad Med 2006;81:20-26. (33) Kaufman A. Leadership and governance. Acad Med 1998;73:S11-S15. (34) Dannefer EF, Johnston MA, Krackov SK. Communication and the process of educational

change. Acad Med 1998;73:S16-S23. (35) Calman KC, Temple JG, Naysmith R, Cairncross RG, Bennett SJ. Reforming higher

specialist training in the United Kingdom--a step along the continuum of medical education. Med Educ 1999;33:28-33.

(36) Dzau VJ, Yoediono Z, Ellaissi WF, Cho AH. Fostering innovation in medicine and health care: what must academic health centers do? Acad Med 2013;88:1424-1429.

(37) Grant JR. Changing postgraduate medical education: a commentary from the United Kingdom. Med J Aust 2007;186:S9-13.

(38) Ludmerer KM, Johns MM. Reforming graduate medical education. JAMA 2005;294:1083-1087.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 6

126

(39) Snelgrove H, Familiari G, Gallo P et al. The challenge of reform: 10 years of curricula change in Italian medical schools. Med Teach 2009;31:1047-1055.

(40) Lillevang G, Bugge L, Beck H, Joost-Rethans J, Ringsted C. Evaluation of a national process of reforming curricula in postgraduate medical education. Med Teach 2009;31:e260-e266.

(41) Malling B, Bonderup T, Mortensen L, Ringsted C, Scherpbier A. Effects of multi-source feedback on developmental plans for leaders of postgraduate medical education. Med Educ 2009;43:159-167.

(42) Wallenburg I, van Exel J, Stolk E, Scheele F, de Bont A, Meurs P. Between Trust and Accountability: Different Perspectives on the Modernization of Postgraduate Medical Training in the Netherlands. Academic Medicine 2010;85:1082-1090.

(43) Varpio L, Bell R, Hollingworth G et al. Is transferring an educational innovation actually a process of transformation? Adv Health Sci Educ Theory Pract 2012;17:357-367.

(44) Ginsburg L, Tregunno D. New approaches to interprofessional education and collaborative practice: lessons from the organizational change literature. J Interprof Care 2005;19 Suppl 1:177-187.

(45) Jippes E. The role of social communication networks in implementing educational innovations in healthcare. 2012

(46) Davis DA, Taylor-Vaisey A. Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ 1997;157:408-416.

(47) Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006;13:547-556.

(48) Grol RPTM, Bosch MC, Hulscher MEJL, Eccles MP, Wensing M. Planning and studying improvement in patient care: the use of theoretical perspectives. Milbank Q 2007;85:93-138.

(49) Jippes E, Van Luijk S, Pols J, Achterkamp M, Brand P, Van Engelen J. Facilitators and barriers to a nationwide implementation of competency-based postgraduate medical curricula: A qualitative study. Med Teach 2012.

(50) Baldwin TT, Ford JK. Transfer of Training: A Review and Directions for Future Research. Personnel Psychology 1988;41:63-105.

(51) van den Eertwegh V, van Dulmen S, van Dalen J, Scherpbier AJJA, van der Vleuten CPM. Learning in context: identifying gaps in research on the transfer of medical communication skills to the clinical workplace. Patient Educ Couns 2013;90:184-192.

(52) Bunniss S, Kelly DR. Research paradigms in medical education research. Med Educ 2010;44:358-366.

(53) May C. Towards a general theory of implementation. Implement Sci 2013;8:18. (54) Bordage G. Conceptual frameworks to illuminate and magnify. Medical Education

2009;43:312-319. (55) Cross RM. Exploring attitudes: the case for Q methodology. Health Education Research

2005;20:206-213. (56) Pronin E, Gilovich T, Ross L. Objectivity in the eye of the beholder: divergent perceptions

of bias in self versus others. Psychol Rev 2004;111:781-799. (57) Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Unexpected collateral

effects of simulation-based medical education. Acad Med 2011;86:1513-1517.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

General discussion

127

6

(58) Mennin S. Complexity and health professions education. J Eval Clin Pract 2010;16:835-837.

(59) van Mierlo B, Arkesteijn M, Leeuwis C. Enhancing the Reflexivity of System Innovation Projects With System Analyses. American Journal of Evaluation 2010;31:143-161.

(60) Athena institute. http://www.falw.vu.nl/nl/onderzoek/athena-institute/research Accessed 12-5-2014.

7Summary

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 7

130

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Summary

131

7


Innovations are being introduced into medical specialty training in response to

changes in healthcare demands and related changes in requirements for physicians’

training. However, bringing about the intended changes in practice turns out to be

challenging and largely dependent on how innovations are handled by the people

involved. The aim of this thesis is to contribute to the knowledge about innovating

medical specialty training. It uses knowledge about innovation processes from other

fields as a basis, and focuses on the experiences of the people who participate in

medical specialty training: residency program directors, consultants, and residents.

The overall question of this thesis is: how do people who participate in medical

specialty training deal with innovations in this training?

Chapter 1 provides the introduction to the research in this thesis. It starts with

explaining that the term innovation in medical education in this thesis indicates

something new or changed to training departments, which requires certain intended

behaviour of the people involved and typically includes new concrete methods.

Then, the changes in the fields of healthcare and medical education of the recent

past are explained. These include an increase of complexity of healthcare, and

societal demand for accountability of costs and of physician performance that

should include general qualities like good communication. These changes have led

to the contemporary view that specialty training should be competency based and

outcome oriented. The chapter continues by sketching the resulting innovations to

specialty training, like workplace-based assessment, and how research up until now

has mainly focused on the innovative applications and on intended and educational

effects, but not yet on the challenging process of innovating. Knowledge about

innovation processes from other fields like business, sociology, and healthcare

is introduced, followed by an outline of the paucity of attention to the process of

innovating medical specialty training. Thus, while it is clear that innovating specialty

training is challenging, it is still largely unclear how innovations in this area should

be dealt with in daily practice. Therefore, insight is needed into how those who

participate in medical specialty training deal with innovations. The chapter rounds

off with an overview of the studies that were conducted in order to address this

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 7

132

topic. Firstly, distinct aspects that are involved with innovation were looked into,

respectively: approaches of program directors responsible for change, the effects of

using an innovation, and the perceptions regarding these effects of those involved in

medical specialty training. Then, these elements were combined by looking into the

effects of an innovation and how those were intertwined with the implementation

approach.

Chapter 2 describes an exploratory qualitative study about program directors’

approaches to change at a training department. The study design was based on

notions from corporate business and social psychology about the roles of change

managers. The specific research questions were: which approaches to change

in specialty training do lead consultants use and what factors influence these

approaches? The study showed that lead consultants described different stages in

change processes, including cause, development of content, and the execution and

evaluation of change. Also, they seemed to use individual change strategies consisting

of elements such as ideas, intentions and behaviour. Factors influencing approaches

to change were: knowledge, ideas and beliefs about change; level of reflection; task

interpretation; personal style, and department culture. However, most consultants

showed limited awareness of their own approaches to change. This finding suggests

that they might adopt a rigid approach, whereas the ability to adapt strategies to

circumstances is considered important to effective change management.

Chapter 3 reports an exploratory qualitative study for establishing what types of

effects of an innovation its users perceive. It focussed on workplace-based assessment

(WBA) as a case of an innovation in specialty training that is widely used. In 2011,

semi-structured interviews were conducted with 17 purposively sampled Dutch

trainees and (lead) consultants in surgical and nonsurgical specialties. To encourage

exploration of effects outside the domain of education, the study design was

informed by sociological theory on the diffusion of innovations. Analysis supported

by the template analysis technique revealed six different, albeit interrelated,

domains of effects of WBA: sentiments (including affinity with the innovation and

emotions); dealing with the innovation; specialty training; teaching and learning;

workload and tasks, and patient care. Users’ affinity with WBA partly determined its

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Summary

133

7

effects on teaching and learning. Organisational support and the match between the

innovation and routine practice were considered important to minimise additional

workload and ensure that WBA was used for relevant rather than easily assessable

training activities. Dealing with WBA stimulated attention for specialty training and

placed specialty training on the agenda of clinical departments. These findings of

substantial effects outside the strictly education-related domain strongly indicate

that the people involved should consider all potential effects when designing and

implementing innovations.

Chapter 4 describes a Q methodological study that was conducted to explore the

distinct perceptions that users of the same innovation can have about its effects, again

focussing on WBA as a case. The specific research question was: what perceptions

of the effects of using WBA exist among its users? Purposively sampled obstetrics–

gynaecology residents and attending physicians (including program directors) at six

hospitals in the Netherlands performed individual Q sorts by ranking 36 statements

concerning WBA and WBA tools according to their level of agreement. By-person

factor analysis was conducted to uncover patterns in the ranking of statements,

followed by interpretation using participant comments about their Q sorts. This

led to identification of five distinct user perceptions regarding the effects of WBA

in practice, which were labelled enthusiasm, compliance, effort, neutrality, and

scepticism. These perceptions were characterized by differences in views on three

main issues: the intended goals of the innovation, its applicability (ease of applying

it to practice), and its actual impact. These findings point out that variance in

perceptions of an innovation’s effects have to be considered when innovating medical

specialty training, and provide insight into the nature of the distinct perceptions that

one could encounter.

Chapter 5 describes a study that aimed to gain insight into effects of an innovation

and how these are influenced by the implementation approach. For this purpose,

a case study was conducted of a Dutch project that tried to improve accountability

and quality of specialty training by introducing transparency and competition. Using

a theory-driven methodology, proceedings of project meetings were thematically

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 7

134

analyzed to identify choices and developments regarding the implementation

approach and to assess the effects of various approaches. This revealed intertwined

effects and features of the implementation approach that could be brought together

in themes called transparency, competition, and obstacles for competition. The

project team’s approach leaned on appealing to professional motivation, but was

supplemented by pressure to participate. This did elicit use of the innovation by

parties who had first resisted, but in a way that was not conducive to the goals.

Furthermore, the approach involved stakeholder involvement. This revealed

obstacles for implementation, to which the approach was then adapted. Attending

to these issues enabled the development of effects that went beyond the intentions,

but that were relevant for training quality. These findings suggest that adapting an

implementation approach to obstacles revealed by involving stakeholders supports

development of fundamental changes, and illustrate the dynamic nature of the

approach and its effects when innovating specialty training.

Finally, chapter 6 synthesises the research from the previous chapters and provides

an answer to the central research question: how do people who participate in

medical specialty training deal with innovations in this training? This answer points

to the need for a different conceptualization of innovating specialty training, which is

proposed subsequently. It entails abandoning the dominant linear perspective where

optimal implementation of an innovation will lead to certain intended effects. The

conceptualization described in this chapter offers a more dynamic model that can

account for the complexities of innovating specialty training. It links the conceptual

foundation of the innovation, its translation to practice, and effects in practice.

It accentuates the notion of translation of a concept to practice, which is formed

by the combination of applications and implementation approach. Following the

explanation of the new conceptualization, it is compared to existing literature. Next,

reflections on the methodology of the work in this thesis are provided. These include

an argument for not striving for a detailed manual for dealing with innovations in

specialty training, reflections on the strengths of this thesis (including the relevance

of this research, the approach, and the methodological rigour) and on the limitations

(including sources of possible bias and the transferability of the findings). Then,

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Summary

135

7

implications of the content of this thesis are set out. These include a shift in focus

for both research and practice from innovative applications to the translation of

innovative concepts that includes implementation approach. For medical education

research, this entails the challenge of taking up methodologies that are fit to study

complexity. For practice, it means that the training programmes of the innovation

professionals of the future, like in educational or management studies, need to

cover a broad terrain. Furthermore, all involved in innovating specialty training need

awareness of the complexity of the process as a basis for an appropriate approach.

The insight provided by this thesis can prevent them from expecting unambiguity,

clear-cut use of applications, and immediately reaching the intended effects. In this

way, this thesis supports realistic expectations and approaches for innovating the

practice of medical specialty training.

8Samenvatting

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 8

138

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Samenvatting

139

8

Het vernieuwen van de praktijk van de medische vervolgopleidingen

In aansluiting op de veranderende maatschappelijke verwachtingen over de

gezondheidszorg en het opleiden van medici worden vernieuwingen ingevoerd

in de opleidingen tot medisch specialist. Het tot stand brengen van de bedoelde

veranderingen blijkt echter een uitdaging te zijn en grotendeels af te hangen van

hoe de betrokkenen omgaan met de voorgestelde vernieuwingen. Het doel van

dit proefschrift is een bijdrage te leveren aan de kennis over het vernieuwen van

de medische vervolgopleidingen. Hierbij werd gebruik gemaakt van kennis over

veranderingsprocessen uit andere vakgebieden. De nadruk is gelegd op de ervaringen

van de mensen die betrokken zijn bij de dagelijkse praktijk van het opleiden: de

formele opleiders, de opleidende medisch specialisten en de arts-assistenten in

opleiding tot medisch specialist (aios). De hoofdvraag van dit proefschrift is: Hoe

gaan de betrokkenen bij de dagelijkse praktijk van de medische vervolgopleiding om

met vernieuwingen in die opleiding?

Hoofdstuk 1 introduceert het onderzoek in dit proefschrift. Het begint met een uitleg

dat de term vernieuwing in medisch onderwijs in dit proefschrift verwijst naar een

verandering of vernieuwing voor een opleidingsafdeling, waarvoor bepaald gedrag

van de betrokkenen wordt verwacht en dat meestal concrete nieuwe methoden en

instrumenten behelst. Vervolgens worden de veranderingen in de gezondheidszorg

en in het medisch onderwijs geschetst. Dit zijn onder andere toenemende

complexiteit van de gezondheidszorg, en maatschappelijke vraag om verantwoording

van kosten. Daarbij wordt van artsen steeds meer verwacht dat zij ook generieke

kwaliteiten, zoals goede communicatievaardigheden, bezitten. Deze veranderingen

hebben geleid tot de huidige visie dat de medische vervolgopleidingen competentie

gebaseerd en uitkomst gericht moeten zijn. Het hoofdstuk geeft vervolgens een

overzicht over de vernieuwingen die zijn ingevoerd in de vervolgopleidingen,

zoals werkplekbeoordelingen. De bespreking van het onderzoek op dit gebied

laat zien dat dit vooral gericht is op nieuwe methoden en instrumenten, en op

bedoelde en onbedoelde opleidingseffecten, maar nog niet op het uitdagende

vernieuwingsproces. Hierop volgt een inleiding in de kennis die er in andere

domeinen is over vernieuwingsprocessen, zoals in de bedrijfskunde, sociologie en

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 8

140

gezondheidszorg, en wordt een indruk gegeven van de beperkte aandacht voor het

proces van vernieuwen van de medische vervolgopleidingen. Dus, alhoewel het

bekend is dat het vernieuwen van de opleidingen tot medisch specialist uitdagend

is, is het nog grotendeels onduidelijk hoe er in de dagelijkse praktijk omgegaan

moet worden met vernieuwingen op dit gebied. Daarom is er inzicht nodig in hoe

de betrokkenen bij de dagelijkse praktijk van de medische vervolgopleiding omgaan

met deze vernieuwingen. Het hoofdstuk sluit af met een overzicht van het onderzoek

dat is beschreven in dit proefschrift. Eerst zijn verschillende aspecten van het

vernieuwingsproces afzonderlijk bekeken, respectievelijk: de aanpak van opleiders

die verantwoordelijk zijn voor het vernieuwen, de effecten van het gebruiken van

een vernieuwing, en de beleving van de betrokkenen over die effecten. Vervolgens

zijn deze elementen gecombineerd in een onderzoek naar de effecten van een

vernieuwing en hoe die effecten samenhingen met de implementatieaanpak.

Hoofdstuk 2 beschrijft een exploratief kwalitatief onderzoek naar de aanpak van

opleiders bij het tot stand brengen van een verandering op hun opleidingsafdeling.

Voor de onderzoeksopzet werd gebruik gemaakt van kennis uit de bedrijfswereld

en de psychologie over de rol van verandermanagers. De onderzoeksvragen waren:

welke aanpak voor verandering in de specialistenopleiding hebben opleiders en

welke factoren beïnvloeden deze aanpak? Het onderzoek liet zien dat opleiders

verschillende stadia beschreven in de veranderprocessen, waaronder de aanleiding,

de ontwikkeling van inhoud, en de uitvoering en evaluatie van de verandering.

Daarbij leken zij persoonlijke veranderingsstrategieën te gebruiken, die elementen

bevatten zoals ideeën, intenties en gedrag. Factoren die de veranderaanpak

beïnvloedden waren: kennis, ideeën en aannames over verandering, reflectieniveau,

taakinterpretatie, persoonlijke stijl, en afdelingscultuur. Echter, het merendeel

van de opleiders leek zich niet of slechts in beperkte mate bewust van de eigen

veranderaanpak. Deze bevinding wijst erop dat opleiders een rigide aanpak kunnen

hanteren, terwijl het voor effectief verandermanagement belangrijk is dat de aanpak

wordt aangepast aan de omstandigheden.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Samenvatting

141

8

Hoofdstuk 3 doet verslag van een exploratief kwalitatief onderzoek om erachter te

komen welke typen effecten van een vernieuwing worden ervaren door de gebruikers

ervan. Dit onderzoek richtte zich op de casus van werkplekbeoordelingen (WPB) als

een vernieuwing in de specialistenopleidingen die nu wijdverbreid wordt gebruikt.

In 2011 werden semi-gestructureerde interviews gehouden met 17 doelgericht

benaderde aios, opleiders en medisch specialisten uit snijdende en niet-snijdende

specialismen. Om exploratie van effecten buiten het opleidingsdomein te stimuleren

werd er bij de onderzoeksopzet gebruik gemaakt van sociologische theorie over

diffusie van innovaties. Analyse met behulp van de template analysis techniek

bracht zes verschillende, doch gerelateerde, domeinen van effecten van WPB aan

het licht: gevoelens (waaronder affiniteit met de innovatie en emoties), omgaan met

verandering, de opleiding, leren en opleiden, werkbelasting en taken, en patiëntenzorg.

De affiniteit van gebruikers met WPB bepaalde deels de effecten op het leren en

opleiden. Organisatorische ondersteuning en de aansluiting van de vernieuwing

bij de dagelijkse gang van zaken werden belangrijk geacht om werkbelasting te

minimaliseren. Dit was ook belangrijk om ervoor te zorgen dat WPB bij de meest

relevante opleidingsactiviteiten werd ingezet, in tegenstelling tot bij gemakkelijk

toegankelijke activiteiten. Het bezig zijn met WPB vergrootte de aandacht voor

opleiden en maakte de opleiding onderwerp van gesprek op opleidingsafdelingen.

De bevinding dat er aanzienlijke effecten zijn buiten het opleidingsdomein wijst erop

dat het belangrijk is om bij het ontwerpen en implementeren van vernieuwingen

aandacht te besteden aan alle mogelijke effecten.

In hoofdstuk 4 wordt een Q methodologisch onderzoek beschreven dat werd

uitgevoerd om te exploreren welke verschillende percepties van de effecten van

een innovatie de gebruikers ervan kunnen hebben, wederom toegespitst op WPB.

De onderzoeksvraag was: welke percepties van de effecten van het gebruik van

werkplekbeoordelingen bestaan er onder de gebruikers? De 65 deelnemende aios,

opleiders en medisch specialisten van zes verschillende obstetrie-gynaecologie

afdelingen waren doelgericht benaderd. Zij maakten individuele Q-sorteringen door

36 stellingen over WPB en WPB-instrumenten te rangschikken op basis van hun mate

van overeenstemming met de stellingen. Met factoranalyse met de participanten

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 8

142

als variabelen werden patronen in de rangschikkingen opgespoord, waarna de

interpretatie hiervan kon worden gedaan met behulp van de toelichtingen die de

participanten op hun rangschikking hadden gegeven. Dit resulteerde in de vondst

van vijf verschillende gebruikerspercepties over de effecten van WPB in de praktijk,

die enthousiasme, meegaandheid, moeite, neutraliteit en scepsis werden genoemd.

Deze percepties worden gekarakteriseerd door verschil in visie op drie gebieden: het

doel van de vernieuwing, de toepasbaarheid en de daadwerkelijke uitwerking in de

praktijk. Deze bevindingen wijzen erop dat het voor het vernieuwen van de medische

vervolgopleidingen belangrijk is om rekening te houden met variatie in de percepties

van effecten van een vernieuwing.

Hoofdstuk 5 beschrijft een onderzoek om inzicht te verkrijgen in de effecten van

een vernieuwing en hoe die worden beïnvloed door de implementatieaanpak.

Hiervoor werd als casus een Nederlands project bestudeerd, waarvan het doel was

de verantwoording over en kwaliteit van de specialistenopleidingen te vergroten

door transparantie en competitie in te voeren. Met een op theorie gebaseerde

methodologie werden notulen van projectvergaderingen thematisch geanalyseerd

om keuzes en ontwikkelingen in de implementatieaanpak op te sporen en om effecten

van verschillende benaderingen te achterhalen. Dit bracht samenhangende effecten

en kenmerken van de implementatieaanpak aan het licht, die werden samengevoegd

in thema’s genaamd transparantie, competitie en obstakels voor competitie. De

aanpak van het projectteam berustte op het appelleren aan professionele motivatie,

maar werd aangevuld met druk om te participeren. Dit lokte inderdaad uit dat

partijen die zich eerder verzetten gebruik gingen maken van de vernieuwing, maar

zij deden dit niet op een manier die bijdroeg aan de doelen. Verder bevatte de

aanpak het betrekken van de belanghebbende partijen. Dit bracht obstakels voor

de implementatie aan het licht, waaraan de aanpak werd aangepast. Dit maakte het

mogelijk dat er effecten ontstonden die verder reikten dan het oorspronkelijke doel,

maar die relevant waren voor opleidingskwaliteit. Deze bevindingen suggereren

dat de ontwikkeling van fundamentele veranderingen kan worden versterkt door

de implementatieaanpak aan te passen aan de obstakels die belanghebbenden

aandragen. Ze illustreren ook de dynamische aard van de aanpak en de effecten van

een vernieuwingsproces in de specialistenopleiding.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Samenvatting

143

8

Tot slot voegt hoofdstuk 6 het onderzoek uit de voorgaande hoofdstukken samen

en geeft antwoord op de centrale onderzoeksvraag: Hoe gaan de betrokkenen bij

de dagelijkse praktijk van de medische vervolgopleiding om met vernieuwingen in

die opleiding? Het antwoord maakt duidelijk dat er een verandering nodig is in de

conceptualisering van het vernieuwen van de medische vervolgopleiding, waarvoor

vervolgens een voorstel wordt gedaan. Het behelst het verlaten van het dominante

lineaire perspectief waarin optimale implementatie van een vernieuwing zal leiden

tot bepaalde bedoelde effecten. De conceptualisering die in dit hoofdstuk wordt

voorgesteld bevat een meer dynamisch model waarin rekening wordt gehouden

met de complexiteit van het vernieuwen van de medische vervolgopleiding. Het

model verbindt de conceptuele basis van de vernieuwing, de vertaling naar de

praktijk en de effecten op de praktijk. Het benadrukt daarbij het begrip van de

vertaling van een concept naar de praktijk, en dat deze vertaling bestaat uit de

combinatie van een nieuw opleidingsinstrument en de implementatieaanpak.

Aansluitend op de uitleg van de nieuw geopperde conceptualisering wordt deze

vergeleken met bestaande literatuur. Het hoofdstuk vervolgt met een reflectie op

de methodologie die bij het werk in dit proefschrift is gehanteerd. Hieronder vallen

een overweging om niet te streven naar een gedetailleerde handleiding voor het

vernieuwen van de specialistenopleidingen, reflecties op de sterke punten van dit

proefschrift (waaronder de relevantie van het onderzoek, de aanpak, en de grondige

methodologie) en op de zwakke punten (waaronder mogelijke bronnen van bias en

de generaliseerbaarheid van de bevindingen). Hierna worden de implicaties van de

inhoud van dit proefschrift besproken. Voor zowel onderzoek als praktijk zijn dit

onder andere het verschuiven van de nadruk op opleidingsinstrumenten naar nadruk

op conceptvertaling waarvan implementatieaanpak deel uitmaakt. Voor onderzoek

naar medisch onderwijs ligt er de uitdaging om methodologieën te gaan gebruiken

die geschikt zijn voor het bestuderen van complexiteit. Voor de praktijk is er de

implicatie dat opleidingen van vernieuwingsprofessionals, zoals onderwijskundige

en management studies, een breed terrein moeten beslaan. Bovendien moeten alle

betrokkenen bij het vernieuwen van de medisch specialistenopleidingen zich bewust

zijn van de complexiteit van het proces, als basis voor een passende benadering

ervan. Het inzicht dat dit proefschrift biedt kan hen behoeden voor het verwachten

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 8

144

van duidelijkheid, of eenduidig gebruik van instrumenten, of dat bedoelde effecten

direct tot stand komen. Op deze manier draagt dit proefschrift bij aan realistische

verwachtingen en benaderingen voor het vernieuwen van de dagelijkse praktijk van

de medische vervolgopleiding.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

9Dankwoord

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 9

148

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Dankwoord

149

9

Ik kijk met heel goede herinneringen terug op de jaren waarin het werk in dit

proefschrift tot stand is gekomen. Dat is niet omdat het altijd makkelijk ging, maar

omdat ik niet alleen was. Er waren veel mensen in mijn persoonlijke en werkomgeving,

van binnen onze onderzoeksgroep tot ver daarbuiten, die zich betrokken hebben

getoond. Zij dachten en puzzelden met mij mee, zij stonden paraat bij vragen en

zorgen, zij daagden mij uit, en ik kon plezier met hen maken over werk en heel andere

zaken. Ik dank hen hiervoor allemaal heel hartelijk. In het bijzonder wil ik noemen:

Fedde Scheele, jij hebt het mogelijk gemaakt dat ik onderzoek kon gaan doen op het

gebied van de medische opleidingen. Op een ongeëvenaarde manier betrek jij mij

en jouw andere promovendi bij jouw avonturen in opleidingsland. Gedurende mijn

hele traject straalde jij vertrouwen in mij uit, ook als ik het zelf niet had. In de grote

‘scharrelruimte’ die jij bij dit alles bood heb jij mij heel veel laten leren.

Joep Dörr kan ik tot mijn grote spijt niet meer persoonlijk bedanken voor zijn

vertrouwen, interesse en stimulerende vragen. Zijn inzet voor het ontwikkelen van

medische opleidingen wordt door mij en vele anderen gemist.

Pim Teunissen, het was fantastisch voor mij dat jij mijn copromotor wilde worden. Zo

mocht ik in alle fasen van het onderzoek een beroep doen op het vangnet van jouw

intellect, tact en humor.

Albert Scherpbier, ik ben vereerd dat jij tot mijn team behoort. Jij hielp mij om zelf

keuzes te maken die ik ingewikkeld vond; niet alleen met jouw expertise, maar ook

door steeds oog te houden voor mijn persoonlijke voorkeur.

Cees van der Vleuten, ook jou om advies te mogen vragen is een groot voorrecht.

Ik kon rekenen op een kritische blik en scherpe vragen, die mij -in tegenstelling tot

angst- juist vertrouwen gaven.

Michiel Westerman, jouw luisterend oor, advies van praktische en filosofische aard,

goede recepten en flauwe grappen bleken vanaf dag één onmisbaar. Het past dus

ook goed dat jij mij als paranimf bijstaat tijdens de verdediging van mijn proefschrift.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 9

150

Nadine van der Lee, het was erg prettig om het ontwikkelen als onderzoeker en

moeder met jou te kunnen delen. Ik bewonder jouw werklust en vind het fantastisch

dat jouw proefschrift ook klaar is.

Emma Paternotte, Noera Kieviet, Tessa van den Beukel, Bert Loosman, Irene

Slootweg, Renée van der Leeuw, Harold Bok, Chantal de Haan, Marianne Kerssens

en andere collega’s van het SLAZ, de Journal Club en elders, jullie hebben mij de

geweldige ervaring gegeven van collega’s die je altijd om hulp kan vragen. Daarbij

was het op het werk en daarbuiten gewoon lang niet zo leuk geweest zonder jullie.

Marjan van Wegen, jouw inzet voor allerlei zaken omtrent mijn bijzondere

dienstverband was fantastisch.

Job van Exel, zonder jouw hulp en bereikbaarheid was het doen van de Q studie niet

alleen onmogelijk geweest, maar zeker ook niet zo leuk en leerzaam.

Alle collega’s van het Etalageproject, jullie hebben mij de mogelijkheid geboden het

vernieuwen van de vervolgopleidingen van dichtbij te bestuderen.

Niels en Annelijn, familieleden, vriendinnen en vrienden, jullie zijn mij ieder op jullie

eigen manier dierbaar. Gelukkig is het voor het afronden van een proefschrift niet

nodig om dit met wetenschappelijke nauwkeurigheid te omschrijven.

Friedolien de Fraiture, ik ben blij dat jij als mijn paranimf de steun van al deze lieve

mensen wilt vertegenwoordigen.

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

10About the author

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

Chapter 10

154

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

R12

R13

R14

R15

R16

R17

R18

R19

R20

R21

R22

R23

R24

R25

R26

R27

R28

R29

R30

R31

R32

R33

R34

About the author

155

10

Joanne Fokkema was born on 26 March 1983 in Groningen as the daughter of Dirk and

Josca Fokkema-Elders, and sister to her brother Sanne. She attended the Praedinius

Gymnasium in Groningen, while the family lived in Beilen. After graduating high

school in 2000, she spent a gap year in New Zealand travelling and instructing school

children at an outdoor activity centre. In 2001, she was selected to study Medicine

at the University of Amsterdam. She was a member of the student counsel in 2003-

2004, of the development committee for the new undergraduate curriculum in 2003-

2005, and a tutor for first-year students in 2004-2005. She did a scientific internship

in Uganda and a surgical internship in Zambia. After her graduation in 2008, she

worked as a surgical house officer at the Sint Lucas Andreas Hospital for a year and

as an assistant general practitioner at the island of Texel during the summer. At that

time, she developed the aspiration to study a medical educational topic in a PhD

trajectory. Fedde Scheele and Joep Dörr created the opportunity for her to do so by

involving her in the innovative “Opleidingsetalage” project for postgraduate medical

education in the Netherlands. Now that her PhD trajectory has come to an end, she

is setting out to become a general practitioner and aims to stay involved with medical

education research.

From early childhood on, Joanne has passed a lot of her spare time sailing and as a

sailing instructor. She started off on her parents’ traditional fishing boat the pluut

HK60 and in dinghies. Nowadays, Joanne lives in Amsterdam with Niels de Graaf and

their daughter Annelijn (2013). Together they enjoy taking out her parents’ current

yacht Staron that is depicted on the cover of this thesis.

INNOVATING THE PRACTICE OF

MEDICAL SPECIALTY TRAININGJoanne P. I. Fokkema

INN

OVATIN

G TH

E PRACTICE OF M

EDICA

L SPECIALTY TRA

ININ

G

Joanne P. I. Fokkema

UITNODIGING

Voor het bijwonen van de openbare verdediging van

het proefschrift

INNOVATING THE PRACTICE OF MEDICAL SPECIALTY TRAINING

door Joanne Fokkema

op woensdag 29 oktober 2014om 13:45 uur in de aula van de

Vrije Universiteit,De Boelelaan 1005,

Amsterdam

en voor de hierop aansluitende receptie in

The Basket,De Boelelaan 1111,

Amsterdam

Paranimfen:Michiel Westerman

0642307595Friedolien de Fraiture

[email protected]

Joanne Fokkema

Tweede Oosterparkstraat 249

1092 BM Amsterdam

0627056411

[email protected]

Proefschrift Fokkema

Documents

Transcript of Proefschrift Fokkema