PREST Module Program

download PREST Module Program

of 84

Transcript of PREST Module Program

  • 8/13/2019 PREST Module Program

    1/84

    PRESTPractitioner Research andEvaluation Skills Training inOpen and Distance Learning a

    Programme evaluation and itsrole in quality assurance

    HANDBOOK

    B1

  • 8/13/2019 PREST Module Program

    2/84

    The PREST training resources aim to help open and distance learning practitioners develop andextend their research and evaluation skills. They can e used on a sel!"study asis or y trainingproviders. The resources consist o! t#o sets o! materials$ a six"module !oundation course inresearch and evaluation skills and six hand ooks in speci!ic research areas o! ODL. There is anaccompanying user guide. % !ull list appears on the ack cover.

    The print" ased materials are !reely do#nloada le !rom the &ommon#ealth o! Learning '&OL(#e site '###.col.org)prest(. Providers #ishing to print and ind copies can apply !or camera"ready copy #hich includes colour covers 'in!o*col.org(. They #ere developed y the +nternationalResearch ,oundation !or Open Learning '###.ir!ol.ac.uk( on ehal! o! &OL.

    The PREST core team&harlotte &reed 'Programme coordinator(Richard ,reeman '+nstructional designer- editor and author(Pro!essor ernadette Ro inson '%cademic editor and author(

    %lan /oodley '%cademic editor and author(

    %dditional mem ersTerry %llsop '&ritical revie#er(

    %licia ,entiman ' asic education adviser(0raham 1iles 'Page layout(1elen Lentell '&ommon#ealth o! Learning Training Programme 2anager(Santosh Panda 'External academic editor(Reehana Ra3a '1igher education adviser(

    Steering groupThe PREST programme has een guided y a distinguished international steering group including$Peter &ookson- Ra4 Dhanara4an- Tony Dodds- Terry Evans- Olug emiro 5egede- David 2urphy-Evie 6onyongo- Santosh Panda and 1ilary Perraton.

    %ckno#ledgements/e are particularly grate!ul to 1ilary Perraton and Ra4 Dhanara4an #ho originally conceived o! thePREST programme and have supported the pro4ect throughout. %mong those to #hom #e areinde ted !or support- in!ormation and ideas are 1onor &arter- 7ate &ro!ts- 5ohn Daniel- 6ick 0ao-5enny 0lennie- 7eith 1arry- &olin Latchem- Lydia 2eister- Roger 2ills- San4aya 2ishra- Ros2orpeth- Rod Tyrer- Paul /est and Dave /ilson. +n developing the materials- #e have dra#ninspiration !rom the lead provided y Roger 2itton in his hand ook- 2itton- R. 89:; Practicalresearch in distance education - &am ridge$ +nternational Extension &ollege.

    1and ook ' http$))creativecommons.org)licenses) y"sa)=.>) (

    +S 6 8":9

  • 8/13/2019 PREST Module Program

    3/84

    &ontents

    Programme evaluation and its role in quality assurance.......................................1

    1and ook overvie# .................................................................................................. 8Learning outcomes ................................................................................................... 81and ook organisation ............................................................................................ ;1o# to use the materials .......................................................................................... =Evaluation examples ................................................................................................ AResources ................................................................................................................ :Status o! the readings .............................................................................................. :

    Unit 1: What is programme evaluation and why is it important?.........................11Bnit overvie# .......................................................................................................... 88Bnit outcomes ........................................................................................................ 8;/hat is programme evaluationC ............................................................................ 8;Evaluation as a !orm o! research ........................................................................... 8@The relationship et#een programme monitoring and evaluation ......................... 89Programme monitoring in ODL ...............................................................................89/hy should distance education practitioners conduct evaluationsC ..................... ;>Bnit summary ......................................................................................................... ;8Re!erences ............................................................................................................. ;;,eed ack to selected activities .............................................................................. ;;

    Unit 2: Evaluation conceptuali ation: de!ining the purpose o! an evaluation....2"Bnit overvie# .......................................................................................................... ;@Bnit outcomes ........................................................................................................ ;@Purposes and types o! evaluation .......................................................................... ;@+denti!ying an evaluation s target audience ............................................................ =>

    ,ormulating key questions .......................... ........................... ................ ....... ...... .. =8&onsultation a key to success ............................................................................ =@Bnit summary ......................................................................................................... =?Re!erences ............................................................................................................. =:,eed ack to selected activities .............................................................................. =9

    Unit #: $esigning and implementing an e!!ective evaluation...............................%#Bnit overvie# ...........................................................................................................

  • 8/13/2019 PREST Module Program

    4/84

    &ontents

    &onnecting evaluation to programme planning and quality assurance processes A:Bnit summary ......................................................................................................... ?=&onclusion ............................................................................................................. ?>= SOMDEL $Somali Distance Education Literacy Programme(Macallin a Raddiya! - report prepared !or %!rica Educational Trust '%ET( y the

    +nternational Research ,oundation !or Open Learning '+R,OL(- &am ridge- B7$+R,OL

    'This #ill e re!erred to as Evaluation " in the rest o! this hand ook.(

    +n 2arch ;>>; the %!rica Educational Trust together #ith the & /orld Trustlaunched a ne# and innovative distance education literacy programme in Somalilandand Somalia. This programme is called SOMDEL # the Somali Distance EducationLiteracy Programme # Li$e S ills %&&roach . The programme !ocuses on theprovision o! literacy- numeracy and li!e skills training to out"o!"school children andadults. SOMDEL involves the use o! radio- print and !ace"to"!ace teaching. +n order

    to assess the e!!ectiveness o! this programme the +nternational Research,oundation !or Open Learning #as commissioned to undertake an externalevaluation.

    The evaluation used the logical !rame#ork o! the programme as the asis on #hichachievements #ere measured. Iarious research methods #ere employed to gatherthe evaluation data- including examination and scrutiny o! statistical records anddocuments- !ocus group discussions- semi"structured intervie#s- case studies-surveys and participant o servation.

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning6

  • 8/13/2019 PREST Module Program

    11/84

    +ntroduction

    Evaluation ;2agagula- &. ;>>; Evaluation re&ort o$ the 'ourse $or Distance Education Policyma ers in Southern %$rica - Iancouver$ &ommon#ealth o! Learning

    'This #ill e re!erred to as Evaluation ) in the rest o! this hand ook.(

    The &ommon#ealth o! Learning '&OL( contracted the evaluator to evaluate anonline course !or policy"makers #ho #ork in ODL institutions in su "Saharan %!rica.This course set out to provide participants #ith$

    a conceptual plat!orm and in"depth introduction to open learning and distanceeducation concepts

    an understanding o! the policy development process

    an understanding o! the implications o! di!!erent policy choices in open learning

    and distance education.The evaluator #as asked to determine #hether the online course met theparticipants needs and also #hether the delivery strategies 'online deliverysupported y a one"#eek !ace"to"!ace #orkshop( #ere e!!ective. To ans#er thesequestions the evaluator used questionnaires- intervie#s- and analysis o! documents-e"mail messages and learning activities.

    &ommon#ealth o! Learning 7

  • 8/13/2019 PREST Module Program

    12/84

    1and ook < Programme evaluation and its role in quality assurance

    ResourcesThe !ollo#ing resources are used in this hand*oo $

    ,esource -ame when

    re!erred to in ourte t

    /ocation

    ,entiman- %. ;>>= SOMDEL+ Somali DistanceEducation Literacy Programme (Macallin aRaddiya! - report prepared !or %!rica EducationalTrust '%ET( y the +nternational Research,oundation !or Open Learning '+R,OL(-&am ridge$ +R,OL

    Evaluation " Resources File

    2agagula- &. ;>>; Evaluation re&ort o$ the'ourse $or Distance Education Policy ma ers inSouthern %$rica - Iancouver$ &ommon#ealth o!Learning

    Evaluation ) Resources File

    7achroo- 7. 8999 GDistance educationprogrammes at college and university levels in5ammu region$ an evaluative study Indian,ournal o$ O&en Learning :- =$ ;9="=>@

    -achroo Resources File

    2ugridge- +. 8999 GJuality assurance in openand distance learning Indian ,ournal o$ O&enLearning :- =$ ;8=";;>

    Mugridge Resources File

    2oore- 2. 8999 GEditorial 2onitoring andevaluation .he %merican ,ournal o$ DistanceEducation 8=- ;$ 8"@

    Moore Resources File

    /hyte- %. ;>>> %ssessing communitytelecentres+ guidelines $or researchers -Otta#a$+nternational Development Research &entre- athttp$))###.eldis.org)static)DO&?:A?.htm

    /hyte htt&+001112eldis2org0static 0DO'34532htm

    These resources are included as an appendix at the end o! the pu lication.

    Status o! the readings+n order to achieve the learning o 4ectives outlined a ove you #ill need to make sure

    to read the essential reading. /e #ill re!er to these evaluation studies throughoutthis hand ook. The optional readings #ill allo# you to !urther develop yourunderstanding o! the concepts presented here. +! possi le- + #ould recommend thatyou try to read these. The !urther readings have een provided to assist you toexplore issues that you !ind particularly interesting or use!ul !or your #ork.

    Essential Reading,entiman- %. ;>>= SOMDEL+ Somali Distance Education Literacy Programme

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning8

  • 8/13/2019 PREST Module Program

    13/84

    +ntroduction

    (Macallin a Raddiya! - report prepared !or %!rica Educational Trust '%ET( y the+nternational Research ,oundation !or Open Learning '+R,OL(- &am ridge$ +R,OL

    7achroo- 7. 8999 GDistance education programmes at college and university levelsin 5ammu region$ an evaluative study Indian ,ournal o$ O&en Learning :- =$ ;9="=>@

    2agagula- &. ;>>; Evaluation re&ort o$ the 'ourse $or Distance Education Policyma ers in Southern %$rica - Iancouver$ &ommon#ealth o! Learning

    Optional reading2ugridge- +. 8999. Juality %ssurance in Open and Distance Learning. Indian ,ournalo$ O&en Learning - :'=(.

    ,urther reading2oore- 2. 8999 GEditorial 2onitoring and evaluation .he %merican ,ournal o$Distance Education 8=- ;$ 8"@

    Rathore- 1. and Schuemer- R. 'eds.( 899: GEvaluation concepts and practice inselected distance education institutions - 6IFF &a&er 8>:- 1agen$ &entral +nstitute!or Distance Education Research- ,ernBniversitat- at http$))###.!ernuni"hagen.de)K+,,)pre!8>:.htm

    /hyte- %. ;>>> %ssessing community telecentres+ guidelines $or researchers -Otta#a$ +nternational Development Research &entre- at http$))#e .idrc.ca)en)ev"

    ;:=>8";>8"8"DO TOP+&.html+ hope you are looking !or#ard to #orking through this hand ook. The 4ourney

    egins y considering #hat programme evaluation is and #hy this !orm o! appliedsocial research is use!ul in the context o! distance education.

    &ommon#ealth o! Learning 9

  • 8/13/2019 PREST Module Program

    14/84

    1and ook < Programme evaluation and its role in quality assurance

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning10

  • 8/13/2019 PREST Module Program

    15/84

    /hat is programme evaluationand #hy is it importantC

    Bnit overvie#This unit !ocuses on helping you to develop a clear understanding o! #hatprogramme evaluation is and #hy it could e use!ul !or you. /e are going to egin

    y looking at #hat programme evaluation is. To do this- #e #ill consider a series o!de!initions o! evaluation and then look in more detail at #hat the concept o! aprogramme includes. Since this hand ook is part o! a series !ocused on researchskills #e #ill rie!ly look at evaluation as a !orm o! research. The unit ends #ith aconsideration o! the relationship et#een programme monitoring and evaluation as#ell as #hy evaluation is use!ul !or distance education programmes.

    2any o! the concepts that you #ill learn here- although covered in the context o!

    distance education- can in !act e used !or designing evaluations o! a #ide range o!social interventions or programmes. So- + hope this unit 'and the #hole hand ook in!act( #ill e a use!ul research tool in many di!!erent research and evaluation contextsthat you might !ind yoursel! in.

    +n this unit you #ill e encouraged to read the t#o evaluation studies rie!lydescri ed in the introduction. Each o! these studies has een chosen to highlightspeci!ic issues that #e #ill cover as #e progress.

    You might like to read the executive summaries of Evaluation 1 and Evaluation2 in the Resources File be ore beginning the handbook to gain a sense o !hatthe" cover# You should not s$end more than about 30 minutes doing this# %s areminder& the t!o studies are'

    &ommon#ealth o! Learning

    B6+T

    1

    ,eading 0 ,esources !ile

    11

  • 8/13/2019 PREST Module Program

    16/84

    1and ook < Programme evaluation and its role in quality assurance

    (entiman& %# 2003 SOMDEL: Somali Distance Education Literacy Programme (Macallin a Raddiya!" re$ort $re$ared or % rica)ducational *rust +%)*, b" the -nternational .esearch (oundation or/$en earning +-.(/ ,& ambridge& ' -.(/ # *his is called

    Evaluation 1 in this handbook# agagula& # 2##2 Evaluation re$ort of t%e &ourse for Distance

    Education Policy'ma ers in Sout%ern frica" ancouver'ommon!ealth o earning# *his is called Evaluation 2 in this

    handbook#

    Bnit outcomes

    /hen you have #orked through this unit- you should e a le to$

    8 state a de!inition o! programme evaluation

    ; explain programme evaluation !rom a research perspective

    = descri e the relationship et#een programme monitoring and evaluation

    < give reasons #hy conducting programme evaluation is important !or distanceeducation programmes and institutions.

    /hat is programme evaluationC/hat do you understand y the concept o! programme evaluation C +s it a term youhave come across e!oreC +n #hat contextsC

    e!ore #e look at some !ormal de!initions o! programme evaluation- #rite do#n yourresponse to the t#o short questions in %ctivity 8.

    1 o! !ould "ou de ine $rogramme evaluation

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning

    ctivity 1 8@ mins

    12

  • 8/13/2019 PREST Module Program

    17/84

    2 ive at least one reason !h" "ou think "ou might !ant to conduct a $rogramme evaluation at "our institution#

    .he $eed*ac to this activity is in the $ollo1ing te7t

    Programme evaluation some de!initions6o# lets see ho# your ideas compare to the de!initions given in the literature.

    2ore de!initions8Evaluation is the &rocess o$ collecting and0or using in$ormation $or the &ur&oses o$determining the value and 1orth 1hileness o$ the su*9ect o$ the evaluation &rocess:' irley M 2orel 899:(.

    8Evaluation is the collection; analysis and inter&retation o$ in$ormation a*out anyas&ect o$ a &rogramme o$ education or training; as &art o$ a recognised &rocess o$

    9udging its e$$ectiveness; its e$$iciency and any other outcomes it may have: 'Thorpe 899=(.

    8Evaluation is the systematic &rocess o$ collecting; analysing and inter&retingin$ormation that ena*les 9udgements to *e made a*out the value o$ a &rogramme(o$ learning! and its e$$ectiveness and0or e$$iciency in achieving a set o$ outcomes:(Dolley "

  • 8/13/2019 PREST Module Program

    18/84

    1and ook < Programme evaluation and its role in quality assurance

    a ie and 2outon ';>>8( present a use!ul conceptual model or map o! aprogramme ',igure 8(.

    Figure " % conce&tual ma& o$ a &rogrammeSource+ >a**ie; E2 ? Mouton; ,2 )@@"; &2A=A2

    This diagram sho#s us #hat a programme Glooks like. The details o! each o! theoxes #ill di!!er depending on the speci!ic programme. The example here is relevant

    to an educational programme. This diagram highlights the core !eatures that shoulde part o! a programme and sho#s the links et#een them.

    1 -n the s$ace belo!& re$roduce this conce$tual model& using a $rogramme that"ou kno! o & or $erha$s& are a $art o & to com$lete the details o each o the

    blocks# *o do this& start !ith the block numbered one and the ill in the

    details o each block in numerical order#

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning

    ctivity 2

  • 8/13/2019 PREST Module Program

    19/84

    2 o! !ell did this model hel$ "ou to ma$ out "our exam$le o a distanceeducation $rogramme

    .he $eed*ac to this activity is at the end o$ the unit

    Evaluation as a !orm o! research0iven that this hand ook is one resource in a series dealing #ith research methods#e should ask #hat the relationship is et#een evaluation and research more

    roadly. %re they one and the same thingC

    To help us egin this discussion- #e re!er to the #ords o! 2ichael Patton taken !roma much"quoted ook on evaluation. 1e says$

    8/hen one e7amines and 9udges accom&lishments and e$$ectiveness; one isengaged in evaluation2 /hen this e7amination o$ e$$ectiveness is conductedsystematically and em&irically through care$ul data collection and thought$ulanalysis; one is engaged in evaluation research: (Patton "

  • 8/13/2019 PREST Module Program

    20/84

    1and ook < Programme evaluation and its role in quality assurance

    evaluation research. Others argue that in !act evaluation is a di!!erent type o! activity!rom research- even though oth make use o! similar methods.

    Bnderstanding this de ate is important ecause your stance #ill a!!ect the #ay thatyou approach the design and implementation o! an evaluation. /hat do you thinkC

    Read through the extract elo# that outlines some possi le similarities anddi!!erences et#een research and evaluation.

    Similarities between research and evaluation

    Same methods, same skills.esearchers and evaluators use the same methods o social science in;uir" andre;uire the same sorts o skills& ;ualities and $rinci$les in the $ractitionersconducting an in;uir"#

    Evaluation and research share the same domain)valuation is $art o the domain o educational research and is not isolated romgeneral develo$ments and issues in educational research& its methodolog" andtheoretical issues#

    Rational re-construction versus reality: a shared challenge

  • 8/13/2019 PREST Module Program

    21/84

    Differences between research and evaluation

    Selecting uestions or problems for investigation% researcher is $robabl" more ree to select his or her o!n research ;uestionsand to seek ans!ers to them in !hatever !a" he or she sees as a$$ro$riate

    $ro essionall"# %n evaluator o ten is commissioned to investigate a $re=determined ;uestion or to$ic# >ome $ractitioners vie! the di erence bet!eenresearch and evaluation as l"ing in the $ur$ose o the t!o activities and thekinds o ;uestion asked# ?hile researchers ma" seek generalisation in their

    indings& and ask ;uestions relating to general $ro$ositions& evaluators ocusmore on s$eci ic activities in $articular contexts +e#g#& ?as this $articular

    $rogramme cost=e ective or this institution using these media :,#

    ! different relationship between the research uestion and methodologyused.esearch ;uestions are likel" to be sha$ed b" the $roblem=solving or in;uir"techni;ues or methods available and considered sound& so an initial research;uestion ma" be amended in the light o !hat it is most $ossible to measure orans!er# earning to ormulate ans!erable research ;uestions is $art o thetraining o researchers#

  • 8/13/2019 PREST Module Program

    22/84

    1and ook < Programme evaluation and its role in quality assurance

    ! more political activity than research)valuation is inesca$abl" a $olitical activit" because o the vested interests othe various stakeholders and the di erent meanings the" $lace on the evaluationactivit" and its indings# >ince evaluations $oint to!ards $ractical action&di erent stakeholders are likel" to be more immediatel" a ected b" the indingsthan b" a $urer: research stud"#

    ! broader set of competences and skills re uired)valuators o ten need to be com$etent in a !ide range o methods andtechni;ues& es$eciall" i !orking alone& !hereas researchers ma" $re er& or aresome!hat more likel" to s$ecialise in one or other a$$roach +;uantitative or;ualitative,# ?here researchers combine methods o research& one a$$roachtends to lead and the other su$$orts +the" are not generall" used in e;ualmeasure,#

    Robinson 2001, pp. 5-7

    So- to summarise the point here evaluation is a speci!ic !orm o! research thatshould !ollo# the same overall guidelines and protocols as any other researchprocess. 1o#ever- !or various reasons listed a ove- an evaluation pro4ect is alsodi!!erent !rom a research pro4ect. %s an evaluator you need to e clear on oth thesesimilarities and di!!erences.

    Evaluations are usually applied research- conducted !or someone or some

    organisation and involve making some !orm o! value 4udgements a out theprogramme in question. +n other #ords- evaluations should al#ays have a clearlyde!ined purpose and target audience. Iery o!ten- evaluations have implications !orresource decisions- most commonly !unding decisions- and so evaluation researchcan e more sensitive than other !orms o! research.

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning18

  • 8/13/2019 PREST Module Program

    23/84

    The relationship et#een programme monitoring andevaluationHou may or may not have #orked through the hand ook on programme monitoring-so 4ust in case- here are the de!initions o! monitoring presented in that hand ook.

    8Monitoring is the continuous assessment o$ theintervention and its environment 1ith regard to the

    &lanned o*9ectives; results; activities; and means2 Itta es &lace at all levels o$ management and uses*oth $ormal re&orting and in$ormal communications2Monitoring ena*les a sta eholder to revie1 &rogressand to &ro&ose action to *e ta en in order to achievethe o*9ectives2 Monitoring identi$ies actual or &otentialsuccesses or $ailures as early as &ossi*le and$acilitates timely ad9ustments to the o&erations:

    2inistry !or ,oreign %!!airs 899:(.

    8Monitoring+ recording and trac ing the &rogress o$ a &rogramme 1hile it is running;1ith a vie1 to 9udging 1hether a &lan is on course and its o*9ectives are on the 1ayto *eing achieved2 In$ormation collected $or monitoring &ur&oses is also use$ul asin$ormation $or evaluation: (Dolley "

  • 8/13/2019 PREST Module Program

    24/84

    1and ook < Programme evaluation and its role in quality assurance

    monitoring is one o! the central means o! systematic data collection. So- programmemonitoring is one o! the tools !or use in programme evaluation. % good programmemonitoring system should provide the researcher #ith much o! the data needed#hen conducting an evaluation. The challenge is to identi!y #hat in!ormation to look!or in order to ans#er your speci!ic evaluation question's(. Deciding on #hichin!ormation to gather #ill e a topic that runs throughout this hand ook. Hou mightalso like to have a look at the hand ook Bsing Programme 2onitoring in Researchand Evaluation.

    /hy should distance education practitioners conductevaluationsCSo !ar #e have considered #hat programme evaluation is- ho# it relates to researchmore roadly and also the role that programme monitoring plays in evaluation.

    ut #hy should you e interested in conducting evaluationsC

    !ctivity # $ hour

    You should use either o the t!o evaluation re$orts& Evaluation 1 or 2& inthe Resources File or this activit"#

    @ote do!n "our res$onse to the ollo!ing ;uestions as "ou read throughthe ull re$ort# + int' *he $ur$ose o the evaluation is usuall" stated earl"

    on in the re$ort,#1 ?hat is the stated $ur$ose o the evaluation

    2 o! is this evaluation use ul or the distance education $rogramme in;uestion

    .he $eed*ac to this activity is at the end o$ the unit

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning

    ctivity # 8 hour

    20

  • 8/13/2019 PREST Module Program

    25/84

    One o! the conclusions to dra# !rom these examples is that in most casesevaluations are a tool !or decision"making- and !or assessing and promoting thequality o! a programme or institution. /hile evaluations provide !indings that canhelp programme planners make decisions- it is also important to note that this doesnot necessarily make decision"making easier. Sometimes evaluation !indings #illrecommend that very di!!icult decisions e made- !or example that a programme isnot #orking and should e discontinued.

    Evaluations have an important role to play in quality assurance processes.Evaluations are a out making 4udgements a out quality and value o! a programme./e #ill look in much more detail at quality assurance in Bnit

  • 8/13/2019 PREST Module Program

    26/84

    1and ook < Programme evaluation and its role in quality assurance

    You ma" need to a$$roach the leader o the $rogramme "ou have chosen ordocumentation or $erha$s a brie intervie! to ind the in ormation that "ou !illneed to hel$ "ou to develo$ "our s$eci ic model#

    Re!erencesa ie- E. and 2outon- 5. ;>>8 .he &ractice o$ social research - &ape To#n$ Ox!ord

    Bniversity Press in South %!rica

    irley- 2. and 2orel- 6. 899: % &ractical guide to academic research - London$7ogan Page

    Dolley- 5. 899< Planning; monitoring and evaluating learning &rogrammes -uckingham$ Open Bniversity Press

    2inistry !or ,oreign %!!airs. 899: Cuidelines $or &rogramme design; monitoring andevaluation - 1elsinki$ Department !or +nternational Development- athttp$))glo al.!inland.!i)4ulkaisut)yleis)pdme)index.html

    Patton- 2. 899> Qualitative evaluation and research methods ';nd edition( London$Sage Pu lications

    Ro inson- . ;>>8 % &ractical guide to evaluation - a #orkshop hand ook produced!or the B6ES&O Learning !or Li!e Pro4ect in Blaan aatar$ 2ongolia

    Ru in- ,. 899@ % *asic guide to evaluation $or develo&ment 1or ers - Ox!ord$ Ox!amPu lications

    Thorpe- 2. 899= GEvaluating open and distance learning - 1arlo#$ Longman- cited inD. Ro#ntree 899: G%ssessing the quality o! materials" ased teaching and learningin %. Tait 'ed.( O&en Learning =$;

    ,eed ack to %ctivity ; %t a !irst glance- the dou le arro#s tell us that each o! the parts making up aprogramme are inter"related and have an e!!ect on each other. ,ollo#ing thenum ers a ove each lock- #e see a direct relationship et#een goals and thetarget group. This is an important point- since a programme is usuallyconceptualised and implemented to meet the needs o! a speci!ic target group. /hothe target group is #ill di!!er !rom programme to programme as #ill the goals. % goodand #ell"de!ined programme #ould operationalise these goals into measura leindicators o! success. 1o#ever- you #ill see as you #ork #ith programmes thatmany do not reach this point- goals remain vague and one is not sure #hether they

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning

    eed3ac+ to selected activities

    22

  • 8/13/2019 PREST Module Program

    27/84

    have een achieved or not. This is one point at #hich programme evaluation is o!central importance. /e #ill return to this issue in later units.

    +n order to achieve the goals o! the programme several components #ill e needed.ox < re!ers to the mechanisms and means o! implementation used to achieve the

    goals o! the programme- #hich- in this example- are di!!erent #ays o! delivering aneducational programme. +n order to e a le to implement the programme- amanagement and a human resource ase #ill e needed. +n the context o! adistance education institution this #ill include institutional management structures-educators- tutors- materials developers- administrators etc. These are not the onlypeople involved in a programmeN there are also various other stakeholders.Stakeholders are the people or organisations #ho have any interest- direct orindirect- in the success o! the programme. +n a distance education setting learnersare an important group o! stakeholders. ,inally- all o! the parts making up aprogramme exist in context. There are several levels o! context that might in!luencea programme. ,or example- a distance education programme exists in the context o!the department and institution providing it- as #ell as in a local- national orinternational context. /hen designing an evaluation it is important to keep all o!these contexts in mind- as they could in!luence the !indings o! your evaluation.

    % programme evaluation- depending on its purpose- #ill consider all or some o!these components. Hou #ill see later that one approach to #riting an evaluation planis to #ork out appropriate research question's( !or each o! the parts making up aprogramme.

    ,eed ack to %ctivity =Hou #ill pro a ly have identi!ied several purposes and various #ays in #hich thereports #ould e use!ul to the organisation. These might include$

    Purpose to analyse the e!!ectiveness o! the programme including its reach o! the target

    audience

    to examine the impact o! the programme

    to decide #hether a !under s money has een #ell spent

    to explore the strengths and #eaknesses o! the distance education approachused

    to make recommendations !or changes to the programme.

    &ommon#ealth o! Learning 23

  • 8/13/2019 PREST Module Program

    28/84

    1and ook < Programme evaluation and its role in quality assurance

    Bse!ul ecause it provides a thorough appraisal o! the #hole programme y an external agent

    it in!orms the quality assurance process.

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning24

  • 8/13/2019 PREST Module Program

    29/84

    Evaluation conceptualisation$de!ining the purpose o! an

    evaluation

    Bnit overvie#The !irst unit o! this hand ook !ocused on setting out de!initions and clari!ying someimportant issues in the !ield o! programme evaluation and ho# this relates toensuring quality #ithin distance education. +n this and the next unit you #ill #orkthrough some concepts and approaches that underlie programme evaluation. +n thisunit #e #ill e concentrating on understanding the di!!erent purposes o! evaluations-ho# #e ensure that the evaluation is use!ul !or those it is eing conducted !or- andalso- ho# #e decide on #hat questions to ask #hen !ocusing our evaluation study.Each o! these is important !or planning an evaluation.

    Bnit outcomes/hen you have #orked through this unit- you should e a le to$

    8 de!ine the purposes and types o! evaluation

    ; state the di!!erence and relationship et#een !ormative and summativeevaluation

    = identi!y the target audience o! an evaluation

    < !ormulate key evaluation questions

    @ descri e #hy consultation is important #hen conducting an evaluation.

    Purposes and types o! evaluation/e ended Bnit 8 noting the importance o! evaluation in supporting e!!orts to ensurequality in ODL programmes. 1ere #e #ill look in more detail at some o! the reasons

    &ommon#ealth o! Learning

    B6+T

    2

    25

  • 8/13/2019 PREST Module Program

    30/84

    1and ook < Programme evaluation and its role in quality assurance

    #hy people conduct evaluations- i.e. the purpose o! the evaluation. Bnderstandingthis is central !or designing and conducting good evaluations. /e #ill then exploreho# the purpose o! an evaluation shapes ho# the evaluation is designed andconducted.

    ist at least three reasons !h" someone might !ant to conduct an evaluation#

    1

    2

    3

    .he $eed*ac to this activity is at the end o$ the unit

    The main purposes o! evaluationsPeople #orking in the !ield o! programme evaluation have attempted to summarise

    this #ide range o! reasons into categories so that it is easier to #rite a out anddiscuss evaluation purposes. /hile there is al#ays some di!!erence et#een theapproaches adopted y di!!erent authors there is also a general acceptance o! themain purposes !or #hich evaluations are conducted. %lthough #e look at only t#ohere- many di!!erent taxonomies o! evaluation have een proposed in the literature.

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning

    ctivity 1 8@ mins

    26

  • 8/13/2019 PREST Module Program

    31/84

    a ie and 2outon ';>>8( use the #ork o! 2ichael Patton to de!ine three mainpurposes o! evaluations as$

    8 to make 4udgements '4udgement"orientated(

    ; to make improvements 'improvement"orientated(= to generate kno#ledge 'kno#ledge"orientated(.

    Similarly ut #ith one addition- ,rancis Ru in '899@(de!ines !our categories$

    8 to improve per!ormance

    ; to make choices and decisions

    = to learn lessons

    < to increase accounta ility.

    /e #ill look in more detail at a ie and 2outon s three categories. ut e!ore #edo- #e should note that the last o! Ru in s points to increase accounta ility is alsoan important purpose o! evaluation- especially in the context o! quality assurance.Evaluation is one means o! assessing #hether people and institutions do #hat theyhave een assigned to do- and so provides a means o! checking sta!! per!ormance-as people #ill have to provide ans#ers !or their actions during an evaluationprocess. %s #e noted in Bnit 8- !or these kinds o! reasons evaluations cansometimes e challenging to conduct- and in such cases evaluators are rarely liked

    y everyoneF Some approaches to evaluation have explicitly sought to address thisissue. ,or example participatory evaluation seeks to involve programme participantsin all parts o! the evaluation and approaches evaluation more as learning opportunitythan an opportunity !or making 4udgements.

    To return to our purposes o! evaluation #e #ill look rie!ly at each o! the threepurposes de!ined a ove. This #ill also provide us #ith an opportunity to look at thedistinction et#een !ormative and summative evaluation- #hich has ecome a#idely accepted distinction #ithin evaluation methodology.

    5udgement"oriented evaluationsThink o! a situation in #hich you might need to conduct an evaluation o! aprogramme that has provided teachers #ith computer training to assess #hetherthey are no# a le to use computers and #hether this has improved their teaching. +nthis instance you #ould e conducting a 4udgement"oriented evaluation. Thiscategory o! evaluation research includes those studies that set out to determine the#orth- value or success o! a programme.

    ut- ho# do you kno# i! the teachers teaching has improvedC &entral to all

    &ommon#ealth o! Learning

    Orientations o!evaluations1 Budgement

    2 -m$rovement

    3 kno!ledge

    27

  • 8/13/2019 PREST Module Program

    32/84

    1and ook < Programme evaluation and its role in quality assurance

    evaluations that set out to make 4udgements is the !ormulation o! criteria against#hich to 4udge the success o! the programme.

    Think ack to Bnit 8- #here #e looked at the structure o! a programme. Do youremem er that #e noted that a good programme #ould have clearly de!ined goals

    and that those goals #ould have een translated into measura le outcomes orcritical success !actorsC /e also noted that- o!ten- programmes do not have cleargoals and measura le outcomes. O!ten- it is only #hen evaluations egin that goalsare made explicit and measura le outcomes de!ined. % core step in conducting a

    4udgement"oriented evaluation is to de!ine these criteria and ho# they #ill emeasured.

    Since #e are measuring #hether a programme meets speci!ied criteria- then #emust e conducting the evaluation a!ter the programme has een implemented orto#ards the end o! implementation- !or example- at the end o! a ne#ly introduced

    distance education course. This is #here #e meet up #ith the concept o! summativeevaluation.

    Summative evaluations

    4ummative evaluation is an evaluation that takesplace to make 4udgements a out the overall successor !ailure o! a programme- and is o!ten related todecisions a out #hether the programme should econtinued or not. Summative evaluation usually takesplace to#ards the end o! a programme or a!terimplementation.

    ,unders o!ten use this kind o! evaluation to test #hether the programme they have!unded has achieved the outcomes it #as supposed to. The ans#er to this questionmay then determine #hether !urther !unding is granted or not. This kind o! evaluationis also conducted to identi!y the lessons that can e learned !rom the experienceand applied to !uture pro4ects or programmes.

    +mprovement"orientated evaluations %s the name suggests- improvement"orientedevaluations are concerned #ith improving theprogramme #hile it is eing implemented rather than

    4udging ho# success!ul it has een duringimplementation or a!ter completion. This kind o!evaluation is usually called a !ormative evaluation .

    1ere the evaluation helps to G!orm the programme y providing a means o!assessing the strengths and #eaknesses o! the programme- looking at ho#implementation takes place- considers the response o! programme participants etc.

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning

    Summative evaluationSummative evaluation is an evaluationthat takes $lace to make Audgementsabout the overall success or ailure o a

    $rogramme#

    ,ormative evaluation Formative evaluation is concerned !ithim$roving a $rogramme !hile it is

    being im$lemented#

    28

  • 8/13/2019 PREST Module Program

    33/84

    The aim here is to identi!y pro lems as they occur so that they can e quicklycorrected. Sometimes more su stantial changes o! approach or ad4ustment o! plansmay e needed to improve the programme.

    7no#ledge"orientated evaluations5udgement and improvement"orientated evaluationsare carried out in order to provide direct input into aspeci!ic programme or to assess #hether or not theprogramme is success!ul. +n other #ords- oth !ormso! evaluation have an applied !ocus. +n contrast-+nowledge5oriented evaluations are conducted toimprove our understanding- to help uild theory andsometimes to in!orm policy"making processes.

    ,or example- you might evaluate a range o! distance education programmes toassess #hich approach led to the est learning outcomes- the results o! #hich couldin!orm the planning o! !uture programmes.

    This purpose !or evaluation is much less common than the t#o descri ed a ove andyou are pro a ly more likely to make use o! !ormative and summative evaluations-or a com ination o! these- in your o#n evaluation #ork.

    alancing !ormative and summative evaluation/e have de!ined summative

    evaluation as evaluation that is usuallyconducted to#ards the end o! aprogramme #ith the purpose o!

    4udging its success or e!!ectiveness.,ormative evaluation in contrast #asde!ined as evaluation conductedduring a programme !or the purpose o!improving it. ,rom these de!initionsyou might have concluded that theset#o !orms o! evaluation stand in

    opposition to each other- that thedi!!erent kinds o! evaluation arede!ined y purpose and point inprogramme development at #hichthey are conducted.

    Rather- in practice these categories overlap. Summative evaluation can econducted during a programme to measure achievement o! speci!ic o 4ectives at apoint in time. Simultaneously- the evaluator may also e using !ormative evaluation

    &ommon#ealth o! Learning

    7no#ledge"orientatedevaluations im$rove our understanding

    build theor"

    in orm $olic"=making

    7no#ledge"generating evaluations%n exam$le o no)ledge'generating evaluation isincluded in the Resources File C see *ac%roo # -n thisstud" the authors evaluated a range o distance learninghigher education $rogrammes in order to makerecommendations or success ul im$lementation odistance education $rogrammes at college and univerist"levels in the Bammu region# *he evaluations are used togenerate kno!ledge about im$lementing distance highereducation $rogrammes in a general sense& rather thanmaking Audgements about or im$roving a s$eci ic

    $rogramme#

    29

  • 8/13/2019 PREST Module Program

    34/84

    1and ook < Programme evaluation and its role in quality assurance

    to evaluate and improve programme processes. These di!!erent kinds o! evaluationscomplement each other and using a com ination o! oth is likely to ensure a morethorough evaluation study. Each type serves a speci!ic purpose in the evaluation o!a programme. %lthough complementary- planning !or !ormative and summativeevaluations usually get done at di!!erent times and may even e conducted ydi!!erent evaluators depending on the speci!ic programme and circumstances.

    *his activit" is based on !hichever o the t!o evaluation exam$les& Evaluations1 or 2 in the Resources File" "ou have chosen to !ork on# -t !ill hel$ "ou thinkabout the extent to !hich a given evaluation is ormative& summative& or acombination o both#

    1 -s this a ormative or summative evaluation& or a combination )x$lain brie l" !h" "ou came to this conclusion#

    .he $eed*ac to this activity is at the end o$ the unit

    +denti!ying an evaluation s target audienceEvaluations are usually applied research except in the case o! kno#ledge"orientedevaluations rie!ly descri ed a ove. +t !ollo#s that this applied research should euse!ul to some ody or some organisation. /hen designing an evaluation you needto clari!y #ho the target audience is. This #ill depend to some extent on #ho askedyou to conduct the evaluation and the type o! evaluation eing conducted and

    #hat the evaluation report #ill e used !or.+n oth o! our evaluation examples- #e have an evaluation consultant conducting anevaluation o! an ODL programme. +n each case the !unders o! the programmecommissioned the evaluation and- as such- are its main target audience. Theprogramme implementers #ould also e target audiences since they #ould einterested in ho# e!!ective their distance education programme #as and #ould liketo kno# #hich areas should e improved. Others #ho may #ish to use similarapproaches- or perhaps replicate a programme in another context- #ould also einterested in the evaluation !indings. ecause o! the range o! audiences the

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning

    ctivity 2 => mins

    30

  • 8/13/2019 PREST Module Program

    35/84

    evaluation #ill need to consider the politics o! the situation. ,or example- the needsand interests o! !unders and pro4ect implementers may not e the same- and may in!act e contradictory i! the !unders are considering a #ithdra#al o! !unding ased onthe evaluation results. oth o! these examples highlight !or us the !act thatevaluation studies and the report thereo!- most o!ten have a range o! targetaudiences.

    /hen designing your evaluation and #hen #ritingyour evaluation report you should ask yoursel! the!ollo#ing questions$

    +s the audience's( !or the evaluation clearlyspeci!iedC

    Does the evaluation address the particularconcerns o! the main target audienceC This isnot an easy question to ans#er. One #ay o!arriving at an ans#er is through a process o!consultation- #hich #e #ill address elo#.

    1o# should the evaluation report e #rittenso that it is appropriate and accessi le !or thetarget audiencesC

    Hou might like to look at the Re&orting on research and evaluation to in$luence andsu&&ort change module- #hich deals in more detail #ith the construction o!

    reports and dissemination strategies.

    ,ormulating key questionsThe core module- Planning research and evaluation- has covered ho# to developresearch questions- and also the importance o! this !or the overall success o! theresearch pro4ect eing conducted. ,or a success!ul evaluation- the clear !ormulationo! your key question's( is !undamental. %t this point it use!ul to recall the point made

    y 7umar '8999 p. ?($

    8% research &ro*lem is li e the $oundation o$ a *uilding2 .he ty&e and design o$ the*uilding is de&endent u&on the $oundation2 I$ the $oundation is 1ell designed andstrong; you can e7&ect the *uilding to *e also2 .he research &ro*lem serves as the$oundation o$ a research study+ i$ it is 1ell $ormulated you can e7&ect a good studyto $ollo12:

    The same principles that you covered !or research questions in general also apply toevaluation research questions. +n addition- the purpose and type o! evaluation eingconducted #ill in!orm your decision a out the key question's(.

    &ommon#ealth o! Learning

    ,ood !or thought*hink back to the institution "ouhave chosen or "our $roAecttasks#

    @o! think about ho! "ou might begin to ans!er these ;uestions in

    that context#

    You !ill return to this belo!!hen "ou go back to "our $roAect!ork#

    31

  • 8/13/2019 PREST Module Program

    36/84

    1and ook < Programme evaluation and its role in quality assurance

    Evaluation types are a slightly di!!erent- although related- distinction to that #emade a ove et#een di!!erent kinds o! purposes !or evaluations. /ithin each o! thepurpose categories it is possi le to conduct di!!erent types o! evaluations. /e #illagain dra# on a ie and 2outon ';>>8( in presenting this section. These authorsdra# on the !our types o! evaluation proposed y Posavac and &arery '899;(although it is important to note that di!!erent authors classi!y these types o!evaluation in other #ays too.

    Type 8$ Evaluation o! need1ere you #ould e interested in evaluating the needs o! an identi!ied target group.O!ten this is related to the provision o! a speci!ic service. ,or example- you mayneed to evaluate #hether students taking part in your distance educationprogramme need additional peer support or not- e!ore implementing a ne# peer"support programme. This type o! evaluation can also e called a needs assessment.

    Juestion examples /hat training do primary teachers need on the ne# school curriculumC

    /hat income"generating activities do #omen in rural communities #ant to learna outC

    /hat kind o! childcare in!ormation do young parents in rural areas needC

    Type ;$ Evaluation o! process+n this case the evaluation #ould !ocus on the design and implementation o! aprogramme. ,or example- you may need to evaluate ho# #ell your institution scurrent administration system #orks #ith respect to student registration- assignmentsu mission and return o! marks and examinations. Hou #ould most likely einterested in understanding the processes !ollo#ed in order to assess #hether this isthe most e!!ective system or #hether improvements are needed.

    Juestion examples +s the programme eing implemented as designedC

    Does the programme serve the target populationC

    %re the necessary administrative and management procedures in place !or theprogramme to run e!!ectivelyC

    Do the materials reach the rural village to coincide #ith the supporting radioroadcastsC

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning32

  • 8/13/2019 PREST Module Program

    37/84

    Type =$ Evaluation o! outcomeOutcome evaluation is one o! the most common !orms o! evaluation. +t assessesthe e!!ects o! the programme. This is usually done y comparing the e!!ects #iththe goals and measura le outcomes de!ined at the outset. To assess outcome an

    evaluator should try to collect aseline data that re!lects the situation e!ore theprogramme #as implemented. y collecting the data a!ter and)or during theprogramme implementation- the evaluator can then assess #hether there have eenany changes due to the programme. /hen considering programme outcomes ore!!ects the evaluator usually looks at oth intended and unintended e!!ects.Evaluators are usually interested in proving that some !orm o! positive change hasoccurred as a result o! the programme. To do this- it must e esta lished that somechange or e!!ect has occurred- and also that this is a result o! the programme " andnot other external !actor's(.

    +t is also important that #e distinguish et#een outcomes and outputs. These areterms that you have pro a ly seen used in a variety o! #ays. %n outcome is the endresult or e!!ect o! a programme- #hile an output is a product o! a programme. ,orexample- in the SO2DEL evaluation example- the programme aims to achieve theoutcome o! providing literacy- numeracy and li!e skills !or out"o!"school children andadults in Somalia and Somaliland. +n striving to achieve this outcome- theprogramme includes several outputs- !or example- print materials- radio roadcasts-and !ace"to"!ace teaching sessions.

    Juestion examples To #hat extent have intended outcomes een achievedC

    To #hat extent have literacy levels een improvedC

    /hat unintended outcomes have een achievedC

    1as the programme achieved the speci!ied goalsC

    Type learners have access to their o#ncomputer- and a common television and video recorder to support a speci!icdistance education course. The pro4ect might sho# that learning outcomes improved

    &ommon#ealth o! Learning 33

  • 8/13/2019 PREST Module Program

    38/84

    1and ook < Programme evaluation and its role in quality assurance

    remarka ly due to the introduction o! +&Ts. One might then e tempted to concludethat this programme #as a success and should e replicated in other environments.1o#ever- an evaluation o! e!!iciency may sho# that the costs o! providing thistechnology per learner #ould more than dou le standard learning costs. +n this caseit #ould e necessary to assess #hether the improved learning outcomes 4usti!iesthis additional cost and also #hether !unds are availa le to implement this on alarger scale.

    Juestion examples /ere !unds spent !or the intended purposeC

    Did the programme achieve success at an accepta le costC

    1o# do the outcomes o! this programme compare to those o! a similarprogramme o! compara le costC

    The range o! terminologies used in this area can e con!using. ,or this reason-some use!ul explanations are provided in Ta le 8.

    6a3le 1 4ome terminology in the evaluation o! e!!iciency

    6erm E planation

    &ost ene!it The relationship et#een the costs o! a programme and its ene!its- in terms o! theactual or estimated monetary value.This contrasts #ith cost"e!!ectivenessmeasurement- #hich may measure ene!its in terms o! di!!erent 'non"monetary(kinds o! outcomes. &ost ene!it results are sho#n as the ratio o! costs to ene!its-or as a !igure representing the net ene!it 'or net loss(.The technique allo#s

    comparisons to e made et#een programmes and policies that have quitedi!!erent outcomes. &osts o! input are typically easier to measure than ene!its o!outcomes. G&ost ene!it is o!ten used loosely in everyday language to re!er to therelationship et#een the cost 'not necessarily !inancial( o! doing something and the

    ene!its 'o! #hatever kind( gained.

    &ost"e!!ectiveness

    The relationship et#een the achievement o! o 4ectives o! a programme- course orpro4ect and their costs. % cost"e!!ective programme is one that can e sho#n toprovide either the maximum gains !or a given level o! resources or a given level o!

    ene!its at the lo#est cost. &ost"e!!ectiveness analysis is most easily applied #henthere is one clearly identi!ia le outcome measure or gain 'this is not al#ays thecase #ith education pro4ects(.

    &ost e!!iciency %n education programme is cost e!!icient i! its outputs cost less per unit o! input-#hen compared #ith alternatives.

    E!!iciency The measurement o! the resources used 'costs( to achieve stated goalsN themeasurement or ratio o! output to input.To e e!!icient- a programme or institutionneeds to demonstrate that good use is eing made o! the resources availa le.E!!iciency is not an a solute measure ut a 4udgement a out relative #orth.

    E!!ectiveness The extent to #hich an education or training programme or intervention candemonstrate that its goals 'explicit ones( have een met.

    Source$ Ro*inson ($orthcoming!

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning34

  • 8/13/2019 PREST Module Program

    39/84

    Summary+n this section #e have looked at the evaluation o! need- the evaluation o! process-the evaluation o! outcome and the evaluation o! e!!iciency. %s #ith the distinction

    et#een !ormative and summative evaluation- there is likely to e overlap et#een

    these types o! evaluation #ithin any one evaluation study. Hou #ill very o!ten !indthat questions !or all o! the a ove types are asked in a comprehensive programmeevaluation.

    -n this activit" "ou !ill take another look at the evaluation exam$le !hich"ou used in %ctivit" 2#

    1 ?hat t"$e o evaluation do "ou think this is

    2 @ote do!n !hat research ;uestions are being ans!ered#

    .he $eed*ac to this activity is at the end o$ the unit

    &onsultation a key to successThis !inal issue that you need to think a out in the design o! your evaluation is the

    relationship you 'as the evaluator( #ill have #ith those involved in the programmeeing evaluated- as #ell as #ith any others #ho have an interest in the evaluation

    'these are the stakeholders(. These could include !unders such as examinationoards or accreditation authorities or pro!essional associations or employers or

    sponsors o! students !or example.

    There has een much de ate concerning the relationships et#een researchers andparticipants and their respective roles in evaluation research. Traditionally-evaluation research assumed that researchers stood ack !rom those eingresearched- collected data !rom them- analysed it and #rote a report. 2any

    &ommon#ealth o! Learning

    ctivity # => mins

    35

  • 8/13/2019 PREST Module Program

    40/84

    1and ook < Programme evaluation and its role in quality assurance

    evaluators no longer accept this and usually the evaluator and programmeparticipants- programme leaders or !unders- must reach some clearly de!inedagreements a out shared expectations o! the purpose o! the evaluation and ho# it#ill e conducted. Bnless the programme is small and !ocused it is unlikely that theevaluator #ill e a le to consult all participants or stakeholders. +n this case- theconsultation usually takes place #ith those requesting the evaluation research. ,orcontract research- this is o!ten clearly stated in a #ritten contract agreement.

    There is an approach to evaluation called Gparticipatory evaluation that is ased onthe principle o! consultation at all levels. %s noted a ove- this approach to evaluation!ocuses on evaluation as a learning process rather than a process o! making

    4udgements. Programme participants make the 4udgements- rather than theevaluator. The role o! the evaluator in participatory evaluation is as a !acilitator ratherthan as the Gexpert researcher . The evaluator !acilitates an evaluation process- utthe !ocus o! the evaluation- ho# it takes place- #hen it takes place- the researchquestions and so on- are determined y the programme participants rather than theevaluator or !under.

    +t is particularly important !or researchers or evaluators to e a#are o! organisationaldynamics and structures- as o!ten the person responsi le !or negotiating a contractor requesting that an evaluation e conducted is not directly involved in the micromanagement o! the programme eing evaluated. +t may also e the case that theevaluation is not #elcomed y the programme providers or y the institution.Throughout the evaluation- expectations o! oth the evaluator and programmeparticipants need to e monitored and managed. This may include returning to the

    !ocus o! the evaluation and revie#ing #hether this still meets expectations- and i!not- #hat should e done a out this. 1o# this is done #ill depend on various !actors-including the kind o! programme eing evaluated- the relationship o! the evaluator tothe programme 'e.g. an employee or contracted in !rom outside(- the scope o! theevaluation- the time!rame etc. ,or large programmes and complex evaluations #ith arange o! stakeholders- it might e use!ul to set up a steering group to ensureconsultation. +t #ill e important that this group is representative o! the range o!stakeholders in the programme. There #ill e di!!erent kinds o! evaluationparticipants or stakeholders. Some #ill e the evaluation !unders- the programme!unders- those responsi le !or the planning and management o! the programme-those #ho #rite distance education materials- the learners- the assessors etc.

    Equally important to managing expectations !rom the outset o! evaluation research-is to esta lish agreement et#een researchers and evaluation participants a outho# data #ill e collected and analysed.

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning36

  • 8/13/2019 PREST Module Program

    41/84

    +n this regard- 2iles and 1u erman '899

  • 8/13/2019 PREST Module Program

    42/84

    1and ook < Programme evaluation and its role in quality assurance

    >o ar in "our $roAect "ou should have'

    identi ied a distance education $rogramme to !ork !ith

    dra!n a sketch o the $rogramme so "ou have a clear idea o !hat "ouare !orking !ith#

    You are no! read" to begin the $re$aration o "our evaluation $lan# se theollo!ing ;uestions to hel$ "ou do this'

    1 ?hat is the $ur$ose o m" evaluation

    2 ?hat are m" research ;uestion+s,

    3 ?hat t"$e o evaluation !ill - be conducting

    4 ?ho is the target audience o m" evaluation

    5 ?ho !ill it be im$ortant to consult in designing this evaluation +You mightlike to arrange a time to brie l" meet !ith one or some o the $eo$le to ask!hat the" think about the evaluation "ou are $lanning#,

    Re!erencesa ie- E. and 2outon- 5. ;>>8 .he &ractice o$ social research - &ape To#n$ Ox!ord

    Bniversity Press in South %!rica 'pp ==?"==9(

    Ru in- ,. 899@ % *asic guide to evaluation $or develo&ment 1or ers - Ox!ord$ Ox!amPu lications 'pp =8(

    7achroo- 7. 8999 GDistance education programmes at college and university levelsin 5ammu region$ an evaluative study Indian ,ournal o$ O&en Learning :- =$;9="=>@

    7umar- R. 8999 Research methodology+ a ste& *y ste& guide $or *eginners - London$

    SagePosovac- E. and &arey- R. 899; Programme evaluation+ methods and case studies -Engle#ood &li!!s$ Prentice 1all

    Ro inson- . '!orthcoming( %chieving uality in o&en and distance learning - Paris$B6ES&O

    2iles- 2. and 1u ermann- %. 899< Qualitative data analysis - London$ Sage 'pp

  • 8/13/2019 PREST Module Program

    43/84

    ,eed ack to %ctivity 8Hou might have included some o! the !ollo#ing reasons !or evaluations in your list$

    !or e!!ective management and planning-

    to measure progress and)or e!!ectiveness

    to learn lessons !or !uture programmes

    ecause !unders made evaluation a condition !or giving money

    as part o! a research pro4ect

    ecause your institution has a policy a out conducting evaluations regularly

    even though sta!! aren t al#ays clear a out #hy evaluations are done or ho#the results #ill e usedF

    ,eed ack to %ctivity ;/e thought that these evaluations could e classi!ied as !ollo#s$

    8 Somali distance education literacy programmeThis evaluation aimed to 8analyse the e$$ectiveness o$ the SOMDEL &rogramme interms o$ reaching its target *ene$iciaries to e7amine the im&act o$ this three 1aya&&roach 1hich incor&orates &rinted materials; radio *roadcasts and $ace to $aceinstruction to discuss the strengths and 1ea nesses o$ this a&&roach 1ith a varietyo$ sta eholders and to ma e recommendations o$ suggestions $or $uture

    &rogrammes: 'p. :(.

    % look at the research design o! this study #ill sho# that the research #asconducted to#ards the end o! the !irst set o! roadcasts !or this programme. /e seethat some o! the aims o! the evaluation #ere to assess the impact and e!!ectiveness

    o! the programmes- i.e. summative evaluation approach. The evaluation also hadthe purpose o! learning lessons !rom this !irst set o! roadcasts !or improving the!uture implementation o! the programme. This aspect o! the evaluation can edescri ed as !ormative evaluation.

    ; &ourse !or distance education policy makers in Southern %!rica

    The aim here #as to 8determine 1hether the online course met course &artici&ants:

    &ommon#ealth o! Learning

    eed3ac+ to selected activities

    39

  • 8/13/2019 PREST Module Program

    44/84

    1and ook < Programme evaluation and its role in quality assurance

    needs and 1hether the delivery strategies 1ere e$$icient and e$$ective: 'p. @(.

    Thus- this study is a out making 4udgements a out the programme and as such isan example o! a summative evaluation. 1o#ever- a closer look at the methodologyused #ill sho# that !ormative evaluation data #as also used in carrying out the

    summative evaluation.

    ,eed ack to %ctivity =

    Evaluation Example One$ SO2DELThis example is an interesting one ecause instead o! !ormulating researchquestions- the evaluation has !ormulated aims and then makes use o! the pro4ect

    log!rame as the !rame#ork o! the evaluation. +n the introduction to this evaluationreport #e read the !ollo#ing$

    .he s&eci$ic aims o$ the evaluation 1ere to analyse the e$$ectiveness o$ theSOMDEL &rogramme in terms o$ reaching its target *ene$iciaries to e7amine theim&act o$ this three 1ay a&&roach 1hich incor&orates &rinted materials; radio*roadcasts and $ace to $ace instruction to discuss the strengths and 1ea nesseso$ this a&&roach 1ith a variety o$ sta eholders and to ma e recommendations orsuggestions $or $uture &rogrammes (&2 4!2

    ased on these aims #e can say that this evaluation is an outcome)impact

    evaluation and also an evaluation o! process. %lthough no speci!ic researchquestions are noted- do you see that #e could re!ormulate these aims as researchquestionsC &onsider the !ollo#ing possi le research questions$

    8 Did the SO2DEL programme reach those it #as intended to reachC Evaluationo! process

    ; /hat #as the impact o! the three"#ay approach adoptedC Evaluation o!outcome

    = /hat are the strengths and #eaknesses o! the SO2DEL programmeCEvaluation o! process

    Evaluation Example T#o$ &ourse !or Distance Education Policy2akersOn page : o! the evaluation report #e !ind the terms o! re!erence !or this study. /esee that- #ith a !ocus on online delivery and the !ace"to"!ace #orkshop- theevaluator #as requested

    to determine 1hether the online course met the &artici&ants: needs (i2e2 o$

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning40

  • 8/13/2019 PREST Module Program

    45/84

  • 8/13/2019 PREST Module Program

    46/84

    1and ook < Programme evaluation and its role in quality assurance

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning42

  • 8/13/2019 PREST Module Program

    47/84

    Designing and implementingan e!!ective evaluation

    Bnit overvie#+n this unit you #ill consider ho# to move !rom the research design phase that youreached in Bnit ; to the practical issues o! choosing research methods- collectingdata- analysing data and #riting up the !inal evaluation report. %lthough #e #ill re!erto di!!erent kinds o! research methods in the context o! conducting an evaluation- #e#ill not cover these in detail here since you can read a out research methods inother hand ooks and modules in the PREST series and other Research2ethodology re!erence ooks.. +n addition- the Resources File includes moredetailed in!ormation.

    /e #ill also- very rie!ly- consider some important issues to keep in mind #hen

    udgeting !or an evaluation. %!ter completing this unit- you #ould have #orkedthrough all the steps required !or planning a good evaluation study. y the end o!this unit- i! you decided to do the pro4ect tasks- you should also have your o#nevaluation plan or proposal.

    Bnit outcomes/hen you have #orked through this unit- you should e a le to$

    8 list the planning questions you need to ans#er #hen designing an evaluation

    ; de!ine evaluation outputs and speci!y activities using the Logical ,rame#orkapproach

    = choose evaluation research methods

    < descri e the logistical issues that need to e taken into account #henconducting an evaluation

    @ outline the udgeting issues !or evaluations.

    &ommon#ealth o! Learning

    B6+T

    #

    43

  • 8/13/2019 PREST Module Program

    48/84

    1and ook < Programme evaluation and its role in quality assurance

    Evaluation planning- !orming a conceptual !rame#ork+n Bnit ; #e spent some time thinking through some o! the theoretical and researchdesign principles important !or conducting an evaluation.

    To summarise much o! #hat has already een covered and some o! #hat is yet tocome in this unit- let us have a look at a set o! key questions that Ru in '899@ p. =>(notes should in!orm the development o! an evaluation design.

    Overall evaluation planning questions to ans#er$

    /hat is the purpose o! the evaluation and !or #hom is it eing conductedC

    /hen in the pro4ect)programme li!ecycle #ill the evaluation e conductedC

    /hat are the scope- o 4ectives and key questions o! the evaluationC

    /ho holds responsi ility !or managing the evaluation- collecting data- #ritingreports etcC

    1o# #ill the evaluation e conducted- #hat procedures #ill e !ollo#edC

    /hat !inancial resources- human resources- time and other resources #ill eneededC

    1o# #ill the evaluation !indings e presented to di!!erent audiencesC

    1o# #ill !ollo#"up take placeC

    +n addition to these questions that help #ith planning evaluation research-esta lishing a conceptual !rame#ork !or an evaluation also requires that theevaluator understands the programme to e evaluated- ho# it is designed and #hatits aims are. This in!ormation might e o tained !rom documentation and)or talkingto those directly involved in the programme. +n many cases there #ill also e atheoretical !rame#ork underlying the design programme and ho# it is implemented.,or example- i! the aim o! the programme has een to introduce ne# teaching andlearning approaches #ithin an existing distance education programme- you #ill !inda range o! literature and other resources that address the issues o! teaching andlearning !or distance education. Perhaps the programme you are evaluating !ocuses

    on the use o! computers to enhance distance education. Hou may !ind a range o!literature on the use o! technology in distance education that could provide help!ul

    ackground !or your evaluation. These resources can e used to help you designyour study.

    +n some instances- esta lishing the conceptual asis #ould also include exploringthe context #ithin #hich the speci!ic programme takes place. ,or example- i! theprogramme #ere seeking to ring current teaching and learning methods in line #ithne# national distance education policy- then it #ould e important !or the evaluatorto understand the policy.

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning44

  • 8/13/2019 PREST Module Program

    49/84

    6ot only #ill articulating the conceptual !rame#ork o! the evaluation provide use!ulinput #hen developing your research questions- ut you may also e a le to raisesome generic issues out o! your !indings- instead o! programme speci!ic !indingsonly. ,or example- you might e a le to make some more general conclusions a outa ne# teaching and learning approach- or perhaps the use o! computers in distanceeducation.

    Evaluation 2 +see "our Resources File , contains a !rite u$ o the conce$tualrame!ork o the evaluation# You !ill ind this in cha$ters 2 +

  • 8/13/2019 PREST Module Program

    50/84

    1and ook < Programme evaluation and its role in quality assurance

    De!ining the evaluation delivera les and activities+n Bnit ; #e looked in some detail at research question !ormulation. +n order toans#er your evaluation questions you #ill need to de!ine the delivera les 'i.e. #hatyou need to produce to ans#er your questions during the evaluation(- that arerequired !rom the evaluation process. +n some cases- the delivera les o! theevaluation are de!ined y the organisation requesting the research and may also ede!ined in your research contract or rie!. ',or an example o! this- have a look at

    %ppendix 8 'p. 9A( o! Evaluation ) in your Resources File . 1ere you #ill see theterms o! re!erence !or this evaluation pro4ect. 6ote that these terms o! re!erencespeci!y exactly #hat the evaluation should cover- and so #hat the delivera les #ill

    e.(

    Once you have de!ined your delivera les- you then need to think #hat speci!icresearch activities are required in order to produce each o! these. There are many

    di!!erent #ays o! doing this- depending on the requirements o! the speci!icevaluation- your research ackground- and your approach to research. This processhas een explored in depth in the core modules. 1ere #e #ill look at t#o methods o!evaluation planning.

    !ctivity % $ hr #& m

    se Evaluation 2 rom "our Resources File or this activit"#

    1 *urn to $age 96 o Evaluation 2 #2 .ead the terms o re erence given to the evaluator#

    3 sing onl" these terms o re erence and& imagining that "ou are theevaluator o this $roAect& com$lete the ollo!ing table'

    /ist the delivera3les !or thisevaluation

    /ist some o! the researchactivities needed

    .he $eed*ac to this activity is at the end o$ the unit

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning

    1ctivity 2 8 hr => mins

    46

  • 8/13/2019 PREST Module Program

    51/84

    Log"!rame approach+n the last activity you sa# a detailed example o! one#ay o! planning your evaluation study. Since in thisseries you have een introduced to the logical

    !rame#ork 'o!ten simply called a log !rame( !orplanning research- #e #ill #ork through a log !rameexample here. % log !rame can e used to plan aresearch pro4ect or the implementation o! aprogramme. Since this !ormat outlines the outputsand activities o! the programme 'note$ not the outputsand activities o! the evaluation( it can e used as theconceptual !rame#ork against #hich to evaluate ho#success!ul the programme is.

    %lthough people o!ten !ind log !rames time consumingand sometimes even intimidating to complete- theyare a use!ul planning tool. +n general- log !ramessummarise in a standard !ormat$

    #hat a programme aims to achieve

    ho# the goals #ill e achieved

    #hat is needed to ensure success 'includingassumptions as #ell as inputs(

    proposed #ays o! measuring progress potential pro lems that might e !aced.

    The !ollo#ing activity is designed !or you to assess ho# #ell you understand the roleo! the di!!erent parts o! a log !rame.

    se Evaluation 2 rom "our Resources File or this activit"#

    *urn to $ages 11=14& !here "ou !ill ind an exam$le o a logicalrame!ork being used or $lanning an evaluation#

    % ter reading through the logical rame!ork& ocus on onl" the irstcolumn that describes the $rogramme being evaluated to ans!er the

    ollo!ing ;uestions#

    1 o! does the ormulation o the goal: and the $ur$ose: o this $rogrammedi er

    &ommon#ealth o! Learning

    Log !rames #hereto !ind out more

    htt$'EE!!!#bond#org#ukE$ubsEi$!#htm

    Do!nload the resource guide&ogical (rame!ork %nal"sis#

    mins

    47

  • 8/13/2019 PREST Module Program

    52/84

    1and ook < Programme evaluation and its role in quality assurance

    2 ?hat does the out$uts section tell us about the $rogramme

    3 -n terms o $rogramme im$lementation& ho! are the activities related to theout$uts

    .he $eed*ac to this activity is at the end o$ the unit

    The other columns o! the log !rame6o# look at the second and third columns o! this log !rame #here you #ill see ho#the evaluator has made use o! this tool !or the purposes o! her evaluation study.Read through the second column. Do you see that the evaluator has documentedthe extent o! achievement !or each aspect o! the programme as de!ined in theprogramme log !rameC This is #hat is documented in column ;. ecause the log!rame outlines exactly #hat the programme should achieve- it is a use!ul tool against

    #hich the success o! the programme can e assessed. Read through the thirdcolumn. /hat is the purpose o! this columnC /ell- this column provides additionaldetails !rom the evaluation research to ack up #hat has een noted in the secondcolumn- #here the extent o! achievement is assessed. &olumn = also providesin!ormation that is help!ul !or making recommendations a out the !uture o! theprogramme.

    To sum up- #e looked at t#o #ays o! planning an evaluation study. Remem er-there are also many other #ays o! presenting your evaluation plan. 7no#ing ho# touse a log !rame is a very use!ul skill since many !unding agencies #ill requestproposals in this !ormat.

    &hoosing evaluation research methodsProgramme evaluation usually requires that several aspects o! the programme itsel!are assessed- and that a range o! programme participants- their roles-responsi ilities and activities are also researched in order to ans#er the evaluationresearch question. 'Hou sa# the complexity o! this in ,igure 8.(

    ecause o! this complexity you #ill usually !ind that a range o! data gathering

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning48

  • 8/13/2019 PREST Module Program

    53/84

    methods- oth quantitative and qualitative are used in evaluation. The rationale !orusing a variety o! methods and sources o! in!ormation is to enrich the quality o! thedata and to ensure sensitivity to the !ull range o! issues and concerns that are likelyto emerge during an evaluation study.

    se Evaluation 1 and Evaluation 2 rom "our Resources File or this activit"#

    1 -n the tables belo!& list belo! the di erent kinds o research methodsused#

    2 @ote !hether these are ;ualitative or ;uantitative#

    3 ist one strength and !eakness o each#

    Evaluation 8 2ethods Jualitative)quantitative Strengths)#eaknesses

    Evaluation ; 2ethods Jualitative)quantitative Strengths)#eaknesses

    .he $eed*ac to this activity is at the end o$ the unit

    &ommon#ealth o! Learning

    ctivity % => mins

    49

  • 8/13/2019 PREST Module Program

    54/84

    1and ook < Programme evaluation and its role in quality assurance

    Bse o! existing in!ormationThe speci!ic research questions eing asked are onein!luence on the choice o! research methods. This #illalso e a!!ected y the kind o! in!ormation that

    already exists #ithin the programme- and that #hichneeds to e speci!ically collected !or the purposes o!the evaluation. oth o! the evaluation examples that#e have looked at have made use o! programmedocumentation in their analyses. This #ill ein!ormation that already exists in the programme andincludes- among others- proposals- programmereports- planning documents- memos- emails-

    udgets- etc. +! you think ack to Bnit 8 you #illremem er that #e noted there that programme

    monitoring systems are an important tool !orevaluation. % good programme monitoring system #illprovide the evaluator #ith much o! the in!ormationneeded- especially in!ormation related to theprocesses and systems #ithin the programme.

    ,or example- you #ill e a le to !ind in!ormation a out the num er o! learnersenrolled- num ers o! learners in each course- lists o! tutors- assignments sent out-assignments received- marks and assessments etc. +n the context o! distanceeducation programmes- a monitoring system should also provide you #ith

    in!ormation that you can use as indicators. ,or example- researchers sometimes usedropout rate as an indicator o! success o! a programme. Once you have esta lished-!or example- the dropout rate o! students #ithin a particular course- you might thenconduct individual intervie#s #ith students to determine their reasons !or#ithdra#ing !rom the course. This #ould provide you #ith in!ormation that themonitoring system could not provide.

    ,or example- a researcher- interested in studying the causes o! student dropout-intervie#ed y telephone all the students 'over 8>> practising teachers(- #ho haddropped out o! one course in a distance learning diploma programme that comprisedo! !our courses 'Ro inson 89:

  • 8/13/2019 PREST Module Program

    55/84

    2ultiple methods research/e noted a ove that the complex evaluations o!ten require the use o! a range o!research methods. The use o! a com ination o! research methods can e termedmulti5method or mi ed5method approaches. These approaches can e more

    rigorous than single method assessments- i! each o! the research methods is usedcorrectly 'Tshakkori and Teddlie 899:(. 1o#ever- one should not assume that usingmore than one method necessarily means a etter study.

    To e e!!ective- multiple methods research must$

    e competently designed and conducted !or all o! the methods used

    e appropriate !or the research question 'collecting Gmore data is not al#aysG etter ( and multiple methods tend to e more expensive to use than singlemethods

    !it together as a set o! methods in relation to the research question

    avoid spreading limited research resources too #idely and thinly in !indingans#ers to the research question

    e conducted y researchers #ho have a su!!icient level o! competency in thevarious research methods involved.

    +n addition- di!!erent research methods can e com ined in di!!erent #ays dependingon the reason !or using multiple methods. ,or example- 1ammersley '899A( notesthe !ollo#ing three #ays in #hich research methods can e com ined.

    6riangulation the use o! quantitative research to corro orate qualitativeresearch !indings or vice versa. ,or example- to determine the success!uloutcome o! a distance learning programme the researcher may use assessmentresults and attendance patterns- together #ith intervie#s #ith learners toexplore reasons !or success and)or !ailure.

    acilitation #hen one research strategy is used to aid research that is usinganother research strategy. e!ore conducting a survey- the researcher mightconduct intervie#s to determine the est #ay o! asking the questions.

    7omplementarity #hen t#o research strategies are used to dovetail di!!erentaspects o! an investigationN !or instance- our example o! the evaluation o! theonline course !or distance education policy makers. This included oth theevaluation o! the online component and the !ace"to"!ace #orkshop thatsupported it. Slightly di!!erent research approaches #ere used !or each- utthese complemented each other and oth #ere used to produce the !inalevaluation report o! the programme.

    &ommon#ealth o! Learning 51

  • 8/13/2019 PREST Module Program

    56/84

    1and ook < Programme evaluation and its role in quality assurance

    Triangulation %re you !amiliar #ith the term triangulationC This is aterm that has ecome increasingly commonplace ineducation and social science research methods so it

    is important !or us to explore it in a little more detailhere. One o! the reasons !or using multiple methods-i! used appropriately as outlined a ove- is that thiscan help to achieve triangulation and so potentiallyenhance the rigour o! the study.

    Triangulation is a term adopted !rom surveying- #here surveyors take readings o! ageographical point !rom several angles to ensure and check the accuracy o! theirconclusions. +t has een used in social science and educational research as a #ayo! increasing con!idence in the researcher s !indings.

    2ethodological triangulation re!ers to the use o! a com ination o! di!!erent researchmethods to study the same thing in order to veri!y the !indings o! each method. +nthis #ay- the same programme- process- person etc is studied !rom more than oneperspective as is highlighted in the !ollo#ing quotation$

    8Firstly the researcher needs to *e con$ident that the data generated are not sim&lyarte$acts o$ one s&eci$ic method o$ collection2 .his con$idence can only *eachieved 1hen di$$erent methods o$ data collection yield su*stantially the sameresults2 Furthermore; the more the methods contrast 1ith each other; the greaterthe researcher:s con$idence2 I$; $or e7am the outcomes o$ a uestionnaire

    survey corres&ond to those o$ an o*servational study o$ the same &henomena; themore the researcher 1ill *e con$ident a*out the $indings: ('ohen ? Manion ">>( notes that-

    8>ecause &rogramme evaluation is ine7trica*ly intert1ined 1ith &olitics and values;and *ecause evaluators must navigate care$ully amid com&eting &olitical and valueagendas; it is essential that evaluators have a diverse set o$ a&&roaches to hel&guide &ractice2:

    Practitioner Research and Evaluation Skills Training in Open and Distance Learning

    Triangulation*riangulation is the use o a

    combination o di erent researchmethods to stud" the same thing inorder to veri " the indings o eachmethod#

    52

  • 8/13/2019 PREST Module Program

    57/84

    udgeting !or evaluations$ keeping expectationsrealisticResearch takes time and is costly. +t is essential that #e remain realistic a out #hat#e can and cannot do #hen planning an evaluation. ,or this reason- care!ul

    udgeting is an important part o! planning !or an evaluation study. The evaluationplan- o 4ectives- delivera les- and methods chosen must e realistically achieva le#ithin oth the time!rame o! the study and the availa le udget. +t is important at theoutset to try to note any constraints or research limitations that might arise due toeither time or !inancial limits.

    There are di!!erent #ays in #hich an evaluation might e !unded and this #ill a!!ectthe udgeting required and the kind o! evaluation undertaken. ,or example- an in"house evaluation o! an ODL programme might e conducted y someone #orking#ithin the programme as an aspect o! his or her #ork. The udgeting !or such anevaluation #ould e quite di!!erent !rom one in #hich an external evaluator iscontracted to do the evaluation and has to travel to another country !or this- as in theSO2DEL example. The kinds o! research methods to e used also a!!ect the

    udgeting. ,or example- conducting a !ocus group rather than many individualintervie#s can reduce costs ' ut #ill provide a di!!erent kind o! data(. Large"scalesurveys- #hilst o!ten use!ul !or evaluations- are costly- unless linked to otherdistri ution channels o! an ODL programme- !or example including a survey #henregistration materials are set out.

    /hen udgeting !or an evaluation 'or any other research pro4ect( you need to think

    care!ully a out all the expenses that are likely to e incurred. This includes !inancialcosts such as salaries- daily expenses- travel costs 'vehicle hire- petrol etc(- costs o!meetings and venues- costs o! materials- production o! research instruments-reports- the costs o! gathering !eed ack !rom participants on the !inal results o! theevaluation- and the time allocation o! sta!! usually employed !or other tasks. Ieryo!ten research udgets are ased on the num er o! days it is estimated that theresearch #ill take.

    There are many di!!erent #ays o! #orking out a udget. Hou may !ind that yourinstitution has a standard #ay in #hich all udgets are calculated. % help!ul tip !or

    creating a udget is to map out your delivera les and activities in ta les as #e dida ove 'see the !eed ack to %ctivity ;(. ,or each o! the activities you can then #orkout ho# much time and other resources #ill e needed- to #hich a monetary valuecan then e attached.

    /hatever method you use- #hen putting together a udget you should al#ays ea#are that costs add up quickly. %lso- unanticipated costs 'such as di!!iculties #ithgaining access to the people #ho need to e intervie#ed- #hich means thatresearch time ecomes longer( o!ten arise. This is particularly relevant to evaluationresearch since you #ill al#ays e #orking closely #ith an organisation or

    &ommon#ealth o! Learning 53

  • 8/13/2019 PREST Module Program

    58/84

    1and ook < Programme evaluation and its role in quality assurance

    organisation's(- programme's(- etc that have their o#n #ay o! !unctioning andprocesses o! doing things #hich might cause delays in your research process. Hour

    udget needs to make provision !or this.

    Evaluation logisticsSo !ar #e have looked at the theoretical- design and methodological issues that areimportant #hen planning an evaluation study. The previous section presented somepractical in!ormation a out udgeting. +n this section #e very rie!ly note some o! thelogistical issues and decisions that you might e required to make #hen planningand conducting an evaluation.

    O viously- each evaluation study #ill have speci!ic logistical requirements. The listelo# outlines some o! these.

    contact stakeholders make contact #ith- and possi ly negotiate terms o!access to- the stakeholders in the evaluation

    pro4ect documentation gain access to all the relevant documentation

    plan data collection plan the most appropriate #ay o! collecting data !romparticipants #ho- in a distance education setting- are dispersed. ,or example-#ould it e etter to use a postal survey- send emails- or do telephoneintervie#sC

    plan timing