KBA Nijmegen - Interventions in education to prevent STEM ... 1.pdf2019 KBA Nijmegen Behoudens de in...

Interventions in education to prevent STEM pipeline leakage

Anniek van den HurkMartina MeelissenAnnemarie van Langen

DEE

LON

DER

ZOEK

1

.

Interventions in education to prevent STEM pipeline leakage

Deelonderzoek 1

Anniek van den Hurk, Martina Meelissen & Annemarie van Langen

Enschede/Nijmegen, 2019

ii

Projectnummer: 2016.555 Illustratie omslag: @ARudolf - Can Stock Photo Inc. 2019 KBA Nijmegen Behoudens de in of krachtens de Auteurswet van 1912 gestelde uitzonderingen mag niets uit deze uitgave worden ver-menigvuldigd en/of openbaar gemaakt door middel van druk, fotokopie, microfilm of op welke andere wijze dan ook, en evenmin in een retrieval systeem worden opgeslagen, zonder de voorafgaande schriftelijke toestemming van het Kennis-centrum Beroepsonderwijs Arbeidsmarkt te Nijmegen. No part of this book/publication may be reproduced in any form, by print, photo print, microfilm or any other means without written permission from the publisher.

Deze uitgave is mede mogelijk gemaakt met financiering van het Nationaal Regieorgaan Onderwijsonderzoek. NRO-projectnummer: 405-16-404

Full Terms & Conditions of access and use can be found athttps://www.tandfonline.com/action/journalInformation?journalCode=tsed20

International Journal of Science Education

ISSN: 0950-0693 (Print) 1464-5289 (Online) Journal homepage: https://www.tandfonline.com/loi/tsed20

Interventions in education to prevent STEMpipeline leakage

Anniek van den Hurk, Martina Meelissen & Annemarie van Langen

To cite this article: Anniek van den Hurk, Martina Meelissen & Annemarie van Langen (2019)Interventions in education to prevent STEM pipeline leakage, International Journal of ScienceEducation, 41:2, 150-164, DOI: 10.1080/09500693.2018.1540897

To link to this article: https://doi.org/10.1080/09500693.2018.1540897

© 2018 The Author(s). Published by InformaUK Limited, trading as Taylor & FrancisGroup

View supplementary material

Published online: 31 Oct 2018.

Submit your article to this journal

Article views: 1560

View Crossmark data

https://www.tandfonline.com/action/journalInformation?journalCode=tsed20https://www.tandfonline.com/loi/tsed20https://www.tandfonline.com/action/showCitFormats?doi=10.1080/09500693.2018.1540897https://doi.org/10.1080/09500693.2018.1540897https://www.tandfonline.com/doi/suppl/10.1080/09500693.2018.1540897https://www.tandfonline.com/doi/suppl/10.1080/09500693.2018.1540897https://www.tandfonline.com/action/authorSubmission?journalCode=tsed20&show=instructionshttps://www.tandfonline.com/action/authorSubmission?journalCode=tsed20&show=instructionshttp://crossmark.crossref.org/dialog/?doi=10.1080/09500693.2018.1540897&domain=pdf&date_stamp=2018-10-31http://crossmark.crossref.org/dialog/?doi=10.1080/09500693.2018.1540897&domain=pdf&date_stamp=2018-10-31

Interventions in education to prevent STEM pipeline leakageAnniek van den Hurk a, Martina Meelissena and Annemarie van Langen b

aDepartment OMD, University of Twente, Enschede, Netherlands; bKBA Nijmegen, Nijmegen, Netherlands

ABSTRACTThe so-called leaking STEM pipeline (dropout in STEM education)has been the subject of many studies. The large interest ofscholars in plausible causes of this leakage has resulted in anumber of meta-reviews describing factors at system, school andstudent level related to interest and persistence in STEMeducation. The STEM pipeline discussion has also resulted in alarge number of programmes aimed at enhancing STEM interestand persistence in STEM education. Although these programmeshave been widely evaluated, there seems to be no consensusabout which interventions are successful in raising interest inSTEM or persistence in STEM education. This study reports theresults of a systematic review of empirical studies in which theeffectiveness of STEM-related interventions are assessed. Initially,538 studies were found. The quality analyses showed that only afew of these evaluation studies are designed in such a way that itis likely that the found effects are caused by the intervention.Although some potentially effective interventions were found, thisreview shows that there is still a need for research into theeffectiveness of those programmes, especially with regard toprogrammes preventing talented and initially motivated STEMstudents to drop out of STEM education.

ARTICLE HISTORYReceived 4 December 2017Accepted 22 October 2018

KEYWORDSSTEM education; STEMinterest; STEM persistence;dropout; interventions

Introduction

Despite several initiatives to increase the interest of Dutch students for STEM (Science,Technology, Engineering, and Mathematics), enrolment of students in STEM educationin the Netherlands is still low compared to that of most other western countries(OECD, 2016, 2017). Recent data have shown that a substantial part of Dutch studentswho initially show talent and interest in STEM, eventually leave education with a non-STEM-related diploma or choose a non-STEM-related programme in tertiary education(TechniekPact & Platform Bèta Techniek, 2016).

Furthermore, in both the Netherlands and other western countries, there are still largegender differences in the participation in STEMeducation and STEM-related careers. Enrol-ment data regarding students at different levels of education show that in most western

© 2018 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis GroupThis is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License(http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in anymedium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way.

CONTACT Anniek van den Hurk [email protected] Department OMD, University of Twente, Postbus 217,7500 AE, Enschede, Netherlands

Supplemental data for this article can be accessed at https://doi.org/10.1080/09500693.2018.1540897.

INTERNATIONAL JOURNAL OF SCIENCE EDUCATION2019, VOL. 41, NO. 2, 150–164https://doi.org/10.1080/09500693.2018.1540897

http://crossmark.crossref.org/dialog/?doi=10.1080/09500693.2018.1540897&domain=pdfhttp://orcid.org/0000-0002-0673-6073http://orcid.org/0000-0001-7576-9816http://creativecommons.org/licenses/by-nc-nd/4.0/mailto:[email protected]://doi.org/10.1080/09500693.2018.1540897http://www.tandfonline.com

countries girls are no longer lagging behind boys in the attained educational level (OECD,2015). In the Netherlands, on average girls attain a higher educational level at the end oftheir school career and show lower retention and dropout rates than boys (Van Langen,Driessen, & Dekkers, 2008). However, figures about the so-called horizontal participationin education demonstrate a very different picture. In all levels of education, female studentsare still underrepresented in Science, Technology, Engineering, and Mathematics (STEM)programmes (Microsoft, 2017; OECD, 2016; TechniekPact & Platform Bèta Techniek,2016). Furthermore, more girls than boys drop out of STEM education during their edu-cational careers (Miller & Wai, 2015; Van Langen, Elfering, & Hilkens, 2018).

This phenomenon is often referred to as ‘the leaking STEM pipeline’ (Alper, 1993; Ber-ryman, 1983). It is a metaphor for the trajectory from early interest in STEM and enrol-ment in STEM programmes to the first steps into the labour market. During theeducational career, there are several formal and informal choices to be made by students,offering the opportunity to switch educational programmes. These choices are often,especially among female students, to the detriment of STEM courses, hence the metaphor‘the leaking STEM pipeline’. Some researchers recommend using different metaphors forexample STEM trajectories or pathways (Cannady, Greenwald, & Harris, 2014). Theyargue that the pipeline metaphor suggests a linear and traditional process and does notjustice to the multidimensionality of the process.

Another point of discussion is how to define STEM. Although the abbreviation clearlydistinguishes what STEM is: Science, Technology, Engineering, and Mathematics, thereseems to be no consensus about which more specific courses and professions are partof STEM. Most researches also place ICT-education and ICT-related jobs in the STEMcategory (Stoeger et al., 2016). Others suggest to include health sciences (e.g. Wallace,Perry, Ferguson, & Jackson, 2015) and even social sciences(e.g. Schultz et al., 2011). Inline with the majority of studies, this study focuses on natural sciences – without healthscience – technological science, engineering, mathematics and ICT or combinations.

Recent data show that in the Netherlands the STEM pipeline is leaking even faster thanin other western countries. The percentages of graduating girls in upper secondary voca-tional STEM-related programmes are below the OECD average (OECD, 2016). This is,even more, the case for the enrolment of girls in tertiary STEM education and the percen-tages of Dutch females leaving tertiary education with a STEM-related diploma (OECD,2016). In today’s society where knowledge and skills in STEM could provide a securefuture on individual, societal, and economical level, and where there is a shortage ofSTEM-trained professionals, it seems a waste that a part of the STEM potential, femaleand male, is ‘lost’ during the school career. Although this is specifically a problem inthe Dutch context, other (western) countries also struggle to motivate young people forSTEM education and careers. This emphasises the need for a broad search into existingresearch to determine what interventions can help to repair this persistent problem.

In the past few decades, there has been an extensive amount of research focusing on theenrolment, attainment and persistence in STEM education (Kelly, 2016; Wang & Degol,2013). The majority of these studies is aimed at finding possible explanations for thelow entry rates and dropout rates, specifically among girls, for STEM education. Thesestudies have resulted in a number of meta-reviews and theoretical models describingfactors at environmental, school and student level that are related with interest and per-sistence in STEM education (e.g. Wang & Degol, 2013). Insight in these factors is an

INTERNATIONAL JOURNAL OF SCIENCE EDUCATION 151

important prerequisite in repairing the leakage in the STEM pipeline, as it can be used forthe development of interventions to raise interest and to prevent dropout of STEM coursesand programmes. However, most studies and meta-reviews in this field offer suggestionsfor such interventions, but do not assess the effects of these interventions (e.g. Eccles &Wang, 2016). By conducting a systematic review of the literature, the aim of this studyis to find interventions that have proved their effectiveness in raising interest and prevent-ing students dropping out of STEM education. The research question of this study is:What kind of interventions are successful in raising interest and persistence in STEM edu-cation among girls and boys, according to the research literature?

This systematic review is based on the method described by Petticrew and Roberts(2006). For the description of the results of this review, a theoretical model was used,based on the outcomes of recent meta-reviews of studies focusing on factors influencingthe interest in STEM and persistence of students to stay in the STEM field.

Theoretical framework

The low participation of especially females in STEM education has been subject of manystudies in the past thirty years. This has resulted in a number of meta-reviews and theor-etical models trying to explain the low interest and persistence in STEM education. Basedon the outcomes of four recent and relevant meta-reviews, this section presents a theor-etical framework to guide the systematic review. The meta-studies are published in thelast twelve years and cover a wide range of research focusing on factors related to(gender differences in) the interests and persistence in STEM education (Blickenstaff,2005; Ceci & Williams, 2010; Eddy & Brownell, 2016; Wang, Eccles, & Kenny, 2013).The factors mentioned in these reviews can be categorised into three levels: environmentalfactors, factors at the school level and factors at the student level. Most scholars assumethat these factors are not only correlating with the interest or persistence in STEM edu-cation but that they are also interrelated (Blickenstaff, 2005).

Environmental factors

Environmental factors steering students in or out of STEM education have been thesubject of several studies. Stereotypes about gender and STEM not only influence the stu-dents themselves as well as their teachers and study advisors (see below), but also peers,family and future employers (Kelly, 2016). As a consequence, girls might experienceless encouragement and support to excel or to choose STEM-related educational pro-grammes and careers. Parental beliefs and behaviour can both promote and discouragestudents into STEM education (Kelly, 2016; Wang & Degol, 2013). The lack of femalerole models can also decrease the sense of belonging in STEM for STEM students and stu-dents considering STEM education (Blickenstaff, 2005). Cultural and societal beliefs,policy and economical and work-related developments should also be considered as aninfluence on students’ behaviour. As several large-scale studies state, the differencesbetween countries in amount of students intending to pursue an education or career inSTEM are visible (e.g. Microsoft, 2017). These cultural differences should also be takeninto account in reviewing the interventions, while some interventions can be effective inone context and have no effect in others.

152 A. VAN DEN HURK ET AL.

Factors at the school level

In the meta-reviews, empirical support was found for the assumption that the curriculumin STEM courses is particularly focused on ‘masculine’ subjects and is therefore lessappealing to female students (Blickenstaff, 2005; Wang & Degol, 2013). Some studies con-clude that the instructional approach of STEM teachers tends to be more in favour of boys.For example, girls are more likely to use cooperative techniques to learn and this is notoften provided in STEM classrooms (Kelly, 2016; Wang & Degol, 2013). Last, STEM-gender stereotypes can influence teachers and study advisors to be (subconsciously oropenly) prejudiced that women do not belong in the STEM world. This biased view canlead to less encouragement for girls to pursue a school career in STEM and a negativesocial atmosphere towards girls following STEM programmes. This so-called chillyclimate has a negative influence on their sense of belonging and of competence (Eddy& Brownell, 2016; Kelly, 2016; Wang & Degol, 2013).

Factors at the student level

The student-related factors are covered in all meta-studies under review. These factorsconsist of cognitive characteristics, such as (academic) ability and achievement, back-ground characteristics (e.g. gender, socioeconomic status), and affective characteristics,such as self-efficacy, motivation, belonging, and engagement (Eddy & Brownell, 2016).

The STEM ability or aptitude of students is probably the most discussed explanation ofthe gender gap in STEM education. The review studies by Blickenstaff (2005), Ceci andWilliams (2010), Eddy and Brownell (2016), and M. T. Wang et al. (2013) hypothesisethat the gender gap in participation in STEM education is caused by the lower performingrates of female students in STEM assessments compared to male students. However, in allfour meta-reviews it is concluded this is not always the case, and that even if a gender gapin STEM ability is found, this difference is too small to have such an impact that it couldexplain the gender differences in STEM participation (Blickenstaff, 2005; Ceci &Williams,2010; Eddy & Brownell, 2016; Wang et al., 2013). Eddy and Brownell (2016) also concludethat the validity of the results in research on STEM ability is questionable. Differentmethods and measurements used in these studies make the data poorly comparable.For example, some studies base their statement about STEM ability on mathematicsresults and others on physics or biology results or a combination. In spite of thesedoubts, international large-scale studies show that the STEM ability of boys and girls inprimary and secondary education is on average equal or slightly in favour of boys(Mullis, Martin, Foy, & Hooper, 2016; OECD, 2015). This observed difference in abilityis too small to account for the large difference in STEM participation. However, studieshave also shown that amongst the top performers in STEM subjects, boys are in themajority (Ceci & Williams, 2010; OECD, 2015), possibly explaining (part of) the gendergap in STEM participation.

Wang et al. (2013) have explored a totally different perspective on the relationshipbetween gender differences in STEM ability and gender participation. They suggestedthat the underrepresentation of females in STEM education can be (partly) explainedby the higher level of verbal ability of girls compared with that of boys. More girls thanboys seem to show a combination of both a high STEM ability and a high verbal


ability. Large-scale international studies such as PIRLS and PISA show that on averagegirls outperform boys in reading literacy in most countries (Mullis, Martin, Foy, &Drucker, 2012; OECD, 2014). This means that talented girls in STEM have moreoptions; they can choose from a wider range of educational programmes and professions(STEM- and non-STEM-related) than talented boys in STEM (Wang et al., 2013).Other factors explained below can make that these girls tend to choose non-STEMpathways.

Next to gender, other background characteristics are also related to the interest and per-sistence in STEM education. It is found that students from minority backgrounds and/orwith low socioeconomic status (SES) are less likely to pursue a career in STEM education.Students with highly-educated parents and/or higher SES more often choose STEMcourses and persist longer in STEM education (Eddy & Brownell, 2016; Wang & Degol,2013; National Science Foundation, 2017). These factors underline the intertwiningwith the environmental factors, as explained by Blickenstaff (2005).

Eddy and Brownell (2016) explored in their meta-review the role of affective factors andengagement. Affective measures include self-efficacy, sense of belonging, and science iden-tity. Wang and Degol (2013) added motivational beliefs, such as occupational and lifevalues as an important part of the decision-making process to pursue STEM educationor STEM-related careers. It is found that the self-efficacy, belonging, and science identityof girls tend to be lower than those of boys, influencing their choice whether or not topursue STEM education (Eddy & Brownell, 2016; Kelly, 2016; Wang & Degol, 2013;OECD, 2015). Female students consistently perceive their STEM competences lowerthan male students (Wang & Degol, 2013). Furthermore, Blickenstaff (2005) and (Micro-soft, 2017)found that these gender differences increase with age. Among primary schoolstudents, the difference in self-concept and science identity between boys and girls doesnot yet exist or is negligible.

Theoretical models for participation or persistence in STEM

Based on the outcomes of their review, both Wang and Degol (2013) and Eddy and Brow-nell (2016) developed a theoretical framework that shows potentially influencing factorson the participation in STEM education.

The meta-reviews (Blickenstaff, 2005; Ceci & Williams, 2010; Eddy & Brownell, 2016;Wang et al., 2013) and the theoretical models offer an overview of factors that are relatedto or may explain (gender) differences in participation in STEM education. A distinctioncan be made between factors that are (to a certain extent) malleable such as school andenvironmental factors, and factors that are not malleable, such as SES or student aptitude.Insight in factors that could be influenced to increase the participation and persistence inSTEM education and in factors that are not malleable but should be taken into account, isan important prerequisite in the development of interventions aiming at increasing theinterest and persistence in STEM.

The focus of this study is to find successful (based on empirical evidence) interventionsat the student, school or environmental level to raise interest in STEM or prevent dropoutin the STEM pipeline. The outcomes of the meta-reviews (Blickenstaff, 2005; Ceci & Wil-liams, 2010; Eddy & Brownell, 2016; Wang et al., 2013) and the existing models (Dekkers,2007; Eddy & Brownell, 2016; Wang & Degol, 2013) were used to develop a framework to


guide the description of the results of this systematic review. The framework used in thisstudy is presented below (Figure 1).

The framework is divided into several sets of factors mediating and moderating theprocess of making educational choices and persisting in STEM education. The arrowsin the framework represent the assumed direction of influence of the sets of factors.The social context domain encompasses national, regional and cultural factors, such asthe national educational policy, cultural environment (Blickenstaff, 2005), andeconomy. It is assumed that the social context influences the academic choices and per-sistence in STEM education by influencing the school context, social environment and/or student characteristics (Dekkers, 2007). Next, the school context refers to the edu-cational policy on the school level, school climate and teaching pedagogy (Dekkers,2007; Wang & Degol, 2013). The school context is assumed to influence the studentcharacteristics. The social environment includes the parents (family) and peers. Boththe support and view of parents and peers, and the background and socioeconomicstatus are taken into account (Wang & Degol, 2013). Then, in the student characteristicsdomain, the motivation, attitude, preferences, performances, and behaviour of the stu-dents are combined. Both the student characteristics and the academic ability and aptitudedirectly influence the educational outcomes (Eccles, 2009). These are the persistence inSTEM education and educational choices. The latter describes the level of educationattained, for example, vocational secondary education or university.

Method

Where many previous studies on the participation, dropout and gender gap in STEM edu-cation can be characterised as correlational or explorative research with suggestions forpossible interventions, this review focused on the effectiveness of programmes aimed atincreasing the interest and persistence in STEM. The research question that guided thissystematic review is: What kind of interventions are successful in raising interest and per-sistence in STEM education among girls and boys, according to the research literature? A

Figure 1. Model for academic choices and persistence in STEM education.Note: This model is constructed based on several (review) studies and models used by other researchers. The arrowsbetween the levels do not imply a causal effect but rather suggest a relationship worth considering in evaluating andresearching this topic.


systematic review is selected as a method to give a comprehensive view of the research oninterventions in STEM education.

The systematic review is based on the method described by Petticrew and Roberts(2006). The review included an extensive search of different research databases, a selectionof publications based on inclusion and exclusion criteria; a quality assessment; and acontent analysis of the studies finally selected. First, the search process, inclusion criteriaand review of the scientific quality will be explained.

Search process

The search for this systematic review was carried out in August 2017. First, an electronicsearch of four databases: ERIC, PsycINFO, Web of Science and Scopus, was conducted.Several combinations of search items were tested preparatory to the final search. Itshowed that these studies use a rich vocabulary. The search keys that seemed to yieldthe most relevant results were Science, Technology, Engineering, Mathematics (STEM),dropout, retention, persistence, motivation, attitude, intervention, programme, evaluationand combinations of these. These search keys were used for each database. Any possibledifferences in spelling (i.e. correct spelling) were also included in the search keys. The elec-tronic search showed that a relatively large amount of intervention studies was reported inthe Journal of Science Education and Technology and in the Journal of STEM Education.Therefore, also a manual search of these two journals was conducted. Finally, the referencesections of the studies that met the criteria for inclusion (see below) were explored. Despitethis, it is still possible that relevant publications are missing in this review. However, thenumber of publications that met the inclusion criteria seems to be sufficient to reportgeneral trends and to be representative of this research field between 2005 and 2017.

Based on the electronic search, a pilot of the review was conducted to evaluate the use-fulness of the systematic review, to assess the number of relevant studies and to evaluatethe usefulness of the selection criteria (Petticrew & Roberts, 2006). The pilot also deter-mined the final order of the criteria and was used to sharpen or broaden these criteria.

Inclusion and exclusion criteria

The possibly relevant publications were then analysed by reading the titles and abstracts.The following inclusion criteria were used to select or exclude the studies.

(1) The study was published after 2004. This criterion was used to diminish the chancethat interventions have become less applicable due to social and educationalchanges over time. For example, before 2005, the Dutch educational system in second-ary education was quite different, resulting in different educational (STEM) pathwaysfor students.

(2) The study focused on primary, secondary and/or tertiary education. The interest ofmost studies in this research field lies with secondary and tertiary education. Blicken-staff (2005) found that the interest in STEM is often equal for boys and girls inprimary education. However, this does not mean that interventions at an early agecould not have an effect on the interest and persistence in STEM later on (Wang,2013). For this reason, primary education was included in the review as well.


(3) The study evaluated the effects of the intervention on the interest in STEM as a schoolsubject or as a future career or evaluated the effects of the intervention on preventingdropout from STEM education. Studies evaluating the intervention by only assessinggrowth in STEM content knowledge or by evaluating experiences of students with theprogramme itself were excluded. Often, these studies argue that improving knowledgeor positive experiences with the programme will increase interest or persistence inSTEM. However, if these effects are not measured (for example by attitude question-naires or retention data) it remains only an assumption. Studies which focused specifi-cally on certain groups such as females, minorities or other underrepresented groupsin STEM were included.

(4) The data of at least 25 respondents (students) were analysed. The pilot showed thatthe number of respondents in some studies was very low. To increase the chancethat the study was able to draw valid conclusions about the intervention effects, itwas decided to exclude the smallest studies beforehand.

(5) The study was published in a peer-reviewed academic journal. Finally, the study hadto be published in a peer-reviewed academic journal, based on information providedby Ulrichs (see http://ulrichsweb.serialssolutions.com) to ensure a certain level ofquality provided in the peer review publication process.

The selected studies that met all five criteria were categorised in type of intervention,research design and used instruments.

Quality assessment

The selected studies were assessed on quality with the use of a data extraction form (Pet-ticrew & Roberts, 2006). The data extraction form that was used for this study consists ofseveral categories. The goal of this review was to find evidenced-based successful interven-tions. With this specific goal in mind, the quality of the study is determined to a largeextent by its research design. In general, only ‘experimental studies’ or ‘quasi-experimentalstudies’ were considered appropriate to show that the effects were caused by the interven-tion. However, the pilot showed that most intervention studies used a ‘pre–post onesample design’, often for practical reasons. The lack of a control group prevents causalinference regarding the effects of the intervention. Pre- and post-scores could differ forreasons unrelated to the intervention such as regression to the mean, instrumentation,history, and maturation (Marsden & Torgerson, 2012; Shadish, Cook, & Campbell,2002). If a pre–post study used a comparison group that is not comparable to the inter-vention group, the effect can also be caused by selection bias (often the interventiongroup is self-selected or selected by teachers), and it remains unclear whether these out-comes are a programme or a selection effect. This means that ‘post-test only’, ‘post-testonly with comparison group’, ‘pre/post-test design without comparison group’ or a‘pre/post-test design with a comparison group not comparable with the interventiongroup’ were classified as ‘weak causal evidence’.

The pilot also showed that some studies combine a ‘pre/post-test design’ with a ‘casestudy design’ (mixed method). Case studies (observations, interviews) can provide veryuseful in-depth information about how the programme is experienced by students, theirbehaviour during the programme or their perception of the influence of the programme


http://ulrichsweb.serialssolutions.com

on their future educational (career). However, because these outcomes are perceptions ofchanges, they cannot provide evidence for actual changes in interest or persistence inSTEM due to the programme. Therefore, such case studies or mixed method studieswith a pre/post-test design were classified as ‘weak causal evidence’ as well.

Two reviewers assessed the quality of the studies independently. The research designwas determined by reading the whole article. Publications, in which the design was notclear or the suitability of the comparison group was questionable, were discussed by thetwo reviewers. If the study design was categorised as ‘weak causal evidence’, the publi-cation was excluded from further analysis.

Results

Search results

The electronic search yielded 918 studies in total. After removal of the duplicates, books,and reports, 538 possible relevant publications were selected. After application of the fiveinclusion criteria, 71 studies remained. The five inclusion criteria were also applied in themanual search of the two journals and the search in the reference sections of the selectedpublications. The manual search added 48 publications. This means that for quality analy-sis a total of 119 publications were reviewed (this list is presented in the supplementarymaterials). Although the systematic search followed the steps of Petticrew and Roberts(2006) carefully, there is always a risk of overlooking relevant studies. However, due tothe large amount of studies that were initially found, it is expected that the studies are asufficient representation.

Most (67%) of the publications found were aimed at evaluating the effects for students’interest in STEM education and/or a STEM career. Only 26% of the interventions weretargeted at preventing dropout in STEM and 7% focused on both interest and persistencein STEM. A small part of the evaluated interventions was directed to STEM in general,however, most interventions were aimed at one or two STEM fields. Engineering- andScience-related programmes seemed to be the most popular. Circa two-thirds of thestudies reported the effects of Summer Camps, covering the largest group of interventions.Other interventions are for example changes in STEM pedagogy and mentoring pro-grammes. Many publications focused specifically on underrepresented groups in STEM:female students, minority students and/or students in rural areas. The outcomes of theinterventions were often measured by (a large diversity of) attitude questionnairestowards the STEM fields in general, towards STEM as school subjects or towards STEMcareers. Some of the studies measured the effects of the interventions (also) by reportingretention or graduation rates.

Quality assessment

As described in the method section, the quality of the publications was reviewed with theuse of data extraction forms. However, the first step in the quality assessment was todescribe the research design of each study.

The quality assessment showed that the majority of the studies used a pre/post-testdesign without control or comparison group. Many of those studies describe the


evaluation of a summer science camp. Participants were asked to fill in an attitude surveybefore or at the start of the summer camp and after or at the end of the summer camp.Based on the growth in the attitude of the participants, the researchers concluded thesummer camp to be effective. Because of the absence of a control group or comparisongroup, there is a risk of several biases; for example, maturation and history. Therefore,those designs were not considered to be able to draw reliable conclusions.

In total, 25% of the studies, were considered evaluation studies using a quasi-experimentalor experimental design, which makes them able to draw those conclusions. These thirtystudies were assessed on the quality of the design based on the following criteria: trustworthi-ness of the executed design, sampling and assignment of the intervention group and controlgroup, and instrumentation used to measure the dependent variable. Only nine studies metthe standards such that solid conclusions could be drawn about the effectiveness of the inter-vention. Most of the excluded studies lacked information about the sampling and/or theassignment of the control group and intervention group. Or the sampling design was con-sidered as weak. For example, by purposeful choice of participants by teachers, or thechoice of an existing group as a control group without controlling for group characteristicssuch as gender, ability and/or motivation, causing possibly unequal groups (internal threat ofselection). Unequal groups make it impossible to compare results and to draw reliable con-clusions about causality. The results from studies using such designs were therefore con-sidered not convincing and were excluded from the final group.

Content analysis

The nine selected studies are summarised and sorted into the categories used in the theor-etical framework (Figure 1). The results are presented in this paragraph.

Social contextStudies were categorised in the social context when the interventions were aimed at alter-ing educational policy and/or cultural and social views. Two of the selected studies in thisreview were aimed at altering the social context, both attempting to influence the studentcharacteristics (Stoeger et al., 2016; Stout, Dasgupta, Hunsinger, & McManus, 2011). Bothstudies were aimed at enhancing women’s motivation and attitude by providing them withfemale role models (as mentors). Both interventions had a positive effect on the dependentvariables.

School contextStudies in the school context encompass interventions in pedagogy, school climate, andschool-level educational policies aiming to increase retention of students in STEM edu-cation and/or increase the STEM attitude. In total, six studies were considered as evalu-ation studies of interventions in the school context domain. In this domain, fourinterventions were found to focus on improving or renewing the STEM pedagogy:Gaspard et al. (2015), Kara and Yesilyurt (2008), Lee and Erdogan (2007), and Prokop,Tuncer, and Kvasnicak (2007). For example, by using (more) ICT (Kara & Yesilyurt,2008) or more hands-on experiences inside and outside the classroom (Lee & Erdogan,2007; Prokop et al., 2007). Two other studies provided an (online) training programmefor women and minorities aimed at retaining the participants in STEM (Bekki, Smith,


Bernstein, & Harrison, 2013; Schultz et al., 2011). Gaspard et al. (2015) focussed in their90-minute lessons on the relevance of STEM to raise awareness amongst students. Allstudies reported a positive effect of the intervention on students’ attitudes or intentionsto pursue a career in STEM. However, one study reported an overall decline in studentinterest over the years albeit less for the intervention group (Schultz et al., 2011).

Social environmentThe social environment of students includes both the familial environment, such asparents and siblings, as well as the students’ peers. It is expected that this environmentinfluences students’ motivation and behaviour to pursue a (educational) career inSTEM (Wang &Degol, 2013). One of the selected studies in this review was aimed at inter-ventions in the social environment.

Harackiewicz, Rozek, Hulleman, and Hyde (2012) focused on the expected influence ofparents in their randomised experiment. They informed parents about the positive aspectsof choosing STEM courses in upper high school by providing brochures and a website,hereby attempting to enhance parents’ view of STEM and increase the communicationon this subject between parents and adolescents. It was concluded that the interventionspositively influence both the mothers’ perception of the value of STEM courses, the con-versations about the value of STEM courses and the number of STEM courses chosen bythe adolescents (p. 904). Subsequently, Harackiewicz et al. (2012) found that students’gender did not have a significant effect and parents’ educational level was a predictorfor the number of STEM courses taken by the children. The authors used the samedata in a follow-up analysis to elaborate on the absence of an effect of gender on the effec-tiveness of the intervention (Rozek, S. Hyde, Svoboda, Hulleman, & Harackiewicz, 2014).These analyses confirmed the positive effect of the intervention on the number of STEMcourses taken by the adolescents.

Conclusion

The leaking STEM pipeline is a significant subject in educational research for severalreasons, one of which is the shortage of STEM professionals in the labour market. Thisphenomenon is a problem in the Netherlands as well as in many western countries. Ithas resulted in a large number of studies exploring the reasons for this leakage, the ten-dency to follow pathways outside STEM education and gender differences. Most ofthese studies have used a correlational design to identify malleable and non-malleablefactors related to (gender differences in) enrolment and persistence in STEM education.Meta-reviews have demonstrated that these factors can be found in social context,school context, social environment, and student characteristics, and that they are ofteninterrelated (Blickenstaff, 2005). These factors can be taken into account when interven-tions to address this problem are developed. However, it seems that despite the large inter-est of scholars in the leaking STEM pipeline, not many robust studies assessing the effectsof interventions are available. The main purpose of this study was to find promising pro-grammes aimed at increasing the interest and persistence in STEM. The study focused onthe question:What kind of interventions are successful in raising interest and persistence inSTEM education among girls and boys, according to the research literature? This researchquestion was answered by using a systematic review of publications about the effects of


interventions aimed at increasing the participation in STEM education and preventingstudents to drop out of STEM-related educational programmes. The results of thesearch of the literature revealed that very few published studies (n = 9) examined theeffects of interventions in such a way that causal effects could be determined.

It is important to note that a systematic review always suffers from publication bias(Petticrew & Roberts, 2006). Studies with negative effects or no effects have a muchsmaller chance to be published, despite the fact that information about what does notwork can also be important for the development of effective interventions. The dataextraction form showed that the results of an even lower number of the peer-reviewedstudies were convincing in terms of internal validity. Furthermore, the heterogeneity ofthese studies made it difficult to compare the results of the studies (Petticrew &Roberts, 2006). For example, different scales were used to measure attitude. Others usedmotivation, career aspirations or perceived compatibility as predicators or outcome vari-ables. None of the studies under review were longitudinal and examined the effects forpotential dropout moments, such as the transition from secondary to tertiary educationor from tertiary education to the labour market. The results were also difficult to generalisebecause many of the studies used a convenience sample, were aimed at a specific group ofstudents (e.g. minorities) or took place in a very specific context.

Despite all these limitations, it seems that programmes focusing on knowledge, ability,motivation and feelings of belonging could increase the interest and persistence in STEMeducation. These interventions can be found in a school context: change of the pedagogicalapproach of teaching STEM, or a support programme in a STEM programme. Other inter-ventions focused on science mentors as ‘role models’ for potential science students toenhance their motivation and belonging in STEM, STEM oriented summer camps orinvolved raising awareness about the importance of STEM education among studentsand parents. However, the number of successful interventions and the limitations men-tioned above, make it difficult to draw any general conclusions about what seems towork, or about which characteristics of an intervention are specifically effective. Allstudies, both selected for this review as well as those who were found to be insufficientlyable to prove causal effect, reported generally (small) positive results. This confirms theimportance of persisting in evaluation studies in STEM education aiming at motivatingstudents and helping them to persist in STEM. Some remarks for further research willbe given in the next section.

Implications for further research

The results of the meta-reviews in this research field as well as this systematic review stressthe need for more empirical experimental research, specifically aimed at the effectiveness ofinterventions focussed on directly or indirectly increasing the interest in STEM and prevent-ing dropout from STEM, on the short-term as well as the long-term. The amount of researchthat was not able to draw conclusions about causal relations on, for example, STEM interestis of such a proportion that there should be more attention towards research methodologywhen evaluating a potentially successful intervention. For example summer camps, being thesubject of a large part of the evaluation studies that were found in this systematic review, arequite difficult to evaluate. In these studies, students are not randomly selected to be part ofthe intervention. Students choose to join and are therefore probably more motivated than


the students who did not decide to enrol. These studies often also lack a control group. Tostrengthen the evaluation method for the effectiveness of summer camps, the researchdesign applied by Stoeger et al. (2016) could be useful. They invited students to join theirmentor programme and randomly selected the participants from the register list. The restof the students were told that they could join the programme next year, but functionedas a waiting list-control group (under the condition that they did not enrol in anotherSTEM-related summer camp). The purpose was to have equally motivated students inboth the experimental group and the control group.

Besides the relevance of more attention to research methodology (before starting theintervention) in order to draw more reliable conclusions, the definition issue shouldprioritised as well. Interventions are often focused on one school subject or universitystudy or the focus is not specified enough. Therefore conclusions about STEM-interven-tions are more difficult to draw, although some researchers do extrapolate the effect of anintervention on one specific subject to STEM as a large group of subjects.

Also, results of several studies have indicated that students fromaminority background arealso more likely to drop out of STEM education and can therefore be an interesting topic forfurther research (e.g. Schultz et al., 2011). Another group that could profit from more atten-tion in this research field are the (female) top performers in STEM. PISA data showed thatthere is a difference in achievement between high-performing male and female mathematicsand science students (OECD, 2016). Whether this is caused by gender differences inability, learning needs or other reasons such as lower interest due to the lack of rolemodels, is not clear. It would also be interesting to explore whether an early positive exposureto STEM and more knowledge of STEM-related education and profession among parentshave positive effects on the long term, especially for girls in STEM education.

Research in the past thirty years has resulted in a lot of knowledge about why many(female) students are less interested and less persistent in STEM education than inother fields of education. However, this knowledge can be more useful for the field of edu-cation if it results in actions focusing on increasing the interest and persistence in STEMeducation and STEM-related professions. In future studies, a comprehensive view usingthe various aspects of the conceptual framework should be the basis for the design of pro-grammes and experiments with them to prevent talented boys and girls from dropping outof STEM education.

Disclosure statement

No potential conflict of interest was reported by the authors.

Funding

This work was supported by the Netherlands Initiative for Educational Research (NRO) [grantnumber 405-16-404].

ORCID

Anniek van den Hurk http://orcid.org/0000-0002-0673-6073Annemarie van Langen http://orcid.org/0000-0001-7576-9816


http://orcid.org/0000-0002-0673-6073http://orcid.org/0000-0001-7576-9816

References

Alper, J. (1993). The pipeline is leaking women all the way along. Science, 260(5106), 409–411.doi:10.1126/science.260.5106.409

Bekki, J. M., Smith, M. L., Bernstein, B. L., & Harrison, C. (2013). Effects of an online personal resi-lience training program for women in stem doctoral programs. Journal of Women and Minoritiesin Science and Engineering, 19(1), 17–35. doi:10.1615/JWomenMinorScienEng.2013005351

Berryman, S. E. (1983). Who will do science? Trends, and their causes in minority and female rep-resentation among holders of advanced degrees in science and mathematics. A special report.New York, NY: Rockefeller Foundation.

Blickenstaff, J. C. (2005). Women and science careers: Leaky pipeline or gender filter? Gender andEducation, 17(4), 369–386. doi:10.1080/09540250500145072

Cannady, M. A., Greenwald, E., & Harris, K. N. (2014). Problematizing the STEM pipeline meta-phor: Is the STEM pipeline metaphor serving our students and the STEM workforce? ScienceEducation, 98(3), 443–460. doi:10.1002/sce.21108

Ceci, S. J., & Williams, W. M. (2010). Sex differences in math-intensive fields. Current Directions inPsychological Science, 19(5), 275–279. doi:10.1177/0963721410383241

Dekkers, H. (2007). Accessible and effective education: New research in a sophisticated theoreticalcontext educational inequality: Persistence and change (pp. 117–134). Dordrecht: Springer.

Eccles, J. (2009). Who am I and what am I going to do with my life? Personal and collective iden-tities as motivators of action. Educational Psychologist, 44(2), 78–89. doi:10.1080/00461520902832368

Eccles, J. S., & Wang, M. T. (2016). What motivates females and males to pursue careers in math-ematics and science? International Journal of Behavioral Development, 40(2), 100–106. doi:10.1177/0165025415616201

Eddy, S. L., & Brownell, S. E. (2016). Beneath the numbers: A review of gender disparities in under-graduate education across science, technology, engineering, and math disciplines. PhysicalReview Physics Education Research, 12(2), doi:10.1103/PhysRevPhysEducRes.12.020106

Gaspard, H., Dicke, A. L., Flunger, B., Brisson, B. M., Hafner, I., Nagengast, B., & Trautwein, U.(2015). Fostering adolescents’ value beliefs for mathematics with a relevance intervention inthe classroom. Developmental Psychology, 51(9), 1226–1240. doi:10.1037/dev0000028

Harackiewicz, J. M., Rozek, C. S., Hulleman, C. S., & Hyde, J. S. (2012). Helping parents to motivateadolescents in mathematics and science: An experimental test of a utility-value intervention.Psychological Science, 23(8), 899–906. doi:10.1177/0956797611435530

Kara, Y., & Yesilyurt, S. (2008). Comparing the impacts of tutorial and edutainment software pro-grams on students’ achievements, misconceptions, and attitudes towards biology. Journal ofScience Education and Technology, 17(1), 32–41. doi:10.1007/s10956-007-9077-z

Kelly, A. M. (2016). Social cognitive perspective of gender disparities in undergraduate physics.Physical Review Physics Education Research, 12(2), doi:10.1103/PhysRevPhysEducRes.12.020116

Lee, M. K., & Erdogan, I. (2007). The effect of science-technology-society teaching on students’ atti-tudes toward science and certain aspects of creativity. International Journal of Science Education,29(11), 1315–1327. doi:10.1080/09500690600972974

Marsden, E., & Torgerson, C. J. (2012). Single group, pre- and post-test research designs: Somemethodological concerns. Oxford Review of Education, 38(5), 583–616. doi:10.1080/03054985.2012.731208

Microsoft. (2017).Why Europe’s girls aren’t studying STEM. Retrieved from https://news.microsoft.com/europe/features/dont-european-girls-like-science-technology/#sm.0001rwq32pg3aeugq6s20umv1vnag

Miller, D. I., & Wai, J. (2015). The bachelor’s to Ph.D. STEM pipeline no longer leaks more womenthan men: A 30-year analysis. Frontiers in Psychology, 6. doi:10.3389/fpsyg.2015.00037.

Mullis, I. V. S., Martin, M. O., Foy, P., & Drucker, K. T. (2012). PIRLS 2011 international results inreading. Chestnut Hill, MA.

Mullis, I. V. S., Martin, M. O., Foy, P., & Hooper, M. (2016). TIMSS 2015 international results inmathematics. Chestnut Hill, MA.


https://doi.org/10.1126/science.260.5106.409https://doi.org/10.1615/JWomenMinorScienEng.2013005351https://doi.org/10.1080/09540250500145072https://doi.org/10.1002/sce.21108https://doi.org/10.1177/0963721410383241https://doi.org/10.1080/00461520902832368https://doi.org/10.1080/00461520902832368https://doi.org/10.1177/0165025415616201https://doi.org/10.1177/0165025415616201https://doi.org/10.1103/PhysRevPhysEducRes.12.020106https://doi.org/10.1037/dev0000028https://doi.org/10.1177/0956797611435530https://doi.org/10.1007/s10956-007-9077-zhttps://doi.org/10.1103/PhysRevPhysEducRes.12.020116https://doi.org/10.1080/09500690600972974https://doi.org/10.1080/03054985.2012.731208https://doi.org/10.1080/03054985.2012.731208https://news.microsoft.com/europe/features/dont-european-girls-like-science-technology/#sm.0001rwq32pg3aeugq6s20umv1vnaghttps://news.microsoft.com/europe/features/dont-european-girls-like-science-technology/#sm.0001rwq32pg3aeugq6s20umv1vnaghttps://news.microsoft.com/europe/features/dont-european-girls-like-science-technology/#sm.0001rwq32pg3aeugq6s20umv1vnaghttps://doi.org/10.3389/fpsyg.2015.00037

National Science Foundation. (2017). Women, minorities, and persons with disabilities in scienceand engineering: 2017. Arlington, VA. www.nsf.gov/statistics/wmpd/

OECD. (2014). PISA 2012 results: What students know and can do (volume I, revised ed., February2014). Paris: Author.

OECD. (2015). The ABC of gender equality in education: Aptitude, behaviour, confidence. Paris:Author.

OECD. (2016). Education at a glance 2016. Paris: Author.OECD. (2017). Education at a glance: OECD indicators. Paris: Author.Petticrew, M., & Roberts, R. (2006). Systematic reviews in the social sciences: A practical guide.

Oxford: Blackwell.Prokop, P., Tuncer, G., & Kvasnicak, R. (2007). Short-term effects of field programme on students’

knowledge and attitude toward biology: A Slovak experience. Journal of Science Education andTechnology, 16(3), 247–255.

Rozek, C., S. Hyde, J., Svoboda, R., Hulleman, C., & Harackiewicz, J. (2014). Gender differences inthe effects of a utility-value intervention to help parents motivate adolescents in mathematics andscience (Vol. 107).

Schultz, W. P., Hernandez, P. R., Woodcock, A., Estrada, M., Chance, R. C., Aguilar, M., & Serpe, R.T. (2011). Patching the pipeline: Reducing educational disparities in the sciences through min-ority training programs. Educational Evaluation and Policy Analysis, 33(1), 95–114.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Quasi-experimental designs that either lack acontrol group orlack pretest observations on the outcome experimental and quasi-experimentaldesigns for generalized causal inference. Boston, MA: Houghton Mifflin Company.

Stoeger, H., Schirner, S., Laemmle, L., Obergriesser, S., Heilemann, M., & Ziegler, A. (2016). A con-textual perspective on talented female participants and their development in extracurricularSTEM programs. Beyond the Iq Test, 1377, 53–66. doi:10.1111/nyas.13116

Stout, J. G., Dasgupta, N., Hunsinger, M., & McManus, M. A. (2011). STEMing the tide: Usingingroup experts to inoculate women’s self-concept in science, technology, engineering, andmathematics (STEM). Journal of Personality and Social Psychology, 100(2), 255–270. doi:10.1037/a0021385

TechniekPact, & Platform Bèta Techniek. (2016). Monitor: Facts & figures bètatechniek 2016[Monitor: Facts & figures STEM 2016]. Den Haag: Ministerie van Economische zaken enPlatform Bèta Techniek.

Van Langen, A., Driessen, G., & Dekkers, H. (2008). Sekseverschillen in onderwijsloopbanen innederland [Gender differences in education careers in the Netherlands]. Pedagogische Studien,85, 3–15.

Van Langen, A., Elfering, S., & Hilkens, T. (2018). Bèta/technische weglek uit het Nederlandse onder-wijs: omvang en kenmerken [STEM leakage from Dutch education: size and characteristics].Nijmegen: KBA.

Wallace, E. W., Perry, J. C., Ferguson, R. L., & Jackson, D. K. (2015). The careers in health andmedical professions program (CHAMPS): An impact study of a university-based STEM +H out-reach program. Journal of Science Education and Technology, 24(4), 484–495.

Wang, X. L. (2013). Why students choose STEM majors: Motivation, high school learning, andpostsecondary context of support. American Educational Research Journal, 50(5), 1081–1121.doi:10.3102/0002831213488622

Wang, M. T., & Degol, J. (2013). Motivational pathways to STEM career choices: Using expectancy-value perspective to understand individual and gender differences in STEM fields. DevelopmentalReview, 33(4), 304–340. doi:10.1016/j.dr.2013.08.001

Wang, M. T., Eccles, J. S., & Kenny, S. (2013). Not lack of ability but more choice: Individual andgender differences in choice of careers in science, technology, engineering, and mathematics.Psychological Science, 24(5), 770–775. doi:10.1177/0956797612458937


http://www.nsf.gov/statistics/wmpd/https://doi.org/10.1111/nyas.13116https://doi.org/10.1037/a0021385https://doi.org/10.1037/a0021385https://doi.org/10.3102/0002831213488622https://doi.org/10.1016/j.dr.2013.08.001https://doi.org/10.1177/0956797612458937

AbstractIntroductionTheoretical frameworkEnvironmental factorsFactors at the school levelFactors at the student levelTheoretical models for participation or persistence in STEM

MethodSearch processInclusion and exclusion criteriaQuality assessment

ResultsSearch resultsQuality assessmentContent analysisSocial contextSchool contextSocial environment

ConclusionImplications for further research

Disclosure statementORCIDReferencesLege paginaLege pagina

KBA Nijmegen - Interventions in education to prevent STEM ... 1.pdf2019 KBA Nijmegen Behoudens de in...

Documents

Transcript of KBA Nijmegen - Interventions in education to prevent STEM ... 1.pdf2019 KBA Nijmegen Behoudens de in...