8/12/2019 13 De Bodt

1/7

Journal of VoiceVol. 11, No. I, pp. 74--80 1997 Lippineott-Ra venPublishers, Philadelphia

Test Retest Study of the GRBAS Scale: Influence ofExperience and Professional Background on Perceptual

Rating of Voice Quality* t M a r c S D e B o d t , * F l or i s L . W u y t s , * P a ul H . V a n d e H e y n i n g , a n d

t ~ t C h r i s t o p h e C r o u x*Department of Otolaryngology, Head and Neck Surgery, and Communication Disorders, University Hospital;J;Department of Mathematics, University of Antwerp, UIA, Belgium; and tBelgian National ScienceFoundation NFWO)

S u m m a ry : P e rc e p t u a l e v a l u a t io n wa s p e r fo rm e d b y m e a n s o f t h e GR B AS s c a l efo r a se r ies o f n ine pa tho log ica l vo ice samples tha t were p rese n ted tw ice to ag ro u p o f 2 3 j u d g e s , c o n s i s ti n g o f e x p e r i e n c e d a n d i n e x p e r i e n c e d o t o l a ry n g o l -o g i s ts E NT s ) a n d s p e e c h - l a n g u a g e p a t h o l o g i s t s S / L P s ) . T h e t i m e i n te rv a lb e t we e n r a t in g s wa s 14 d a y s . T e s t - r e t e s t r e l i ab i l i ty wa s m o d e ra t e . T h e b e s ta g re e m e n t b e t we e n t h e o b s e rv e r s wa s o b ta i n e d fo r t h e G g ra d e ) p a ra m e t e ra n d t h e wo r s t f o r t h e S s t r a in e d ) p a ra m e t e r . C o n s i d e r i n g th e m e d i a n s o f t h eGR B A S ra t i ng s , n o s i g ni f i ca n t i n f lu e n c e wa s m e a s u re d fo r l e v e l o f e x p e r i e n c eo r p ro fe s s i o n a l b a c k g ro u n d . Ye t , b a s e d o n m e a n s , t h e r e s u l t s s h o w t h a t p ro -f e s s i o n a l b a c k g ro u n d h a s a g r e a t e r i m p a c t o n p e rc e p t u a l r a t i n g t h a n e x p e r i -e n c e. K e y W or d s: P e r c e p tu a l e v a l u a t i o n - - V o i c e q u a l i t y - - T e s t - r e t e s t - -G R B A S s c a l e .

Perceptual evaluat ion is fundamental to assess-ment of vocal qual i ty , overal l severi ty of defects ,and their impact on communicat ion ski l ls . Never-theless , i t is one of the m ost controversial subjectsin vo ice eva lua t ion , because of poor cor re la t ionamong raters . Review of the l i terature reveals awide v ariety of rat ing scales and reliabil i ty data thatf luctuate from s tudy to s tudy (1-3). Mult iple factorscont r ibu te to th is pheno meno n.Firs t , there is inadequate internat ional ly acceptedterminology in describing voice qual i ty (4) . Someresearchers have t r ied to make order f rom chaos .Severa l o f the be t te r known percep tua l judgmentprotocols are thos e of Ha mm arbe rg (5) , Wilson (6) ,Laver (7,8) , Gelfer (9) , and the Japan Society ofLogopaedics and Phoniatr ics (10) . At the present ,

Accepted July 2, 1996.Addre ss correspondence and reprint requests to Dr. M. S. DeBodt at U niversity Hospital Antwerp, D epartment of Otolaryn-go logy and Communica t ion Diso rders , Wi l r i jk s t raa t 10 ,B 265 EDEGEM, Belgium.

there is no universal ly accepted system for the per-cep tua l assessment o f vo ice qua l i ty . The GRBASsca le of the Japan Soc ie ty of Logopaedics and Pho-niatr ics received i ts internat ional introduct ion andreputat ion through the publicat ion of Hirano 's Clinical examination of voice in 1981 (10). It seems tohave become the mos t wide ly used sca l ing method(11).The GR BAS sys tem for descr ib ing voca l qua l itycontains f ive well-defined param eter s (10,12): G(overa l l g rade of hoarseness ) ; R ( roughness ) ; B(breathiness); A (asthenic) , an d S (s trained qual i ty)of the voice. A four-point scale , f ro m 0 to 3, is usedfor each parameter . 0 equa ls nonhoarse or nor -mal , 1 s l ight , 2 mo derate , and 3 severe.Results are deno ted as G~RIBIAoS 0. The GRB ASscale is probably the most c om pac t of al l perceptu alrat ing systems and was conceived for dai ly use bya ll members of the vo ice team. The Japan Soc ie tyhas made a s tandardized tape with typical voicesamples represen ted by the GRBAS sca le .

74

8/12/2019 13 De Bodt

2/7

GRB SSC LE

A second issue is the difference in perceptuals t ra teg ies be t ween judges as d iscussed by Kre imanet a l. (13). M ost pe rceptua l tasks involve comparingvoice s to internal s tandards. The se internal s tan-dards are f luctuating and unstable and cause vari-ability in rating. Kreiman et al. (13) suggest the useof exp l ic it comp ar isons be twe en s t imul i (anchoredprotocols) to improve rel iabil i ty . I t is not c lear howthese modif ica t ions a re to be opera t iona l ized inclinical prac tice.

F ina l ly , in te r ra te r agreem ent i s in f luenced bym e th o d o lo g ic a l a s p e c t s s u c h a s t y p e s o f s c a l e,speech sample , and ra te r .To date , research concerning factors influencingpercep tua l ra tings o f vo ice qua l i ty has foc used on

training scales , l is tening experience, cultural back-ground , and speech f ragments . Severa l au thors em-phasize the im portan ce of tra ining and experience(3,4,12). Anders et a l . (14) s tudied the effects oftraining ( i .e . , c lass es of auditors) and of culture( language) on the percep t ion o f hoarseness . Theyfound small but not s ignif icant differences betweenclasse s of l is teners in favor of tra ined l is teners .They conclude that t ra ining and professional andcu l tu ra l ba ckground do no t have major in fluence onpercep tua l ra t ing . Exper ienced l i s teners ( speech-l a n g u a ge p a th o lo g i s t s ) i n G e l f e r ' s s t u d y ju d g e dsomewhat more cons is ten t ly than un t ra ined ra te rs(9) . In contrast , Kreiman et a l . (15) showed thatnaive and expert l is teners use different perceptuals t ra teg ies . They found tha t exper t l i s teners showless agreement than naive l is teners about the rela-t ive impor tance o f var ious aspec ts o f vo ice qua l ity .They sugges t tha t ca re mus t be taken when us ingdata av erag ed a cross all c linicians (15). B assich andLu dlo w (3) reporte d that 8 h of tra ining we re re-quired for previously inexperienced l is teners to at-tain 80% interjudge reliability using a 13-dimensionpercep tua l ra t ing sys tem.

With regard to the GRBAS scale , Abe et a l . (16)found that reproducibil i ty differs according to l is-teners and scales . High reproducibil i ty is possiblewhen l i s teners a re se lec ted p ruden t ly . Recen t workof Dejon cker e et a l. (17) sho wed that in a perceptualscore l i s t o f 15 i tems , the GRB AS sca le parametersappear to be the mos t in te res t ing , because theyshow fav orab le combina t ions o f low int ra - and in -ter judge variance and high intervoice variance. Int h e c o m p a r i s o n o f j u d g m e n t s o f t w o d i f f e re n tjudges , the bes t cor re la t ion was found fo r G (p =0.7) . Impressions about A and S were less consis-tent .

The GRBAS sca le was in t roduced in our vo iceclinic to s tandardize p erceptu al evalu ation of voicequa l i ty and to improve the descr ip t ive communica-t ion be tween speech- language pa tho log is ts (S /LPs)and otolaryngologists (ENTs) with differing levelsof experien ce. T he curren t l i terature is not c leara b o u t t h e e x t e n t t o w h ic h p e rc e p tu a l r a t i n g b ymeans o f the GRB AS sca le can be in f luenced byexper ience and profess ion . The p r imary purpose o fthis s tudy was to determine the test-retest re l iabil i tyof S /LPs and EN Ts ' judgm ents o f vo ice us ing theGRB AS sca le . A second purpose was to inves t iga tethe in f luence o f exper ience and profess iona l back-ground.

MA T E R IA L S A N D ME T H O D SVoice SamplesRecord ings were made o f 12 pa t ien ts f rom thevoice cl inic . We used three recordings as tr ia ls andthe remaining nine for the s tudy. All patients (3m e n , 6 w o m e n ) c o m p la in e d o f d y s p h o n ia . A g eranged between 22 and 65 years . The voice sampleconsisted of sustained vowels /a / and / i / and thereading of a phonetically b alance d tex t (115 word s) .We ins t ruc ted pa t ien ts to read a t a comfor tab lepitch- and loudness level . High-quali ty recordingwas made in a s i lent room with an ambient noiseleve l 2 year sof experien ce. All S/LP s had inte nsive v oice train-ing. This is not characteris t ic of the training given tothe EN Ts in th is s tudy . A l though a ll judge s as -sessed vo ice d isorders on a regu la r bas is , mos t o fthem were no t fami l ia r w i th the GRBAS sys tem.For the s ta t i st ica l ana lys is , we d enoted two groups :

exp er ie nce d and in exp er ie nce d con tain ing 12Journa l of Voice Vol. 11 No . 1 1997

8/12/2019 13 De Bodt

3/7

76 M. S. DE BODT ET AL.

T A B L E 1 . V o i c e s a m p l e c h a r a c t e r is t ic sF o J i t t er S h i m m e r H / N r a ti o

S a m p l e G e n d e r A g e H z ( % ) ( d B ) ( d B ) G a R " B a A a S ~I M 57 129 4 .53 8 1 .252 - I 1 .682 3 2 3 3 12 M 55 140 6 .65 0 1 .645 - 13 .187 I 1 1 03 F 22 187 0 .59 2 0 .621 6 .77 8 2 2 1 1 24 M 5 3 1 4 3 0 . 6 3 7 0 . 4 4 2 5 . 9 9 6 3 2 2 1 35 F 70 251 0 .88 2 0 .683 - 1 .318 I 1 2 I I6 F 58 258 0 .48 6 0 .30 7 I 1 .451 3 2 3 3 27 F 65 188 3 .107 1 .037 - 7 .097 I 0 1 0 08 F 5 1 2 0 0 0 . 5 8 0 0 . 2 8 4 8 . 5 4 7 2 2 2 1 29 F 6 4 2 5 3 1 . 4 6 0 0 . 8 2 2 7 . 3 9 8 2 2 1 I 2

R a t i n g s c a l e : G , g r a d e ; R , r o u g h ; B , b r e a t h i n e s s ; A , a s t h e n i c ; S , s t r a i n e d .a M e d i a n s c a l c u l a t e d f o r t h e " e x p e r i e n c e d " g r o u p . F o u r - p o i n t g r ad i n g : 0 , n o r m a l ;

2 , m o d e r a t e ; 3 , s e v e r e .I , s l i g h t ;

and 11 judges, respectively. Judges were consid-ered experienced when they had been confrontedwith voice pathology over at least one third of theirprofessional life and during t>3 years. Otherwise,we considered them inexperienced. Both groupscontained ENTs as well as S/LPs.

istening TaskBefore the first rating session, all judges receivedtheoretical and clinical instructions about theGRBAS system and the scoring procedure from anexperienced instructor. A training tape composedby the same instructor, using Dutch spoken speechsamples, was used to ensure that judges were lis-tening to the same percept. The rating of the ninedifferent samples selected for this study was pre-ceded by three trial judgments of samples that werenot included in the study. Judges were allowed toask questions during the trials. We presented allnine patient voices successively to the entire groupat one time. The voice samples were presented onlyonce to the listeners and interrupted by a shortbreak allowing the judges to score the sample. Wedid not allow contact or discussion between judgesduring the listening task.All judges used the same preprinted rating sheetthat included a brief specification of the GRBASparameters as described by Hirano 10). A secondrating, occurring 14 days after the first rating andusing an identical procedure, took place. None ofthe judges were informed about their first rating.

RESULTSThe GRBAS ratings were evaluated by means of

nonparametric statistics because they are on an or-dinal scale. Compa rison o f related groups was per-

formed using the Wilcoxon Matched-Pairs Signed-Ranks Test and comparison of independent groupsby the Wilcoxon Mann-Whitney statistic. Test-retest reliability was measured by the kappa statis-tic 18,19); this method gives the most appropriatemeasure of reliability for ordinal data 20).

The data for statistical analysis consisted of 2,070ratings: 5 GRBAS parameter scores for 9 voices by23 judges at two different times. We divided thejudges into four subsets according to their level ofexperience in voice rating: experienced E) n = 12)or inexperienced I) n = 1 I); and according to theirprofessional background: ENT n = 13) or S/LP n= 10). The data are equally distributed betweenboth profess iona l groups 2: p = 0.28).The first goal of this study was the asse ssment oftest-retest reliability between the first and secondratings. This reliability was evaluated by the kappastatistic. The calculations were based on frequencytables as in Table 2, where the scores are shown forthe G parameter, rated by all judges . The Tableillustrates on the diagonal how many times the firstand second scores were equal. Of the 207 G ratingsmade on the nine voices by the 23 judges, G3 wasassigned 48 times both on the first rating and thesecond rating done 2 weeks later. The kappa statis-

TABLE 2. Frequency table for the first and secondrating of G assessed by all judgesS e c o n d r a ti n g

F i r s t r a t i n g 0 1 2 30 3 3 0 01 4 50 15 02 0 9 50 153 0 0 10 48

G , g r a d e .

Journal of Voice Vol . 11 N o. 1 1997

8/12/2019 13 De Bodt

4/7

G R B A S S C A L E 77

T~tBLE 3 . Tes t -re tes t re l iab il i ty va lue K) be tw een thef i r s t a n d s e c o n d s c o r e , a s s e s s e d b y e x p e r i e n c e d E ) a n di n e x p e r i e n c e d I ) j u d g e s , o t o la r y n g o lo g i s ts E N T s ) a n ds p e e c h - l a n g u a g e p a t h o l o g i s ts S / L P s ) , a n d a l le x a mi n e r s t o g e t h e rK I E EN T S/LP AllG 0.50 0.70 0.62 0.58 0.60R 0.29 0.40 0.27 0.43 0.35B 0.29 0.45 0.30 0.46 0.38A 0.51 0.28 0.42 0.36 0.39S 0.30 0.36 0.32 0.35 0.34GR BA S 0.40 0.45 0.41 0.44 0.43

Rating scale: G, grade; R, rough; B, breathiness; A, asthenic;S, strained.

t i c w a s u s e d t o e v a l u a t e t h e t e s t - r e t e s t r e l i a b i l i t y o ft h e G R B A S s c a l e b e t w e e n t h e f ir s t a n d s e c o n d r a t -i ng s . E a c h i n d i v i d ua l G R B A S p a r a m e t e r , a s w e ll a st h e e n t ir e s c a l e , w a s e v a l u a t e d , a n d t h e j u d g e s w e r ed i v i d e d i n to f o u r g r o u p s a c c o r d i n g t o e x p e r i e n c ea n d p r o f e s s i o n a l b a c k g r o u n d . W e a l s o c o n s i d e r e dt h e m a l l t o g e t h e r i n o n e p o o l . T h e s e r e s u l t s a r e i nT a b l e 3 . I n t h is t a b l e , t h e K v a l u e s i n d i c a t e t h ea g r e e m e n t b e t w e e n t h e s c o r e s f r o m b o t h t ri a ls . I nt h e c a s e o f t h e e x p e r i e n c e d l i s t e n e r s , K e q u a l s 0 . 7f o r G , w h i c h r e f l e c t s t h a t 7 0 o f t h e d i f f e r e n c e b e -t w e e n p e r f e c t a g r e e m e n t (K = 1) a n d t h e re a l a g r e e -m e n t e x p e c t e d p u r e l y b y c h a n c e (K = 0 ) is a c -c o u n t e d f o r b y th e a g r e e m e n t b e t w e e n t h e s c o r e sf o r t r i a l 1 a n d t r i a l 2 . A l t m a n ( 2 1) p r o v i d e s a g o o dd i s c u s s i o n o f i n t e r p r e t a t i o n o f K, a s e x p r e s s e d i nT a b l e 4 .

T h e a g r e e m e n t b e t w e e n t h e 23 o b s e r v e r s w a se q u a l l y c a l c u l a t e d b y m e a n s o f th e k a p p a s t a t is t ic( 18 ). A g a i n w e p e r f o r m e d t h e c a l c u l a t i o n s f o r f o u ro b s e r v e r c a t e g o r i e s : e x p e r i e n c e d v e r s u s i n e x p e ri -e n c e d l i s t e n e rs , E N T v e r s u s S P . S u c c e s s i v e l y , w ep o o l e d a ll g r o u p s t o g e t h e r . A s a n e x a m p l e , t h e r a t -i n g s o f t h e 1 2 e x p e r i e n c e d l i s t e n e r s a r e l i s t e d in T a -b l e 5 , in d i c a t i n g t h a t f o r t h e s e c o n d v o i c e s a m p l e ,o n e o b s e r v e r a s s e s s e d t h is v o i c e a s n o r m a l ( G o) ,n i n e j u d g e s a s s i g n e d a G I , a n d t w o j u d g e s a s s i g n e da G2 .B a s e d o n t h is m e t h o d , a g r e e m e n t w a s c a lc u l a t e d

TA B LE 4 . K in terpre ta t ion tab leValue of K Strength of agreement

8/12/2019 13 De Bodt

5/7

78 M S DE BOD T ET AL

cance levels: G, p = 0.36; R, p = 0.11 ; B, p = 0.18;A, p = 0.32; and S, p = 0.11, indicat ing no influ-ence of professional background on the medians forthe GRBAS scale.We also calculated the mean score of four newjudge groups: experienced and inexperienced ENTsas well as experienced and inexperienced S/LPs.For these groups, we calculated an average basedon all GRBAS scores attributed to all patients. Al-though these averages have no physiological mean-ing because all voices were averaged, it was ex-pected that the averages of the four groups wouldbe distributed closely around the grand average.This was not the case. Figure l shows the grandaverage (-+95 conf idence interval) of all scores forall GRBAS parameters and voice samples by theentire collection of judges , as well as the subgroupaverages and their standard errors.Figure 2 shows, for example, the average G scoreof the nine voices, ordered by their G rating for thefour subgroups of judges.

1 8

A wide variety of perceptual rating scales hasbeen described in literature. In comparison withother systems, the GRBAS scale has the advantageof practical use in a daily clinical environment. Thisis mainly because o f the simplicity of the scale, withonly five parameters (GRBAS) and four categories:normal, slight, moderate, and severe. This advan-

O ENT Sn,

- 9 1 . 7

~ 1.6

1.5m 1.48~ 1 3

1.2

DISCUSSION

n=1035I I

Inexp Exp

Overall Mean+95 CI

L e v e l o f e x p e r i e n c eF I G . 1 . M e a n s c o re s o f t h e fo u r s u b g r o u p s o f j u d g e s , c o m p a r e dt o t h e o v e r a l l m e a n o f t h e G R B A S s c o re f o r a l l v o ic e s . G , g r a d e ;R , rough ; B , b rea t h i nes s ; A , as t hen i c ; S , s t ra i ned ; E N T , o t o -l a r y ngo l og i s t s ; S P , s peec h - l anguage pa t ho l og i s t s .

Oo 2.9

>12 weeks, and bythe test-retest reliability among 23 judges. The in-fluence on the variability of the GRBAS scale byprofessional background as well as experience levelwas also investigated.

The overall test-retest reliability of the GRBASscale when assessing scores at two distinct timeswas moderate (K = 0.43). Professional backgroundor experience level did not influence this reliability.

Journal of Voice Vol. I1 No. 1 1997

8/12/2019 13 De Bodt

6/7

GRB SSC LE

Considered apart, the reliability between time 1 andtime 2 was good for the assessment of the overallgrade of voice disorder (G). Moreover, both profes-sional groups showed similar test-retest reliability.Yet, the influence of experience level is more prom-inent on the test-retest reliability, ranging frommoderate (0.5) to good (0.7). For roughness (R) andbreathiness (B), S/LPs seemed to be more reliablein their judgment than ENTs. Reliability for theseparameters increased with the level of experience.The tes t-retest reliability decreased for the asthenicproperty of the voice with increasing experience.Clearly, this indicates that the A parameter is sus-ceptible to interpretation error. Loss of reliabilitywith experience was also described by Kreiman etal. (15). In their study, clinical training and experi-ence caused listeners to differ more. The strainedproperty (S) of the voices also showed rather lowtest-retest reliability.The best test-retest reliability can be expected ina group of experienced li steners (4,14,22), whichwas confirmed in our study (Table 3). The parame-ters G, R, B, and S were reliable, whereas Ashowed much more variability over time. Dejonc-kere et al. (17) have described similar findings. Thisresult indicated that the interpretation of G, B, R,and S was less ambiguous among the judges thanthe interpretation of A.Another quality of the GRBAS scale was inves-tigated by measuring the agreement among the dif-ferent judges. Table 6 indicates that the G parame-ter was most unambiguously interpreted among thelisteners. Furthermore, the agreement among thejudges was not influenced by level of experience orprofessional background. These findings are in ac-cord with t hose o f Dejonckere (17) and Anders et al.(14).In clinical application, it is of interest to knowwhether two judges would similarly rate a certainpatient's voice. Therefore, we calculated the medi-ans (discrete variable), because the scores arewhole integers and not a figure in between, as is thecase in visual analog scales. The statistical resultsshowed that there was no difference between thesemedians in the different groups of judges, irrespec-tive of their profession or experience. Thus, mostobservers gave the same categorical value of 0 to 3to the same patients for each of the GRBAS param-eters.Although no statistically significant differenceswere found, we demonstrated different tendenciesamong the four classes of listeners when the aver-

ages, based on a large data pool, were calculated.Figure 1 illustrates the deviation from the grandaverage of GRBAS scores for experienced and in-experienced ENTs and S/LPs. Overall, the experi-enced listeners scored higher than the inexperi-enced listeners, suggesting that the later underes -timated and the former ove res tim ate d theseverity of the voice disorder. Also, using this cri-terion, ENTs unde rest imat ed and S/LPs over-estima ted the dysphonia. Moreover, the differ-ence between experienced and inexperienced lis-teners was smaller than that between ENTs andS/LPs. It should be emphasized that the absolutedeviations around the grand average are only 3% ofthe full scale (0-3). Figure I illustrates that the ex-perience factor is more influential in the ENT thanin the SLP group. In their rating, experienced E NTsclearly best approached the overall mean.The results of this study show that test-retest re-liability of the GRBAS scale is moderate. The Gparameter seems to be the most reliable. Based onthe statistical results of the medians, we concludedthat no statistically significant differences were ev-ident between different judges, irrespective of theirexperience or profession. Based on analysis of themeans, small differences appeared among judge sdepending on level of experience and professionalbackground. These differences are small enough(Fig. 2) to permit useful evaluation and reporting ofvoice quality by clinicians with different levels ofexperience or professional background using theGRBAS scale.

REFERENCESI . K r e i m a n J , G e r a t t B R , K e m p s t e r G B , E r m a n A , B e r k e G S .P e r c e p t u a l e v a l u a t i o n o f v o i c e q u a l i t y : r e v i e w , t u t o r i a l , a n da f r a m e w o r k f o r f u t u r e r e s e a r c h . J S p e e c h H e a r R e s 1993;36:21-40.2 . B l a u s t e i n S , A s h e r B . R e l i a b i l i ty o f p e r c e p t u a l v o i c e a s s e s s -m e n t . J C o m m u n D i s o r d 1983;16:157--61.3 . B a s s i c h C J , L u d l o w C L . T h e u s e o f p e r c e p t u a l m e t h o d s b yn e w c l i n i c i a n s f o r a s s e s s i n g v o i c e q u a l i t y . J S p e e c h H e a rD i s o r d 1986;51:125-33.4 . F e x S . P e r c e p t u a l e v a l u a t i o n . J V o i c e 1992;6:155-8 .5 . H a m m a r b e r g B . P a t h o l o g ic a l v o ic e q u a l it i e s. P e r c e p t u a l a n da c o u s t i c c h a r a c t e r i s t i c s o f a s e t o f S w e d i s h r e f e r e n c ev o i c e s . B u l l A u d i o p h o n o l o g y 1 9 9 2 ; 8 : 3 9 - 5 26 . W i l s o n D K . V o i c e p r o b l e m s o f c h i ld r e n . B a l t i m o r e :W i l l i ams & W i lk ins , 1987.7 . L a v e r J . T h e p h o n e t i c d e s c r i p t io n o f v o i c e q u a l i t y . C a m -b r i d g e U n i v e r s i t y P r e s s , 1 9 80 .8 . W i r z S , M a c k e n z i e B e c k J . A s s e s s m e n t o f v o i c e q u a l it y : t h eV o c a l P r o f i l e s A n a l y s i s S c h e m e . I n : W i r z S , e d . P e r c e p t u a la p p r o a c h t o c o m m u n i c a t i o n d i s o r d e rs . L o n d o n : W h u r r P u b -l i sher s , 1995:39-55.

Journal of Voice Vol . 11 N o. 1 1997

8/12/2019 13 De Bodt

7/7

80 M S DE BODT ET AL

9 . G e l f e r M P . P e r c e p t u a l a t t r ib u t e s o f v o i c e : d e v e l o p m e n t a n du s e o f r a t in g s c a l e s . J V o i c e 1988;2:320-326.1 0 . H i r a n o M . C l i n ic a l e x a m i n a t i o n o f v o i c e . N e w Y o r k :Spr inger Ver lag , 1981 :81-4 .I I . D e j o n c k e r e P H , W i e n e k e G H . G R B A S - s c a l in g o f p at h o lo g -ica l vo ices : r e l i ab i l i ty , c l in ica l r e levance and d i f fe ren t ia tedc o r r e l a t io n w i t h a c o u s t i c m e a s u r e m e n t s , e s p e c i a ll y w i t hc e p s t r al m e a s u r e m e n t s . P r o c e e d i n g s o f t h e 2 2 n d I A L P C o n -g r e s s . H a n n o v e r , G e r m a n y : 1 9 9 2 .1 2. H i r a n o M . C l i n i c a l a p p l i c a t i o n s o f v o i c e t e s t s , a s s e s s m e n t o fs p e e c h a n d v o i c e p r o d u c t i o n : r e s e a r c h a n d c l i n ic a l a p p l i ca -t i o n s . N I D C D m o n o g r a p h . P r o c e e d i n g s o f a c o n f e r en c e o ft h e N a t i o n a l I n s t i t u t e o f H e a l t h . Be thes da: 1990: 96-203.1 3 . K r e i m a n J , G e r a t t B R , P r e c o d a K , B e r k e G S . I n d i v i d u a ld i f f e r e n c e s i n v o i c e q u a l i t y p e r c e p t i o n . J S p e e c h H e a r R e s1992;35:512-20.1 4. A n d e r s L C , H o l l ie n H , H u r m e P , S o n i n n e n A . W e n d l e r J.P e r c e p t i o n o f h o a r s e n e s s b y s e v e r a l c l a s s e s o f l is t e n e r s . F o -l i a P h o n i a t r i c a 1988;40:91-100.1 5. K r e i m a n J , G e r a t t B R , P r e c o d a K . L i s t e n e r e x p e r i e n c e a n d

p e r c e p t i o n o f v o i c e q u a l i t y . J S p e e c h H e a r R e s 1990;33:103-15.1 6. A b e H , Y o n e k a w a H , O h t q F , I m a i z u m i S . R e p r o d u c i b i l i t yo f h o a r s e v o i c e p s y c h o a c o u s t i c e v a l u a t io n . J p n J L o g o pP h o n i a t r 1986;27:168-77.1 7. D e j o n c k er e P H , O b b e n s C , D e M o o r G M , W i e n e k e G H .P e r c e p t u a l e v a l u a t i o n o f d y s p h o n i a : r e l i a b i l i ty a n d r e l e -v a n c e . F o l i a P h o n i a t r 1993;45:76-831 8. C o h e n J . A c o e f f i c i e n t o f a g r e e m e n t f o r n o m i n a l s c a l e s .E d u c P s y c h M e a s u r e m e n t 1960;20:37--46.19 . S iege l S , Cas te l l an JN . N o n p a r a m e t r i c s t a t i s t i c s f o r t h e b e -h a v i o r a l s c i e n c e s . N e w Y o r k , M c G r a w - H i l l , 1 9 8 8 .20 . F le i s JL . T h e d e s i g n a n d a n a l y s i s o f c l i n i ca l e x p e r i m e n t s .New York : Wi ley , 1986 :1 -32 .2 1 . A l t m a n D G . P r a c t i c a l s t a ti s t i c s f o r m e d i c a l r e s e a r ch . L o n -d o n : C h a p m a n & H a l l , 1 9 9 1 .2 2. R a b i n o v C R , K r e i m a n J , G e r a t t B R , B i e l a m o w i c s S . C o m -p a r i n g r e l ia b i l i t y o f p e r c e p t u a l r a t i n g s o f r o u g h n e s s a n da c o u s t i c m e a s u r e s o f j i t t e r . J S p e e c h H e a r R e s 1995;38:26--32.

Journal of Voice Vol . I I No . 1 1997