Con Galt On

8/3/2019 Con Galt On

1/12

R E M O T E S E N S . E N V I R O N . 3 7 : 3 5 - 4 6 ( 1 9 9 1 )

A R eview of A ssess ing the Accuracy ofClass ifications of R em otely S en sed D ataRussell G. CongaltonDep artme nt of Forestry an d Resource M anagement, University o f California, Berkeley

T h i s paper reviews the necessary considerat ionsand available techniques for assessing the accuracyof remotely sensed data. Included in this revieware the classification system, the sampling scheme,the sample size, spatial autocorrelation, and theassessment techniques. All analysis is based on theuse o f an error ma trix or contingency table. Exa m-ple matrices and results of the analysis are pre-sented. Future trends including the need fo r assess-me nt of other spatial data are also discussed.

I N T R O D U C T I O NW i t h t h e a d v e n t o f m o r e a d v a n c e d d i g i ta l s a te l li t er e m o t e s e n s i n g t e c h n i q u e s , t h e n e c e s s i t y o f p e r -f o r m i n g a n a c c u r a c y a s s e s s m e n t h a s r e c e i v e d r e -n e w e d i n t e r e s t . T h i s i s n o t t o s a y t h a t a c c u r a c ya s s e s s m e n t i s u n i m p o r t a n t f o r t h e m o r e t r a d i ti o n a lr e m o t e s e n s i n g t e c h n i q u e s . H o w e v e r , g i v e n t h ec o m p l e x i t y o f d i g it a l c l a ss i fi c a ti o n , t h e r e i s m o r e o fa n e e d t o a s se s s t h e r e l i a b i l it y o f t h e r e s u l ts .T r a d i ti o n a ll y , t h e a c c u r a c y o f p h o t o i n t e r p r e t a t i o nh a s b e e n a c c e p t e d a s c o r r e c t w i t h o u t a n y c o n f i r -m at ion . In f ac t , d ig i t a l c las s i f i ca t ions a re o f tena s s e s s e d w i t h r e f e r e n c e t o p h o t o i n t e r p r e t a t i o n . A n

o b v i o u s a s s u m p t i o n m a d e h e r e i s t h a t t h e p h o t o i n -t e r p r e t a t i o n i s 1 0 0 % c o r r e c t . T h i s a s s u m p t i o n i sr a r e l y v a l i d a n d c a n l e a d t o a r a t h e r p o o r a n du n f a i r a s s e s s m e n t o f t h e d i g i t a l c l a s s i fi c a ti o n( B i g i n g a n d C o n g a l t o n , 1 9 8 9 ) .

T h e r e f o r e , i t i s e s s e n t i a l t h a t r e s e a r c h e r s a n du s e r s o f r e m o t e l y s e n s e d d a t a h a v e a s t r o n g k n o w l -e d g e o f b o t h t h e f a c to r s n e e d e d t o b e c o n s i d e r e da s w e l l a s t h e t e c h n i q u e s u s e d i n p e r f o r m i n g a n ya c c u r a c y a s s e s s m e n t . F a i l u r e t o k n o w t h e s e t e c h -n i q u e s a n d c o n s i d e r a t i o n s c a n s e v e r e l y l i m i t o n e ' sa b i l it y t o e f f e c ti v e l y u s e r e m o t e l y s e n s e d d a t a . T h eo b j e c t i v e o f t h i s p a p e r i s t o p r o v i d e a r e v i e w o f t h ea p p r o p r i a t e a n a l y s i s t e c h n i q u e s a n d a d i s c u s s i o n o ft h e f a ct o rs t h a t m u s t b e c o n s i d e r e d w h e n p e r f o r m -i n g a n y a c c u r a c y a s s e s s m e n t . M a n y a n a l y s i s t e c h -n i q u e s h a v e b e e n p u b l i s h e d i n t h e l i te r a t u re ; h o w -e v e r , I b e l i e v e t h a t i t w i l l b e h e l p f u l t o m a n yn o v i c e a n d e s t a b l i s h e d u s e r s o f re m o t e l y s e n s e dd a t a t o h a v e a l l t h e s t a n d a r d t e c h n i q u e s s u m m a -r i z e d i n a s i n g l e p a p e r . I n a d d i t i o n , i t i s i m p o r t a n tt o u n d e r s t a n d t h e a n a l y s i s t e c h n i q u e s i n o r d e r t of u l ly re a l i z e t h e i m p o r t a n c e o f t h e v a r i o u s o t h e rc o n s i d e r a t i o n s f o r a c c u r a c y a s s e s s m e n t d i s c u s s e di n t h i s p a p e r .

Address correspondence to R. G. Congalton, 145 Mulford Hall,Depar tment of Fores try and Resource Management, Univer s i ty ofCalifornia, Berkeley, CA 94720.R ece ived 1 5 Oc to b er 1990; rev i sed 14 Ap r i l 1991 .oo34-42s7/91/$3.50

T E C H N I Q U E SU n t i l r e c e n t l y , t h e i d e a o f a s s e s s i n g t h e c l as s if ic a -t i o n a c c u ra c y o f r e m o t e l y s e n s e d d a t a w a s t r e a t e d

3 5


2/12

36 Congalton

m o r e a s a n a f t e r t h o u g h t t h a n a s a n i n t e g r a l p a r t o fa n y p r o j e c t . I n f a c t , a s r e c e n t l y a s t h e e a r l y 1 9 8 0 sm a n y s t u d ie s w o u l d s i m p l y re p o r t a s i n gl e n u m b e rt o e x p r e s s t h e a c c u r a c y o f a c l a s si f ic a t io n . I n m a n yo f t h e s e c a s es t h e a c c u r a c y r e p o r t e d w a s w h a t i sc a l l e d n o n - s i t e - s p e c i f i c a c c u r a c y . I n a n o n - s i t e -s p e c if i c a c c u r a c y a s s e s s m e n t , l o c a t io n a l a c c u r a c y i sc o m p l e t e l y i g n o r e d . I n o t h e r w o r d s , o n l y t o t a la m o u n t s o f a c a t e g o r y a re c o n s i d e r e d w i t h o u t r e -g a r d f o r t h e l o c a ti o n . I f a ll t h e e r r o r s b a l a n c e o u t ,a n o n - s i t e - s p e c i f i c a c c u r a c y a s s e s s m e n t w i l l y i e l dv e r y h i g h b u t m i s l e a d i n g r e s u l ts . I n a d d i t i o n , m o s ta s s e s s m e n t s w e r e c o n d u c t e d u s i n g t h e s a m e d a t ase t a s was u sed to t r a in th e c l a ss i f i e r . Th i s t r a in in ga n d t e s t i n g o n t h e s a m e d a t a s e t a l s o r e s u l t s i no v e r e s t i m a t e s o f c l as s i fi c a t io n a c c u r a c y .

O n c e t h e s e p r o b l e m s w e r e r e c o g n i z e d , m a n ym o r e s i te s p e c i fi c a c c u r a c y a ss e s s m e n t s w e r e p e r -f o r m e d u s i n g a n i n d e p e n d e n t d a t a s e t. F o r t h e s ea s s e s s m e n t s , t h e m o s t c o m m o n w a y t o r e p r e s e n tt h e c l a s si f ic a t io n a c c u r a c y o f r e m o t e l y s e n s e d d a t ai s in t h e f o r m o f a n e r r o r m a t r i x . U s i n g a n e r r o rm a t r ix t o r e p r e s e n t a c c u r a c y h a s b e e n r e c o m -m e n d e d b y m a n y r e s e a r c h e r s a n d s h o u l d b ea d o p t e d a s t h e s t a n d a r d r e p o r t i n g c o n v e n t io n . T h er e a s o n s f o r c h o o s i n g t h e e r r o r m a t r i x a s t h e s t a n -d a r d a r e c l e a rl y d e m o n s t r a t e d i n t h i s p a p e r.

A n e r r o r m a t r i x is a s q u a r e a r r a y o f n u m b e r ss e t o u t i n r o w s a n d c o l u m n s w h i c h e x p r e s s t h en u m b e r o f s a m p l e u n i t s ( i .e . , p i x el s , c l u s t e r s o fp i x e l s , o r p o l y g o n s ) a s s i g n e d t o a p a r t i c u l a r c a t e -

g o r y r e l a t i v e t o t h e a c t u a l c a t e g o r y a s v e r i f i e d o nt h e g r o u n d ( T a b l e 1 ) . T h e c o l u m n s u s u a l l y r e p r e -s e n t t h e r e f e r e n c e d a t a w h i l e t h e r o w s i n d i c a te t h ec l a s s i f i c a t i o n g e n e r a t e d f r o m t h e r e m o t e l y s e n s e dd a ta . A n e r r o r m a t r i x i s a v e r y e f f e c t i v e w a y t or e p r e s e n t a c c u r a c y i n t h a t t h e a c c u r a c ie s o f e a c hc a t e g o r y a r e p l a i n ly d e s c r i b e d a l o n g w i t h b o t h t h ee r r o r s o f i n c l u s i o n ( c o m m i s s i o n e r r o r s ) a n d e r r o r so f e x c l u s i o n (o m i s s i o n e r r o r s ) p r e s e n t i n t h e c l a s s i-f ica t ion .

Descriptive TechniquesT h e e r r o r m a t r i x c a n t h e n b e u s e d a s a s t a r t i n gp o i n t f o r a s e r i e s o f d e s c r i p t i v e a n d a n a l y t i c a ls t a t i s t i c a l t e c h n i q u e s . P e r h a p s t h e s i m p l e s t d e -s c r i p t iv e s t a ti s t ic i s o v e r a l l a c c u r a c y w h i c h i s c o m -p u t e d b y d i v i d i n g t h e t o t a l c o r r e c t ( i . e . , t h e s u m o ft h e m a j o r d i a g o n a l ) b y t h e t o t a l n u m b e r o f p i x e l si n t h e e r r o r m a t r i x . I n a d d i t i o n , a c c u r a c i e s o fi n d i v i d u a l c a t e g o r i e s c a n b e c o m p u t e d i n a s i m i l a rm a n n e r . H o w e v e r , t h i s c a s e i s a l i t t l e m o r e c o m -p l e x i n t h a t o n e h a s a c h o i c e o f d i v i d i n g t h en u m b e r o f c o r r e c t p ix e ls i n t h a t c a t e g o r y b y e i t h e rt h e t o t al n u m b e r o f p i xe ls i n t h e c o r r e s p o n d i n gr o w o r t h e c o r r e s p o n d i n g c o l u m n . T r a d i t i o n a l l y ,t h e t o t a l n u m b e r o f c o r r e c t p i x e l s i n a c a t e g o r y i sd i v i d e d b y t h e t o ta l n u m b e r o f p i xe l s o f t h a tc a t e g o r y as d e r i v e d f r o m t h e r e f e r e n c e d a t a ( i. e. ,t h e c o l u m n t o t al ). T h i s a c c u r a c y m e a s u r e i n d i c at e st h e p r o b a b i l i t y o f a r e f e r e n c e p i x e l b e i n g c o r r e c t l y

Table 1. A n E x a m p l e E r r o r M a t r i x

R e f e r e n c e D a t aD C B A S B

D 6 5 4 2 2 2 4C 6 81 5 8B A 0 1 1 8 5 1 9S B 4 7 3 9 0c o l u m n 7 5 1 0 3 1 1 5 1 4 1t o t a l

PRODUCER'S ACCURACYD=65/75= 87%C= 81/1 03 = 79%

BA = 85 /11 5= 74%SB =90 /14 1= 64%

lO Wt o t a l1 1 5 Land Cover Categories1 0 0 D = d e c i d u o u s1 1 5 C = c o n i f e r1 0 4 B A = b a r r e n4 3 4 S B = s h r u b

O V E R A L L A C C U R A C Y =3 2 1 / 4 3 4 = 74 %

USER'S ACCURACYD=65/115= 57%C= 81/1 00= 81%

BA=85/115= 74%SB = 90/ 104 = 87%


3/12

Review: Assessing Classification Accuracy 3 7

c lass i fi ed and i s r ea l ly a meas ure o f omiss ion e r ro r .T h i s a c c u r a c y m e a s u r e i s o f t e n c a l l e d " p r o d u c e r ' sa c c u r a c y " b e c a u s e t h e p r o d u c e r o f t h e c la s si fi ca -t i o n is i n t e r e s t e d i n h o w w e l l a c e r ta i n a r e a c a n b ec l as s if ie d . O n t h e o t h e r h a n d , i f t h e t o ta l n u m b e rof co r r ec t p ixe l s i n a ca t ego ry is d iv id ed by th et o ta l n u m b e r o f p i x el s t h a t w e r e c l a ss i fi e d i n t h a tc a t e g o ry , t h e n t h is r e s u l t i s a m e a s u r e o f c o m m i s -s i o n e r r o r . T h i s m e a s u r e , c a l l e d " u s e r ' s a c c u r a c y "or r e l i ab i li t y , is i nd ica t ive o f the p robab i l i t y tha t ap i x e l c l a s s i f i e d o n t h e m a p / i m a g e a c t u a l l y r e p r e -s e n t s t h a t c a t e g o r y o n t h e g r o u n d ( S t o r y a n dConga l ton , 1986) .

A v e r y s i m p l e e x a m p l e q u i c k l y s h o w s t h e a d -v a n t a g e s o f c o n s i d e r i n g o v e r a ll a c c u r a c y , " p r o -d u c e r ' s a c c u r a c y , " a n d " u s e r ' s a c c u ra c y .' " T h e e r -r o r m a t r i x s h o w n i n T a b l e 1 i n d i c a t e s a n o v e r a l lm a p a c c u r a c y o f 7 4% . H o w e v e r , s u p p o s e w e a r em o s t i n t e r e s t e d i n t h e a b i li ty t o c l a s si fy d e c i d u o u sf o r e s t s . W e c a n c a l c u l a t e a " p r o d u c e r ' s a c c u r a c y "f o r t h i s c a t e g o r y b y d i v i d i n g t h e t o t a l n u m b e r o fc o r r e c t p i x e l s i n t h e d e c i d u o u s c a t e g o r y ( 6 5 ) b yt h e t o ta l n u m b e r o f d e c i d u o u s p i x el s a s i n d i c a t e dby the r e f e r ence da ta (75 ) . Th i s d iv i s ion r esu l t s i na " p r o d u c e r ' s a c c u r a c y " o f 8 7% , w h i c h i s q u i t eg o o d. I f w e s t o p p e d h e r e , o n e m i g h t c o n c l u d etha t , a l though th i s c l ass i f i ca t ion has an overa l laccuracy tha t i s on ly f a i r ( 74%) , i t i s adequa te fo rt h e d e c i d u o u s c a t e g o r y . M a k i n g s u c h a c o n c l u s i o ncou ld be a very se r ious mis t ake . A qu ick ca lcu la -t i o n o f t h e " u s e r ' s a c c u r a c y " c o m p u t e d b y d i v i d i n gt h e t o t a l n u m b e r o f c o r r e c t p ix e l s i n t h e d e c i d u o u sca teg ory (65 ) by the to t al nu m be r o f p ixe ls c l ass i-f i ed as dec iduo us (115 ) r evea l s a va lu e o f 57%. Ino t h e r w o r d s , a lt h o u g h 8 7 % o f t h e d e c i d u o u s a r e a sh a v e b e e n c o r r e c t l y i d e n t i f i e d a s d e c i d u o u s , o n l y5 7 % o f t h e a r e a s c a l l e d d e c i d u o u s a r e a c t u a l lyd e c i d u o u s . A m o r e c a r e f u l l o o k a t t h e e r r o r m a t r i xr evea l s tha t t he re i s s ign i f i can t confus ion in d i s -c r i m i n a t i n g d e c i d u o u s f r o m b a r r e n a n d s h r u b .T h e r e f o r e , a l t h o u g h t h e p r o d u c e r o f t h is m a p c a nc l a i m t h a t 8 7 % o f t h e t i m e a n a r e a t h a t w a sd e c i d u o u s w a s i d e n t i f i e d a s s u c h , a u s e r o f t h i sm a p w i l l f i n d t h a t o n l y 5 7 % o f t h e t i m e w i l l a narea he v i s i t s t ha t t he map says i s dec iduous wi l la c t u a l ly b e d e c i d u o u s .

Analytical TechniquesI n a d d i t i o n t o t h e s e d e s c r i p t i v e t e c h n i q u e s , a ne r r o r m a t r i x i s a n a p p r o p r i a t e b e g i n n i n g f o r m a n y

ana ly t i ca l s t a t i s t i ca l t echn iques . Th i s i s e spec ia l lyt r u e o f t h e d i s c r e t e m u l t i v a r i a te t e c h n i q u e s . S t a rt -ing wi th Conga l ton e t a l . ( 1983) , d i sc r e t e mul t i -v a r i a t e t e c h n i q u e s h a v e b e e n u s e d f o r p e r f o r m i n gs ta t i s t i ca l t e s t s on the c l ass i f i ca t ion accuracy o fd i g it a l r e m o t e l y s e n s e d d a t a . S i n c e t h a t t i m e m a n yo t h e r s h a v e a d o p t e d t h e s e t e c h n i q u e s a s t h e s t a n -dard accuracy assessment too l s ( e .g . , Rosenf i e lda n d F i t z p a t r i c k - L i n s , 1 9 8 6 ; H u d s o n a n d R a m m ,1987 ; Campbel l , 1987 ) . D i sc r e t e mul t iva r i a t e t ech-n i q u e s a r e a p p r o p r i a t e b e c a u s e r e m o t e l y s e n s e dd a t a a r e d i s c r e t e r a t h e r t h a n c o n t i n u o u s . T h e d a t aa r e a l s o b i n o m i a l l y o r m u l t i n o m i a l l y d i s t r i b u t e dr a t h e r t h a n n o r m a l l y d i s t r ib u t e d . T h e r e f o r e , m a n yc o m m o n n o r m a l t h e o r y s t a t i s t i c a l t e c h n i q u e s d on o t a p p l y. T h e f o ll o w i n g e x a m p l e p r e s e n t e d i nT a b l e s 2 - 9 d e m o n s t r a t e s t h e p o w e r o f t h e s e d i s-c r e t e m u l t i v a r i a te t e c h n i q u e s . T h e e x a m p l e b e g i n sw i t h t h r e e e r r o r m a t r i c e s a n d p r e s e n t s t h e r e s u l t so f the ana lys i s t echn iques .

T a b l e 2 p r e s e n t s t h e e r r o r m a t r i c e s g e n e r a t e df rom us ing th r ee d i f f e r en t c l ass i f i ca t ion a lgor i thmst o m a p a s m a l l a r e a o f B e r k e l e y a n d O a k l a n d ,C a l i fo r n i a s u r r o u n d i n g t h e U n i v e r s i t y o f C a l i fo r n iacampus f rom SPOT sa te l l i t e da ta . The th r ee c l ass i -f i ca t ion a lgor i thms used inc luded a t r ad i t iona l su -p e r v i s e d a p p r o a c h , a t r a d i t i o n a l u n s u p e r v i s e d a p -p r o a c h , a n d a m o d i f i e d a p p r o a c h t h a t c o m b i n e sthe superv i sed and unsuperv i sed c l ass i f i ca t ions to -g e t h e r t o m a x i m i z e t h e a d v a n t a g e s o f e a c h(Chuv ieco and Conga l ton , 1988) . The c l ass i f i ca t ionwas a s imple one us ing on ly four ca t egor i es ; f o r es t(F ) , i ndus t r i a l ( I ) , u rban (U) , and wate r (W) . Al lth r ee c l ass i f i ca t ions were per fo rmed by a s ing leana lys t . I n add i t ion , Tab le 3 p r esen t s the e r ro rm a t r i x g e n e r a t e d f o r t h e s a m e a r e a u s i n g o n l y t h emodi f i ed c l ass i f i ca t ion approach by a second ana-lys t . Each ana lys t was r espons ib le fo r pe r fo rminga n a c c u r a c y a s s e s sm e n t . T h e r e f o r e , d i f f e re n t n u m -ber s o f samples and d i f f e r en t sample loca t ionsw e r e s e l e c te d b y e a c h .

The nex t ana ly t i ca l s t ep i s t o "normal i ze" o rs t a n d a r d i z e t h e e r r o r m a t r ic e s . T h i s t e c h n i q u e u s e sa n i t e r a t i v e p r o p o r t i o n a l f i t t i n g p r o c e d u r e w h i c hf o r c e s e a c h r o w a n d c o l u m n i n t h e m a t r i x t o s u mto one . I n th i s way , d i f f e r ences in sample s i zesu s e d t o g e n e r a t e t h e m a t r i c e s a r e e l i m i n a t e d a n d ,there fo re , i nd iv idua l ce l l va lues wi th in the mat r ixa r e d i r e c t l y c o m p a r a b l e . I n a d d i t i o n , b e c a u s e a sp a r t o f t h e i t e r at i v e p r o c e ss t h e r o w s a n d c o l u m n sare to t a l ed ( i . e . , marg ina l s ) , t he r esu l t ing normal -


4/12

38 Congalton

Table 2. E r r o r M a t r i c e s f o r t h e T h r e e C l a s s i f i c a t io n A p p r o a c h e s f r o m A n a l y s t # 1

Classi fiedData

ClassifiedData

ClassifiedData

Supervised ApproachReference Data

F I U WF 68 7 3 0I 12 112 15 10U 3 9 8 9 0W 0 2 5 5 6

Unsupervised ApproachReference Data

F I U WF 6 0 11 3 4I 15 102 14 8U 6 1 3 9 0 2W 2 4 5 5 2

M odi f i e d ApproachReference Data

F I U WF 7 5 6 1 0I 4 116 11 3U 3 7 9 6 2W 1 1 4 61

Overal l Accuracy =3 2 5 / 3 9 1 = 8 3 %

Overal l Accuracy =3 0 4 / 3 9 1 = 7 8 %

Overall Accuracy =3 4 8 / 3 9 1 = 8 9 %

Table 3. E r r o r M a t r i x f o r t h e M o d i f i e d C l a s s i f i c a t i o n A p p r o a c h f r o m A n a l y s t # 2

ClassifiedData

ModifiedA p p r o a c h - - A n a l y s t # 2Reference Data

F A G U WF 3 5 6 1 0A G 3 8 2 5 1 0U 4 2 5 4 0W 0 5 2 37

overall accuracy =2 0 8 / 2 4 6 = 8 5 %

i z ed ma tr ix i s mo re i nd i ca t i v e o f the o f f -d i a g o na lce l l v a l ues ( i . e ., the erro rs o f o mi s s i o n a n d co m-mi s s i o n) . I n o ther w o rds , a l l the v a l ues i n thema tr i x a re i t era t i v e l y b a l a nced b y ro w a nd co l umnthereb y i nco rp o ra t i ng i n fo rma t i o n f ro m tha t ro wa nd co l umn i n to ea ch i nd i v i dua l ce l l v a l ue . T hi sp ro ces s then cha ng es the ce l l v a l ues a l o ng themajor diagona l o f the matrix (correct c lass i f i ca-t i o ns ) a nd there fo re a no rma l i zed o v era l l a ccura cyc a n b e c o m p u t e d f o r e a c h m a t r i x b y s u m m i n g t h e

ma jo r d i a g o na l a nd d i v i d i ng b y the to ta l o f theent i re ma tr i x . C o ns eq uent l y , o ne co u l d a rg ue tha tthe no rma l i zed a ccura cy i s a b e t t er rep res enta t i o no f a ccura cy tha n i s the o v era l l a ccura cy co mp utedfro m the o r i g i na l ma tr i x b eca us e i t co nta i ns i n fo r -ma t i o n a b o ut the o f f -d i a g o na l ce l l v a l ues . T a b l e 4p res ent s the no rma l i zed ma tr i ces f ro m the s a methree c l a s s i f i ca t i o n a l g o r i thms fo r a na l y s t # 1 g en-e r a te d u s i n g a c o m p u t e r p r o g r a m c a ll e d M A R G F I T(ma rg i na l f i t t i ng ) . T a b l e 5 p res ent s the no rma l i zed


5/12

Review : Assessing Classification Accu racy 39

Table 4. N o r m a l i z e d E r r o r M a t r ic e s f o r t h e T h r e e C l a s s if i c at i o n A ppr o a c h e s f r om A n a l y s t # 1

ClassifiedData

ClassifiedData

ClassifiedData

FIUW

FIUW

FIUW

Supervised ApproachReference Data

F I U W0.8652 0.0940 0.0331 0.00730.0845 0.7547 0.0784 0.08240.0435 0.1171 0.8319 0.00720.0069 0.0342 00567 0.9031

Unsupervised ApproachReference Data

F I U W0.7734 0.1256 0.0387 0.06220.1242 0.7014 0.1006 0.08240.0656 0.1163 0.7094 0.02730.0369 0.0567 0.0702 0.8370

Modified ApproachReference Data

F I U W0.9080 0.0687 0.0152 0.00760.0372 0.8460 0.0801 0.03660.0370 0.0697 0.8598 0.03340.0178 0.0156 0.6450 0.9224

Normalized Accuracy =3.3549/4 = 84%



Table 5. N o r m a l i z e d E r r or M a tr ix f o r t h e M o d i f ie d A ppr o a c h f r o m A n a l y s t # 2

ClassifiedDataFAGUW

Modified Approach-- Analyst #2Reference Data

AG U W0.8519 0.1090 0.0287 0.01130.0464 0.7641 0.0581 0.13130.0897 0.0348 0.8655 0.00940.0120 0.0921 0.0477 0.8480

Overall Accuracy =3.3295/4 = 83%

I

m a t r ix f o r t he m o di f i ed a p p ro a ch p er f o rm ed bya na ly s t #2 .

I n a ddi t io n t o co m p ut ing a no rm a l i zed a ccu-racy , the normalized matrix can a lso be used tod irec t ly co m p a re ce l l v a lues be t ween m a t r ices .Fo r ex a m p le , we m a y be in t eres t ed in co m p a r ingthe accuracy each analyst obta ined for the forestcategory using the modif ied c lassi f icat ion ap-p ro a ch . F ro m t he o r ig ina l m a t r ices we ca n see t ha ta na ly s t #1 c la ss i f i ed 75 sa m p le uni t s co rrec t ly

whi l e a na ly s t #2 c la ss i f i ed 3 5 co rrec t ly . N e i t her o ft h e s e n u m b e r s m e a n s m u c h b e c a u s e t h e y a r e n o td irec t ly co m p a ra b le due t o t he d i f f erences in t henum ber o f sa m p les used t o g enera te t he errormatrix by each analyst. Instead, these num berswo uld need t o be co nv er t ed int o p ercent so t ha t aco m p a r i so n co uld be m a de . H ere a no t her p ro b lemarises: Do we div ide the tota l correct by the rowtota l (user 's accuracy) or by the co lumn tota l (pro-ducer 's a ccura cy )? W e co u ld ca lcu lat e bo t h a nd


6/12

40 Congalton

compare the results or we could use the cell valuein the normalized matrix. Because of the iterativeproportional fitting routine, each cell value in thematrix has been balanced by the other values in itscorresponding row and column. This balancing hasthe effect of incorporating producer's and user'saccuracies together. Also since each row and col-umn add to 1, an individual cell value can quicklybe converted to a percentage by multiplying by100. Therefore , the normalization process providesa convenient way of comparing individual cellvalues between error matrices regardless of thenumber of samples used to derive the matrix.

Another discrete multivariate technique of usein accuracy assessment is called KAPPA (Cohen,1960). The result of performing a KAPPA analysisis a KHAT statistic (an estimate of KAPPA), whichis another measure of agreement or accuracy. TheKHAT statistic is computed as

/~= i=1 i=1F

N 2 - E ( x , + x + , )i=1where r is the numbe r of rows in the matrix, xii isthe number of observations in row i and column i,x i+ and x +i are the marginal totals of row i andcolumn i, respectively, and N is the total numberof observations (Bishop et al., 1975). The KHATequation is published in this paper to clear upsome confusion caused by a typographical error inCongalton et al. (1983), who originally proposedthe use of this statistic for remotely sensed data.Since that time, numerous papers have been pub-lished recommending this technique. The equa-tions for computing the variance of the KHATstatistic and the standard normal deviate can befound in Congalton et al. (1983), Rosenfield andFitzpatriek-Lins (1986), and Hudson and Ramm(1987) to list just a few. It should be noted thatthe KHAT equation assumes a multinomial sam-pling model and that the variance is derived usingthe Delta method.

Table 6 provides a comparison of the overallaccuracy, the normalized accuracy, and the KHATstatistic for the three classification algorithms usedby analyst # 1. In this particular example, all threemeasures of accuracy agree about the relativeranking of the results. However, it is possible forthese rankings to disagree simply because each

Table 6. A Comparison of the Three Accuracy Measures forthe Three Classification ApproachesClassification Overall KHA T NormalizedAlgorithm Accuracy Ac cu ra cy AccuracySupervisedapproach 84% 77% 83%Unsupervisedapproach 78% 70% 78%Modifiedapproach 88% 85% 89%

measure incorporates various levels of informationfrom the error matrix into its computations. Over-all accuracy only incorporates the major diagonaland excludes the omission and commission errors.As already described, normalized accuracy directlyincludes the off-diagonal elements (omission andcommission errors) because of the iterative pro-portional ftting procedure. As shown in the KHATequation, KHAT accuracy indirectly incorporatesthe off-diagonal elements as a product of the rowand column marginals. Therefore, depending onthe amount of error included in the matrix, thesethree measures may not agree. It is not possible togive cleareut rules as to when each measure shouldbe used. Each accuracy measure incorporates dif-ferent information about the error matrix andtherefore must be examined as different computa-tions attempting to explain the error. My experi-ence has shown that if the error matrix tends tohave a great many off-diagonal cell values withzeros in them, then the normalized results tend todisagree with the overall and Kappa results. Manyzeros occur in a matrix when an insuflqeient sam-ple has been taken or when the classification isexceptionally good. Because of the iterative pro-portional fitting routine, these zeros tend to takeon positive values in the normalization processshowing that some error could be expected. Thenormalization process then tends to reduce theaccuracy because of these positive values in theoff-diagonal cells. If a large number of off-diagonalcells do not contain zeros, then the results of thethree measures tend to agree. There are also timeswhen the Kappa measure will disagree with theother two measures. Because of the ease of com-puting all three measures (software is availablefrom the author) and because each measure re-fleets different information contained within theerror matrix, I recommend an analysis such as the


7/12

Review: Assessing Classification Accuracy 41

Table 7. R e s u l t s o f t h e K A P P A A n a l y s i s T e s t o f S i g n i fi c a n c ef o r I n d i v i d u a l E r r o r M a t r i c e s

T es t o f S i gni fi cance o f E ach E r r or M at r ixClassification Alg ori thm s KHA T Statistic Z Sta tistic Result"Superv ised approach .7687 29.41 S bUns upervise d approach .6956 24.04 SModified approach .8501 39.23 S

~At the 95% conf idence leve l .bS = significant.

Table 8. R e s u l t s o f K A P P A A n a l y si s f o r C o m p a r i s o n b e -t w e e n E r r o r M a t r i c e s f o r A n a l y s t # 1Test of S igni f icant Di f ferences between Error Matr ices

Comparison Z Statistic Result ~Superv ised vs. unsup ervised 1.8753 NS/~Superv ised vs. modif ied 2.3968 SUns upervise d vs. modif ied 4.2741 S

At the 95% conf idence level .I~S = sign ificant, NS = n ot sign ificant.

o n e p e r f o r m e d h e r e t o g l e a n a s m u c h i n f o r m a t i o nf r o m t h e e r r o r m a t r i x a s p o s s i b l e .

I n a d d i t i o n t o b e i n g a t h i r d m e a s u r e o f a c c u -r a c y , K A P P A i s a l s o a p o w e r f u l t e c h n i q u e i n i t sa b i l i ty t o p r o v i d e i n f o r m a t i o n a b o u t a s i n g l e m a t r i xa s w e l l a s to s t a t is t i c al l y c o m p a r e m a t r i c e s . T a b l e 7p r e s e n t s t h e r e s u l t s o f t h e K A P P A a n a l y s is t o t e s tt h e s i g n i f i c a n c e o f e a c h m a t r i x a l o n e . I n o t h e rw o r d s , t h i s t e s t d e t e r m i n e s w h e t h e r t h e r e s u l t sp r e s e n t e d i n t h e e r r o r m a t r i x a r e s i g n if i c a n t ly b e t -t e r t h a n a r a n d o m r e s u l t ( i . e . , t h e n u l l h y p o t h e s i s :K H A T = 0 ). T a b l e 8 p r e s e n t s t h e r e s u l t s o f t h eK A P P A a n a l y s i s t h a t c o m p a r e s t h e e r r o r m a t r i c e st w o a t a t i m e t o d e t e r m i n e i f t h e y a r e s i g n i fi c a n t l yd i f f e r e n t . T h i s t e s t i s b a s e d o n t h e s t a n d a r d n o r m a ld e v i a t e a n d t h e f a c t th a t , a l t h o u g h r e m o t e l y s e n s e dd a t a a r e d i s c r e t e , t h e K H A T s t a t i s t i c i s a s y m p t o t i -c a l l y n o r m a l l y d i s t r i b u t e d . A q u i c k l o o k a t T a b l e 8s h o w s w h y t h i s t e s t i s s o i m p o r t a n t . D e s p i t e t h eo v e r a ll a c c u r a c y o f t h e s u p e r v i s e d a p p r o a c h b e i n g6 % h i g h e r t h a n t h e u n s u p e r v i s e d a p p r o a c h ( 8 4 %- 7 8 % = 6 % ) , t h e r e s u l ts o f t h e K A P P A a n a l y si ss h o w t h a t t h e s e t w o a p p r o a c h e s a r e n o t s i g n i f i -c a n t l y d i f f e r e n t . T h e r e f o r e , g i v e n t h e c h o i c e o fo n l y t h e s e t w o a p p r o a c h e s , o n e s h o u l d u s e t h ee a s i e r, q u i c k e r , o r m o r e e f f i c i e n t a p p r o a c h b e c a u s et h e a c c u r a c y w i l l n o t b e t h e d e c i d i n g f ac t o r. S i m i -l a r r e s u lt s a r e p r e s e n t e d i n T a b l e 9 c o m p a r i n g t h em o d i f i e d c l a ss i f ic a t i o n a p p r o a c h f o r a n a l y s t # 1w i t h a n a l y s t # 2 .

T a b l e 9 . R e s u l t s o f K A P P A A n a l y s is f o r C o m p a r i s o n b e -t w e e n M o d i f i e d A p p r o a c h f o r A n a l y s t # 1 v s . A n a l y s t # 2

Test of S igni f icant Di f ferences between Error Matr icesComparison Z Statistic Result ~

Modif ied #1 vs. modif ied #2 1.6774 NS bAt the 95% conf idence level ./'NS = not significant.

I n a d d i t i o n t o t h e d i s c r e t e m u l t i v a r i a t e t e c h -n i q u e s j u s t p r e s e n t e d , o t h e r t e c h n i q u e s f o r as se s s-i n g t h e a c c u r a c y o f r e m o t e l y s e n s e d d a t a h a v e a l s ob e e n s u g g e s t e d . R o s e n f ie l d ( 1 98 1 ) p r o p o s e d t h eu s e o f a n a l y s is o f v a r i a n c e t e c h n i q u e s f o r a c c u r a c ya s s e s s m e n t . H o w e v e r , v i o l a t io n o f t h e n o r m a l t h e -o r y a s s u m p t i o n a n d i n d e p e n d e n c e a s s u m p t i o nw h e n a p p l y i n g t h i s t e c h n i q u e t o r e m o t e l y s e n s e dd a t a h a s s e v e r e l y l i m i t e d i t s a p p l i c a t i o n . A r o n o f f( 1 98 5 ) s u g g e s t e d t h e u s e o f a m i n i m u m a c c u r a c yv a l u e a s a n i n d e x o f c la s s if i c a ti o n a c c u r a c y . T h i sa p p r o a c h i s b a s e d o n t h e b i n o m i a l d i s t r i b u t i o n o ft h e d a t a a n d i s t h e r e f o r e v e r y a p p r o p r i a t e f o rr e m o t e l y s e n s e d d a t a . T h e m a j o r d i s a d v a n t a g e o ft h e A r o n o f f a p p r o a c h i s t h a t i t i s l i m i t e d t o a s i n g l eo v e r al l a c c u ra c y v a l u e r a t h e r t h a n u s i n g t h e e n t i r ee r r o r m a t r i x . H o w e v e r , i t i s u s e f u l i n t h a t i t t h i si n d e x d o e s e x p r e s s s t a t i s t i c a l l y t h e u n c e r t a i n t yi n v o l v e d i n a n y a c c u r a c y a s s e s s m e n t . F i n a l l y ,S k i d m o r e a n d T u r n e r ( 1 9 89 ) h a v e b e g u n w o r k o nt e c h n i q u e s f o r a s s e s s i n g e r r o r a s i t a c c u m u l a t e st h r o u g h m a n y s p a t ia l l a y e r s o f i n f o r m a t i o n i n aG I S , i n c l u d i n g r e m o t e l y s e n s e d d a t a . T h e s e t e c h -n i q u e s h a v e i n c l u d e d u s i n g a l i n e s a m p l i n g m e t h o df o r a c c u r a c y a s s e s s m e n t a s w e l l a s p r o b a b i l i t y t h e -o r y t o a c c u m u l a t e e r r o r f r o m l a y e r t o l a y e r. I t i s i nt h is a r e a o f e r r o r a n a ly s is t h a t m u c h n e w w o r kn e e d s t o b e p e r f o r m e d .

C O N S I D E R A T I O N SA l o n g w i t h t h e a c t u a l a n a l y si s t e c h n i q u e s , t h e r ea r e m a n y o t h e r c o n s i d e r a t i o n s t o n o t e w h e n p e r -f o r m i n g a n a c c u r a c y a s s e s s m e n t . I n r e a l i t y , t h et e c h n i q u e s a r e o f l i t tl e v a l u e i f t h e s e o t h e r f a c to r sa r e n o t c o n s i d e r e d b e c a u s e a c r it ic a l a s s u m p t i o n o fa l l t h e a n a l y s i s d e s c r i b e d a b o v e i s t h a t t h e e r r o rm a t r i x i s tr u l y r e p r e s e n t a t i v e o f t h e e n t i r e c l a ss i -f i ca t io n . I f t h e m a t r i x i s i m p r o p e r l y g e n e r a t e d ,t h e n a l l t h e a n a l y s i s is m e a n i n g l e s s . T h e r e f o r e , t h e


8/12

42 Congalton

f o l l o w i n g f a c t o r s m u s t b e c o n s i d e r e d : g r o u n d d a t aco l l ec t io n , c l a s s i f i ca t io n sch em e , sp a t i a l au to co r r e -l a t i o n , s a m p l e s i z e , a n d s a m p l i n g s c h e m e . E a c h o ft h e s e f a c t o r s p r o v i d e e s s e n t i a l i n f o r m a t i o n f o r t h ea s s e s s m e n t a n d f a i l u r e t o c o n s i d e r e v e n o n e o ft h e m c o u l d l e a d t o s e r i o u s s h o r t c o m i n g s i n t h ea s s e s s m e n t p r o c es s .

G r o u n d D a t a C o l l e c t i o nI t i s o b v i o u s t h a t i n o r d e r t o a d e q u a t e l y a s s e s s t h ea c c u r a c y o f t h e r e m o t e l y s e n s e d c l a ss i fi c a ti o n , a c-c u r a t e g r o u n d , o r r e f e r e n c e d a t a m u s t b e c o l-l e c te d . H o w e v e r , t h e a c c u r a c y o f t h e g r o u n d d a t ai s r a r e l y k n o w n n o r i s t h e l e v e l o f e f f o rt n e e d e d t oc o l l e c t t h e a p p r o p r i a t e d a t a c l e a r l y u n d e r s t o o d .D e p e n d i n g o n t h e l e v e l o f d e t a i l i n t h e c l a s si fi c a -t i o n ( i . e . , c l a s s i f i ca t io n sch em e) , co l l ec t in g r e f e r -e n c e d a t a c a n b e a v e r y d i ff i c ul t t as k . F o r e x a m p l e ,i n a s im p l e c l a s s i fi c a ti o n s c h e m e t h e r e q u i r e d l e v e lo f d e t a i l m a y b e o n l y t o d i s t i n g u i s h r e s i d e n t i a lf r om c o m m e r c i a l a re a s. C o l l e c t in g r e f e r e n c e d a t am a y b e a s s i m p l e a s o b t a i n i n g a c o u n t y z o n i n gm a p . H o w e v e r , a m o r e c o m p l e x f o r e s t c l a s s i f i c a -t io n s c h e m e m a y i n v o l v e c o ll e c ti n g r e f e r e n c e d a t af o r n o t o n ly sp ec ie s o f t r ee , b u t s i ze c la s s , an dc r o w n c l o s u r e a s w e l l . S i z e c l a s s i n v o l v e s m e a s u r -i n g t h e d i a m e t e r s o f t r e e s a n d t h e r e f o r e a g r e a tm a n y t re e s m a y h a v e to b e m e a s u r e d t o e s t im a t et h e s i z e c la s s f o r e a c h p i xe l . C r o w n c l o s u r e i s e v e nm o r e d i f f ic u l t t o m e a s u r e . T h e r e f o r e , i n t h is c a s e ,c o l l e c t i n g a c c u r a t e r e f e r e n c e d a t a c a n b e d i f f i c u l t .

A t r a d i t i o n a l s o l u t i o n t o t h i s p r o b l e m h a s b e e nf o r t h e p r o d u c e r a n d u s e r o f t h e c l a ss i f ic a t io n t oa s s u m e t h a t s o m e r e f e r e n c e d a t a s e t i s c o r r e c t . F o re x a m p l e , t h e r e s u l t s o f s o m e p h o t o i n t e r p r e t a t i o no r a e ri a l r e c o n n a i s s a n c e m a y b e u s e d a s t h e r e f e r -e n c e d a t a . H o w e v e r , e r r o r s i n t h e i n t e r p r e t a t i o nw o u l d t h e n b e b l a m e d o n t h e d i g i t a l c l a s s i f i c a t i o n ,t h e r e b y w r o n g l y l o w e r i n g t h e d i g i t a l c l a s s i f i c a t i o na c c u r a c y . I t i s e x a c t l y t h is p r o b l e m t h a t h a s c a u s e dt h e l a c k o f a c c e p t a n c e o f d ig i t a l s a t e l li t e d a t a f o rm a n y a p p l i c a t i o n s . A l t h o u g h n o r e f e r e n c e d a t a s e tm a y b e c o m p l e t e l y a c c u r a t e , it is i m p o r t a n t t h a tt h e r e f e r e n c e d a t a h a v e h i g h a c c u r a c y o r e l s e i t isn o t a f a i r a s se ssm en t . Th e r e f o r e , i t i s c r i t i c a l t h a tt h e g r o u n d o r r e f e r e n c e d a t a c o l l e c t i o n b e c a r e -f u ll y c o n s i d e r e d i n a n y a c c u r a c y a s s e s sm e n t . M u c hw o r k i s y e t t o b e d o n e t o d e t e r m i n e t h e p r o p e rl e v e l o f e f f o rt a n d c o l l e c t i o n t e c h n i q u e s n e c e s s a r yto p r o v id e th i s v i t a l i n f o r m a t io n .

C lass i f i ca t ion SchemeW h e n p l a n n i n g a p r o j e c t in v o l vi n g r e m o t e l y s e n s e dd a ta , i t is v e r y im p o r t a n t t h a t su f f i c i en t e ff o r t b eg i v e n t o t h e c l a s s i f i c a t i o n s c h e m e t o b e u s e d .I n m a n y i n s t a n c e s , t h is s c h e m e i s a n e x i s t i n go n e s u c h a s t h e A n d e r s o n c l a s s i f i c a t i o n s y s t e m( An d er so n e t a l . , 1 9 76 ) . I n o th e r ca se s , t h e c l a s s i -f i ca t io n s c h e m e i s d i c t a t e d b y t h e o b j e c t i v e s o f t h ep r o j e c t o r b y t h e s p e c i fi c a t io n s o f t h e c o n t r a c t. I na l l s i t u a t i o n s a f e w s i m p l e g u i d e l i n e s s h o u l d b ef o l lo wed . F i r s t o f a l l, an y c l a ss i f ica t io n sch em es h o u l d b e m u t u a l l y e x c l u s i v e a n d t o t a l l y e x h a u s -t i v e . I n o t h e r w o r d s , a n y a r e a t o b e c l a s s i f i e ds h o u l d f a l l i n t o o n e a n d o n l y o n e c a t e g o r y o r c l as s .I n a d d i t i o n , e v e r y a r e a s h o u l d b e i n c l u d e d i n t h ec l a ss i fi ca t io n . F in a l ly , i f p o ss ib l e , i t i s v e r y ad v an -t ag eo u s to u se a c l a s s i f i ca t io n sch em e th a t i s h i e r -a r c h i c a l in n a t u r e . I f s u c h a s c h e m e i s u s e d , c e r -t a i n c a t e g o r i e s w i t h i n t h e c l a s si f ic a t io n s c h e m e c a nb e c o l l a p s e d t o f o r m m o r e g e n e r a l c a t e g o r i e s . T h i sa b i l i t y i s e s p e c i a l l y i m p o r t a n t w h e n t r y i n g t o m e e tp r e d e t e r m i n e d a c c u r a c y s t a n d a r d s . T w o o r m o r ed e t a i l e d c a t eg o r i e s o f l o w e r t h a n t h e m i n i m u mr e q u i r e d a c c u r a c y m a y n e e d t o b e g r o u p e d t o-g e t h e r ( c o l l a p s e d ) t o f o r m a m o r e g e n e r a l c a t e g o r yt h a t e x c e e d s t h e m i n i m u m a c c u r a c y r e q u i r e m e n t .F o r e x a m p l e , i t m a y b e i m p o s s i b l e t o s e p a r a t ei n t e r i o r l i v e o a k f r o m c a n y o n l i v e o a k . T h e r e f o r e ,t h e s e t w o c a t e g o r i e s m a y h a v e t o b e c o l l a p s e d t of o r m a li v e o a k c a t e g o r y to m e e t t h e r e q u i r e da c c u r a c y s t a n d a r d .

B e c a u s e t h e c l a s s i f i c a t i o n s c h e m e i s s o i m p o r -t an t , n o w o r k s h o u l d b e g i n o n t h e r e m o t e l y s e n s e dd a t a u n ti l t h e s c h e m e h a s b e e n t h o r o u g h l y re -v i e w e d a n d a s m a n y p r o b l e m s a s p os s i bl e i d e n t i -f i ed . I t i s e sp ec ia l ly h e lp f u l i f t h e ca t eg o r i e s i n t h es c h e m e c a n b e l o g i c al l y e x p l a i n e d . T h e d i f f e r e n c eb e t w e e n D o u g l a s f i r a n d P o n d e r o s a p i n e i s e a sy tou n d e r s t a n d ; h o w e v e r , t h e d i f f e r e n c e b e t w e e nD e n s i t y C l a ss 3 ( 5 0 - 7 0 % c r o w n c l o s u re ) a n d D e n -s i ty C l a ss 4 ( > 7 0 % c r o w n c l o s u r e ) m a y n o t b e . I nf a c t , m a n y t i m e s t h e s e c l a s s e s a r e r a t h e r a r t i f i c i a la n d o n e c a n e x p e c t t o f i n d c o n f u s i o n b e t w e e n af o r e s t s t a n d w i t h a c r o w n c l o s u r e o f 6 7 % t h a tb e l o n g s i n C l as s 3 a n d a s t a n d o f 7 3 % t h a t b e l o n g si n C l a s s 4 . S o m e t i m e s t h e r e i s l i t t l e t h a t c a n b ed o n e ab o u t t h e a r t i f i c i a l d e l in ea t io n s in t h e c l a ss i -f i c a t i o n s c h e m e ; o t h e r t i m e s t h e s c h e m e c a n b em o d i f i e d t o b e t t e r r e p r e s e n t n a t u r a l b r e a k s . H o w -ev e r , tZa ilur e to t r y to u n d e r s t an d th e c l a ss i f i ca tio n


9/12

Review: Assessing Classification Accuracy 4 3

s c h e m e f r o m t h e e v e r y b e g i n n i n g w i l l c e r t a i n l yr e s u l t i n a g r e a t l o s s o f t i m e a n d m u c h f r u s t r a t i o ni n t h e e n d .

Spatial AutocorrelationS p a t i a l a u t o e o r r e l a t i o n i s s a i d t o o c c u r w h e n t h ep r e s e n c e , a b s e n c e , o r d e g r e e o f a c e r t a i n c h a r ac -t e r i s t i c a f f e c t s t h e p r e s e n c e , a b s e n c e , o r d e g r e e o ft h e s a m e c h a r a c t e r i s ti c i n n e i g h b o r i n g u n i t s ( C l if fa n d O r d , 1 9 7 3 ). T h i s c o n d i t i o n is p a r t i c u l a r ly i m -p o r t a n t i n a c c u r a c y a s s e s s m e n t i f a n e r r o r i n ac e r t a i n l o c a t i o n c a n b e f o u n d t o p o s i t i v e l y o r n e g a -t i v e l y i n f l u e n c e e r r o r s i n s u r r o u n d i n g l o c a t i o n s( C a m p b e l l , 1 9 8 1) . W o r k b y C o n g a l t o n (1 9 8 8 a ) o nL a n d s a t M S S d a t a f r o m t h r e e a r e a s o f v a r y i n gspa t ia l d iver s i ty ( i . e . , an ag r icu l tu re , a r ange , and af o r e s t s i t e ) s h o w e d a p o s i t i v e i n f l u e n c e a s m u c h a s3 0 p i x e l s ( o v e r 1 m i l e ) a w a y . T h e s e r e s u l t s a r ee x p l a i n a b le i n a n a g r i c u lt u r a l e n v i r o n m e n t w h e r ef i e l d s i z e s a r e l a r g e a n d t y p i c a l m i s c l a s s i f i c a t i o nw o u l d b e t o m a k e a n e r r o r i n l a b e l i n g t h e e n t i r ef i e l d . H o w e v e r , t h e s e r e s u l t s a r e m o r e s u r p r i s i n gf o r r a n g e l a n d a n d f o r e s t e d s i t e s . S u r e l y t h e s e r e -s u i t s s h o u l d a f f e c t t h e s a m p l e s i z e a n d e s p e c i a l l yt h e s a m p l i n g s c h e m e u s e d i n a c c u r a c y a s s e ss m e n t ,e s p e c i a l l y i n t h e w a y t h i s a u t o c o r r e l a t i o n a f f e c t st h e a s s u m p t i o n o f s a m p l e i n d e p e n d e n c e . T h i s a u -t o c o r r e la t i o n m a y t h e n b e r e s p o n s i b l e f o r p e r i o d i c -i t y in t h e d a t a t h a t c o u l d e f f e c t t h e r e s u l t s o f a n yt y p e o f s y s t e m a t i c s a m p l e . I n a d d i t i o n , t h e s i z e o ft h e c l u s t e r u s e d i n c l u s t e r s a m p l i n g w o u l d a l s o b ee f f e c t e d b e c a u s e e a c h n e w p i x e l w o u l d n o t b ec o n t r ib u t i n g i n d e p e n d e n t i n fo r m a ti o n.

Sample SizeS a m p l e s i z e i s a n o t h e r i m p o r t a n t c o n s i d e r a t i o nw h e n a s s e s si n g t h e a c c u r a c y o f r e m o t e l y s e n s e dd a ta . E a c h s a m p l e p o i n t c o l l e c t e d is e x p e n s i v e a n dt h e r e f o r e s a m p l e s i z e m u s t b e k e p t t o a m i n i m u ma n d y e t i t i s c r i t i c a l t o m a i n t a i n a l a r g e e n o u g hs a m p l e s i z e s o t h a t a n y a n a l y s i s p e r f o r m e d i ss t a t i s t i c a l l y v a l i d . O f a l l t h e c o n s i d e r a t i o n s d i s -c u s s e d i n t h i s p a p e r , t h e m o s t h a s p r o b a b l y b e e nw r i t t e n a b o u t s a m p l e s i z e . M a n y r e s e a r c h e r s , n o -t a b ly v a n G e n d e r e n a n d L o c k (1 9 7 7) , H a y ( 1 97 9 ),H o r d a n d B r o o n e r (1 9 7 6 ) , R o s e n f i e l d e t a l. ( 1 98 2 ) ,a n d C o n g a l t o n ( 19 8 8 b ), h a v e p u b l i s h e d e q u a t i o n sa n d g u i d e l i n e s f o r c h o o s i n g t h e a p p r o p r i a t e s a m -p l e s i ze . T h e m a j o r i t y o f r e se a r c h e r s h a v e u s e d a n

e q u a t i o n b a s e d o n t h e b i n o m i a l d i s tr i b u t i o n o r t h en o r m a l a p p r o x i m a t i o n t o t h e b i n o m i a l d i s t r i b u t io nt o c o m p u t e t h e r e q u i r e d s a m p l e s i z e . T h e s e t e c h -n i q u e s a r e s ta t is t ic a l ly s o u n d f o r c o m p u t i n g t h es a m p l e s i z e n e e d e d t o c o m p u t e t h e o v e r a l l a c c u -r a c y o f a c l a ss i fi c a ti o n o r e v e n t h e o v e r a l l a c c u r a c yo f a s i n g l e c a t e g o r y . T h e e q u a t i o n s a r e b a s e d o nt h e p r o p o r t i o n o f c o r r e c t l y c l a s s if i e d s a m p l e s( p i x e l s , c l u s t e r s , o r p o l y g o n s ) a n d o n s o m e a l l o w -a b l e e r r o r . H o w e v e r , t h e s e t e c h n i q u e s w e r e n o td e s i g n e d t o c h o s e a s a m p l e s i z e f o r f i l l i n g i n a ne r r o r m a t r ix . I n t h e c a s e o f a n e r r o r m a t ri x , i t isn o t s i m p l y a m a t t e r o f c o r r e c t o r i n c o r r e c t . I t i s am a t t e r o f w h i c h e r r o r o r, i n o t h e r w o r d s , w h i c hc a t e g o r i e s a r e b e i n g c o n f u s e d . S u f f i c i e n t s a m p l e sm u s t b e a c q u i r e d t o b e a b l e t o a d e q u a t e l y r e p r e -s e n t t h i s c o n f u s i o n . T h e r e f o r e , t h e u s e o f t h e s et e c h n i q u e s f o r d e t e r m i n i n g t h e s a m p l e s i z e f o r a ne r r o r m a t r i x i s n o t a p p r o p r i a t e . F i t z p a t r i c k - L i n s( 1 98 1 ) u s e d t h e n o r m a l a p p r o x i m a t i o n e q u a t i o n t oc o m p u t e t h e s a m p l e s i z e fo r a s s e ss i n g a l a n d u s e /l a n d c o v e r m a p o f T a m p a , F l o r i da . T h e r e s u l ts o ft h e c o m p u t a t i o n s h o w e d t h a t 3 1 9 s a m p l e s n e e d e dt o b e t a k e n f o r a c l a s s i f i c a t i o n w i t h a n e x p e c t e da c c u r a c y o f 8 5 % a n d a n a l l o w a b l e e r r o r o f 4 % . S h ee n d e d u p t a k i n g 3 54 s a m p l e s a n d f i l li n g i n a ne r r o r m a t r i x t h a t h a d 3 0 c a t e g o r i e s i n i t ( i . e . , am a t r ix o f 3 0 r o w s 3 0 c o l u m n s o r 9 0 0 p o s s i b l ece l l s ) . A l though th i s s am ple s i ze i s su f f i c ien t fo rc o m p u t i n g o v e r a l l a c c u r a c y , i t i s o b v i o u s l y m u c ht o o s m a l l t o b e r e p r e s e n t e d i n a m a t r i x . O n l y 3 5 o ft h e 9 0 0 c e l l s h a d a v a l u e g r e a t e r t h a n z e r o . O t h e rr e s e a r c h e r s h a v e u s e d t h e e q u a t i o n t o c o m p u t e t h es a m p l e s i z e f o r e a c h c a t e g o r y . A l t h o u g h r e s u l t i n gi n a l a r g e r s a m p l e , t h e e q u a t i o n s t i l l d o e s n o ta c c o u n t f o r t h e c o n f u s i o n b e t w e e n c a t e g o ri e s .

B e c a u s e o f t h e l a r g e n u m b e r o f p i x el s i n ar e m o t e l y s e n s e d i m a g e , t r a d i t i o n a l t h i n k i n g a b o u ts a m p l i n g d o e s n o t o f t e n a p p l y . E v e n a o n e - h a l fp e r c e n t s a m p l e o f a s i n g le T h e m a t i c M a p p e r s c e n ec a n b e o v e r 3 0 0 , 0 0 0 p i x e ls . N o t a l l a s s e s s m e n t s a r ep e r f o r m e d o n a p e r p i x e l b a s i s , b u t t h e s a m er e l at i v e a r g u m e n t h o l d s t r u e i f t h e s a m p l e u n i t i s ac l u s t e r o f p i x e l s o r a p o l y g o n . T h e r e f o r e , p r a c t i c a lc o n s i d e r a t i o n s m o r e o f t e n d i c t a t e t h e s a m p l e s i z es e l e c t i o n . A b a l a n c e b e t w e e n w h a t i s s t a t i s t i c a l l ys o u n d a n d w h a t i s p r a c t i c a l l y a t t a i n a b l e m u s t b ef o u n d . I t ha s b e e n m y e x p e r i e n c e t h a t a g o o d r u l eo f t h u m b s e e m s t o b e c o l le c t in g a m i n i m u m o f 5 0s a m p l e s f o r e a c h v e g e t a t i o n o r l a n d u s e c a t e g o r yi n t h e e r r o r m a t ri x . I f t h e a r e a i s e s p e c i a l l y l a r g e


10/12

44 Congalton

( i .e . , more than a mi l l ion acres) or the classi f ica-t io n h a s a l a r g e n u m b e r o f v e g e t a t i o n o r l a n d u s eca tegor i es ( i .e . , m ore than 12 ca tegor i es ) , t he m in i -m u m n u m b e r o f sa m p l es s h o u l d b e i n c r e a s e d to 7 5o r 1 00 s a m p l e s p e r c a t e g o ry . T h e n u m b e r o f s a m -p l e s f o r e a c h c a t e g o r y c a n a l s o b e a d j u s t e d b a s e do n t h e r e l a ti v e i m p o r t a n c e o f t h a t c a t e g o r y w i t h i nt h e o b j e c t iv e s o f t h e m a p p i n g o r b y t h e i n h e r e n tvar i ab i li t y wi th in each o f the ca t ego r i es . Some-t imes i t i s be t t e r t o concen t r a t e the sampl ing ont h e c a t e g o r i e s o f i n t e r e s t a n d i n c r e a s e t h e i r n u m -b e r o f s am p l es w h i l e r e d u c i n g th e n u m b e r o fsamples t aken in the l ess impor t an t ca t egor i es .Al so i t may be use fu l t o t ake f ewer samples inca tegor i es tha t show l i t t l e va r i ab i l i t y such as wate ro r fo r es t p l an ta t ions and increase the sampl ing inthe ca t egor i es tha t a r e more var i ab le such as un-even-aged fo res t s o r r ipa r i an a r eas . Aga in , t heob jec t he re i s t o ba lance the s t a t i s t i ca l r ecommen-d a t i o n s i n o r d e r t o g e t a n a d e q u a t e s a m p l e t og e n e r a t e a n a p p r o p r i a t e e r r o r m a t r i x w i t h t h e t i m e ,cos t , and p rac t i ca l l imi t a t ions assoc ia t ed wi th anyv i a b le r e m o t e s e n s i n g p r o j ec t .

Sampling SchemeIn add i t ion to the cons idera t ions a l r eady d i s -c u s s e d , s a m p l i n g s c h e m e i s a n i m p o r t a n t p a r t o fa n y a c c u r a c y a s s e s s m e n t . S e l e c t i o n o f t h e p r o p e rscheme i s abso lu te ly c r i t i ca l t o genera t ing an e r ro rmat r ix tha t i s r ep res en ta t iv e o f the en t i r e c l ass i fi edi m a g e . P o o r c h o i c e i n s a m p l i n g s c h e m e c a n r e s u l tin s ign i f i can t b i ases be ing in t roduced in to thee r r o r m a t r ix w h i c h m a y o v e r o r u n d e r e s t i m a t e t h et r u e a c c u r a c y . I n a d d i t io n , u s e o f t h e p r o p e r s a m -p l i n g s c h e m e m a y b e e s s e n t i a l d e p e n d i n g o n t h eana lys i s t echn iques to be app l i ed to the e r ro rmatr ix .

M a n y r e s e a r c h e r s h a v e e x p r e s s e d o p i n i o n sa b o u t t h e p r o p e r s a m p l i n g s c h e m e t o u s e ( e . g . ,Hord and Brooner , 1976 ; G inevan , 1979 ; Rhode ,1978; Fi tzpat r ick-Lins, 1981) . These opinions varyg r e a t l y a m o n g r e s e a r c h e r s a n d i n c l u d e e v e r y t h i n gf rom s imple r andom sampl ing to s t r a t i f i ed sys t em-a t i c una l igned sampl ing . Desp i t e a l l t hese op in -ions , ve ry l i t t l e work has ac tua l ly been per fo rmedi n t h i s a r e a . C o n g a l t o n ( 1 9 8 8 b ) p e r f o r m e d s a m -p l ing s imula t ions on th r ee spa t i a l ly d iver se a r easa n d c o n c l u d e d t h a t i n a l l c a s e s s i m p l e r a n d o mw i t h o u t r e p l a c e m e n t a n d s t r a t i f i e d r a n d o m s a m -p l ing p rov ided sa t i s f ac to ry r esu l t s . Desp i t e the

n ice s t a t is t i ca l p roper t i es o f s imple r an dom sam-p l ing , t h i s sampl ing scheme i s no t a lways tha tp r a c t i c a l t o a p p l y . S i m p l e r a n d o m s a m p l i n g t e n d st o u n d e r s a m p l e s m a l l b u t p o s s i b l y v e r y i m p o r t a n ta r eas un less the sam ple s i ze i s s ign if i can tly in -c r eased . For th i s r eason , s t r a t i f i ed r andom sam-p ii n g is r e c o m m e n d e d w h e r e a m i n i m u m n u m b e rof samples a r e se l ec t ed f rom ea ch s t r a t a ( i. e .,c a t e go r y ) . E v e n s t ra t if ie d r a n d o m s a m p l i n g c a n b es o m e w h a t i m p r a c t i c a l b e c a u s e o f h a v i n g t o c o l l e c tg r o u n d i n f o rm a t i o n f o r t h e a c c u r a c y a s s e s s m e n t a tr a n d o m l o c a t i o n s o n t h e g r o u n d . T h e p r o b l e m sw i t h r a n d o m l o c a t i o n s a r e t h a t t h e y c a n b e i np laces wi th very d i f f i cu l t access and they can on lybe se l ec t ed a f t e r t he c l ass i f i ca t ion has been per -fo rmed . Th i s l imi t s t he accuracy assessm ent da tat o b e i n g c o l l e c t e d l a te i n t h e p r o j e c t i n s t e a d o f i ncon junc t ion wi th the t r a in ing da ta co l l ec t ion ,the re by increa s ing the cos t s o f t he p ro jec t . I na d d i t i o n , i n s o m e p r o j e c t s t h e t i m e b e t w e e n t h ep r o j e c t b e g i n n i n g a n d t h e a c c u r a c y a s s e s s m e n tmay be so long as to cause t empora l p rob lems inc o l l e c t i n g g r o u n d r e f e r e n c e d a t a . I n o t h e r w o r d s ,t h e g r o u n d m a y c h a n g e ( i. e. , t h e f o r e st h a r v e s t e d )b e t w e e n t h e t i m e t h e p r o j e c t i s s t a r t e d a n d t h ea c c u r a c y a s s e s s m e n t i s b e g u n .

T h e r e f o r e , s o m e s y s t e m a t i c a p p r o a c h w o u l dcer t a in ly he lp make th i s g round co l l ec t ion e f fo r tmore e f f i c i en t by mak ing i t eas i e r t o loca te thep o i n ts o f t h e g r o u n d a n d a l l o w i n g d a t a t o b eco l l ec t ed s imul t aneous ly fo r t r a in ing and assess -m e n t . H o w e v e r , r e s u l ts o f C o n g a l t o n ( 1 9 8 8a )s h o w e d t h a t p e r i o d i c i t y i n t h e e r r o r s a s m e a s u r e db y t h e a u t o c o r r e la t i o n a n al y s is c o u l d m a k e t h e u s eof sys t emat i c sampl ing r i sky fo r accuracy assess -m e n t . T h e r e f o r e , p e r h a p s s o m e c o m b i n a t i o n o fr a n d o m a n d s y s t e m a t i c s a m p l i n g w o u l d p r o v i d et h e b e s t b a l a n c e b e t w e e n s t a t i s t i c a l v a l i d i t y a n dprac t i ca l app l i ca t ion . Such a sys t em may employs y s t e m a t i c s a m p l i n g t o c o l l e c t s o m e a s s e s s m e n td a t a e a r l y i n a p r o j e c t w h i l e r a n d o m s a m p l i n gwi th in s t r a t a wou ld be used a f t e r t he c l ass i f i ca t ioni s c o m p l e t e d t o a s s u r e t h a t e n o u g h s a m p l e s w e r ec o l l e c t e d f o r e a c h c a t e g o r y a n d t o m i n i m i z e a n yper iod ic i ty in the da ta .I n a d d i t i o n t o t h e s a m p l i n g s c h e m e s a l r e a d yd i s c u s s e d , c l u s t e r s a m p l i n g h a s a l s o b e e n f r e -q u e n t l y u s e d i n a s s e s s i n g t h e a c c u r a c y o f r e m o t e l ysensed da ta , e spec ia l ly to co l l ec t i n fo rmat ion onm a n y p i x e l s v e r y q u i c k l y . H o w e v e r , c l u s t e r s a m -p i i n g m u s t b e u s e d i n t e l l i g e n t l y . S i m p l y u s i n g


11/12

R eview : Assess ing C lass if i ca tion Accu racy 4 5

v e r y l a r g e c l u s t e r s i s n o t a v a l i d m e t h o d o f c o l l e c t -i n g d a t a b e c a u s e e a c h p i x e l is n o t i n d e p e n d e n t o ft h e o t h e r a n d a d d s v e r y l i tt l e i n f o rm a t i o n t o t h ec l u s te r . C o n g a l t o n (1 9 8 8 b ) r e c o m m e n d e d th a t n oc l u s t e r s l a r g e r t h a n 1 0 p i x e l s a n d c e r t a i n l y n o tl a r g e r t h a n 2 5 p i x e ls b e u s e d b e c a u s e o f t h e l a c ko f in f o r m a ti o n a d d e d b y e a c h p i x e l b e y o n d t h e s ec lus te r s i zes .

F i n a l l y , s o m e a n a l y t i c t e c h n i q u e s a s s u m e t h a tc e r t a i n s a m p l i n g s c h e m e s w e r e u s e d t o o b t a i n t h ed a t a . F o r e x a m p l e , u s e o f t h e K a p p a a n a l y s is a s -s u m e s a m u l t i n o m i a l sa m p l i n g m o d e l . O n l y s i m p l er a n d o m s a m p l i n g c o m p l e t e l y s a ti sf ie s t h is a s s u m p -t io n . T h e e f fe c t o f u s i n g a n o t h e r o f t h e s a m p l i n gs c h e m e s d i s c u s s e d h e r e i s u n k n o w n . A n i n t e r e s t -i n g p r o j e c t w o u l d b e t o t e s t t h e e f f e c t o n t h eK a p p a a n a ly s is o f u s i n g a s a m p l i n g s c h e m e o t h e rt h a n s i m p l e r a n d o m s a m p l i n g . I f t h e e f fe c t i sf o u n d t o b e s m a ll , t h e n t h e s c h e m e m a y b e a p p r o -p r i a t e t o u s e w i t h i n t h e c o n d i t i o n s d i sc u s s e d a b o v e .I f t h e e f f e c t i s f o u n d t o b e l a r g e , t h e n t h a t s a m -p l i n g s c h e m e s h o u l d n o t b e u s e d t o p e rf o r m K a p p aa n a ly s is . T o c o n c l u d e t h a t s o m e s a m p l i n g s c h e m e sc a n b e u s e d f o r d e s c r i p t i v e t e c h n i q u e s a n d o t h e r sf o r a n a l y t i c a l t e c h n i q u e s s e e m s i m p r a c t i c a l . A c c u -r a c y a s s e s s m e n t i s e x p e n s i v e a n d n o o n e i s g o i n gt o c o l l e c t d a t a f o r o n l y d e s c r i p t i v e u s e . E v e n t u a l l y ,s o m e o n e w i l l u s e t h a t m a t r i x f o r s o m e a n a l y t i c a lt e c h n i q u e .

C O N C L U S I O N ST h i s p a p e r h a s r e v i e w e d t h e f a c t o r s a n d t e c h -n i q u e s t o b e c o n s i d e r e d w h e n a s s e s s in g t h e a c c u -r a c y o f c la s s if i c a ti o n s o f r e m o t e l y s e n s e d d a t a . T h ew o r k h a s r e a l l y j u s t b e g u n . T h e f a c t o r s d i s c u s s e dh e r e a r e c e r t a i n l y n o t f u l l y u n d e r s t o o d . T h e b a s i ci s s ue s o f s a m p l e s i z e a n d s a m p l i n g s c h e m e h a v en o t b e e n r e s o l v e d . S p a t i a l a u t o c o r r e l a t i o n a n a l y s i sh a s r a r e ly b e e n a p p l i e d t o a n y s t u d y . E x a c tl y w h a tc o n s t i t u t e s g r o u n d o r r e f e r e n c e d a t a a n d t h e l e v e lo f ef fo r t n e e d e d t o c o l l ec t i t m u s t b e s t u d i e d .R e s e a r c h n e e d s t o c o n t i n u e i n o r d e r t o b a l a n c ew h a t i s s t a ti s t ic a l l y v a l i d w i t h i n t h e r e a l m o f p r a c -t i c a l a p p l i c a t i o n . T h i s n e e d b e c o m e s i n c r e a s i n g l yi m p o r t a n t a s t e c h n i q u e s a r e d e v e l o p e d t o u s e r e -m o t e l y s e n s e d d a t a o v e r l a r g e r e g i o n a l a n d g l o b a ld o m a i n s . W h a t i s v a l i d a n d p r a c t i c a l o v e r a s m a l la r e a m a y n o t a p p l y t o r e g i o n a l o r g l o b a l p r o j e c t s .U p t o n o w , t h e l i tt l e e x p e r i e n c e w e h a v e h a s b e e n

o n r e l a t i v e l y s m a l l r e m o t e s e n s i n g p r o j e c t s . H o w -e v e r , t h e r e i s a n e e d t o u s e r e m o t e s e n s i n g f o rm u c h l a r g e r p r o j e c t s s u c h a s m o n i t o r i n g g lo b a lw a r m i n g , d e f o r e s t a t i o n , a n d e n v i r o n m e n t a l d e g r a -d a t io n . W e d o n o t k n o w a l l t h e p r o b l e m s t h a t w i lla r i s e w h e n d e a l i n g w i t h s u c h l a r g e a r e a s . C e r -t a i n l y , t h e t e c h n i q u e s d e s c r i b e d m u s t b e e x t e n d e da n d r e f i n e d to b e t t e r m e e t t h e s e a s s e s s m e n t n e e d s .I t i s c ri t ic a l t h a t t h i s w o r k a n d t h e u s e o f q u a n t i t a -t iv e a n a l y si s o f r e m o t e l y s e n s e d d a t a c o n t i n u e . W eh a v e s u f f e re d t o o l o n g b e c a u s e o f t h e o v e r s e ll o ft h e t e c h n o l o g y a n d t h e u n d e r u t i l i z a t i o n o f a n yq u a n t i t a t i v e a n a l y s i s e a r l y i n t h e d i g i t a l r e m o t es e n s i n g e r a . P a p e r s s u c h a s M e y e r a n d W e r t h( 1 9 9 0 ) t h a t s t a t e t h a t t h e d i g i t a l r e m o t e s e n s i n g i sn o t a v i a b l e t o o l f o r m o s t r e s o u r c e a p p l i c a t i o n sc o n t i n u e t o d e m o n s t r a t e t h e p r o b l e m s w e h a v ec r e a t e d b y n o t q u a n t i t a t iv e l y d o c u m e n t i n g o u rw o r k . W e m u s t p u t a s i d e t h e d a y s o f a c a s u ala s s e s s m e n t o f o u r c l a ss i fi c a ti o n . " I t l o o k s g o o d " i sn o t a v a l i d a c c u r a c y s t a t e m e n t . A c l a s s i f i c a t i o n i sn o t c o m p l e t e u n t il i t h a s b e e n a s s es s e d . T h e n a n do n l y t h e n c a n t h e d e c i s i o n s m a d e b a s e d o n t h a ti n f o r m a t i o n h a v e a n y v a l i d i t y .

I n a d d i t i o n , w e m u s t n o t f o r g e t t h a t r e m o t e l ys e n s e d d a t a i s j u s t a s m a l l s u b s e t o f sp a t i a l d a t ac u r r e n t l y b e i n g u s e d i n g e o g r a p h i c i n f o r m a t i o ns y s t e m s ( G I S ) . T h e t e c h n i q u e s a n d c o n s i d e r a t i o n sd i s c u s s e d h e r e n e e d t o b e a p p l i e d o v e r a ll s p a ti a ld a t a . T e c h n i q u e s d e v e l o p e d f o r o t h e r s p a t i a l d a t an e e d t o b e t e s t e d f o r u s e w i t h r e m o t e l y s e n s e dd a ta . T h e w o r k h a s j u s t b e g u n , a n d i f w e a r e g o i n gt o u s e s p a t i a l d a t a t o h e l p u s m a k e d e c i s i o n s , a n dw e s h o u l d , t h e n w e m u s t k n o w a b o u t t h e a c c u r a c yo f t h is i n f o r m a t i o n .The author would like to thank Greg Biging and Craig Olsonfor their helpful reviews of this paper. Thanks also to the twoanonymous reviewers whose comments significantly improvedthis manuscript.

R E F E R E N C E SAnderson, J. R., Ha rdy, E. E ., Roa ch, J. T., and W itmer, R. E.(1976), A l and use and lan d cover classification system foruse with rem ote sensor data, U.S. Geol. Survey Prof. Paper964, 28 pp.Aronoff, Stan (1985), The min imu m accuracy value as anindex of classification accuracy, Photogramm. Eng. RemoteSens. 51(1):99-111.Biging, G. and Conga lton, R. (1989 ), Advan ces in forestinventory using adv anced digital imagery, in Proceedings


12/12

46 Congalton

of Global Natural Research Monitoring and Assessments':Preparing fo r the 21st Century, Venice, I ta ly , Sep tember ,Vol . 3 , pp . 1241-1249 .

Bishop, Y., Fienberg, S., and Holland, P. (1975), DiscreteMultivaria te Analysis--Theory and Practice, MIT Press ,Cambridge, MA, 575 pp .

Campbell , J. (1981), Spatial autocorrelation effects upon theaccuracy of superv ised class if ication of land cover , Ph o -togramm. Eng. Remote Sens. 47(3 ) : 355-363 .

Campbell , J. (1987), Introduction to Remote Sensing, Guil fordPress, N ew Y ork, 551 pp.

Chuvieco, E., and Congalton, R. (1988), Using cluster analysisto improv e the selection o f training statist ics in classifyingremotely sensed data , Photogramm. Eng. Remote Sens.54(9) : 1275-128 1 .

Cliff, A. D., and Ord, J. K. (1973), Spatia l Autocorrelation,Pion, London, 178 pp.

Cohen , J . (1960), A coeff icien t o f agree me nt fo r nominalscales, Educ. Psychol. Measurement 20 (1 ) : 37 -46 .Congalton, R. G. (1988a), Using spatial autocorrelation analy-

s is to exp lore er rors in maps generated f rom remotelysensed data , Photogramm. Eng. Remote Sens . 54(5):5 8 7 - 5 9 2 .

Congal ton , R . G. (1988b), A compar ison of sampl ing schem esused in generat ing er ror nmtr ices fo r assess ing the accu-r acy o f map s g en era t ed f ro m remo te ly sen sed d a ta , Pho-togramnt. Eng. Remote Sens. 54(5 ) : 59 3 -60 0 .

Congalton, R. G., Oderwald, R. G., and Mead, R. A. (1983),Asse ssing Lands at classification acc urac y using discretemultivariate statist ical techniques, Photogramm. Eng. Re-mote Sens . 49(12) :1671-1678 .

Fi tzpat r ick-Lins , K. (1981), Com par ison of sampl ing proce-dures and data analysis fo r a land-use and land-cover map ,Photogramm. Eng. Remote Sens. 47(3 ) : 343 -351 .

Ginevan , M. E. (1979) , Test ing land-use map accuracy : an-other look, Photogramm. Eng. Remote Sens . 45(10):1 371 -1 377 .

Hay , A. M. (1979) , Sampl ing designs to tes t land-use mapaccuracy , Photogramm. Eng. Remote Sens. 45(4 ) : 529 -533 .

Hord , R . M., and Brooner , W. (1976) , Land use map accuracycriteria, Photogramm. Eng. Remote Sens. 42(5 ) : 671 -677 .

Hudson , W., and Ram m, C. (1987), C orrect fo rmu lat ion of thekappa coeff icien t o f agreeme nt , Photogramm. Eng. RemoteSens. 53 (4 ) : 421 -422 .

Meyer , M., and Werth , k (1990), Satell ite data: mana gem entpanacea o f po ten t ial p rob lem?, J. Forestry 88(9) :10-13.

Rhode, W. G. (1978), Digital image analysis techniques fornatural resource inven tory , in National Computer Confer-ence Proceedings, p p . 43 -1 0 6 .

Rosenfield, G. (1981), Analysis of varianc e of them atic m ap-p ing exper iment data , Photogramm . Eng. Remote Sens .47(12) :1685-1692.

Rosenfield, G ., and Fitzpatric k-Lin s, K. (1986), A coefficientof agre em ent as a mea sure o f thema t ic class if ication accu-racy, Photogramm. Eng. Remote Sens. 52(2 ) : 223 -227 .

Rosenfield, G. H., Fitzpatriek-Lins, K., and Ling, H. (1982),Sampl ing tb r themat ic map accuracy tes t ing , Photogramm.Eng. Remote Sens . 48(1) :131-137.

Skidmore, A., and Turner, B. (1989), Assessing the accuracyof resource inven tory maps, in Proceedings of GlobalNatural Resource Monitoring and Assessments: Preparingfor the 21st Century, Venice, I ta ly , Sep tem ber , Vol . 2 , pp .5 2 4 - 5 3 5 .

Story, M ., and Co ngalton, R. (1986), Accu racy asses sme nt: auser 's perspect ive, Ph o to g ra m m . En g . Rem o te Sen s .52(3 ) : 39 7-39 9 .

van Genderen, J. L., and Lock, B. F. (1977), Testing land usem a p a c c u r a c y , Ph o to g ra m m . En g . Rem o te Sens .43 (9 ) :1 1 35-1 1 37 .

Con Galt On

Documents

Transcript of Con Galt On