fCONTENT-VALIDITY-University-of-Florida內容效度佛羅里達大學課件

CONTENTVALIDITYJeffreyM.MillerNovember,2003CONTENTVALIDITYJeffreyM.Mil1OriginsContentvalidityreferstothedegreetowhichthecontentoftheitemsreflectsthecontentdomainofinterest(APA,1954)Isthecontentaboutwhatwesaythetestisabout?OriginsContentvalidityrefers2DistinctorSubsumed?Guion’s(1980)HolyTrinity 1.Criterion-related(Predictive/Concurrent) 2.Construct 3.CriterionCronbach(1984)/Messick(1989)–ThethreearedifferentmethodsofinquirysubsumedbytheoverarchingconstructvalidityDistinctorSubsumed?Guion’s(3CurrentDefinition“Validityreferstothedegreetowhichevidenceandtheorysupporttheinterpretationsoftestscoresentailedbyproposedusesoftests(AERA/APA/NCME,1999)CurrentDefinition“Validityre4SoDoesContentMatter???Contentisnotapartofthescoresoitisnotapartofvalidity(Messick,1975;Tenopyr,1977)Contentisaprecursortodrawingascore-basedinference.Itisevidence-in-waiting(Shepard,1993;Yalow&Popham,1983)Contentisafeatureofthetest,notthescoreSoDoesContentMatter???Conte5Precursorsto“SloppyValidation”?Theoverarchingconstructvalidityparadigmrelegatesthestatusofcontentvalidityandjustifiespoorimplementation

Thecurrentdefinitionofvalidityrelegatesthestatusofcontentvalidityandjustifiespoorimplementation

Intendedorunintended,whatthenhappenstothevalidationofcontent?Precursorsto“SloppyValidati6ProphecyFulfilled?“Wefearthateffortstowithdrawthelegitimacyofcontentrepresentativenessasaformofvaliditymay,intime,substantiallyreduceattentiontotheimportofcontentcoverage(Yalow&Popham,1983).”“Unfortunately,inmanytechnicalmanuals,contentrepresentationisdealtwithinaparagraph,indicatingthatselectedpanelsofsubjectmatterexperts(SMEs)reviewedthetestcontent,ormappedtheitemstothecontentstandards–andalliswell(Crocker,2003)”ProphecyFulfilled?“Wefearth7RecentArgument“Contentrepresentationistheonlyaspectofvalidationthatcanbecompletedpriortoadministeringthetestandreportingresults.Ifthisprocessyieldsdisappointingresults,thereisstilltimetorecoup”(Crocker,2003)RecentArgument“Contentrepres8TheStandardProcedureCrocker&Algina(1986)DefinetheperformancedomainofinterestSelectapanelofqualifiedexpertsinthecontentdomainProvideastructuredframeworkfortheprocessofmatchingitemstotheperformancedomainCollectandsummarizedatafromthematchingprocessTheStandardProcedureCrocker9Hambleton’s(1980)12StepsPrepareandselectobjectiveordomainspecificationsClarifytest’spurposes,desirableformats,numberofitems,instructionforwritingWriteitemstomeasuretheobjectivesItemwritersperformtheinitialeditSystematicallyassessitemmatchtoobjectivestodeterminerepresentativenessPerformadditionalitemeditingHambleton’s(1980)12StepsPre10Hambleton’s(1980)12StepsAssemblethetestSelectandimplementmethodforsettingstandardsforinterpretingperformanceAdministerthetestCollectdataaddressingreliability,validity,andnormsPrepareuser’smanual/technicalmanualConductongoingstudiesrelatingtesttodifferentsituationsandpopulationsHambleton’s(1980)12StepsAss11Beyond“TheExpertsAgreed”Althoughtheproceduresareexplicitanddetailed,ultimateassuranceofcontentvalidityisbasedonthemethodonauthority

Ourtrainingintheimportanceofthescientificmethodmayexplainwhy“Theexpertsagreed”doesn’tsettlewell.Wehavethequantitativeitemanalysis,factoranalysis,IRT,andCronbach’salphainthesamereportasthequalitativeexpertagreementBeyond“TheExpertsAgreed”Alt12Katz’sPercentage(1958)Usingthismethod,expertsratewhetherornottheitemtapstheobjectiveonayesornodichotomousscaleLetyes=1andno=0Thenletn=thenumberof1’sforaparticularraterTheproportionissimplythesumofthen’sacrossallratersdividedbytheproductofthetotalnumberofitems(N)andthetotalnumberofraters(J)P=sumofn/(N*J)Katz’sPercentage(1958)Using13Theobviouslimitationsare:Influencebythenumberofitemsand/orratersDichotomousdecision(hencenodegreeofcertainty/uncertainty)Inclusionofallitems(hencenoregardforindividualitemweighting)NoinclusionofobjectivesthatareNOTintendedtobemeasuredand/ormultipleobjectivesTheobviouslimitationsare:14Klein&Kosecoff’sCorrelation(1975)Expertsratetheimportanceoftheobjectiveona1to5pointLikertscaleThemeanormedianisusedasanindexofrelativeimportanceforanitemThen,judgesratehowwelltheitemmatcheseachobjectiveonayes(1)/no(0)scale.Letp=theproportionofjudgeswhoassigna1toanitemononeobjectiveLetP=thesumofthep’sforallitemsmeasuringaparticularobjectivePearson’sristhencomputedusingthePofobjectiveimportanceandthePofitemtoobjectivematchKlein&Kosecoff’sCorrelation15AlthoughthistechniquetriestocontroltheproblemofindividualitemweightingviarankingsofimportanceANDincludesthepossibilityofmultipleobjectives,thelimitationsareAgain,sensitivitytothenumberofitemsandthenumberofjudgesThepossibilityofahighrwhenitemsdonotmatchanyobjectiveAlthoughthistechniquetries16Aiken’sV(1985)content-validitycoefficientnexpertsratethedegreetowhichtheitemtapsanobjectiveona1tocLikert-scaleLetlo=thelowestpossiblevalidityrating(usually,thisis1ontheLikert-scale)Letr=theratingbyanexpertLets=r–loLetS=thesumofsforthenratersAiken’sVisthenV=S/[n*(c-1)]Therangewillbefrom0to1.0Ascoreof1.0isinterpretedasallratersgivingtheitemthehighestpossibleratingAiken’sV(1985)content-valid17Aiken’sVcanbeusedwitharight-tailedbinomialprobabilitytabletoobtainstatisticalsignificanceAiken’sVdoesnotinclude 1.ObjectivesthatareNOTintendedtobemeasured 2.MultipleobjectivesAiken’sVcanbeusedwithar18Rovinelli&Hambleton’sIndexofItem-ObjectiveCongruence(1977)

Contentexpertsrateitemsregardinghowwelltheydo(ordonot)taptheestablishedobjectivesTheratingsare:1:itemclearlytapsobjective0:unsure/unclear-1:itemclearlydoesnottapobjectiveSeveralcompetingobjectivesareprovidedforeachitemAstatisticalformula(orSASprogram)isthenappliedtotheratingsofeachitemacrossraters.Theresultisanindexrangingfrom–1to+1Rovinelli&Hambleton’sIndex19Anindexof–1canbeinterpretedascompleteagreementbyallexpertsthattheitemismeasuringallthewrongobjectivesAnindexof+1canbeinterpretedascompleteagreementbyallexpertsthattheitemisonlymeasuringthecorrectobjectiveAnindexof–1canbeinterpre20Theindexofitem-objectivecongruenceassumesthattheitemtapsoneandonlyoneobjectiveHowever,thereisaformula(andSAScode)forsituationswhenanitemtapsmorethanoneobjective.Theindexofitem-objectiveco21Penfield’s(2003)ScoreIntervalManyofthequantificationproceduresaddressthemeanratingforanitemAnimprovementwouldbetoconstructaconfidenceintervalforthemeanratingofanitem.Wecouldthensaythat,givenameanratingof3.42ona4-pointLikert-scale,weare95%certainthatthetruepopulationmeanratingisbetween1.2and3.5orthatitisbetween3.4and3.5anddeterminetheaccuracyofexpertagreement.Penfield’s(2003)ScoreInterv22Thetraditionalconfidenceintervalassumesanormaldistributionforthesamplemeanofaratingscale.However,theassumptionofpopulationnormalitycannotbejustifiedwhenanalyzingthemeanofanindividualscaleitembecause1.)theoutcomesoftheitemsarediscrete,and2.)theitemsareboundedbythelimitsoftheLikert-scale.Thetraditionalconfidenceint23TheScoreconfidenceintervaltreatsratingscalevariablesasoutcomesofabinomialdistribution.Thisasymmetricintervalwasshowntoberobusttoalackoffittoabinomialdistributionespeciallywhenthesamplesizeand/orthenumberofscalecategoriesissmall(e.g.,lessthanorequaltofive).TheScoreconfidenceinterval24ConclusionContentvalidityaddressestheadequacyandrepresentativenessoftheitemstothedomainoftestingpurposesContentvalidityisnotusuallyquantifiedpossiblydueto1.)subsumingitwithinconstructvalidity;2.)ignoringitasimportant;and/or3.)relyingonacceptedexpertagreementproceduresIndicesareavailable,andthereisapushtowardsimprovingthereportingofcontentvalidationproceduresConclusionContentvalidityadd25CONTENTVALIDITYJeffreyM.MillerNovember,2003CONTENTVALIDITYJeffreyM.Mil26OriginsContentvalidityreferstothedegreetowhichthecontentoftheitemsreflectsthecontentdomainofinterest(APA,1954)Isthecontentaboutwhatwesaythetestisabout?OriginsContentvalidityrefers27DistinctorSubsumed?Guion’s(1980)HolyTrinity 1.Criterion-related(Predictive/Concurrent) 2.Construct 3.CriterionCronbach(1984)/Messick(1989)–ThethreearedifferentmethodsofinquirysubsumedbytheoverarchingconstructvalidityDistinctorSubsumed?Guion’s(28CurrentDefinition“Validityreferstothedegreetowhichevidenceandtheorysupporttheinterpretationsoftestscoresentailedbyproposedusesoftests(AERA/APA/NCME,1999)CurrentDefinition“Validityre29SoDoesContentMatter???Contentisnotapartofthescoresoitisnotapartofvalidity(Messick,1975;Tenopyr,1977)Contentisaprecursortodrawingascore-basedinference.Itisevidence-in-waiting(Shepard,1993;Yalow&Popham,1983)Contentisafeatureofthetest,notthescoreSoDoesContentMatter???Conte30Precursorsto“SloppyValidation”?Theoverarchingconstructvalidityparadigmrelegatesthestatusofcontentvalidityandjustifiespoorimplementation

Thecurrentdefinitionofvalidityrelegatesthestatusofcontentvalidityandjustifiespoorimplementation

Intendedorunintended,whatthenhappenstothevalidationofcontent?Precursorsto“SloppyValidati31ProphecyFulfilled?“Wefearthateffortstowithdrawthelegitimacyofcontentrepresentativenessasaformofvaliditymay,intime,substantiallyreduceattentiontotheimportofcontentcoverage(Yalow&Popham,1983).”“Unfortunately,inmanytechnicalmanuals,contentrepresentationisdealtwithinaparagraph,indicatingthatselectedpanelsofsubjectmatterexperts(SMEs)reviewedthetestcontent,ormappedtheitemstothecontentstandards–andalliswell(Crocker,2003)”ProphecyFulfilled?“Wefearth32RecentArgument“Contentrepresentationistheonlyaspectofvalidationthatcanbecompletedpriortoadministeringthetestandreportingresults.Ifthisprocessyieldsdisappointingresults,thereisstilltimetorecoup”(Crocker,2003)RecentArgument“Contentrepres33TheStandardProcedureCrocker&Algina(1986)DefinetheperformancedomainofinterestSelectapanelofqualifiedexpertsinthecontentdomainProvideastructuredframeworkfortheprocessofmatchingitemstotheperformancedomainCollectandsummarizedatafromthematchingprocessTheStandardProcedureCrocker34Hambleton’s(1980)12StepsPrepareandselectobjectiveordomainspecificationsClarifytest’spurposes,desirableformats,numberofitems,instructionforwritingWriteitemstomeasuretheobjectivesItemwritersperformtheinitialeditSystematicallyassessitemmatchtoobjectivestodeterminerepresentativenessPerformadditionalitemeditingHambleton’s(1980)12StepsPre35Hambleton’s(1980)12StepsAssemblethetestSelectandimplementmethodforsettingstandardsforinterpretingperformanceAdministerthetestCollectdataaddressingreliability,validity,andnormsPrepareuser’smanual/technicalmanualConductongoingstudiesrelatingtesttodifferentsituationsandpopulationsHambleton’s(1980)12StepsAss36Beyond“TheExpertsAgreed”Althoughtheproceduresareexplicitanddetailed,ultimateassuranceofcontentvalidityisbasedonthemethodonauthority

Ourtrainingintheimportanceofthescientificmethodmayexplainwhy“Theexpertsagreed”doesn’tsettlewell.Wehavethequantitativeitemanalysis,factoranalysis,IRT,andCronbach’salphainthesamereportasthequalitativeexpertagreementBeyond“TheExpertsAgreed”Alt37Katz’sPercentage(1958)Usingthismethod,expertsratewhetherornottheitemtapstheobjectiveonayesornodichotomousscaleLetyes=1andno=0Thenletn=thenumberof1’sforaparticularraterTheproportionissimplythesumofthen’sacrossallratersdividedbytheproductofthetotalnumberofitems(N)andthetotalnumberofraters(J)P=sumofn/(N*J)Katz’sPercentage(1958)Using38Theobviouslimitationsare:Influencebythenumberofitemsand/orratersDichotomousdecision(hencenodegreeofcertainty/uncertainty)Inclusionofallitems(hencenoregardforindividualitemweighting)NoinclusionofobjectivesthatareNOTintendedtobemeasuredand/ormultipleobjectivesTheobviouslimitationsare:39Klein&Kosecoff’sCorrelation(1975)Expertsratetheimportanceoftheobjectiveona1to5pointLikertscaleThemeanormedianisusedasanindexofrelativeimportanceforanitemThen,judgesratehowwelltheitemmatcheseachobjectiveonayes(1)/no(0)scale.Letp=theproportionofjudgeswhoassigna1toanitemononeobjectiveLetP=thesumofthep’sforallitemsmeasuringaparticularobjectivePearson’sristhencomputedusingthePofobjectiveimportanceandthePofitemtoobjectivematchKlein&Kosecoff’sCorrelation40AlthoughthistechniquetriestocontroltheproblemofindividualitemweightingviarankingsofimportanceANDincludesthepossibilityofmultipleobjectives,thelimitationsareAgain,sensitivitytothenumberofitemsandthenumberofjudgesThepossibilityofahighrwhenitemsdonotmatchanyobjectiveAlthoughthistechniquetries41Aiken’sV(1985)content-validitycoefficientnexpertsratethedegreetowhichtheitemtapsanobjectiveona1tocLikert-scaleLetlo=thelowestpossiblevalidityrating(usually,thisis1ontheLikert-scale)Letr=theratingbyanexpertLets=r–loLetS=thesumofsforthenratersAiken’sVisthenV=S/[n*(c-1)]Therangewillbefrom0to1.0Ascoreof1.0isinterpretedasallratersgivingtheitemthehighestpossibleratingAiken’sV(1985)content-valid42Aiken’sVcanbeusedwitharight-tailedbinomialprobabilitytabletoobtainstatisticalsignificanceAiken’sVdoesnotinclude 1.ObjectivesthatareNOTintendedtobemeasured 2.MultipleobjectivesAiken’sVcanbeusedwithar43Rovinelli&Hambleton’sIndexofItem-ObjectiveCongruence(1977)

Contentexpertsrateitemsregardinghowwelltheydo(ordonot)taptheestablishedobjectivesTheratingsare:1:itemclearlytapsobjective0:unsure/unclear-1:itemclearlydoesnottapobjectiveSeveralcompetingobjectivesareprovidedforeachitemAstatisticalformula(orSASprogram)isthenappliedtotheratingsofeachitemacrossraters.Theresultisanindexrangingfrom–1to+1Rovinelli&Hambleton’sIndex44Anindexof–1canbeinterpretedascompleteagreementbyallexpertsthattheitemismeasuringallthewrongobjectivesAnindexof+1canbeinterpretedascompleteagreementbyallexpertsthattheitemisonlymeasuringthecorrectobjectiveAnindexof–1canbeinterpre45Theindexofitem-objectivecongruenceassumesthattheitemtapsoneandonlyoneobjectiveHowever,there