Examining Method Effect of Synonym and Antonym Test in Verbal Abilities Measure

13 Pages·2015·0.85 MB·English

by  Wahyu Widhiarso, Haryanta

Europe'sJournalofPsychology
ejop.psychopen.eu|1841-0413
Research Reports
Examining Method Effect of Synonym and Antonym Test in Verbal Abilities
Measure
Wahyu Widhiarso*a, Haryantaa
[a]UniversitasGadjahMada,Yogyakarta,Indonesia.
Abstract
Manyresearchershaveassumedthatdifferentmethodscouldbesubstitutedtomeasurethesameattributesinassessment.Variousmodels
havebeendevelopedtoaccommodatetheamountofvarianceattributabletothemethodsbutthesemodelsapplicationinempiricalresearch
israre.Thepresentstudyappliedoneofthosemodelstoexaminewhethermethodeffectswerepresentsinsynonymandantonymtests.
Studyparticipantswere3,469applicantstograduateschool.TheinstrumentusedwastheGraduateAcademicPotentialTest(PAPS),which
includessynonymandantonymquestionstomeasureverbalabilities.Ouranalysisshowedthatmeasurementmodelsthatusingcorrelated
trait–correlatedmethodsminusone,CT-C(M–1),thatseparatedtraitandmethodeffectintodistinctlatentconstructsyieldedslightlybetter
valuesformultiplegoodness-of-fitindicesthanonefactormodel.However,eitherforthesynonymorantonymitems,theproportionofvariance
accountedforbythemethodissmallerthantraitvariance.Thecorrelationbetweenfactorscoresofbothmethodsishigh(r=0.994).These
findingsconfirmthatsynonymandantonymtestsrepresentthesameattributesothatbothtestscannotbetreatedastwouniquemethods
formeasuringverbalability.
Keywords:methodvariance,confirmatoryfactoranalysis,synonymsandantonymstest
Europe'sJournalofPsychology,2015,Vol.11(3),419–431,doi:10.5964/ejop.v11i3.865
Received:2014-09-17.Accepted:2015-05-25.Published(VoR):2015-08-20.
HandlingEditors:VladGlăveanu,AalborgUniversity,Aalborg,Denmark;MaciejKarwowski,AcademyofSpecialEducation,Warsaw,Poland
*Correspondingauthorat:FacultyofPsychology,UniversitasGadjahMada,Jl.HumanioraNo.1Bulaksumur,Yogyakarta,Indonesia.E-mail:
[email protected]
ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionLicense
(http://creativecommons.org/licenses/by/3.0),whichpermitsunrestricteduse,distribution,andreproductioninanymedium,providedthe
originalworkisproperlycited.
Measurementprocesscomprisesanumberofelements,includingtheattributebeingmeasured,theinstrument
being used to measure, the methods of using the instrument, and the unit of measurement. In contrast to the
physicalsciences,whichgenerallyhaveestablishedandconsistentmeasurementmethods,measurementinthe
socialsciencesisstillevolving,andthereisthereforenosingleexactmethodformeasuringindividualattributes
(Urbina, 2004). Individual attributes such as, for example, intelligence, can be measured by using a number of
differentinstrumentsandmethods.Toassessanindividual'struepsychologicalcomplexity,multipleinstruments
that employ a variety of methods for collecting data are used. For example, in assessing job performance, the
primary instrument that measures employee performance may be supplemented by information from external
sourcessuchaspeersandsupervisors.
In the field of measurement, method has a wide meaning, comprising all ways of measuring (Kline, 2011). The
methodcanrefertothesourceofinformation(e.g.,self,others)(Urbina,2004)theformatscale(e.g.,Likert,se-
manticdifferential)(Kenny,1976),thedirectionofthestatementsinthescale(e.g.,positiveornegative)(Marsh,
MethodEffectofSynonymandAntonymTest 420
1996),ortheoverallformatoftheinstrument(Marsh,Asci,&Thomas,2002).Forthisreason,differentinstruments
thatmeasurethesameconstruct(e.g.,self-esteem)—suchasRosenberg’sSelf-EsteemInventoryandCopper-
smith’sSelf-EsteemInventory—couldbeperceivedasdifferentmethodsformeasuringself-esteem.
TheexistenceofthesevariousmeasurementmethodsinspiredCampbellandFiske(1959)todevelopthemulti-
traitmulti-method(MTMM)analysisofconstructvalidity.Aninstrumentisassumedtoattainhighconstructvalidity
if there is a strong correlation between two instruments that measure similar attributes using different methods,
while different instruments that measure different attributes have low correlations. This type of testing assumes
thatthemethodofmeasurementshouldnotaffectthescoringoftheattributeofinterest:iftheobtainedcorrelation
ishigh,thenbothinstrumentscanbeassumedtobevalid.Thisapproachimpliesthatmethodsofmeasurement
areinterchangeable.
This concept of interchangeability of method was challenged by the discovery of the person specific method,
meaningthatindividualsresponddifferentlytodifferentmethods.Forexample,someindividualsarelikelytoget
lowerscoresonpaper-and-pencilteststhanoncomputer-basedtest,meaningthatdifferentmethodsmayaffect
theindividuals’obtainedscore.Theassumptionthatmethodswerecompletelyinterchangeablewasalsochallenged
by the discovery of method effects, meaning that using different methods may unintentionally require different
skills.Forexample,atestofliteracymaygiveverydifferentresultsifadministeredinapaper-and-pencilformor
onacomputer.
Another crucial method effect was introduced by Marsh (1996), who found that instruments for measuring self-
esteemgaveverydifferentresultsiftheitemswerewordedpositively(e.g.,“Iamproudofmyself”)ornegatively
(e.g.,“Iamnotproudofmyself”).Currently,inthescaledevelopmentliterature,differentitemstemswithdifferent
intensities and emphases are used to minimize response style bias (e.g., acquiescence bias). For this reason,
assessmentsofmood,forexample,shouldnotonlyincludeitemsthatmeasurepositivemood(e.g.,“Ifeelhappy”)
but also negative mood (e.g., “I feel sad”). Both items are parallel and interchangeable—individuals who truly
haveapositivemoodstatewillendorsethefirstitembutnotendorsethesecond—thismixofpositivelyandneg-
ativelyphraseditemsshouldreduceresponsebias.
However, this technique assumes that mood follows a bipolar continuum, which has not found to be strictly the
case in bi-dimensional models, as happiness and sadness, for example, can occur simultaneously (Steyer &
Riedl,2004).Moodisnottheonlyexampleofthismeasurement-relatedconcern;inmanypsychologicalmeasure-
ments,attributesthattheoreticallystandontheoppositeendsofacontinuumcanoccursimultaneously.Established
examplesofthisincludetrustanddistrust(Lewicki,McAllister,&Bies,1998)aswellasmasculinityandfemininity
(Udry,2000).Factoranalysisconductedonsuchscalesrevealtheexistenceofitemgroupsbasedonthetypeof
item,asopposedtoontheattributeofinterest.ThesefindingssupportMarsh’s(1996)findingthatitemspresented
positivelyornegativelydonotonlyfunctionasadifferentmeasurementmethods,butmayalsoassessdifferent
attributes. Clearly, different measurement methods may result in bias, which can sometimes only be detected
afterwhenthemeasurementinstrumenthasbeenadministered.
When testing the construct validity of a new instrument, researchers must distinguish between method effects
(differences due to the use of different methods) and construct effects (differences due to actual differences in
theattributeofinterest).Methodeffectsareoftenalsoreferredtoasmethod-relatedsourcesofvariance,which
helpresearcherstopinpointwhatvarianceinthemeasurecanbeattributedtothemethodologyorothertypesof
systematicmeasurementerror(Podsakoff,MacKenzie,Lee,&Podsakoff,2003).Methodvarianceisoperationalized
Europe's Journal of Psychology
2015, Vol. 11(3), 419–431
doi:10.5964/ejop.v11i3.865
Widhiarso&Haryanta 421
ascorrelationsamongtheuniqueness’sofvariablesmeasuredbythesamemethod(Conway,2002)thatrepresents
formofsystematicmeasurementerrorthatcontaminatesthevalidityofmeasurements.
Under MTMM construct validity testing, method-related variance is seen as a residual that must be minimized;
thisformofvaliditytestingattemptstomaximizeconstruct-relatedvarianceandminimizemethod-relatedvariance.
Method-relatedvarianceisseenasaresidualthatisunrelatedtoconstruct-relatedvariance.Thisissimilartore-
gressionanalysis,whichalsoseekstominimizeresiduals.Theresidualsinregressionanalysisarerandom,and
not correlated with true score variance. This explains the effect of the predictor on the criterion. Following the
principle that residuals must be random and do not correlate to any variable, method-related variance must not
relate to any variable, including the construct measured. A low correlation between the construct and method
varianceimpliesthattheconstructisindependentfromtheselectedmethodofmeasure;inthiscase,themethod
usedisjustonealternativeamongmanyalternativemethodsthatcouldbeusedtoassesstheconstruct,andthis
methodcouldbereplacedbyothers.Incontrast,ahighcorrelationbetweentheconstructandmethodvariance
means that the construct must be assessed using the same method that was already used, as changing the
methodwillreducetheinformationprovidedabouttheconstructbeingmeasured.Researchersseektominimize
theroleofmethodeffectsduetoconceptualandmethodologicalconstructs,soastoisolatethetrueunderlying
variance.
Recentliteraturesuggeststhatusingdifferentmethodscanstronglyimpacttheconstructbeingmeasured(Pace,
2010).Forexample,researchersmayusetwomethods,self-reportandpeerjudgment,tomeasuresocialsupport.
Thesemethodsarenotparallel,andcannotreplaceeachother,meaningthateliminatingoneofthosemethods
willchangetheoperationaldefinitionofsocialsupport.Forexample,removingthepeerjudgmentinthepreceding
examplewillchangetheattributebeingmeasurestoperceivedsocialsupport,asitisbeingassessedonlythrough
subjective self-report. This example illustrates how different methods sometimes cannot be eliminated or even
separatedwhenmeasuringpsychologicalattributes.Researchersinthisareathereforeneedananalyticalprocedure
toassessthecorrelationbetweentheconstructsbeingmeasuredandthemethodsbeingemployed.
SynonymandAntonymTest
Tests measuring cognitive abilities usually focus on the ability involved in thinking (i.e., reasoning, perception,
memory, verbal ability, math, and problem solving). Such tests pose questions designed to assess test-takers’
potential ability to use mental processes to solve school- or work-related problems. In the work-related context,
cognitiveabilitytestingcandeterminetheextenttowhichanindividual’sperformanceonthejobisrelatedtotheir
learning,problem-solvinganddecision-makingcapabilities(Allworth&Passmore,2008).
Verbal abilities tests usually measure verbal knowledge and verbal reasoning. There are many types of tests
available to measure verbal abilities (e.g., synonym, antonym, analogy, reading comprehension, and sentence
completion). All sub-types are not used in every test measuring cognitive abilities; two well-known examples of
this are the Scholastic Aptitude Test (SAT) and the Graduate Record Examination (GRE). The selection of the
typesofsub-testsdependsonthetheory,purpose,andtest-takercharacteristicschosenbytestdevelopers.For
example, in 1994 major changes in both content and procedures were implemented in the SAT: antonym items
wereeliminatedfromtheverbalsection,readingcomprehensionitemsweremademorecomplex,andsentence
completionitemswereadded(Zwick,2007).
Europe's Journal of Psychology
2015, Vol. 11(3), 419–431
doi:10.5964/ejop.v11i3.865
MethodEffectofSynonymandAntonymTest 422
The present study focused on two verbal abilities tests: synonyms and antonyms. Although prior studies have
assumed that synonym and antonym tests are substitutive methods that both measure verbal comprehension
(e.g., Ward, 1982), we believe that these two sub-tests measure different domains. Two words are synonyms if
theyhavesimilarmeanings.Inasynonymquestion,respondentsmustchoosethewordthatisclosestinmeaning
tothegivenword(e.g.,happy–glad).Individualsareinstructedtolookforthebestmatchamongthechoicesgiven,
notnecessarilyaperfectmatch.Antonymsaretwowordsthathaveoppositemeanings.Inanantonymquestion,
individuals must choose the word or phrase that most nearly means the opposite of the given word (e.g.,
happy–sad). Synonyms and antonyms have paradigmatic and syntagmatic relationships, respectively, between
thetwomatcheditems.AccordingtoRapp(2002),wordswithaparadigmaticrelationshiparenormallythesame
part of speech, whereas words with a syntagmatic relation can be (but need not be) the same part of speech.
Further, although the definitions of synonym and antonym are very similar, the strategy for handling antonym
questionsisdifferentfromthatforansweringsynonymquestions.Synonymyandantonymyaresemanticrelations
ofa very differentlogicalnature,oppositionofmeaningis notsimply theextremecase ofdifferencein meaning
(Lyons, 1968). The opposite of synonymity is heteronymy while the opposite of antonymy is lack of opposition
(Hermann,Conti,Peters,Robbins,&Chaffin,1979).
Thereareseveralconsiderationsinassessingverbalabilitiesusingwordrelationshipquestions:thekeyanswer
should be most similar in meaning or most opposite in meaning, excerpted text should not be used when one
mustanalyzewordrelationshipswithinanswerchoices,onlygrade-levelappropriatewordsfoundwithinthetext
shouldbeassessed,andthequestionshouldcontainclearandsufficientcontextfordeterminingthemeaningof
theassessedwords(FloridaDepartmentofEducation,2010).Theclarityofwordcontextdiffersbetweensynonym
and antonym questions, with synonym questions providing a clearer verbal context and more specificity than
antonyms(Freedle&Kostin,1990).Antonymsquestionsisnotmerelydescribedsetphrases(e.g.,longandshort,
thickandthin)butantonympairsco-occuracrossalargerangeofdifferentphrases(Paradis&Willners,2011).
Thesynonymandantonymtestsareuniqueinmeasuringverbalcomprehension.Thisuniquenessisassociated
with the differences in the skills individuals need to be able to take these two tests. At a certain level, individual
onlyrequiresknowledgeaboutvocabularytosuccessfullytakethesynonymtest;however,individualsneedboth
knowledge of vocabulary and reasoning to succeed with the antonym test, as the antonym test assesses one’s
knowledgeaboutvocabularyspecifictothecontextandmeaningoftheword.Thisnotionissupportedbyseveral
authors. Many tests employ synonym test purposed to measure breadth of vocabulary whereas antonyms
measureanalogicalthinking(Arjona-Tseng,1993).Solvingantonymquestionsrequirebroaddomainofcognitive
skills(Phillips,2013),whereassolvingsynonymquestionrequirenarrowdomainofcognitiveskills.Astudycon-
ducted by Jeon and colleagues (2009) examined different brain activity when individual solving synonym and
antonym questions. Solving synonym activated brain that associated with mental processes of searching and
selectingtargetwordsthathavesimilarfeaturesincertaindimensions.Inothersides,solvingantonymquestions
activatedbrainrepresentmentalprocessofreversingthesemanticmeaninginonedimensionandfindingopposite
features. As a consequence due to differences in measurement domain, items of both tests have different level
ofdifficulty.Antonymitemsmaybeeasiertobeprocessedthansynonymitems(Sabourin,1998).
Synonymandantonymtestarenon-substitutablemethods,sincetheyarebothassociatedwithdifferentdomain
attribute of verbal abilities being measured. We therefore hypothesized that the synonym and antonym cannot
beunderstoodasinterchangeablemethodsinmeasuringverbalability,sincebothtestmeasuredifferentdomain
ofverbalability.Totestourhypothesis,weemployedseveralanalyses:comparingmodelfitofCFAmodelsthat
Europe's Journal of Psychology
2015, Vol. 11(3), 419–431
doi:10.5964/ejop.v11i3.865
Widhiarso&Haryanta 423
assumingsynonymandantonymisthesameversusdifferentmethodandexaminingthecorrelationbetweentrue
scoreofsynonymandantonymtest.
Method
Participants
Participantsofthepresentstudywere3,469graduateschoolapplicants(Agerange:24–46years)forUniversitas
GadjahMada.Dataweretakenbetween2010to2011insixcitiesinIndonesia:Yogyakarta,Jakarta,Balikpapan,
Makassar,Mataram,andSurabaya.
Instruments
TheinstrumentusedinthepresentstudywastheAcademicPotentialTestforGraduateStudents(PAPS)Form
A1,developedbyAzwar,Suhapti,Haryanta,andWidhiarso(2009).ThePAPSaimstoassessstudents’potential
tosucceedinanacademicsetting,andconsistsofthreecomponents,withatotalof120items:verbal,quantitative,
and logical. The verbal component contains synonym, antonym, analogy, and reading comprehension items
(Azwar et al., 2009); however, the present study looked only at the synonym and antonym components. Each
subtest consists of 10 items, with five alternative answers for each item. For the synonym test, participants are
instructedtochoosethewordthathasmostnearlythesamemeaningasthetargetword.Fortheantonymtest,
participantsareinstructedtochoosethewordthathasthemostnearlyoppositemeaningtothetargetword.For
bothofthesetests,thetotalscoreisthetotalnumberofitemsthathavebeenansweredcorrectly.Amongthefive
possibleanswers,onlyoneansweriscorrect.
AnalysisProcedure
Wehypothesizedthatthesetwotestsrepresentedtwodistinctmethodsandmeasureddifferentdomainsofverbal
abilities.AnalyseswereperformedusingCFA,usingmodelsderivedfromthefourmodelspresentedinFigure1.
Three models were tested: Model 1 is a null model that involves a single factor simultaneously comprised of
synonymandantonymitems.Model2consistoftwofactors;thefirstfactorconsistsofallsynonymandantonym
items (verbal abilities factor), while second factor consists of only items from the same method either synonym
orantonym(methodfactor).IntheModel2(a),antonymitemswereselectedasthereferencemethodsothatthe
model containsa non-reference(method)factorfor the synonym items.In the Model 2(b), synonym items were
selected as the reference method so that the model contains a non-reference (method) factor for the antonym
items.
Europe's Journal of Psychology
2015, Vol. 11(3), 419–431
doi:10.5964/ejop.v11i3.865
MethodEffectofSynonymandAntonymTest 424
Figure1.PathdiagramofaCT-C(M–1)modelfortheVerbalAbilitydata.
Note.Items1-10aresynonymitems,anditems11-20areantonym.Allloadingsarefreelyestimated.Adouble-headedarrow
onasinglevariableindicatesitsvariance.Forthesakeofsimplify,measurementerroroneachitemisomitted.Themodelis
identifiedbyfixingthevarianceofthereferencefactorandtheloadingofoneitemonthenon-referencefactorto1.0.
We estimated chi-square (χ2) statistics representing the Likelihood Ratio (LR) that compares the relative fit of
saturated model and restricted model (model being tested). Smaller χ2 values indicate a slightly better fit than
largerχ2values.Sincethistestissensitivetosamplesize,itisimportanttorelyonamultiple-indexfittingstrategy
(Jöreskog&Sörbom,1988).Besides(χ2),therearemanyindicesproposedbyauthors(seeHu&Bentler,1999).
ThisstudyusedTheComparativeFitIndex(CFI)andTheTuckerLewisIndex(TLI)withrangingfrom0to1.High
value of these indexes indicating a great degree of variance in the data accounted for by the model. This study
also used the Root Mean Square Error of Approximation (RMSEA), an index that represented represents the
amountofvariancenotaccountedforbythemodel.TheRMSEAvaluebelow0.06suggestssatisfactorilymodel
fit.
Results
ModelTestingandComparison
MPLUS6.12(Muthén&Muthén,2005)wasusedtoanalyzeallmodelsusingweightedleastsquaresmeansand
varianceadjusted(WLSMV).Becauseemptyanswersweretreatedasincorrectanswers,thenthereisnomissing
valueavailableinthedata.Theresultsindicatedthatallmodelsyieldedacceptablefit.TheRMSEAvaluesforall
models were below .06, CFI and TLI values were both above .90 (Hu & Bentler, 1999). Since chi-square test is
easilyaffectedbysamplesize,theevaluationofmodel–datafitwasbasedonthesefourgoodness-of-fitindices
(RMSEA,CFI,andTLI).Analysisshowedthatthethreetestedmodelsproducedsatisfactoryfit-indices(seeTable
1).Theresultsfromtestcomparisonbetweenthemodelusingthechi-squaretestfordifferencetestinginMPLUS
(i.e.,DIFFTEST)foundmodelfitdifferences,theCT(CM-1)eitheruseasynonym(∆χ2=44.389;p<.01)orantonym
Europe's Journal of Psychology
2015, Vol. 11(3), 419–431
doi:10.5964/ejop.v11i3.865
Widhiarso&Haryanta 425
(∆χ2 = 69.690; p < .01) items as a reference, produces a better fit value. Results from the model comparison
showedthatallalternativemodels(Model-2)providedslightlyabetterfitthanthebaselinemodel(Model1).
Table1
StatisticsforFitIndicesoftheThreeTestedConfirmatoryAnalysisModels
Model χ2 df P RMSEA CFI TLI WRMR
Model-1 One-factormodel 378.907 170 0.019(0.016-0.021) 0.955 0.950 1.259
Model-2a CT-C(M-1)model 335.597 160 p<0.01 0.018(0.015-0.020) 0.962 0.955 1.180
Model-2b CT-C(M-1)model 301.466 160 p<0.01 0.016(0.013-0.019) 0.970 0.964 1.107
Note.RMSEA=rootmeansquareerrorofapproximation;CFI=comparativefitindex;TLI=TuckerLewisIndex;WRMR=weightedroot
meansquareresidual.
FactorLoading
A more precise assessment of trait and method-related variance can be established by examining individual
parameter estimates (i.e., factor loading) (Byrne, 2009). Hence, this section will compare standardized factor
loadingsbetweentraitandmethodfactor.Standardizedloadingscanbeinterpretedas correlationsbetweenin-
dicators and factors in the CC(M-1) model, because the reference factor is uncorrelated with all method factors
pertainingto the same trait-methodunitin this model (Geiser, Eid,West,Lischetzke,& Nussbeck,2012). In the
CTC(M-1)model,onemethodischosenasstandard(reference)andthespecificfactorsaretreatedasmethod
factors. The factor loadings obtained when the antonym items (Model 2a) and synonym items (Model 2b) are
usedasreferencemethodareshowninTable2.ResultfromModel2asuggeststhathalfofsynonymitemsfactor
loadingsareallsmallerthanthetraitfactorloadings.ResultfromModel2bsuggeststhatallantonymitemsfactor
loading are smaller than trait factor loadings. These imply that either for the synonym and antonym items, the
proportion of variance accounted for by the method is smaller than the proportion of variance accounted for by
theverbalability.
Since in the CTC(M-1) model, the high factor loadings of the non-reference indicators on the reference factor
showthedegreeofconvergentvaliditybetweenmethodsusedformeasure,ourresultssuggeststhatsynonym
andantonymtesthaveachievedconvergentvalidity.Resultsofanalysisshowthatindividual’sscoreofeachtest
representstraitvariancemorethanmethodvariance.
Europe's Journal of Psychology
2015, Vol. 11(3), 419–431
doi:10.5964/ejop.v11i3.865
MethodEffectofSynonymandAntonymTest 426
Table2
FactorLoadingsforeachTestedCFAModel
Baseline Model2(a) Model2(b)
No. VerbalAbility VerbalAbility MethodFactor VerbalAbility MethodFactor
I1 0.34 (0.027) 0.36 (0.03) 0.19 (0.06) 0.34 (0.03)
I2 0.46 (0.026) 0.48 (0.03) 0.07* (0.06) 0.47 (0.03)
I3 0.37 (0.037) 0.38 (0.04) 0.10* (0.06) 0.35 (0.04)
I4 0.01* (0.035) 0.02* (0.04) 0.18* (0.08) -0.03* (0.04)
I5 0.06* (0.027) 0.06* (0.03) 0.34 (0.06) 0.05* (0.03)
I6 0.20 (0.025) 0.21 (0.03) 0.07* (0.08) 0.19 (0.03)
I7 0.16 (0.024) 0.17 (0.03) 0.09* (0.07) 0.15 (0.03)
I8 -0.43 (0.098) -0.44 (0.10) 0.33 (0.06) -0.45 (0.10)
I9 0.14 (0.034) 0.14 (0.04) 0.32 (0.06) 0.13 (0.03)
I10 0.27 (0.024) 0.29 (0.03) 0.25 (0.06) 0.26 (0.02)
I11 0.23 (0.027) 0.18 (0.03) 0.23 (0.03) 0.04* (0.05)
I12 0.32 (0.024) 0.32 (0.03) 0.32 (0.02) -0.08* (0.05)
I13 0.47 (0.024) 0.46 (0.03) 0.47 (0.02) 0.33 (0.06)
I14 0.91 (0.027) 0.89 (0.03) 0.91 (0.03) 0.47 (0.08)
I15 0.41 (0.022) 0.34 (0.03) 0.41 (0.02) 0.11* (0.05)
I16 0.83 (0.026) 0.84 (0.03) 0.83 (0.03) 0.20 (0.05)
I17 0.70 (0.024) 0.70 (0.03) 0.70 (0.02) 0.13 (0.05)
I18 0.46 (0.023) 0.39 (0.03) 0.47 (0.02) 0.21* (0.17)
I19 0.38 (0.024) 0.31 (0.03) 0.38 (0.02) 0.09* (0.06)
I20 0.40 (0.021) 0.35 (0.03) 0.41 (0.02) 0.33 (0.06)
Note.Standarderrorsforestimatedparametersareshowninparentheses.
*Factorloadingsarenotsignificantlydifferentfromzero(p<.05).
CorrelationBetweenTests
In order to test the correlation between individual’s latent score of synonym and antonym test, we fitted model
thatconsistsoftwofactors(synonymandantonym)tothedata.ThetestedmodelyieldsagoodfittothePAPS
data:χ2(41,N=3.469)=379.098;p<.05;RMSEA=0.019(0.016-0.021)CFI=0.955;TLI=0.949.Thecorrel-
ationbetweensynonymandantonymfactorisveryhigh(r=0.994).Thishighcorrelationsuggeststhatbetween
synonymandantonymitemshavelittleuniquenessinadditiontotheirsharedconstructofverbalability.Thecross
plotting the person scores from the two test is depicted in Figure 2. From this figure we can see that the points
are scattered closely to a straight line and the correlation is high, thus it can be concluded that the two sets of
itemsdoindeedmeasurethesameconstruct.Usingthismodel,wefoundthatsynonymitemsweremoredifficult
(M=0.070)antonym(M=-0.496).
Europe's Journal of Psychology
2015, Vol. 11(3), 419–431
doi:10.5964/ejop.v11i3.865
Widhiarso&Haryanta 427
Figure2.ScatterplotoffactorscoresfromSynonymandAntonymTest.
Discussion
Synonymsandantonymsaretwotestormethodsformeasuringverbalabilities(verbalknowledgeandreasoning).
The aim of the present study was to examine whether these two test measure different abilities. If the synonym
andantonymmeasuredifferentattributesthentheycanbeseenastwocomplementarymethodsformeasuring
verbalability.Ifbothmeasurethesameattributesthentheyhavethepossibilitytobeseenastwomethodsthat
canbeinterchangeable.Ourhypothesiswasnotsupported,astheanalysisconfirmedthatsynonymsandantonyms
are two the same methods that measure similar domains of cognitive abilities. Although, measurement models
thatseparatedsynonymandantonymsasdistinctlatentconstructsyieldedslightlybettergoodness-of-fitindices
than the null model, a comparison of factor loadings across traits and methods suggests that the proportion of
traitvarianceexceedsmethodvariancein themostoftheitems.Thenullmodelin thisstudywas a one-dimen-
sionalmodelandthealternativemodelswereCTC(M-1).Thus,althoughevidenceofdiscriminantvalidityappeared
to be fairly good at the matrix level since model fit comparison suggest CTC(M-1) model better than one factor
model;amorein-depthexaminationattheindividualparameterlevelrevealslowmethodeffectassociatedwith
synonymandantonymtest.
Synonymtestsfollowthesynonymyconcept;twowordsaresaidtobesynonymsifonecanbeusedinastatement
in place of the other without changing the meaning of the statement (Jarmasz & Szpakowicz, 2003). There are
manystrategiesathowtochoosethebestanswer,themostcommonbeingtointerpretthedistancebetweenthe
nodes that correspond to the items being compared: the shorter the path, the more similar the items (Resnik,
1995). Our study findings were supported by the fact that synonym tests are more complicated than antonym
tests, as finding an opposite meaning requires strong reasoning skills as well as synonym test. This was why
separatingthesynonymandantonymtestsproducedthemodelwiththebestfit.Synonymitemsareparticularly
easy to answer because they are modeled by preference materials of all kinds, in which a main word is paired
Europe's Journal of Psychology
2015, Vol. 11(3), 419–431
doi:10.5964/ejop.v11i3.865
MethodEffectofSynonymandAntonymTest 428
withthewordssimilartoit.However,therelationshipsunderlyingantonymitemsarequiteabitmorevaried,and
arenotsystematicallycompiled(Turney,Littman,Bigham,&Shnayder,2003).
Thestrategiesforhandlingantonymquestionsarequitedifferentthanthoseforhandlingsynonymquestions,an
oppositemeaningcanbefoundinseveralways.Theusualwayistoinstantlylocatetheoppositemeaningdirectly,
basedoncolloquiallanguage.However,thismethodishamperedbythepresenceofdistractoritemsinthepossible
responses, which are deliberately chosen to catch test-takers using this strategy. This example illustrates how
the procedures needed to answer antonym-type questions require different cognitive effort than synonym-type
questions.Antonymquestionhavedifferentpatternsthansynonymquestions.Thesepatternscanbeunderstood
by people who have large vocabularies, good reasoning skills, and the ability to connect these two skills. Test-
takersthenhavetothinkofsynonymsofquick,suchasfast,andthenfindtheantonymofthatsecondaryword
amongthelistofpossibleresponses.Whenindividualsareaskedtoarticulatetherelationshipbetweenthewords
broad and road, they might consider a number of possibilities. Many possible relationships would need to be
considered,dependingon the context(Turney et al.,2003), unlike for synonym questions.Antonymitems have
limitedverbalcontexts(Freedle&Kostin,1990),requiringindividualstointerpretwhatcontextmightbeappropriate
toconnecttheterminthequestionandpossibleanswers.
Becausethisresearchfoundthatvarianceassociatedwithspecificmethodsinsynonymandantonymtestdonot
exceed variance associated with attribute of interest (i.e., trait variance), we conclude that the unique variance
proportion produced by the two tests is small. Although synonym and antonym tests were found to be mutually
substituted, using these types of tests in a measurement instrument should be done only after considering the
different skills and attributes measured by these two types of tests. While both tests can be used to measure
cognitive abilities—synonym and antonym tests are among the best measures of verbal reasoning, they are
complementary.Individualslearnmostnewwordsbyinferringtheirmeaningsfromthecontextsinwhichthewords
areembedded,andthenrememberingandrevisingtheirunderstandingsastheyencounterthewordsrepeatedly,
indifferentcontexts.
Funding
Theauthorshavenofundingtoreport.
CompetingInterests
Theauthorshavedeclaredthatnocompetinginterestsexist.
Acknowledgments
WewishtothankProf.Dr.SaifuddinAzwar,CoordinatorofPAPSdevelopmentandDra.SriHartati,M.Si,headofPsychological
TestDevelopmentUnit(UPAP),FacultyofPsychology,UGMforsupplyingdatasetforthisstudy.
References
Allworth,E.,&Passmore,J.(2008).Usingpsychometricsandpsychologicaltoolsincoaching.InJ.Passmore(Ed.),
Psychometricsincoaching:Usingpsychologicalandpsychometrictoolsfordevelopment(pp.7-24).London,United
Kingdom:KoganPage.
Europe's Journal of Psychology
2015, Vol. 11(3), 419–431
doi:10.5964/ejop.v11i3.865