Reliability & Validity 2014 Seminar (abridged

20 Pages·2014·0.66 MB·English

by  Dean Mckenzie

how good is
the
Instrument?
Dr Dean McKenzie
BA(Hons) (Psychology)
PhD (Psych Epidemiology)
Senior Research Fellow
(Abridged Version)
Full version to be presented July 2014

__ deanmckenzieconsulting.com (c)2014 1
Goals
To briefly summarize
the basic tools and concepts
needed to answer the question
‘Is this instrument any good?’
Mainly Psych instrument examples
(my
but also applies to instruments
background)
from other health areas

__ deanmck enzieconsulting.com (c)2014 2
Psycho-Statistical Inventions
Cohen’s effect size / kappa / power analysis
Cronbach’s alpha (internal consistency)
factor analysis (1904)
item response theory, Rasch modelling, partial credit
Likert scale
, semantic differential
McNemar’s test
Spearman correlation
Test reliability
Types of data: nominal, ordinal, interval, ratio
(Stanley Smith Stephens, Science 1946, 103: 677-680)
(list partly based on Cohen et al (2013)
__ deanmckenzieconsulting.com (c)2014 3
numbers into
Information
F
First female A ‘coxcomb’ or ‘polar area’
diagram
member of the
Royal
Rehmeyer J (2008) Florence Nightingale: the passionate
Statistical
statistician. ScienceNews, 26 November. 2008
Society
__ deanmckenzieconsulting.com (c)2014 4
A typical Instrument, the Kessler
Psychological Distress Scale (K10)
Developed in US , tested in Australia,
by Ronald Kessler et al
employed in Victorian Population Health Survey,
Australian National Health Surveys etc
10 questions such as ‘in the past 4 weeks, about how
often did you feel worthless’ and ‘in the past 4
weeks, about how often did you feel nervous’
Similar in scope GHQ (General Health
to David Goldberg’s
Questionnaire, 1972) but K10 is public domain
(GHQ licensed by NFER-Nelson)
Andrews G, Slade T (2001) Australian and New Zealand Journal of Public Health, 25, 6: 494-497
http://www.blackdoginstitute.org.au/docs/5.k10withinstructions.pdf
__ deanmckenzieconsulting.com (c)2014 5
Determining instrument/test
Quality
• Is this instrument any good?
Reliability: Does it give consistent results, across different raters /
situations?
Validity: Does it measure what it is supposed to?

• Does instrument measure what it is supposed to,
consistently and well, in the population(s) that it is
applied to, here & now

__ deanmckenzieconsulting.com (c)2014 6
Test Reliability
= Repeatability or consistency
• Test-retest: Should achieve same scores, each
time
(if testing characteristics that are stable across time, such as intelligence,
personality, rather than less stable characteristics, such as mood, pain
levels
but even then, stability may be an unrealistic assumption, )
Cohen et al 2013
• Inter-rater: Different interviewers / raters should reach
same conclusions / score levels

__ deanmckenzieconsulting.com (c)2014 7
Measuring Test Reliability
• Measured by intraclass* correlation, should be 0.70 and
above (1.0 is maximum)

• (e.g. if rater A’s scores are 1,3,5 and rater B’s scores are 2,4,6
(Karl) Pearson correlation would be 1.0 (perfect), but the raters
never actually assign same score!)
intraclass correlation often used for reliability & (Jacob) Cohen’s
kappa* chance-corrected for binary/categorical measures

(*please see Streiner & Norman 2008 for more details)
__ deanmckenzieconsulting.com (c)2014 8
Reliability? Internal consistency?
• Split half: correlation between first half of test/instrument
and second half, or between odd numbered and even
numbered items, should be 0.70 or above

• (Lee) Cronbach’s alpha: equivalent to the average of all
possible split halves of instrument, should be 0.70 or above

• Cronbach’s alpha size is affected by number of items, and
although everyone reports it, may not be sure what it actually
means!

__ deanmckenzieconsulting.com (c)2014 9
Internal consistency: Alpha

‘Virtually every published measure … will have adequate to good
internal consistency’ (alpha)
Miles J & Gilbert P (2005) A handbook of research
methods for clinical & health psychology. Oxford. p. 100.

Choosing a test merely on basis of Cronbach’s alpha is a bit like
choosing a restaurant on basis of whether the chefs wash
their hands
(can only hope that they all do so!)
__ deanmckenzieconsulting.com (c)2014 10