ERIC EJ1152459: Validity Issues in Assessing Dispositions: The Confirmatory Factor Analysis of a Teacher Dispositions Form

2017·0.86 MB·English

by  ERIC

Validity Issues in Assessing Dispositions: The Confirmatory
Factor Analysis of A Teacher Dispositions Form
Chunling Niu University of Kentucky
Kimberlee Everson Western Kentucky University
Sylvia Dietrich Western Kentucky University
Cassie Zippay Western Kentucky University
Critics against the inclusion of dispositions as part of the teacher education accreditation
focus on the dearth of empirical literature on reliably and validly accessing dispositions
(Borko, Liston, & Whitcomb, 2007). In this study, a confirmatory factor analysis (CFA)
was performed to test the factorial validity of a teacher dispositions form (3 factors and 12
indicators) employed by a southern American university to assess teacher candidates from
2006 to 2016. The initial CFA results revealed highly unsatisfactory model fit statistics.
Further model modifications were then implemented to remove two less relevant indicators
(Diversity and Collaboration) which significantly improved the model fit.
Si nce its first introduction as one of teachers’ dispositions ratings and their
the new standards by the National teaching effectiveness (Lay, 2016;
Council for Accreditation of O’Neill, Hansen, & Lewis, 2014).
Teacher Education (NCATE) in 2000, the Dispositions ratings attempt to identify
role of dispositions in teacher education the abilities of teacher candidates to
has been a topic of heated debates interact confidently and respectfully with
(NCATE, 2002). Controversies mainly others who are either similar or dissimilar
center on two questions in the existing to their belief and value systems.
related literature in teacher education. However, the lack of psychometric
First, exactly what are dispositions? validation of dispositions measures has
Teacher educators and researchers have made it challenging to integrate
used similar but confusing terms dispositions into teacher preparation
regarding teachers’ dispositions in the constructively and convincingly.
past decade; it has been difficult to
generate a consensus within the teaching Theoretical Framework
profession on an operational definition of
dispositions as a construct that can be Historically, a qualified teacher
taught, assessed, and improved. must possess the “right kind” of
Second, how are dispositions “character, values, and beliefs,” relevant
measured? Partly due to the lack of knowledge, and pedagogical skills
clarity in defining dispositions, many satisfactorily assessed by various
critics complain that it is nearly “experts” such as members of the clergy
impossible to create a reliable scale to and faculty in particular subjects (Murray,
conduct any meaningful empirical 2007, p. 381). The equivalent of teachers’
research related to dispositions, such as “character, values, and beliefs” has been
investigating the relationships between referred to as “attitudes” in the published
SRATE Journal Summer 2017/Volume 26(2) 41
NCATE Standards until 2000 when the “professional dispositions” has been
term “dispositions” was first used and significantly improved in the following
broadly defined as follows (as cited in two ways: (1) “observable behaviors in
Borko, Liston, & Whitcomb, 2007, p. educational settings” were clearly
360): specified as the indicators to measure
Dispositions are guided relevant professional dispositions; and (2)
by beliefs and attitudes “fairness and the belief that all students
related to values such as can learn” were identified as the two main
caring, fairness, honesty, categories of the teacher dispositions to
responsibility, and social be assessed.
justice. For example, they While the operational definition
might include a belief that of dispositions has become less
all students can learn, a ambiguous, a number of “methodological
vision of high and obstacles” still remain in dispositions
challenging standards, or assessment, such as low inter-rater
a commitment to a safe agreement, the tendency of the raters to
and supportive learning rate either too high or too low, and
environment. (NCATE, psychometric properties of the
2002, p. 53; 2006, p. 53) dispositions scales falling below the
The vagueness of such a acceptable level (Choi, Benson, &
definition and the lack of operational Shudak, 2016; Welch et al., 2010).
guidelines in assessing dispositions led to Despite the lack of solid empirical
a variety of issues in the attempts of many evidence to support the validity and
teacher preparation programs to adopt reliability of various self-developed
dispositions as an effective program dispositions scales, teacher preparation
admission and exit standard along with programs (e.g.,The Renaissance Group)
teaching knowledge and skills. For are still using these instruments without
instance, Murray (2007) argued that rigorous validation processes. Many of
dispositions cited in the teacher education the self-developed or adapted
literature should be considered as no dispositions scales merely went through
more than a “superfluous construct” faculty approval and basic calculation of
(2007, p. 381). Likewise, Jung and the internal consistency of the scales
Rhodes (2008, p. 647) also noted that “the using Cronbach’s Alpha without careful
current approach to dispositions examination of validity (Almerico,
assessment in the United States focuses Johnson, Henriott, & Shapiro, 2011). In
on personal characteristics and character- this sense, the adaptation and utilization
related dispositions and is frequently used of the teacher dispositions scale in this
as a sorting device to identify those who study was no exception. Thus this study
appear to be inadequately disposed to focused on conducting a confirmatory
teaching.” In 2007, to address such factor analysis (CFA) to consider one
methodological concerns about form of construct validity, factorial
dispositions assessment, NCATE made validity, for the scale adapted and used by
several significant revisions to its a southern American university since
operational definition as found in the 2006 to assess teacher candidates’
Glossary of NCATE Terms (NCATE, professional dispositionss (See Appendix
2016). The clarity of the definition of A).
SRATE Journal Summer 2017/Volume 26(2) 42
The purposes of this study were (1)
to add validity evidence for the use of an
original 3-factor teacher candidate
dispositions form using the CFA method,
(2) to understand the extent to which the
model fit with the longitudinal
dispositions data, and (3) to improve the
model fit of the current dispositions form
by modifying the CFA model, as needed.

Methods

Sampling and Protocol. To Step 1. Data preparation and clean
examine the factorial validity of the 3- up of three samples containing 152, 196,
factor structure of the Teacher Candidate and 349 observations that were randomly
Dispositions Form, a confirmatory factor selected from the original dataset with
analysis (CFA) was conducted to 18,769 observed cases (since clean-up of
examine whether the scale structure fit the full data set was both time-consuming
well with the 10-year longitudinal data; if and unnecessary for the purpose of this
the results yielded unsatisfactory model study). The purpose of this procedure
fit statistics, further investigation would was to use different samples to test
be carried out using modification ot the modified models if needed.
model based on theoretical Step 2. An initial CFA was
considerations and model modification performed on Sample 1 to produce model
indices to find out what might have fit statistics for the 3-factor instrument as
caused the poor model fit. it is.
According to Hoyle, confirmatory Step 3. If the initial model fit
factor analysis (CFA), also known as statistics proved unacceptable,
restricted factor analysis or the modifications would have been made to
measurement model, is often used “in a the original structure of the scale based on
deductive mode to test hypotheses theoretical considerations and
regarding unmeasured sources of modification indices.
variability responsible for the Step 4. A CFA of the modified
commonality among a set of scores” model was performed on Sample 2 to see
(2000, p. 465). To put it simply, CFA is whether any improvement occurred in the
a suitable statistical method to test if (or model fit.
how well) a scale developed from an Step 5. If the model fit statistics
existing theory accurately reflects the from Step 4 still proved unsatisfactory,
intended factor structure. Step 3 and 4 would be repeated based on
As shown in Table 1, the general Sample 3.
procedures in performing CFA to validate All CFA analyses were performed using
the Dispositions Form included the the Stata software (Version 14).
following five steps:

SRATE Journal Summer 2017/Volume 26(2) 43
Measure. The Teacher Candidate Results
Dispositions Form (hereafter referred to
as the Form) was adapted from the The descriptive statistics for each
dispositions rubric (including 5 values: of the three selected samples showed that
Learning & Knowledge, Diversity, the means of all three samples were
Collaboration, Professionalism, and around 4, suggesting an unusually large
Personal Integrity) used in Wayda & number of higher ratings on the Likert
Lund (2005, p. 36) and has been used by scale from 1 to 5.
the School of Teacher Education to assess The skewness and kurtosis of the
teacher candidates’ professional data were also calculated, suggesting all
dispositions since 2006. The Form was three samples were notably skewed to the
initially approved by all program faculty high end of the scale, and were not
and then adopted through the normally distributed.
Professional Education Council (PEC) An initial CFA was performed on
process; however, no formal validation Sample 1 using the robust maximum
research had been conducted to examine likelihood estimation method with
the validity and reliability of the scale Setorra-Bentler correction for data non-
until the current study. normality (See Figure 2), and yielded
The 5-point-Likert-Scale Form unsatisfactory model fit statistics of
has 3 factors (i.e., Learning Attitudes, RMSEA=0.192, CFI=0.815, TLI=0.749,
Professionalism, Personal Integrity, in considering the conventional cutoff
addition to Diversity and Collaboration values are .06, .95, and .95 for RMSEA,
each measured by a single indicator) CFI, and TLI respectively (Hu & Bentler,
measured by twelve indicators (See 1999). The overall model chi square
Figure 1). The 10-year longitudinal value Χ2(48) = 312.10, p < 0.001.
dataset based on the Form yielded 18,769
observed cases for the teacher candidates
(TCs). Each TC was rated multiple times

by multiple raters (e.g., facilitators,
faculty, P-12 practitioners, self, and peers)
throughout their teacher preparation
program. For convenience, this study
only focused on faculty ratings since they
accounted for the majority of the
observations in the dataset.
Based on the model modification
indices combined with theoretical
considerations, covariance between
Collaboration and Professionalism and
covariance between Learning Attitudes
and Diversity, as well as correlated error
terms of two indicators of Personal
Integrity (Profess2 and Integ2) were
added to CFA Model 2 (See Figure 3).

Another CFA analysis of Model 2 was
SRATE Journal Summer 2017/Volume 26(2) 44
performed on Sample 2. This
modification resulted in substantially
improved model fit statistics
(RMSEA=0.065, CFI=0.966,
TLI=0.951), although the RMSEA (0.065)
value was still slightly above the cutoff
value .06. The overall model chi square
value for Model 2 was Χ2(45) = 80.57, p
< 0.001.
Comparison of factor loadings
and covariances/correlations between
Model 1 and 2 revealed that Diversity and
Collaboration, each measured by a single
indicator, had the weakest Thus we decided to conduct a CFA model
covariances/correlations with the 3 that removed these two indicators from
exisitng factors (Professionalism, the original scale of the Form (See Figure
Learning, and Personal Integrity) 4). This final modification produced
compared to the other indicators (See significantly improved model fit statistics
Table 2). (RMSEA=0.066, CFI=0.976, TLI=0.965,
Χ2(31) = 75.81, p < 0.001).
In conclusion, the results of this
CFA validation study of the Form
showed that it was very hard to measure
SRATE Journal Summer 2017/Volume 26(2) 45
certain constructs such as Diversity and two dispositions (Diversity and
Collaboration accurately by a single Collaboration).
survey item/indicator; and by so doing, Several meaningful empirical attempts
the overall psychometric properties, have recently tackled this issue. For
especially the content and construct instance, Kapner (2013) developed a
validity, of the dispositions scale are new Valuing Diversity rubric for
significantly weakened. assessors to validly identify and give
Discussion numeral scores on teacher candidates’
Supported by previous performance related to valuing diversity.
dispositions studies (Brewer, Lindquist, Street (2014) adopted the Diversity
& Altemueller, 2011; Ruitenberg, 2010), Disposition Index (DDI) survey
the current study contributes to the (developed and validated by Schulte,
existing knowledge base by adding Edwards, & Edick, 2008) to measure the
important empirical evidence from a construct of self-perceived teachers’
local case study which found that dispositions towards diversity based on a
dispositions instruments in teacher 3-factor structure: Educators’ Skills in
education need rigorous validation for Helping Students Gain Knowledge (18
effective dispositions assessment. items), Educators’ Beliefs and Attitudes
Furthermore, this study reveals an about Students and Teaching/Learning
existing tendency in the current (16 items), and Educators’ Connections
approach adopted by many American with the Community (9 items).
teacher education institutions to Future research directions include (1)
oversimplify complex dispositions modifying the Dispositions Form to
constructs in the development and use of include an adequate number of
their own dispositions scales. The indicators based on theoretical
results of this study point out the considerations for the two dispositions,
potential risks that threaten the validity Diversity and Collaboration, to improve
of the overall dispositions scale. the content and construct validity of the
Theoretically, it seemed highly measurement; (2) running the dyadic
questionable that the Diversity and model analysis to test if there is a
Collaboration dispositions could be significant difference between the
validly observed and measured by a faculty raters and P-12 practitioner raters
single indicator. Just like for the same teacher candidates; and (3)
Professionalism, Learning, and Personal conducting a growth model analysis
Integrity, both Diversity and based on the longitudinal data.
Collaboration are complex concepts and
embody a rich array of values, beliefs,
References
and observable behaviors,which could
hardly warrant accepatable face and Almerico, G., Johnston, P., Henriott, D.,
content validity of such measurements & Shapiro, M. (2011).
(Castro, 2010; Reiter & Davis, 2011). Dispositions assessment in
Empirically, a single indicator could not teacher education: Developing an
even be considered a factor/latent assessment instrument for the
variable in factor analysis, thus it was college classroom and the
impossible to measure the statistical field. Research in Higher
reliability of the data collected for the Education Journal, 11, 1.
SRATE Journal Summer 2017/Volume 26(2) 46
Borko, H., Liston, D., & Whitcomb, J. dissertation, University of
A. (2007). Apples and Southern California).
fishes. Journal of Teacher Lay, C. S. (2016). The effectiveness of
Education, 58(5), 359-64. assessing teacher dispositions in
Brewer, R. D., Lindquist, C., & the teacher selection processes of
Altemueller, L. (2011). The Taiwan's junior high
dispositions improvement schools (Doctoral dissertation,
process. Online Submission, 4(2), Morehead State University).
51-68. Murray, F. B. (2007). Dispositions: A
Castro, A. J. (2010). Themes in the superfluous construct in teacher
research on preservice teachers’ education. Journal of teacher
views of cultural diversity education, 58(5), 381-387.
implications for researching National Council for Accreditation of
millennial preservice Teacher Education (NCATE).
teachers. Educational (2002). Professional standards
Researcher, 39(3), 198-210. for the accreditation of schools,
Choi, H. S., Benson, N. F., & Shudak, colleges, and departments of
N. J. (2016). Assessment of education. Washington, DC:
teacher candidate dispositions: NCATE.
evidence of reliability and National Council for Accreditation of
validity. Teacher Education Teacher Education (NCATE).
Quarterly, 43(3), 71. (2006). Professional standards
Hoyle, R. H. (2000). Confirmatory for the accreditation of schools,
factor analysis. Handbook of colleges, and departments of
applied multivariate statistics education. Washington, DC:
and mathematical modeling, 465- NCATE.
497. National Council for Accreditation of
Hu, L. T., & Bentler, P. M. (1999). Teacher Education (NCATE).
Cutoff criteria for fit indexes in (2016). Professional standards
covariance structure analysis: for the accreditation of schools,
Conventional criteria versus new colleges, and departments of
alternatives. Structural equation education. Washington, DC:
modeling: a multidisciplinary NCATE.
journal, 6(1), 1-55. O’Neill, J., Hansen, S., & Lewis, E.
Jung, E., & Rhodes, D. M. (2008). (2014). Dispositions to teach:
Revisiting dispositions Review and synthesis of current
assessment in teacher education: components and applications,
Broadening the and evidence of impact.
focus. Assessment & Evaluation Retrieved from
in Higher Education, 33(6), 647- https://education.govt.nz/assets/U
660. ploads/DispositionsReportFinal1
Kapner, L. S. (2013). Assessing 00914.pdf
Dispositions Towards Diversity Reiter, A. B., & Davis, S. N. (2011).
in Math and Science Submissions Factors influencing pre-service
of the Performance Assessment teachers' beliefs about student
for California Teachers(Doctoral achievement: evaluation of a pre-
SRATE Journal Summer 2017/Volume 26(2) 47
service teacher diversity
awareness Dr. Sylvia Dietrich serves as the
program. Multicultural Director of the School of Teacher
Education, 18(3), 41. Education at Western Kentucky
Ruitenberg, C. W. (2011). The trouble University. Early childhood and
with dispositions: A critical personnel preparation, including clinical
examination of personal beliefs, models comprise her research interests.
professional commitments and
actual conduct in teacher Dr. Cassie Zippay is the clinical model
education. Ethics and project director for Western Kentucky
Education, 6(1), 41-52. University’s School of Teacher
Street, A. (2014). A comparison of Education. Her research interests
dispositions for educating include culturally responsive instruction
diverse students(Doctoral and clinical teacher preparation.
dissertation, Dallas Baptist
University).
Wayda, V., & Lund, J. (2005). Assessing
dispositionss: An unresolved
challenge in teacher
education. Journal of Physical
Education, Recreation &
Dance, 76(1), 34-41.
Welch, F. C., Pitts, R. E., Tenini, K. J.,
Kuenlen, M. G., & Wood, S. G.
(2010). Significant issues in
defining and assessing teacher
dispositions. The Teacher
Educator, 45(3), 179-201.
Dr. Chunling Niu is Research
Administrative Principal Coordinator at
Training Resource Center, College of
Social Work, University of Kentucky.
Her research interests include school
performance and improvement, teacher
preparation, interdisciplinary studies
involving school intervention and child
welfare.
Dr. Kimberlee Everson is an Assistant
Professor of the Educational
Administration, Leadership, and
Research Department at Western
Kentucky University. Dr. Everson is a
quantitative methodologist whose
research focuses on teacher evaluation.
SRATE Journal Summer 2017/Volume 26(2) 48
Appendix A. The Teacher Candidate Dispositions Form
Scoring Scale: (X=Not Observed; 1=Little/No Value for Dispositions; 2=Improvement Needed in Value for Dispositions;
3=Moderate Value for Dispositions; 4=Above Average Understanding Value for Dispositions; 5=Well-established and Consistent
Behavior which Demonstrates Value for Dispositions)
Dispositionss Descriptors Holistic Scoring Scale
Values Learning: ♦Contacts Instructor X 1 2 3 4
Attendance ♦Attends Regularly 5
♦On Time Comments
♦Stays Full Time
Values Learning: ♦Attentive in Class X 1 2 3 4
Class Participation ♦Engaged/Interested in Activities 5
♦Responds Appropriately to Questions Comments
♦Participates in Discussions
Values Learning: ♦Reliable X 1 2 3 4
Class Preparation ♦Flexible 5
♦Work Completed on Time Comments
♦Shows Diversity of Curriculum Design
Values Learning: ♦Work Shows Effort X 1 2 3 4
Communication ♦Uses Correct Grammar 5
♦Expresses Ideas/Self Well Comments
♦Listens Thoughtfully and Responsibly
Values Personal ♦Controls Temper X 1 2 3 4
Integrity: ♦Shows Enthusiasm 5
Emotional Control ♦Takes Personal Responsibility for Own Comments
Behaviors
♦Respects Others
♦Takes Criticism Openly
Values Personal ♦Keeps Confidentiality X 1 2 3 4
Integrity: ♦Uses Truth and Honesty 5
Ethical Behavior ♦Trustworthy/Keeps Promises Comments
♦Shows Dignity and Integrity
Values Diversity ♦Provides Equal Learning for All X 1 2 3 4
♦Treats Students in a Non-discriminatory Manner 5
♦Endeavors to Understand Community and Home Comments
Cultures
Values Collaboration ♦Is Collegial, Cooperates with Faculty/Peers X 1 2 3 4
♦Demonstrates Flexibility 5
♦Exhibits Openness to Change Comments
♦Displays Willingness to Revise
Values Professionalism: ♦Dresses Appropriately X 1 2 3 4
Respects School Rules ♦Protects the Health, Safety, and Emotional Well- 5
and Policies Being of Students Comments
Values Professionalism: ♦Shows Commitment to Reflection and On-Going X 1 2 3 4
Commitment to Self- Learning 5
Reflection and Growth ♦Exhibits Initiative, Self-direction Comments
Values Professionalism: ♦Seeks to Improve Teaching Skills X 1 2 3 4
Shows Involvement and ♦Assumes Responsibility for Classroom Climate 5
Professional ♦Assumes Responsibility for Quality of Comments
Development Instruction
Values Professionalism: ♦Demonstrates Belief that All Children Can Learn X 1 2 3 4
Professional at High Levels 5
Responsibility ♦Assumes Responsibility for Personal Actions Comments
♦Uses Educational Technology Appropriately
♦Practices Ongoing Assessment
SRATE Journal Summer 2017/Volume 26(2) 49