Calibration and Validation of Instruments measuring Academic Ability in Physics using Item ...

123 Pages·2017·1.94 MB·English

by  Kapsalas Ioannis

University of Ioannina
Department of Primary School Education
Postgraduate Studies Program “Science in Education”

Calibration and Validation
of Instruments measuring
Academic Ability in Physics
using Item Response Theory

Kapsalas Ioannis

Postgraduate Thesis

Supervisor: Mavridis Dimitrios

September 2017
Calibration and Validation of Instruments measuring Academic Ability
in Physics using Item Response Theory 1

Calibration and Validation of Instruments
measuring Academic Ability in Physics using
Item Response Theory

Καψάλας Ιωάννης

Μεταπτυχιακή Εργασία υποβληθείσα για την εκπλήρωση των προϋποθέσεων
απονομής Μεταπτυχιακού Διπλώματος Ειδίκευσης στις Επιστήμες της Αγωγής
του Π.Τ.Δ.Ε. του Πανεπιστημίου Ιωαννίνων

2017
© Καψάλας Ιωάννης
2 Kapsalas Ioannis
Calibration and Validation of Instruments measuring Academic Ability
in Physics using Item Response Theory 3

Contents

Acknowledgments .................................................................................................................... 5
Abstract ..................................................................................................................................... 7
1. Introduction .................................................................................................................... 11
2. Theoretical Context ........................................................................................................ 15
2.1. Educational Measurement ...................................................................................... 15
2.2. Classical Test Theory (CTT) .................................................................................. 15
2.2.1. CTT Terminology ......................................................................................... 16
2.2.2. Advantages and Disadvantages of CTT......................................................... 18
2.3. Item Response Theory (IRT) .................................................................................. 20
2.3.1. IRT Assumptions and Terminology .......................................................... 21
2.3.2. IRT Models ............................................................................................... 26
2.4. Comparing CTT and IRT ...................................................................................... 30
2.5. Test Calibration ....................................................................................................... 31
2.6. Validity and Validation ........................................................................................... 31
3. Aim and Hypotheses ....................................................................................................... 33
4. Method ............................................................................................................................. 35
5. Results .............................................................................................................................. 41
5.1. Primary School ........................................................................................................ 41
5.2. Junior High School .................................................................................................. 50
5.3. Senior High School .................................................................................................. 59
5.4. Students .................................................................................................................... 60
5.5. Teachers ................................................................................................................... 68
6. Discussion and Conclusions ........................................................................................... 77
References ............................................................................................................................... 81
Appendix ................................................................................................................................. 89
4 Kapsalas Ioannis
Calibration and Validation of Instruments measuring Academic Ability
in Physics using Item Response Theory 5
Acknowledgments

Special thanks to my family, for the support through all the years of my
studies, that made it possible for me to reach this level of academic education
and also for the great help with the accomplishment of this thesis.
I would also like to thank my supervisor, Mr. Mavridis, for the support
and guidance and Mr. Kotsis for generously providing me with the data of his
research.
Finally, I would like to thank each one of the participants of my research
for their willingness to take some of their personal time to fill up my
questionnaires.
6 Kapsalas Ioannis
Calibration and Validation of Instruments measuring Academic Ability
in Physics using Item Response Theory 7
Abstract

Educational measurement typically aims at evaluating the abilities and
knowledge of students in various fields such as mathematics, language, science,
physics etc., using tests, questionnaires and other instruments. The aim of this
thesis is to create valid instruments that measure the academic ability in
mechanics through calibrating already used ones. An instrument that was used
for similar purposes in other studies was analyzed separately for each category
of the sample, by fitting the data into IRT models with the statistical software
Stata 14. Based on the results of the analysis, a new instrument was created for
each category, with only one exception, where this was not possible due to
serious statistical problems. The four new instruments were distributed to a total
of 489 subjects and subsequently they were analyzed in the same way, in order
to examine their improvement. All of them appeared to be improved, each one
at a different degree. In this way, it was once again highlighted how IRT can be
of great importance at the development of instruments in the area of Physics
research (and furthermore of educational research). The implementation of this
theory can lead to more accurate measurement instruments and consequently to
more accurate measurement and conclusions.
8 Kapsalas Ioannis
Calibration and Validation of Instruments measuring Academic Ability
in Physics using Item Response Theory 9
Περίληψη

Η εκπαιδευτική μέτρηση στοχεύει τυπικά στην αξιολόγηση των
ικανοτήτων και των γνώσεων των μαθητών σε διάφορους τομείς όπως τα
μαθηματικά, η γλώσσα, η φυσική κλπ., χρησιμοποιώντας τεστ, ερωτηματολόγια
και άλλα όργανα. Ο σκοπός αυτής της εργασίας είναι να δημιουργήσει έγκυρα
όργανα που μετρούν την ακαδημαϊκή ικανότητα στη Μηχανική μέσω της
βαθμονόμησης ήδη χρησιμοποιημένων. Ένα όργανο που χρησιμοποιήθηκε για
παρόμοιους σκοπούς σε άλλες μελέτες, αναλύθηκε ξεχωριστά για κάθε
κατηγορία του δείγματος, προσαρμόζοντας τα δεδομένα σε μοντέλα της
Θεωρίας Απόκρισης Ερωτήματος (ΘΕΑ) χρησιμοποιώντας το στατιστικό
λογισμικό STATA 14. Με βάση τα αποτελέσματα της ανάλυσης, ένα νέο
όργανο δημιουργήθηκε για κάθε κατηγορία, με μόνο μια εξαίρεση, όπου αυτό
δεν κατέστη δυνατό λόγω σοβαρών στατιστικών προβλημάτων. Τα τέσσερα νέα
ερωτηματολόγια μοιράσθηκαν σε ένα δείγμα 489 υποκειμένων συνολικά και
στη συνέχεια αναλύθηκαν με τον ίδιο τρόπο, έτσι ώστε να εξεταστεί ο βαθμός
βελτίωσής τους. Όλα τα όργανα παρουσίασαν βελτίωση, το καθένα σε
διαφορετικό βαθμό. Μέσω αυτού, αναδεικνύεται η σημαντικότητα της ΘΕΑ
στην κατασκευή οργάνων μέτρησης στον τομέα της έρευνας της Φυσικής (και
κατ’ επέκταση της εκπαιδευτικής έρευνας). Η εφαρμογή αυτής της θεωρίας
μπορεί να οδηγήσει σε πιο ακριβή όργανα μέτρησης και κατά συνέπεια σε πιο
ακριβή μέτρηση και συμπεράσματα.