Technology Enhanced Assessment: 20th International Conference, TEA 2017, Barcelona, Spain, October 5–6, 2017, Revised Selected Papers

245 Pages·2018·15.561 MB·English

by  Eric Ras, Ana Elena Guerrero Roldán

Eric Ras
Ana Elena Guerrero Roldán (Eds.)
Communications in Computer and Information Science 829
Technology Enhanced
Assessment
20th International Conference, TEA 2017
Barcelona, Spain, October 5–6, 2017
Revised Selected Papers
123
Communications
in Computer and Information Science 829
Commenced Publication in 2007
Founding and Former Series Editors:
Phoebe Chen, Alfredo Cuzzocrea, Xiaoyong Du, Orhun Kara, Ting Liu,
Dominik Ślęzak, and Xiaokang Yang
Editorial Board
Simone Diniz Junqueira Barbosa
Pontifical Catholic University of Rio de Janeiro (PUC-Rio),
Rio de Janeiro, Brazil
Joaquim Filipe
Polytechnic Institute of Setúbal, Setúbal, Portugal
Igor Kotenko
St. Petersburg Institute for Informatics and Automation of the Russian
Academy of Sciences, St. Petersburg, Russia
Krishna M. Sivalingam
Indian Institute of Technology Madras, Chennai, India
Takashi Washio
Osaka University, Osaka, Japan
Junsong Yuan
University at Buffalo, The State University of New York, Buffalo, USA
Lizhu Zhou
Tsinghua University, Beijing, China
More information about this series at http://www.springer.com/series/7899
á
Eric Ras Ana Elena Guerrero Rold n (Eds.)
(cid:129)
Technology Enhanced
Assessment
20th International Conference, TEA 2017
–
Barcelona, Spain, October 5 6, 2017
Revised Selected Papers
123
Editors
EricRas Ana ElenaGuerrero Roldán
Luxembourg Institute of Science Faculty of Computer Science, Multimedia
andTechnology andTelecommunications
Esch-sur-Alzette Universitat Oberta deCatalunya
Luxembourg Barcelona
Barcelona
Spain
ISSN 1865-0929 ISSN 1865-0937 (electronic)
Communications in Computer andInformation Science
ISBN 978-3-319-97806-2 ISBN978-3-319-97807-9 (eBook)
https://doi.org/10.1007/978-3-319-97807-9
LibraryofCongressControlNumber:2018950660
©SpringerNatureSwitzerlandAG2018
Chapter“StudentPerceptionofScalablePeer-FeedbackDesigninMassiveOpenOnlineCourses”islicensed
underthetermsoftheCreativeCommonsAttribution4.0InternationalLicense(http://creativecommons.org/
licenses/by/4.0/).Forfurtherdetailsseelicenseinformationinthechapter.
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartofthe
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodologynow
knownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbookare
believedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsortheeditors
give a warranty, express or implied, with respect to the material contained herein or for any errors or
omissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictionalclaimsin
publishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
The objective of the International Technology-Enhanced Assessment Conference
(TEA) is to bring together researchers and practitioners with innovative ideas and
research on this important topic. This volume of conference proceedings provides an
opportunity for readers to engage with refereed research papers that were presented
during the 20th edition of this conference, which took place in Barcelona, Spain, at
Casa Macaya. Each paper has been reviewed by at least three experts and the authors
revised their papers based on these comments and discussions during the conference.
In total, 17 submissions from 59 authors were selected to be published in this
volume. These publications show interesting examples of current developments in
technology-enhanced assessment research. Technology is gaining more and more
importance in all phases of assessment as well as in the many difference assessment
domains(i.e.,schooleducation,highereducation,andperformancemeasurementatthe
workplace).
We see a progression in research and technologies that automatize phases in
assessment: Several contributions focused on using natural language processing tech-
niquestoautomaticallyanalyzewrittenessaysoropentextanswers;presentationswere
giventoshowhowreportscouldbeautomaticallygeneratedfromscoringdata;andlast
butnotleast,approacheswereexplainedofhowtoautomaticallygeneratefeedbackin
the context offormative assessment. Complementary to the automatizing approaches,
means were elaborated to raise the engagement of students in assessment as well as
approachesforonlineproctoring.Likelastyear’sconference,severalsubmissionsdealt
with the topic of higher-order skills, such as collaborative problem solving or pre-
sentation skills, but also with the development of tools for assessors. Since last year,
assessment inMOOC hasbeen includedand during this year’s conferencewe learned
how to use our own device for assessment purposes (i.e., BYOD) to handle huge
numbers of students in the same course.
Thepaperswillbeofinterestforeducationalscientistsandpractitionerswhowantto
be informed about recent innovations and obtain insights into technology-enhanced
assessment. We thank all reviewers, contributing authors, keynote speakers, and the
sponsoring institutions for their support.
April 2018 Eric Ras
Ana Elena Guerrero Roldán
Organization
The International Technology-Enhanced Assessment Conference 2017 was organized
byUniversitatObertadeCatalunyaandtheEmbeddedAssessmentResearchGroupof
LIST, the Luxembourg Institute of Science and Technology.
Executive Committee
Conference Chairs
Eric Ras Luxembourg Institute of Science and Technology,
Luxembourg
Ana Elena Guerrero Roldán Universitat Oberta de Catalunya, Spain
Local Organizing Committee
Eric Ras Luxembourg Institute of Science and Technology,
Luxembourg
Ana Elena Guerrero Roldán Universitat Oberta de Catalunya, Spain
Hélène Mayer Luxembourg Institute of Science and Technology,
Luxembourg
Nuria Hierro Maldonado Universitat Oberta de Catalunya, Spain
Marc Vila Bosch Universitat Oberta de Catalunya, Spain
Cristina Ruiz Cespedosa Universitat Oberta de Catalunya, Spain
Program Committee
Santi Caballe Universitat Oberta de Catalunya, Spain
Geoffrey Crisp University of New South Wales, Australia
Jeroen Donkers University of Maastricht, The Netherlands
Silvester Draaijer Vrije Universiteit Amsterdam, The Netherlands
Teresa Guasch Universitat Oberta de Catalunya, Spain
Ana Elena Guerrero Roldán Universitat Oberta de Catalunya, Spain
David Griffiths University of Bolton, UK
José Jansen Open University, The Netherlands
Eka Jeladze Tallinn University, Estonia
Desirée Joosten-ten Brinke Welten Institute, Open University, The Netherlands
Marco Kalz Welten Institute, Open University, The Netherlands
Ivana Marenzi Leibniz Universität Hannover, Germany
Hélène Mayer Luxembourg Institute of Science and Technology,
Luxembourg
Rob Nadolski Open University, The Netherlands
Ingrid Noguera Universitat Oberta de Catalunya, Spain
Hans Põldoja Tallinn University, Estonia
VIII Organization
Luis P. Prieto Tallinn University, Estonia
James Sunney Quaicoe Tallinn University, Estonia
Eric Ras Luxembourg Institute of Science and Technology,
Luxembourg
M. Elena Rodriguez Universitat Oberta de Catalunya, Spain
María Jesús Tallinn University, Estonia
Rodríguez-Triana
Peter Van Rosmalen Open University, The Netherlands
Ellen Rusman Welten Institute, Open University, The Netherlands
Christian Saul Fraunhofer IDMT, Germany
Marieke van der Schaaf University of Utrecht, The Netherlands
Sonia Sousa Tallinn University, Estonia
Slavi Stoyanov Open University, The Netherlands
Esther Tan Open University, The Netherlands
William Warburton Southampton University, UK
Denise Whitelock Open University, UK
Sponsoring Institutions
Universitat Oberta de Catalunya, Barcelona, Spain
Luxembourg Institute of Science and Technology, Esch-sur-Alzette, Luxembourg
H2020 Project TESLA http://tesla-project.eu/, TeSLA is coordinated by Universitat
Oberta de Catalunya (UOC) and funded by the European Commission’s Horizon
2020 ICT Programme
Contents
What Does a ‘Good’ Essay Look Like? Rainbow Diagrams Representing
Essay Quality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
DeniseWhitelock,AlisonTwiner,JohnT.E.Richardson,DeboraField,
and Stephen Pulman
How to Obtain Efficient High Reliabilities in Assessing Texts: Rubrics
vs Comparative Judgement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Maarten Goossens and Sven De Maeyer
Semi-automatic Generation of Competency Self-assessments
for Performance Appraisal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Alexandre Baudet, Eric Ras, and Thibaud Latour
Case Study Analysis on Blended and Online Institutions by Using
a Trustworthy System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
M. Elena Rodríguez, David Baneres, Malinka Ivanova,
and Mariana Durcheva
Student Perception of Scalable Peer-Feedback Design in Massive Open
Online Courses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Julia Kasch, Peter van Rosmalen, Ansje Löhr, Ad Ragas,
and Marco Kalz
Improving Diagram Assessment in Mooshak . . . . . . . . . . . . . . . . . . . . . . . 69
Helder Correia, José Paulo Leal, and José Carlos Paiva
A Framework for e-Assessment on Students’ Devices:
Technical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Bastian Küppers and Ulrik Schroeder
Online Proctoring for Remote Examination: A State of Play in Higher
Education in the EU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Silvester Draaijer, Amanda Jefferies, and Gwendoline Somers
Student Acceptance of Online Assessment with e-Authentication
in the UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Alexandra Okada, Denise Whitelock, Wayne Holmes,
and Chris Edwards
The Dilemmas of Formulating Theory-Informed Design Guidelines
for a Video Enhanced Rubric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Kevin Ackermans, Ellen Rusman, Saskia Brand-Gruwel,
and Marcus Specht
X Contents
Rubric to Assess Evidence-Based Dialogue of Socio-Scientific Issues
with LiteMap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Ana Karine Loula Torres Rocha, Ana Beatriz L. T. Rocha,
and Alexandra Okada
Assessment of Engagement: Using Microlevel Student Engagement as
a Form of Continuous Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Isuru Balasooriya, M. Elena Rodríguez, and Enric Mor
Assessment of Relations Between Communications and Visual Focus
in Dynamic Positioning Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
YushanPan,GuoyuanLi,ThiagoGabrielMonteiro,HansPetterHildre,
and Steinar Nistad
On Improving Automated Self-assessment with Moodle Quizzes:
Experiences from a Cryptography Course. . . . . . . . . . . . . . . . . . . . . . . . . . 176
Cristina Pérez-Solà, Jordi Herrera-Joancomartí, and Helena Rifà-Pous
Pathways to Successful Online Testing: eExams with the “Secure Exam
Environment” (SEE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
Gabriele Frankl, Sebastian Napetschnig, and Peter Schartner
Calculating the Random Guess Score of Multiple-Response and Matching
Test Items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Silvester Draaijer, Sally Jordan, and Helen Ogden
Designing a Collaborative Problem Solving Task in the Context
of Urban Planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Lou Schwartz, Eric Ras, Dimitra Anastasiou, Thibaud Latour,
and Valérie Maquil
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235