Michail Giannakos · Daniel Spikol · Daniele Di Mitri · Kshitij Sharma · Xavier Ochoa · Rawad Hammad Editors The Multimodal Learning Analytics Handbook The Multimodal Learning Analytics Handbook Michail Giannakos • Daniel Spikol (cid:129) Daniele Di Mitri (cid:129) Kshitij Sharma (cid:129) Xavier Ochoa (cid:129) Rawad Hammad Editors The Multimodal Learning Analytics Handbook Editors MichailGiannakos DanielSpikol NorwegianUniversityofScienceand UniversityofCopenhagen Technology Copenhagen,Denmark Trondheim,Norway KshitijSharma DanieleDiMitri NorwegianUniversityofScienceand DIPF,FrankfurtamMain Technology Germany Trondheim,Norway XavierOchoa RawadHammad NewYorkUniversity UniversityofEastLondon NewYork,NY,USA London,UK ISBN978-3-031-08075-3 ISBN978-3-031-08076-0 (eBook) https://doi.org/10.1007/978-3-031-08076-0 ©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerland AG2022 Chapter11islicensedunderthetermsoftheCreativeCommonsAttribution4.0InternationalLicense (http://creativecommons.org/licenses/by/4.0/).Forfurtherdetailsseelicenseinformationinthechapter. Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar ordissimilarmethodologynowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland Contents PartI IntroductiontoMMLA IntroductiontoMultimodalLearningAnalytics ............................. 3 MichailGiannakos,DanielSpikol,DanieleDiMitri,KshitijSharma, XavierOchoa,andRawadHammad PartII MMLAforDesign MultimodalLearningAnalyticsandtheDesignofLearningSpaces....... 31 Milica Vujovic, Davinia Hernández-Leo, Roberto Martinez-Maldonado,MutluCukurova,andDanielSpikol PartIII MMLAforFeedbackandRegulation MultimodalSystemsforAutomatedOralPresentationFeedback: AComparativeAnalysis......................................................... 53 XavierOchoa ModelingtheComplexInterplayBetweenMonitoringEventsfor RegulatedLearningwithPsychologicalNetworks ........................... 79 Jonna Malmberg, Mohammed Saqr, Hanna Järvenoja, Eetu Haataja, HéctorJ.Pijeira-Díaz,andSannaJärvelä The Role of Metacognition and Self-regulation on Clinical Reasoning: Leveraging Multimodal Learning Analytics to TransformMedicalEducation.................................................. 105 ElizabethB.Cloude,MeganD.Wiedbusch,DarynA.Dever,DarioTorre, andRogerAzevedo v vi Contents PartIV MMLAtoSupportTheoryandPedagogy IntermodalityinMultimodalLearningAnalyticsforCognitive Theory Development: A Case from Embodied Design for MathematicsLearning........................................................... 133 Sofia Tancredi, Rotem Abdu, Ramesh Balasubramaniam, andDorAbrahamson Bridging the Gap Between Informal Learning Pedagogy andMultimodalLearningAnalytics........................................... 159 RawadHammad,MohammedBahja,andMohammadAminKuhail PartV MMLAApproaches,ArchitecturesandMethodologies MultimodalLearningExperienceforDeliberatePractice .................. 183 DanieleDiMitri,JanSchneider, Bibeg Limbu,KhaleelAsyraafMat Sanusi,andRolandKlemke CDM4MMLA: Contextualized Data Model for MultiModal LearningAnalytics............................................................... 205 Shashi Kant Shankar, María Jesús Rodríguez-Triana, Luis P. Prieto, AdolfoRuiz-Calleja,andPankajChejara APhysiology-AwareLearningAnalyticsFramework........................ 231 MelanieBleckandNguyen-ThinhLe PartVI MMLAandAffectiveStates OnceMorewithFeeling:EmotionsinMultimodalLearningAnalytics ... 261 MarcusKubsch,DanielaCaballero,andPabloUribe PartVII PrivacyandEthicsofMMLA The Evidence of Impact and Ethical Considerations ofMultimodalLearningAnalytics:ASystematicLiteratureReview...... 289 Haifa Alwahaby, Mutlu Cukurova, Zacharoula Papamitsiou, andMichailGiannakos PartVIII ThePast,Present,andFutureofMMLA Sensor-BasedAnalyticsinEducation:LessonsLearnedfrom ResearchinMultimodalLearningAnalytics.................................. 329 MichailGiannakos,MutluCukurova,andSofiaPapavlasopoulou FramingtheFutureofMultimodalLearningAnalytics..................... 359 MarceloWorsley Index............................................................................... 371 Part I Introduction to MMLA Introduction to Multimodal Learning Analytics MichailGiannakos ,DanielSpikol ,DanieleDiMitri ,KshitijSharma, XavierOchoa ,andRawadHammad Abstract This chapter provides an introduction and an overview of this edited book on Multimodal Learning Analytics (MMLA). The goal of this book is to introducethereadertothefieldofMMLAandprovideacomprehensiveoverview ofcontemporaryMMLAresearch.Thecontributionscomefromdiversecontextsto supportdifferentobjectivesandstakeholders(e.g.,learningscientists,policymakers, technologists). In this first introductory chapter, we present the history of MMLA and the various ongoing challenges, giving a brief overview of the contributions ofthebook,andconcludebyhighlightingthepotentialemergingtechnologiesand practicesconnectedwithMMLA. Keywords Multimodallearninganalytics · Learninganalytics · Sensordata · AI ineducation 1 Introduction The intersection of data coming from different modalities (multimodal data) and advanced computational analyses has the ability to improve our understanding on how humans learn, but also provide novel affordances that enhance our learning M.Giannakos((cid:2))·K.Sharma NorwegianUniversityofScienceandTechnology,Trondheim,Norway e-mail:[email protected] D.Spikol UniversityofCopenhagen,Copenhagen,Denmark D.DiMitri DIPF,FrankfurtamMain,Germany X.Ochoa NewYorkUniversity,NewYork,NY,USA R.Hammad UniversityofEastLondon,London,UK ©TheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerlandAG2022 3 M.Giannakosetal.(eds.),TheMultimodalLearningAnalyticsHandbook, https://doi.org/10.1007/978-3-031-08076-0_1 4 M.Giannakosetal. capacities (e.g.,affectivelearning(D’Mello,2017),embodied learning(Abraham- son et al., 2021)). During the last years we have seen an increasing research interestinthecollectionandanalysisofrichdataleveragingmultipledatachannels fromvarioussourcesandindifferentmodalities,i.e.,multimodallearninganalytics (MMLA) (e.g., Blikstein & Worsley, 2016; Worsley, 2018; Sharma & Giannakos, 2020). MMLA maintains Learning Analytics’ overarching goal of understanding and improving learning in all the different environments where it occurs, by exploiting new opportunities once we capture new forms of digital data from students’ learning activity, and by using computational analysis techniques from data science and artificial intelligence (AI).1 At the same time MMLA expands Learning Analytics methodologies, tools and potential implications (Giannakos et al., 2020a, b), by leveraging advances in machine learning (ML) and affordable sensor technologies to act as a virtual observer and analyst of learning activities (Ochoaetal.,2017). Multimodal data can complement our understanding on how humans learn, providing more information on (meta)cognitive, affective and behavioural aspects oflearning(Cukurovaetal.,2020).Multimodaldatacanenrichthedigitalrepresen- tationofthelearnerinthecomputer(DiMitrietal.,2018).Recentworkshighlight howMMLAresearchenablesustoextractinsightsfromtext,speech,gesture,affect, or gaze analysis (e.g., Blikstein & Worsley, 2016; Sharma & Giannakos, 2020; Spikol et al., 2018), and utilize those insights and indicators to provide automated feedbacktothelearnerstostimulatetheirawarenessandreflection(e.g.,Ochoaet al., 2018; Ochoa, 2022; Di Mitri et al., 2022). With today’s increased availability andcomplexityindata,andadvanceddataanalysistechniquesnewchallengesand opportunitiesalsoarise.Asintraditionallearningsciences,thegoalofMMLAisto understand and explain the phenomena of how humans learn. At the same time, MMLA research holds significant potential for advancing learning sciences and supporting both empirical and theoretical research by utilizing its capabilities to observethelearningactivityatthemicro-leveland“sense”humans’cognitiveand affectiveandpsychomotorfactors. The introduction of MMLA research also signals the need for advancing our understanding about how new sources of information can support our understand- ing. Unlike qualitative data (e.g., interviews, observations) or even mainstream quantitative data (e.g., questionnaires and clickstreams) where the connection of these data with their meaning is evident, the original form of multimodal data (i.e.signalscaptured)aretypicallytoocomplextocontributetotheunderstanding of the researcher without intensive data analysis and data processing steps. For instance, the extraction of specific data features is usually needed (e.g., ampli- tude, temporality, or frequency) to formulate meaningful constructs which can be employed as input for statistical analyses and future validation or rejection of hypotheses and research questions. In other words, the process of inspecting, cleaning, transforming, converting, aggregating, and modeling data in order to 1WhatisLearningAnalytics?:https://www.solaresearch.org/about/what-is-learning-analytics/ IntroductiontoMultimodalLearningAnalytics 5 developmeaningfulinformationthatcanhelpustofurtherourknowledgeanddraw conclusions(e.g.,respondtoaresearchquestion),includesmorecomplexprocesses and multidimensional methodological decisions compared to traditional learning analytics and learning technology research. Those decisions concern different methodologicalsteps,suchasdatagathering,pre-processing,analysis,annotation, modelingandsense-making,inawaythatismeaningfulforlearningscientistsand learningtechnologists,aswellasotherstakeholders(e.g.,studentsorteachers),and posechallengesandopportunitiesintheemergentfieldofMMLA.Thisbookaims toserveasauniqueresourceforstate-of-the-artmethods,processesandchallenges, butalsohighlightpromisingareasforfuturework. In particular, the goal of this book is to introduce the reader to contemporary MMLA research and practice. The contributions come from diverse contexts to support different objectives and stakeholders such as learning scientists, poli- cymakers or learning technologists. In this opening chapter, we introduce the reader to the book, by presenting the history of MMLA and the various ongoing challenges, giving a brief overview of the contributions of the book, and conclude byhighlightingthepotentialofMMLAresearchtoadvancelearningresearch. 2 Background andBrief History:What HaveWe Learned andWhere DoWe GoNow? 2.1 MMLA BriefHistoryandCommunity Development Learningtechnologyresearchfrequentlyutiliseslearningtracestointerpretcomplex learning phenomena, and to support teaching and learning processes. Learning traces are left behind while learners’ interact with technologies and the learning context in general (e.g., other learners, an instructor, and non-digital learning materials). Such interactions produce data that range from low level constructs of simple interpretation (e.g., response times and correctness of multiple-choice questions) to higher level constructs of more complex interpretation (e.g., reading strategiesandcollaborativelearningpatterns).“Traditional”analyticsrepresentsuch interactionsasasequenceofactionsandindexeswhichistypicallycontainedina logfileandvisualisedwithanavigationgraphthatshowsthelearningperformance andothermeaningfulinformation.Theseanalyticsinformthelearnerortheteacher through different representations such as graphs and diagrams, and typically are included as an add-on to the learning system for increasing its functionalities and intelligence(e.g.,recommendersystems,adaptivesupportofindividual’sprogress). Thishasimplicationstobothpedagogyinallowinglearnersandteacherstobeaware of their progress and make informed decisions as well as in learning technology design,asforexample,whichservicestoaddinalearningsystem. In recent years, advances in sensor data, social signal processing and compu- tational analyses have demonstrated the potential to help us understand learning