Multimodality: Disciplinary Thoughts and the Challenge of Diversity

332 Pages·2019·11.131 MB·English

by  Janina Wildfeuer, Jana Pflaeging, John Bateman, Ognyan Seizov, Chiao-I Tseng

Multimodality

Multimodality
Disciplinary Thoughts and the Challenge of Diversity
Edited by
Janina Wildfeuer, Jana Pflaeging, John Bateman,
Ognyan Seizov and Chiao-I Tseng
ISBN 978-3-11-060798-7
e-ISBN (PDF) 978-3-11-060869-4
e-ISBN (EPUB) 978-3-11-060805-2
Library of Congress Control Number: 2018946711
Bibliographic information published by the Deutsche Nationalbibliothek
The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie;
detailed bibliographic data are available on the Internet at http://dnb.dnb.de.
© 2019 Walter de Gruyter GmbH, Berlin/Boston
Printing and binding: CPI books GmbH, Leck
www.degruyter.com
Preface and Acknowledgements
Theeditorsofthisvolumewouldliketothank,firstandforemost,theauthorsfor
theirinsightfulcontributionsandhardworkduringthereviewingprocess.We
arealsogratefulforthevaluablesuggestionsmadebyananonymousreviewer.
Withouttheirdedicationandresponsiveness,thisvolumewouldnothavebeen
possible.
We also want to extend our thanks to our authors as well as to the other
colleaguesfortakingpartin#BreMM17:ThirdBremenConferenceonMultimodality:
TowardsaNewDisciplineon20–22September2017inBremen,Germany.Their
creativityandactiveparticipationintheinterdisciplinarydialoguethroughoutthe
eventsparkedourenthusiasmtodocumentsomeoftheconference’smajorthemes
anddirectionsinthisvolume.
TheBremenConferencesonMultimodalityhavebecomeafirmpillarinthemul-
timodalitycontext.Startingin2014,BreMM14wasdedicatedtobuildingbridges
betweenvariousmultimodality-readydisciplines,BreMM15continuedwiththeo-
reticalandmethodologicalexplorationsin2015andbuiltfirstfoundationsforthe
discipline-buildingendeavoratBreMM17andinthisbook.BreMM19,inSeptem-
ber2019,focusesonempiricalinroadsinmultimodalityresearchandwiththis
furtherestablishesamorerobustdisciplinarycore.Wehopetobeabletokeepthis
highlevelofcontinuityandfruitfulexchangeinoneformoranotherandpush
multimodality’sdevelopmentfurther.
TheBreMM17conferencewasasuccessalsothankstothefinancialsupport
oftheGermanResearchFoundation(DeutscheForschungsgemeinschaft–DFG)
aswellasthelogisticalsupportoftheUniversityofBremen,andwegratefullyac-
knowledgetheircontributionstothisfirststeptowardsthevolume’sdevelopment.
JaninaWildfeuer,JanaPflaeging,JohnA.Bateman,
OgnyanSeizov,andChiao-ITseng
Groningen,Salzburg,Bremen,andBerlin,September2019
https://doi.org/10.1515/9783110608694-202

Contents
PrefaceandAcknowledgements|V
PartI: Introduction
JaninaWildfeuer,JanaPflaeging,JohnA.Bateman,OgnyanSeizov,andChiao-I
Tseng
Multimodality:DisciplinaryThoughtsandtheChallengeofDiversity–
Introduction|3
PartII: DisciplinaryThoughts
HartmutStöckl
LinguisticMultimodality–MultimodalLinguistics:
AState-of-the-ArtSketch|41
MartinThomas
MakingaVirtueofMaterialValues:
TacticalandStrategicBenefitsforScalingMultimodalAnalysis|69
KayL.O’Halloran,SabineTan,PeterWignell,RuiWang,KevinChai,andRebecca
Lange
TowardsaDisciplineofMultimodality:
ParallelstoMathematicsandLinguisticsandNewWaysForward|93
PartIII: Diversity
AxelSchmidtandKonstanzeMarx
MultimodalityasChallenge:
YouTubeDatainLinguisticCorpora|115
LaurenO’Hagan
Class,Culture,andConflictintheEdwardianBookInscription:
AMultimodalEthnohistoricalApproach|145
VIII | Contents
JohnHarnett
CognitivePathfinders:
HighlightingCross-ModalInteractionandtheOrchestrationofMemoryin
Comics|171
ChristopherTaylor
AudioDescription:
AMultimodalPracticeinExpansion|195
EdwardLarkey
NarrativeasaModeofCommunication:
ComparingTVFormatAdaptationswithMultimodalandNarratological
Approaches|219
WendyNielsen,PaulineJones,HelenGeorgiou,AnnetteTurney,andMary
Macken-Horarik
LearningSciencethroughGeneratingMultimodalDigitalExplanations:
ContributionstoMultimodalityinEducationalPractice|247
DušanStamenkovićandMilanJaćević
VideoGamesandMultimodality:
ExploringInterfacesandAnalyzingVideoScreensUsingtheGeM
Model|277
PartIV: MoreDisciplinaryThoughts
JohnA.Bateman
Afterword:LegitimatingMultimodality|297
ListofContributors|323
Index|329
JaninaWildfeuer,JanaPflaeging,JohnA.Bateman,Ognyan
Seizov,andChiao-ITseng
Multimodality: Disciplinary Thoughts and
the Challenge of Diversity – Introduction
Abstract:Inthisintroduction,wediscusstheideaofestablishingadisciplineof
multimodality,consideringbothhowthismightbedefinedandpotentialbenefits
andchallengesofattainingsuchanindependentstatus.Thisbuildsonprevious
roundsofdiscussionwithintheBremenConferencesonMultimodality(BreMM)
seriesconcerningthisissue,wherediverseapproachestothestudyofmultimodal-
ityhavecometogethertocreateabroad-basedsetofagendaitemswhereamore
systematicengagementwiththephenomenaofmultimodalityiskey.
Keywords:multimodality,discipline,interdisciplinarity,transdisciplinarity,multi-
disciplinarity,semiotics
1 Conceptualizing Multimodality’s Disciplinary
Status and Challenges
Thisbookpresentsacollectionofpositionarticlesandreportsofempiricaland
theoreticalresultsinmultimodalitydrawnfromthemostrecentofourseriesof
internationalconferencesonmultimodality.Thebroadrangeoftopics,approaches,
andresearchobjectscoveredbythecontributionsinthisvolumeattestsclearlyto
thegrowingdiversityofwhatistypicallysubsumedunderthelabelofmultimodality.
Notonlydoesourvolumeshowcasethisremarkablediversity,includingmany
innovativetransdisciplinaryandcross-nationalresearchprojects,italsoseeksto
drawattentiontotheinclusivityandflexibilityofcurrentapproachesinourfield.
Beitindifferentways,bothofthesecharacteristicsencourageamoreexplicit
andmorethoroughdiscussionofmultimodality’scurrentdisciplinarystatus,and
anyfuturedevelopmentsthatresultfromit:Ontheonehand,theoverlapinre-
searchinterestsandsimilarityofapproachesseemstojustifythinkingabouta
commonlysharedname,concepts,orinstitutionalization.Ontheotherhand,and
perhapsmoreimportantly,theprevalentdiversityofapproachesandperspectives
suggestsconsideringtheunifyingpotentialofadisciplinary‘home’,whichpro-
motesanongoingexchangeofideasandmitigatelessproductiveivergencesand
fragmentation.
https://doi.org/10.1515/9783110608694-001
Brought to you by | Ludwig-Maximilians-Universität München Universitätsbibliothek (LMU)
Authenticated
Download Date | 11/23/19 10:37 AM
4 | JaninaWildfeueretal.
Asdocumentedbythisvolume’scontributions,scholarshipwithinthe‘broad
contextofmultimodality’hasrecentlybeenexpandingintoawiderangeofareas
in which an even broader variety of methods is adopted. This developmental
trajectoryindicatesthatmultimodalitymayindeedbeatriskoffragmentationif
(meta-)disciplinaryguidelinesarenotadoptedthatfullyembracethenecessityof
associativeandhierarchicalapproaches,andchannelstocommunicate.Asafield
naturallypoisedforfurthergrowth,multimodalityneedstobewell-preparedto
tackleahostofdiversetheoreticalandempiricalchallenges,bothnowandinthe
future.Webelievethatachievingamorematurestatusofcriticalself-reflection
thatwillultimatelybeabletoofferbettermutualsupportacrossresearchinitiatives
andmethods,aswellasstrengtheninginteractionacrosssuchinitiatives,willbe
instrumentalinmovingforwardsuccessfully.
Thisintroduction,inparticular,isentirelydevotedtothepressingdiscipline-
related questions pointed out, placing a particular focus on the diversity and
breadthofmultimodalresearchandcriticallyconceptualizingthequestionof
whethermultimodalitycould,andperhapsevenshould,beseenasadisciplinein
itsownright.Inaddition,someofthediscussionsledinparticularchaptersofthis
bookalsoconsider,eithertheoreticallyorbydrawingontheresultsofempirical
research,whetheramovetowardsestablishingmultimodalityasastand-alone
disciplineisdesirableand,ifso,whatwouldbenecessarytoaccomplishit.
Ourdisciplinarythoughtshavebeenheavilyinfluencedbycontributionsfrom
theinternationalcommunityofmultimodalityresearchers,particularlythoseschol-
arsthatsharedtheirworkatoneofthepreviousBremenConferencesonMulti-
modality.Thesevoicesstronglysupportourassumptionthatthe“broadcontextof
multimodalityresearchisnowonitswaytowardformingitsownfullyacknowl-
edgeddiscipline”(Wildfeuer&Seizov,2017,279).
Asourconferencesaswellasthemanyhandbooksandintroductionstothe
fieldcontinuouslyshow,activitiesgroupedundertheterm‘multimodality’con-
stituteoneofthemostinfluentialcontemporaryapproachestothestudyofall
kindsofcommunicativeartifactsandperformances.Multimodalperspectiveson
suchresearchobjectscannowbefoundinalmosteverydisciplineinterestedin
theanalysisofcommunication.Thispopularityeventranscendssuchbroadorien-
tationsasthe‘humanities’andincludesscholarsandpractitionersfromdiverse
fieldswhereverthegoalistoscrutinizecombinationsofcommunicativeforms
andthemeaning-makingmechanismsdeployed.Asaconsequence,theseefforts
readilyemploytoolsandframeworksthathaveevolved,andwhichcontinueto
evolve,drawnfromdiverseresearchareasaswell.Inthissense,multimodality
canalreadybeseentobefarmoremultidisciplinarythanmanyotherfieldsof
research;vanLeeuwen(2005,1),forexample,evenseesmultidisciplinarityasan
Brought to you by | Ludwig-Maximilians-Universität München Universitätsbibliothek (LMU)
Authenticated
Download Date | 11/23/19 10:37 AM