Analysing Multimodal Documents: A Foundation for the Systematic Analysis of Multimodal Documents

333 Pages·2008·3.31 MB·English

by  John Bateman

Multimodality and Genre
A Foundation for the Systematic Analysis
of Multimodal Documents
John A. Bateman
Multimodality and Genre
This page intentionally left blank
Multimodality and Genre
A Foundation for the Systematic Analysis
of Multimodal Documents
John A. Bateman
University of Bremen
© John A. Bateman 2008
All rights reserved. No reproduction, copy or transmission of this
publication may be made without written permission.
No paragraph of this publication may be reproduced, copied or transmit-
ted save with written permission or in accordance with the provisions of
the Copyright, Designs and Patents Act 1988, or under the terms of any
licence permitting limited copying issued by the Copyright Licensing
Agency, 90 Tottenham Court Road, London W1T 4LP.
Any person who does any unauthorised act in relation to this publication
may be liable to criminal prosecution and civil claims for damages.
The author has asserted his right to be identified as the author of this
work in accordance with the Copyright, Designs and Patents Act 1988.
First published 2008 by
PALGRAVE MACMILLAN
Houndmills, Basingstoke, Hampshire RG21 6XS and
175 Fifth Avenue, New York, N. Y. 10010
Companies and representatives throughout the world
PALGRAVE MACMILLAN is the global academic imprint of the Palgrave
Macmillan division of St. Martin’s Press, LLC and of Palgrave Macmillan Ltd.
Macmillan® is a registered trademark in the United States, United
Kingdom and other countries. Palgrave is a registered trademark in the
European Union and other countries.
ISBN-13: 978–0–230–00256–2 hardback
ISBN-10: 0–230–00256–0 hardback
This book is printed on paper suitable for recycling and made from fully
managed and sustained forest sources. Logging, pulping and manufactur-
ing processes are expected to conform to the environmental regulations
of the country of origin.
A catalogue record for this book is available from the British Library.
A catalog record for this book is available from the Library of Congress.
10 9 8 7 6 5 4 3 2 1
17 16 15 14 13 12 11 10 09 08
Printed and bound in Great Britain by
CPI Antony Rowe, Chippenham and Eastbourne
(cid:2) (cid:2)
“gem-book-final” — 2007/12/16 — 17:04 — page v — #4
(cid:2) (cid:2)
Contents
ListofTables ix
ListofFigures xi
Acknowledgements xvii
Preface xix
1 Introduction: FourWhysandaHow 1
1.1 Learningtowalk: framingissuesandanalyticfocus . . . . 3
1.1.1 Whymultimodality? . . . . . . . . . . . . . . . . 3
1.1.2 Why‘documents’? . . . . . . . . . . . . . . . . . 7
1.1.3 Whygenre? . . . . . . . . . . . . . . . . . . . . . 9
1.1.4 Whyanalysis? . . . . . . . . . . . . . . . . . . . 11
1.2 Howcanweanalysemultimodaldocuments? . . . . . . . 13
1.2.1 Anorientationforanalysis: empiricallinguistics . 14
1.2.2 Aframeworkforempiricalanalysis: theGeMmodel 15
1.2.3 Structureofthebook . . . . . . . . . . . . . . . . 19
2 MultimodalDocumentsandtheirComponents 21
2.1 Startingpoints: howtofinddocumentparts? . . . . . . . . 24
2.2 Thepageasanobjectofinterpretation . . . . . . . . . . . 27
2.2.1 Interpretationwithindocumentdesign . . . . . . . 28
2.2.2 Multimodallinguistics . . . . . . . . . . . . . . . 38
2.3 Thepageasobjectofperception . . . . . . . . . . . . . . 57
2.4 Pageassignal . . . . . . . . . . . . . . . . . . . . . . . . 65
2.5 ThePageasobjectofproduction . . . . . . . . . . . . . . 74
2.5.1 Describingapagefordesign . . . . . . . . . . . . 75
2.5.2 Describingapageforrendering . . . . . . . . . . 85
2.5.3 Producingapagefromintentions: automaticdocu-
mentgeneration. . . . . . . . . . . . . . . . . . . 91
2.6 Combiningviewpointsondocumentparts . . . . . . . . . 103
3 TheGeMModel: TreatingtheMultimodalPageasaMultilay-
eredSemioticArtefact 107
3.1 TheGeMModel: thebaselayer . . . . . . . . . . . . . . 110
3.2 TheGeMpresentationlayers: thelayoutbase . . . . . . . 115
3.2.1 Layoutsegmentation: identificationoflayoutunits 116
v
(cid:2) (cid:2)
(cid:2) (cid:2)
(cid:2) (cid:2)
“gem-book-final” — 2007/12/16 — 17:04 — page vi — #5
(cid:2) (cid:2)
vi Contents
3.2.2 Realisationinformation. . . . . . . . . . . . . . . 117
3.2.3 Layoutstructure . . . . . . . . . . . . . . . . . . 121
3.3 Amorecomplicatedexampleoflayoutanalysis . . . . . . 129
3.3.1 ThepartsoftheLouvre . . . . . . . . . . . . . . . 130
3.3.2 ThelayoutoftheLouvre . . . . . . . . . . . . . . 134
3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 142
4 TheRhetoricalOrganisationofMultimodalDocuments 143
4.1 Rhetoricandmultimodaldocuments: ourstartingpoints. . 144
4.2 AbriefintroductiontoRhetoricalStructureTheory . . . . 146
4.2.1 TheRSTrhetoricalrelations . . . . . . . . . . . . 147
4.2.2 TheRSTrhetoricalstructure . . . . . . . . . . . . 150
4.3 ThemovetomultimodalRST:theGeMrhetoricallayer . . 151
4.3.1 Andre´’sextensionofRST . . . . . . . . . . . . . 152
4.3.2 ProblemswithtraditionalmultimodalRST . . . . 155
4.3.3 Multimodalrelationals: subnuclearelaboration . . 160
4.4 Exampleanalyses: rhetoricalrelationsbetweenlayoutunits 163
4.4.1 Mismatchesbetweenlayoutstructureandintended
rhetoricalstructure . . . . . . . . . . . . . . . . . 166
4.4.2 Explaininghowtouseatelephone . . . . . . . . . 171
4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 174
5 MultimodalDocumentsandGenre 177
5.1 Perspectivesongenre . . . . . . . . . . . . . . . . . . . . 183
5.1.1 Genreassocialsemiotic . . . . . . . . . . . . . . 184
5.1.2 Genreassocialaction . . . . . . . . . . . . . . . 188
5.1.3 Genre: theneedforfine-graineddescriptions . . . 194
5.2 Themovetomultimodalgenre . . . . . . . . . . . . . . . 196
5.2.1 Multimodalmoveswithinlinguisticandrhetorical
approachestogenre. . . . . . . . . . . . . . . . . 197
5.2.2 Movinginongenrefromthevisual . . . . . . . . 201
5.2.3 Cybergenres: abriefcritique . . . . . . . . . . . . 209
5.3 Representinggenre . . . . . . . . . . . . . . . . . . . . . 217
5.3.1 Genretypology . . . . . . . . . . . . . . . . . . . 219
5.3.2 Genretopology . . . . . . . . . . . . . . . . . . . 223
5.4 Themultimodalgenrespace . . . . . . . . . . . . . . . . 225
5.5 Illustrationsofgenre: trackingchange . . . . . . . . . . . 229
5.5.1 Fieldguidesacrosstime . . . . . . . . . . . . . . 229
5.5.2 Wildlifefactfilesacrosstime. . . . . . . . . . . . 240
5.6 Discussionandconclusion . . . . . . . . . . . . . . . . . 246
(cid:2) (cid:2)
(cid:2) (cid:2)
(cid:2) (cid:2)
“gem-book-final” — 2007/12/16 — 17:04 — page vii — #6
(cid:2) (cid:2)
Contents vii
6 BuildingMultimodalDocumentCorpora: theStateoftheArt 249
6.1 Corpus-basedlinguistics . . . . . . . . . . . . . . . . . . 250
6.2 Theoriginandrepresentationofannotatedcorpora . . . . 252
6.2.1 Annotatedcorpora: earlydays . . . . . . . . . . . 252
6.2.2 ApplyingXMLtocorpusdesign . . . . . . . . . . 254
6.2.3 Annotationproblemswithcomplexdata . . . . . . 260
6.3 Themovetomultimodalcorpora . . . . . . . . . . . . . . 264
6.4 TheGeMmodelasacorpusannotationscheme . . . . . . 267
6.5 Conclusionsandrecommendations . . . . . . . . . . . . . 272
7 ConclusionsandOutlook: WhatNext? 273
Bibliography 279
AuthorIndex 301
SubjectIndex 307
(cid:2) (cid:2)
(cid:2) (cid:2)
This page intentionally left blank
(cid:2) (cid:2)
“gem-book-final” — 2007/12/16 — 17:04 — page ix — #8
(cid:2) (cid:2)
List of Tables
1.1 The primary sources of constraints adopted by the Genre
andMultimodalityframework . . . . . . . . . . . . . . . 18
1.2 TheprimarylayersoftheGenreandMultimodalityframe-
work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.1 TheprimaryGestaltlawsofperception. . . . . . . . . . . 61
3.1 Tableofpageelementstobeidentifiedasbaseunitsduring
analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
3.2 BaseunitsoftheGannetpage . . . . . . . . . . . . . . . 113
3.3 BaseunitsoftheLouvretwo-pagespread . . . . . . . . . 133
3.4 Extract of the correspondence between layout units and
baseunits . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4.1 List of RST relations with the constraints they impose on
theirrespectivenucleiandsatellites . . . . . . . . . . . . 149
5.1 Extract from Orlikowski and Yates’s definition of coding
criteriafortheirinvestigationofgenresdeployedwithinan
extendedemail-mediatedorganisationaleffort(Orlikowski
andYates1994,p552). . . . . . . . . . . . . . . . . . . . 195
5.2 Selection of the results of Shepherd and Watters’ analysis
of 96 websites according to their content, form and func-
tionalityscheme . . . . . . . . . . . . . . . . . . . . . . . 213
ix
(cid:2) (cid:2)
(cid:2) (cid:2)