What You See Is What You Hear: Creativity And Communication In Audiovisual Texts

293 Pages·2020·23.438 MB·English

by  Dario Martinelli

Dario Martinelli
What You See Is
What You Hear
Creativity and Communication in Audiovisual Texts
What You See Is What You Hear
Dario Martinelli
What You See Is What
You Hear
Creativity and Communication in Audiovisual Texts
Dario Martinelli
Vilnius Gediminas Technical University
Vilnius, Lithuania
ISBN 978-3-030-32593-0 ISBN 978-3-030-32594-7 (eBook)
https://doi.org/10.1007/978-3-030-32594-7
© Springer Nature Switzerland AG 2020
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned,
specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or
in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software,
or by similar or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply,
even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations
and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be
true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or
implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher
remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
V
To my son Elmis.
In memory of Gino Stefani (1929–2019).
Preface
This monograph allows me to pursue a system- have overloaded courses with all manner of addi-
atic, comprehensive, and coherent approach, tional material and activities. This is a misguided
which is intended to acquaint the reader with attempt to steer so-called millennial students
my theories on audiovisuality, as well as leave through that terrible and dangerous jungle of…
them satisfied that they have received a gen- of what—really? Of a written text! My goodness,
eral introduction that is fair and clear. I wrote what a shock: a written text! We schematize and
with two main purposes in mind, and in the sterilize the information because we assume that
same spirit that inspired my recent book on the students might find the narrative too difficult.
the subject of animal communication (Mar- Unsurprisingly, we disavow the natural time flow
tinelli 2017), from which I intend to bor- of data, and instead become more and more like
row—or rather steal—a few words by way of YouTube tutorials. In other words, we assume that
introduction. they are idiots.
First, influential in my work is a deep, profes- My hunch is that it is we, the academic commu-
sional interest in audiovisual communication nity with our style of pedagogical management,
through the study of various paradigms, which who has made them so. Today, the narrative (the
include, among other things: communication good old fat book) has been demonized in most
studies, film semiotics, creativity studies, nar- academic courses, and while it has saved time for
ratology, media studies, auteur theory, film and students it has also eroded their ability to develop
TV studies, and multimodality studies. My vital skills in information processing and critical
engagement also extends to topics of a varied analysis. The analogue is like eating artificial vita-
and idiosyncratic nature such as the audiovisual min integrators rather than enjoying a balanced
representation of animals; Alfred Hitchcock; the diet that contains fruit and vegetables. We save
relationship between audiovisuality and popular time, yet we kill the “narrative” of food intake, the
music; genres and schools (western, sci-fi, Italian one that suggests what food to eat, and with what,
cinema, and Hollywood); politics and ideology in how much, when, and so forth.
audiovisual texts; and more besides. All of these I
have expressed in articles in various publications, By the same token, have we ever stopped to con-
congress papers, and lecture courses. Notwith- sider that students might like to sit on a sofa with
standing, I have never taken the opportunity to a cup of tea, and actually read a book, instead of a
construct a plain, systematic overview on audio- collection of schemes and tutorials? Have we ever
visual communication in such a way that both my thought that a well-written book is a pleasant and
personal research and the works of my influential more effective way to study? Don’t get me wrong,
and learned colleagues could be applied. I have no argument with the need to develop and
freshen up didactic strategies, as well as any other
The second purpose of this book is to bring academic activities. I would hope that my previ-
together material of pedagogical value that can be ous publications (see Martinelli 2016) demon-
used as either a student textbook or supporting strate this fact. If anything, I regard myself as an
text. I have no wish to write a standard contem- advocate of “appropriate technologies”, and here
porary textbook (see Martinelli 2017) featuring I merely suggest that a textbook, as such, should
the full catalogue of “student services” or course adopt a “narrative” strategy, because along with
material for teachers looking to guide their class- the teacher’s lectures this is the very source of
room activities. Typically, the latter would include information to process. By packaging the latter
chapters of the same length with an introductory we offer preprocessed information to the student
summary, a schematic list of the topics, a final that is almost certain to be poor quality, rather
summary, a self-assessment test, etc. Frankly, I like those ready-made frozen dishes, which can
think it is time to stop mollycoddling university never be as good as those prepared with fresh
students and treat them as primary school chil- ingredients. In reality, any phenomenon of “learn-
dren. Students enrolling in a higher education ing” needs to go through this two-step proce-
institution are young adults: if we insist they act dure. First, we absorb information in a “narrative”
as such then we should also treat them as young by seeing the whole picture, the natural flow of
adults with all the expectations that go with it. My time, the many variables. Next, we process it in a
belief is that in the last two decades universities schematic way, distinguishing the more relevant
VII
Preface
from the less, generalizing the single elements Please don’t mistake my intention as conservative.
into groups or patterns, and so forth. Hence, the I prefer the experimental and have no argument
role of the teacher is to help the student along with this new regime although it adds greatly to
this path, rather than walk in their place. I would my workload. Certainly, I could use that time in
suggest this is a much better way for our teaching more useful ways for society—including students
profession to understand student potential and themselves. I would just like to perform an exper-
help us to bring that out. iment to reestablish a direct dialogue between
course material and students. I suspect that (a)
Perhaps the main challenge in writing a book the general decrease in interest and preparation
such as this is how to offer a clear and engaging we are witnessing in today’s students has noth-
reading experience—clarity being the very reason ing to do with a “traditional” textbook, espe-
why schematizations and tutorials came to replace cially if the latter is clear and engaging and (b) a
narratives, and engagement being the very reason storytelling-oriented approach to teaching is not
why students are less and less eager to invest time more difficult or time-wasting than the schematic
in study. This monograph accepts the challenge, approach. Rather, it is simpler.
and just like “fresh food” it leaves the important
task of “processing the information” to the reader In conclusion, my intention is not to make a
in the belief that this will stimulate greater under- case for choice. Indeed, this preface makes clear
standing of the concepts. Besides, this is why I my viewpoint, which is where such commen-
have written this book in more informal (if not tary belongs and nowhere else. The subtitle of
colloquial) tones, rather than follow the formal, my book is Creativity and communication in
but often dry and cold, academic style. audiovisual texts, which is all I intend to con-
vey here, rather than the injunction, “towards a
I hasten to add that my teacher colleagues—those comeback of good old textbooks,” which is some-
generous enough to adopt this book in their what presumptuous. If the truth be told the book
courses—are not my servants. I have no right to is ordered logically by providing a glossary, ad
point out those parts I consider to be relevant or hoc references, and numerous descriptive tables
not, or how they should phrase them. Certainly, and illustrations. Regarding this last point I must
as a teacher I would reject any imposition of this offer an apology. I could find only a limited num-
kind. Rather, a monograph used as a textbook ber of useable images sourced from the audio-
must be a tool for the lecturer, and not a bible. The visual material (movies, TV series, videos, etc.)
lecturer should be able to do whatever they like, in my analysis. Ideally, I would have used any
for example, whether to exclude parts that I might image I needed to best illustrate particular notions
consider crucial. Just like the student, so it is with under discussion, but this was not possible due
the lecturer who must work on the book and not to copyright restrictions imposed by commer-
rely on ready-made information. The abovemen- cial organizations. Instead, I resorted to images
tioned information processing is, or should be, available in the public domain, in the Creative
cooperation between teacher and student. Commons (bless them!), or to personally prepared
illustrations, which was limiting. “Fair use” is a
Finally, it is my belief that academic courses have poorly defined notion, and an archaic expression
become somewhat “messy”. For those of us old that translates into: “put that image if you must, but
enough to remember there was a time when a then just pray that you won’t be sued.”
course consisted of a limited number of compo-
nents. Let us consider a prototypical situation: the So, loud and clear, I would like to state that it is a
teacher arrives in the classroom, does most of the shame that researchers and educators are bound
talking, then offers a list of books for students to to accept what amounts to the greed of copyright
read and carefully study as preparation for the holders determined to profit from the everyday,
exam. End of story. None of the people of my gen- over which they hold only a spurious claim. Little
eration appear to have suffered serious trauma or no profit attaches to an academic monograph
under this regime. Instead, what we now have is such as this; in any case to make an artwork more
a glorious parade of handouts, Moodle accounts, visible to the readership can only be beneficial for
midterm exams, mock exams, workshops, discus- that artwork. Its inclusion could never inflict rep-
sion groups, self-assessment processes, and so on utational damage. Scholarly work aims to produce
and so forth. Anything goes, except a clear and knowledge and understanding, and knowledge and
engaging textbook—at least most of the time. understanding produce a better world. Anything
that gets in its way should be scorned and averred.
VIII Preface
On a more practical side, I am also saying this, helpful referents at Springer, and to the two anon-
because I would like to invite you to implement ymous reviewers who have helped me improve
the reading of this book by searching for those the text with their critical recommendations.
images and videos that I mention and could not Immense gratitude to my wonderful son Elmis
provide adequate illustration for. A lot of such for all the reasons in the world, and for being a
material can be easily found on the web, so there great model in some of the pictures you see in the
should be no need to invest extra budget to fully book.
enjoy this text.
I hope the reading of this volume will be inspiring
To conclude, this book was written in a period of and enjoyable for you all. Thank you.
tough personal transitions for me, and while I do
not intend to delve into matters concerning my Dario Martinelli
private life that should have no place in any pub- Vilnius, Lithuania
lic platform, I wish to express my deep gratitude 2019
to my whole family. Thank you also to the affili-
ation I proudly represent in this book, Vilnius
Gediminas Technical University, and particularly References
the whole staff at the Faculty of Creative Indus-
tries. Thanks to Ron Ringer for his impeccable Martinelli, Dario. 2016. Arts and humanities in progress: A
language editing and the rhymed messages we manifesto of numanities. Berlin/New York: Springer.
exchanged in the process, to my ever friendly and Martinelli, Dario. 2017. Basics of animal communication.
Newcastle: Cambridge Scholars Publishing.
IX
Introduction
This monograph, as declared in the subtitle, is the presence of a “screen” as a symbolic threshold
about creativity and communication in audio- between the actual item and the witness. For this
visual texts and will be subsequently referred to reason, I have devoted a specific section to the par-
as audiovisual creative communication (AVCC). ticular notion of “screen” for it is this very object
This expression of the universe of audiovisual- that determines the correct circumscription of our
ity is potentially enormous, and so I must use field of inquiry. The “audiovisual texts” we refer
this introduction to explain my terminological to in this book are “screen-mediated audiovisual
choices in order to define the field of investigation texts”.
in the clearest possible way.
Having introduced the word, “text”, perhaps the
First, “audiovisual” (adj.) lies at the heart of this reader might wonder what that is? A “text” is any
enterprise. I have assigned this word to describe item of any size and length that conveys meaning.
any material or item that conveys any type of So, a movie, a TV program, a commercial, a music
information by using at the same time the visual video, a color music artifact, etc. are all “texts”.
and the acoustic channels. By this I mean images Texts are usually thought of, produced, and inter-
and sounds, and technological tools and plat- preted in accordance with conventions associated
forms used to produce and display such material, with a style, a culture, a genre, a society, or other
e.g., camera, TV, cinema hall, Internet, etc. There contexts, and within a particular medium of com-
is no requirement that these channels must be munication (in our case, cinema, TV, Internet,
together all of the time; throughout the same item etc). Unsurprisingly, in this book you will find
we can have specific moments when only images numerous references to “audiovisual text”, which
and/or sounds appear, although in its totality we abbreviate to AVT(s). Indeed, unless we talk
both must be present. That is why, a radio pro- about notions that apply to only one type of text,
gram or a book, for example, is not audiovisual e.g., the use of drawings as something that char-
material, because they contain only acoustic and acterizes a cartoon, though not a live action film,
visual information, respectively. Throughout the we refer to “texts” in order to imply that the notion
text, I will abbreviate this adjective to “AV” (AV in question can be applied to any medium, or any
genres, AV culture, etc.). Consequently, the noun, genre, or any item, e.g., the notion of “montage”.
“Audiovisuality”, is used to refer to the “quality”,
the “experience”, and the “phenomenon” of being When we use the word, “communication”, we
audiovisual. refer to the process of transmission and reception
of a message, so that some “sense”—meaning—is
From the outset I draw attention to the fact that conveyed, exchanged, understood, or misunder-
audiovisuality involves what we also call “techno- stood. With this in mind, what interests us about
logical tools and platforms”, because, taken to the AVTs are the meanings they convey, and how the
letter, the practice of merging visual and acoustic AVT(s) convey them. Perhaps, the construction
channels into a communication item can also be of such meanings is ensured by every single step
referred to “flesh and blood” live actions, such of the creative process, with little, if anything, left
as a theater play, a church mass, or a conference. to chance.
Quite simply and ultimately, it can also refer to
an ordinary linguistic conversation between two Finally, by “creative/creativity”, we refer to the
people, which, yes, involves in principle other ability to transform ideas into meaningful and
channels if the communication is of a “linguis- possibly original texts. These can be artistic,
tic” type, then it is mostly based on sound and although I prefer to avoid this adjective in order
visual stimuli such as spoken words and gestures, to avoid entanglement in a redundant discus-
respectively. sion about what is art and what is not. Can a TV
commercial be considered art? Is a mainstream
Given the scope of this volume and its obvious crowd-pleasing movie art? Are soap operas art?
limitations in terms of extent, the book is not While certifying the “artistic” status of these texts
just about any form of communication. The type may be challenging to some people, we should
of audiovisuality we are talking about here is a confine ourselves to understanding how they are
“mechanically-reproduced” (whether analogical or products of “creativity”. At the same time, the
digital) audiovisuality, which, put simply, requires presence of the “creative” element in our scope
X Introduction
is important if we are to filter the types of AVTs children’s education the importance of installing
under consideration. When we Skype with a concepts of “yes” and “no” (so-called “propae-
friend, or present some data and sounds on Pow- deutic” concepts), so that they are able to under-
erPoint, we create AVTs, yet this falls outside the stand what they should or should not do. Only
scope of this monograph. Hopefully, the notion of at a later stage can we introduce the concepts of
creativity helps us to draw the line. Some might “maybe”, “probably not”, “who knows”, “almost”,
say that “entertainment” would probably do a bet- “it depends”, and so forth.
ter job in emphasizing this distinction, but it is
important to note that we are also interested in a That said there is also a deeper, cognitive reason
great many AVTs, and among these are the exper- for such a choice, which also explains why in so
imental and the avant-garde that are hardly meant many of life’s situations we tend to categorize the
to be “entertaining”. world in dichotomies, or very few variables. Our
interaction with the reality that surrounds us con-
The preface stated clearly that this monograph sists of two important stages, one that we may call
does not offer a general overview of existing the- “perceptive-behavioral”, and another one that we
ories, but rather an approach to AV studies that is may call “mnemonic-organizational”. The former
not likely to be found elsewhere. Hopefully, this is of an analogical nature, whereas the latter is in
book offers innovative components at both the contrast of a digital one. For example, when we
methodological and the theoretical levels. There walk we act analogically in a continuous man-
is little by way of repetition/synthesis of existing ner, whereby our movements flow naturally and
analytical models, except for introductory pur- are not marked by discreet stages. Yet, when we
poses, but more in terms of ideas and concepts classify the act of walking in our mind, we tend
I developed during my research. Of course, this to segment it into discreet units, just as if it was
does not mean that we will not use other para- composed of separate moments, like “back-
digms, for there will be many of those. Informing ward”, “forward”, “slow”, “fast”, and so forth. Sim-
us are writers working in the areas of film and ilar forms of cognition occur roughly in every
media studies, multimodality studies, structural- moment of our life, and apply to the most diverse
ism, narratology, “auteur theory” in a broad sense, objects, events, people, groups, etc. This is quite
and so forth. By combining and reinterpreting understandable, because it is an impossible task
these existing works, and by adding new mate- to perceive analogical information and process it
rial, I am proposing an analytical interdisciplinary analogically. Not even computers are able to do
model which, hopefully, can help us to view AVTs this, instead their enormous potential to store
from a fresh and easily understandable perspec- information is due to repeating a multitude of
tive. separate acts of digitalization (bits), so numerous
as to “resemble” analogical processes. However,
I hope that all the notions you will learn in this as we know, they are never so, and the example
book are useful and important, although I do of the increasing perfection and quantity of pix-
not claim that the way you will be informed here els in an image is paradigmatic: a high-resolution
is any more useful or more important than exist- image can look like a real one, but it is still com-
ing books and existing theories. My intention is posed of pixels, that is, digital units. As humans,
to avoid oversimplifying concepts by presenting we, too, need to somehow digitalize information,
(packaging) them in “binary” forms, but rather let decompose it, divide it into stages, rather than
you, the reader, make up your own mind. Let us coping with a continuous flux of events. And, of
avoid basic binary oppositions—black or white, course, the fewer the units the easier it is to man-
yes or no, and speak more about a “time and age information. So, dividing the world and its
space” dichotomy, “realism versus fiction”, and components into very few parts (usually two) is
so forth. There may be occasions where you may typical of human beings who need to understand
regard some of the concepts as having been pack- and store information. It is no coincidence that
aged in an oversimplified manner: we shall have Claude Lévi-Strauss, one of the greatest anthro-
dichotomies such as “time and space” and “real- pologists in history approached the culture of
ism versus fiction”, and so forth. those communities that Western people barely
knew the existence of in the same way. Lévi-
Of course, this is an intentional reductionist Strauss knew that in order “to begin to under-
approach to concepts that are much more com- stand” these unknown cultures, it was important
plex, but we shall perform this action for purely to describe them in a clear and simple man-
pedagogical purposes. We already know from ner—a “binary” manner indeed. He called this