Image, Language, Brain: Papers from the First Mind Articulation Project Symposium

284 Pages·2000·5.62 MB·English

by  Alec Marantz

Preface
The chapters in this volume,1 now extensively revised for publication, were first pre-
sented in Tokyo 16–17 November, 1998, at the first symposium of the Mind Articu-
lation Project, a joint MIT/JST (Japan Science and Technology) research project
conductedundertheauspicesofJST’sprogramofinternationalcooperativeresearch
(ICORP).
The Mind Articulation Project, codirected by two of the editors of these proceed-
ings (Yasushi Miyashita and Wayne O’Neil), arose largely through the e¤orts of
thethirdeditor,AlecMarantz.Thisfirstsymposium,ajointundertakingofthethree
of us, brought together eleven linguists and cognitive neuroscientists (in addition to
Marantz, Miyashita, and O’Neil): Noam Chomsky and Edward Gibson (both of
MIT), Robert Desimone (NIMH), Richard Frackowiak (University College
London), Angela Friederici (Max Planck Institute/Leipzig), Masao Ito (RIKEN),
Willem Levelt (Max Planck Institute/Nijmegen), Jacques Mehler (CNRS), Helen
Neville (University of Oregon), David Poeppel (University of Maryland at College
Park), andHiroshi Shibasaki(Kyoto University).
One goal of the symposium was to examine the recent attempts to unify linguistic
theory and brain science that have grown out of the increasing awareness that a
proper understanding of language in the brain must reflect the steady advances in
linguistic theory of the past forty years. How can the understanding of language
provided by linguistic research be transformed through the study of the biological
basis of language? How can our understanding of the brain be transformed through
the same research?The best modelof suchinteraction betweencognitive science and
neurobiology is research on vision. Abilities such as visual awareness, attention, and
imagerygenerationarethefirsthighercognitiveabilitiestohavebeenfirmlylocalized
in the brain. Key findings on these matters have been enhanced by the recent explo-
sion of noninvasive brain-monitoring techniques that have been most successfully
applied intheinvestigation of the functionalanatomy ofthevisual system.
Thelong-termgoalsoftheprojectare,ontheonehand,tointegratelinguisticsand
brain science in the way that has been attempted in the study of the visual system,
viii Preface
and,ontheotherhand,toformulateacognitivetheorythatmorestronglyconstrains
visual neuroscience.
The chapters in this volume explore some of the topics just sketched, addressing
thegoalsbothofthesymposiumandoftheproject.Thus,somechaptersexaminethe
currentstatusofthecognitive/neurosciencesynthesisinthestudyofthevisualsystem.
Other chapters address whether and how linguistics and neuroscience can be inte-
grated, given the paradigm of cognitive neuroscience emerging from work on vision.
Still other chapters, though focused on vision, are primarily concerned to illustrate
‘‘how integrative brain mechanisms can be studied’’ through the use of the various
noninvasive brain-imaging techniques (seechapter 8).
For example, in chapter 11, Reynolds and Desimone integrate these approaches
and propose that attentional modulation depends on two interacting neural mecha-
nisms: one, an automatic competitive mechanism within a cortical area, and the
other, feedback from outside a cortical area that biases the competition in favor of
the neuronal population activated by the attended stimulus. They present a simple
computational circuit that provides a good fit to the data and makes easily testable
predictions.
Miyashita begins chapter 12, on the neural mechanism of visual imagery, by
quoting Alain: ‘‘Create an image of the Pantheon in your mind. Can you count the
columnsthatsupportitspediment?’’Thebasicneuralcircuitofimagerygenerationis
drawn in comparison with the cognitive model of Stephen Kosslyn. Two predictions
ofthecircuitweredirectlytestedbyneurophysiologicalapproachesinanimalmodels:
activation of internal representation of objects in the inferotemporal cortex and its
executive control by the prefrontal cortex.
Asforthechaptersonlanguage,wewillnot comment onallofthem,but letusat
least note that Levelt and Indefrey’s reanalysis (in chapter 4) of fifty-eight reports of
brain-imaging experiments on word production from the point of view of an inde-
pendently arrived at theory of lexical access in speech production reveals a remark-
able convergence between the theory and the experimental data. This convergence
leadsthemtowonderabout‘‘howmuchmorecanbeachievedifaprocessingtheory
is used beforehand to guide the planning of functional brain-imaging studies of lan-
guage?’’ And in chapter 6, Friederici presents ‘‘a tentative model of the neuronal
dynamics of auditory language comprehension’’ according to which ‘‘early syntactic
processing is independent of lexical-semantic information [, which] only comes into
play when thematic roles are assigned. If initial syntactic structure and thematic
structure map well, comprehension has taken place adequately.’’ Her results have
clear implications forpsycholinguistic models of language comprehension.
Finally, consider the research by Poeppel and Marantz on the perception of the
sounds of language, presented in detail in chapter 2. The preliminary results of this
Preface ix
work, firmly based in the theory of phonology, suggest that the interface between
the sounds of language and the cognitive system of language is embedded in the
articulatory-perceptual system. Part of auditory cortex thus appears to be language
dedicated, a surprising result ifsupported by furtherresearch.
In chapter 1, Chomsky expresses some caution, however. Commenting on the
work of the pastfifty years, he points out thatalthough
there has been intensive and often highly productive inquiry into the brain, behavior, and
cognitivefacultiesofmanyorganisms[,t]hegoalthathasarousedthemostenthusiasmisalso
likelytobethemostremote,probablybyordersofmagnitude:anunderstandingofthehuman
brainandhumanhighermentalfaculties,theirnature,andthewaystheyenterintoactionand
interaction.
From the outset, there has been no shortage of optimistic forecasts, even declarations by
distinguished researchers that the mind-body problem has been solved by advances in com-
putation, or that everything is essentially understood apart from the ‘‘hard problem’’ of
consciousness. Such conclusions surely do not withstand analysis. To an objective outside
observer—say,ascientistfromMars—theoptimismtoomightseemratherstrange,sincethere
isalsonoshortageofmuchsimplerproblemsthatarepoorlyunderstood,ornotatall.
Chapter 5, by Gibson, is in part an illustration of Chomsky’s last point. Working
on problems of complex sentence processing that Chomsky, together with George
Miller (in Miller and Chomsky 1963), addressed over thirty-five years ago, Gibson
reports progress that results entirely from behavioral studies, and not at all from
brain-imaging technology.
Thus,theconclusiontoChomsky’spaperservesasafittingandchallengingendto
this prefatory essay as well:
Exactly how the story unfolds from here depends on the actual facts of the matter.... A pri-
marygoalistobringthebodiesofdoctrineconcerninglanguageintocloserrelationwiththose
emergingfromthebrainsciencesandotherperspectives.Wemayanticipatethatricherbodies
of doctrine will interact, setting significant conditions from one level of analysis for another,
perhapsultimatelyconvergingintrueunification.Butweshouldnottaketruismsforsubstan-
tivetheses,andthereisnoplacefordogmatismastohowtheissuesmightmovetowardreso-
lution.Weknowfartoolittleforthat,andthehistoryofmodernscienceteachesuslessonsthat
Ithinkshouldnotbeignored.
To say that the chapters of this volume originated at the first Mind Articulation
Projectsymposiumistopresupposethattherewillbeatleastasecondsymposium,at
which time we hope to know better how far we have traveled along the path toward
the unification of the brain and cognitivescience, ifat all.
Wayne O’Neil (MIT)
Yasushi Miyashita(University of Tokyo)
x Preface
Note
1. This volume contains one paper not presented at the symposium, the paper by Kensuke
Sekihara, David Poeppel, Alec Marantz, and Yasushi Miyashita, and lacks a paper that was
presented,apaperbyHelenNeville.
Reference
Miller, George A., and Noam Chomsky. 1963. In Handbook of Mathematical Psychology,
volume 2,ed.R.DuncanLuce,RobertR.Bush,andEugeneGalanter,419–491.NewYork:
JohnWileyandSons.
Introduction: Mind Articulation
Alec Marantz, Yasushi Miyashita, and Wayne O’Neil
Sincethetermmindarticulationmayseemvagueorevenopaquetosome—suggesting
anumberofinterrelatedtopics—weturnfirsttoanexplicationoftherelevantmean-
ings the term has for the Mind Articulation Project.
Perhaps foremost is the notion of ‘‘articulating the mind’’—that is, investigating
the functional anatomy of the mind and the brain, figuring out what the functional
pieces of the mind are and how they work in the brain. We ‘‘articulate the structure
of the mind’’ by identifying specialized modules of computational machinery. For
example, there seems to be aspecial moduleof mind devoted to recognizing faces.
For linguists, mind articulation also suggests the ‘‘articulating’’ or ‘‘speaking’’
mind, the study of language in the minds and brains of humans. Since language is a
uniquely human mental capacity, mind articulation is then the investigation of what
makes us human, of what makes all humans the same. Among the complex mental
capacities of humans or of any species, language is perhaps the best understood; the
cognitive models that linguists and psycholinguists have developed for language are
the richest cognitive models for any cognitive function. Studying the articulating
mind in the articulating brain allows highly developed cognitive theories to help us
understand the brain as acomputationalmachine.
Human Brains/Animal Brains
The capacity for language is unique to humans—a product of our genetic endow-
ment. However, humans are genetically quite close to other primates, the genetic
distancebetweenahumanandachimpanzeebeingsmall.Theevidencesuggeststhat
the specialized human capacity for language involves a small number of genes that
haveane¤ectonthefunctionalstructureofthebrainandperhapsalsoonthemotor
and perceptual systems. The nature of the genetic basis for language provides a spe-
cial role foranimal studiesin theMind Articulation Project. Since other species lack
human language, investigations of the articulating mind must use humans and
mustfindmethodstoinvestigatethehumanarticulating brain—methodssuchasthe
various noninvasive brain-imaging techniques employed in the research reported on
2 Marantz,Miyashita,andO’Neil
in many of the chapters that follow: positron emission tomography (PET), func-
tional magnetic resonance imaging (fMRI), magnetoencephalography (MEG), and
event-related (brain) potential (ERP). However, the small genetic di¤erence between
humans and other primates suggests that, in evolution, language built heavily on
preexistingstructuresandcapacities.Animalmodelsshouldilluminatewhichaspects
oflanguagearepartofasharedprimatestructureandwhicharelanguage-particular
innovations.
Language as aSymbolic ComputationalSystem
Linguists study language as a symbolic computational system, internal to the minds
and brains of humans. Language connects form and meaning—where form is, at
least, the sounds of spoken language or the movements of the body that are the
expression of sign languages. As is well known, the units of language do not involve
an iconic relation between form and meaning: the sound of the word cat does not
soundlikeanyaspectofacat,notitssize,color,shape,orvocalizations.Thesmallest
units of language, then, are noniconic symbols. Small pieces of language like words
are combined into larger constructions like phrases and sentences. Combinations of
words have meanings that are also noniconic. If one considers the hierarchical
structureofasentenceasin(1),thereisnothingaboutthestructuralrelationsamong
the elements in the sentence that transparently or iconically reflects the meaningful
relations among the constituents. For example, there is nothing about the juxtaposi-
tion of the and cat to suggest the semantic e¤ect that the definite determiner the has
on the common noun cat.
(1) a. The man saw the cat.
b.
the man saw
the cat
The computational system of language has thestructure shown in (2):
(2) Motor commands (forspeech) Syntax (combinatory system)
(cid:131)(cid:131)(cid:131)(cid:131)(cid:131)(cid:131)(cid:131) (cid:131) (cid:131) (cid:131) (cid:131) (cid:131) (cid:131)(cid:131)(cid:131)(cid:131)(cid:131)(cid:131)(cid:131)(cid:131)(cid:131)(cid:131)!
Sound interpretation (form) Meaninginterpretation (conceptual system)
(cid:131) (cid:131) (cid:131) (cid:131) (cid:131) (cid:131)(cid:131)
Acoustic perception
Introduction 3
There is a combinatoric engine, called the syntax, that puts the elementary pieces
of language together into the kind of hierarchical structure shown in the tree graph
(1) for the English sentence The man saw the cat. The units combined by the syntax
havefeaturesorpropertiesthatmakethemsubjecttointerpretationbothinformand
in meaning. Structures built by the syntax are thus subject to phonological or sound
interpretation, either in terms of the sounds or acoustics of language or in terms of
the motor commands necessary to produce speech (or sign language). The same
structures are subject to interpretation in terms of the meaningful concepts of our
mental life. The representations of language interface with nonlinguistic systems,
then, at two points or interfaces—with the perceptual and motor systems involved
in speech production and perception and with the conceptual system involved in
meaningful language comprehension.
There is some question about the capacity of nonhumans to use symbols at all,
whether iconic or noniconic. But the special nature of human language goes beyond
the simple use of symbols. The combination of units in language is recursive: in (1),
when we put the and cat together, the result of the combination becomes a unit for
further combination, here with the word saw. The recursive nature of linguistic
computations gives rise to what Chomsky has called the ‘‘discrete infinity’’ of lan-
guage: the potentially infinite use of finite means. In addition, although the units of
language have features interpretable in terms of sound (or movements) and in terms
of meaningful concepts, the combinatoric system of language combines the units to
a large extent independently of their interpretation. Thus, although the units them-
selves consist largely of features with concrete interpretations, the computational
system of language treats these units as abstract symbols. Compare how computers
manipulatecommandsincomputer languages.Such commandsmight beinterpreted
as instructions to draw, say, a particular character on the computer screen, but
internal to the computations of the computer, the commands are strings of 1s and
0s, manipulated as binary numbers. Similarly, the elements of language are treated
internal to the linguistic system according to their internal form—that is, according
to their feature structure considered abstractly, independent of their interpretation
in sound or meaning.
TransformationalGrammar/Articulating Minds
The theory of language schematically diagrammed in (2) holds important implica-
tions for how language works in the brain and for how brains work at language.
Notethatinthisdiagram,thecombinatoricorgenerativeengineoflanguageisinthe
syntax, which creates structures subject to interpretation in sound and meaning.
Thereisnoindependentcombinatoricengineoneitherthesoundorthemeaningside
of the diagram. Theories of language that contain this single generative engine have
beencalled transformational.