Machine Learning of Natural Language PDF
Preview Machine Learning of Natural Language
Machine Learning of Natural Language David M. W. Powers Macquarie University, Sydney Christopher C. R. Turk University ofWales, College ofCardiff Machine Learning of Natural Language ~ L-v"-_~ Springer-Verlag London Berlin Heidelberg New York Paris Tokyo Hong Kong David M. W. Powers, ThC(Hons), PhD, MACS, MIEEE Senior Research Fellow, FB Informatik, University of Kaiserslautern; Honorary Associate in Computing, MPCE, Macquarie University, Director, IMPACTLtd, 259ATrafalgar St., Petersham, New South Wales 2049, Australia Christopher C. R. Turk, MA (Cantab), DPhil (Sussex) 'Bentwys', Llanbair Discoed, NearChepstow, Gwent NB6 6LX, UK BritishLibraryCataloguinginPublicationData Powers,David Machinelearningofnaturallanguage. 1. Nationallanguagecomputersystems I. Title II.Turk,Christopher(ChristopherC. R.) 006.3'5 ISBN-13:978-3-540-19557-3 e-ISBN-13:978-1-4471-1697-4 DOl:10.1007/978-1-4471-1697-4 LibraryofCongressCataloging-in-PublicationData Powers,David(DavidM. W.) Machinelearningofnaturallanguage/DavidPowersandChristopherTurk. p. cm. Includesbibliographicalreferences. 1. Naturallanguageprocessing(Computerscience) 2. Machinelearning. I.Turk,Christopher. II.Title. QA76.9.N38P68 1989 006.3'4-dc20 89--38887 CIP Apartfrom anyfairdealingfor thepurposesofresearchorprivatestudy,orcriticism orreview, aspermittedundertheCopyright,DesignsandPatentsAct, 1988,this publicationmayonlybereproduced,storedortransmitted,inanyformorbyany means, with thepriorpermissioninwritingofthepublishers,orinthecaseof reprographicreproductioninaccordancewiththetermsoflicencesissuedbythe CopyrightLicensingAgency. Enquiriesconcerningreproductionoutsidethose terms shouldbesenttothepublishers. ©Springer-VerlagLondon Limited1989 Theuseofregistered names, trademarksetc,inthispublicationdoesnotimply,even in theabsenceofaspecificstatement,thatsuchnamesareexemptfrom therelevant lawsand regulationsandthereforefree forgeneraluse. 2128/3916-543210 Printedonacid-freepaper Preface We met because we both share the same views of language. Language is alivingorganism, producedby neural mechanisms relatinginlarge numbers as asociety. Language existsbetween minds, as a way of communicating between them, not as an autonomous process. The logical 'rules' seem to us an epiphe nomena·of the neural mechanism, rather than an essential component in language. This view of language has been advocated by an increasing number of workers, as the view that language is simply a collection of logical rules has had less and less success. People like Yorick Wilks have been able to show in paper after paper thatalmost anyrule whichcanbedevisedcanbeshownto have exceptions. The meaning does not lie in the rules. David Powers is a teacher ofcomputer science. Christopher Turk, like many workers who have come into the field of AI (Artificial Intelligence) was originally trained in literature. He moved into linguistics, and then into computational linguistics. In 1983 he took asabbatical in Roger Shank'sAI projectin the Computer Science Department at Yale University. Like an earlier visitor to the project, John Searle from California, Christopher Turk was increasingly uneasy at the view of language which was used at Yale. His attention was drawn to an article by DavidPowers in the SIGART Journal which seemed to reflect the same views. An extendedcorrespondencethenfollowed, whichculminatedin a Ph.D. written by David. Thiswas marked by Christopher, and David spent his sabbatical in 1988 at the University of Wales, Cardiff, working with Christopher. The idea of the book had been discussed before, and during this visit it became reality. The basic academic research was done by David Powers as part of his successful Ph.D. thesis. Christopher Turk added some material, and re-wrote most ofthe book. Butit reflects a stronglyheld jointviewoflanguage, andajointviewoftheway forward in computational 'understanding' of language. In recent years it has become obvious that a living organism such as human language can only be re-produced by a mechanism which follows the same path of learning. We are still at an interim stage in this development. But detailed understanding ofthe neural mechanisms, and the humanlearningprocess, are theonlywayweshallbeabletoreproducehumanlanguageona mechanical system. Ourbookseeslanguageascomplexanddynamic.Thereisfar more subtlety in language than can be uncovered by hand crafted grammars, and programs to re-produce them. We take the path of engineering by plagiarism from nature. We try to glean as much as possible from the natural, biological and behavioural sciences. We also try to learn as much as we can about the neural mechanismswhichsupportlanguagefrom the Philosophy of Science. Our focus is on the way achild learns language. The work is directed atanengineeringsolution, butwefeel thatthe natural way natural language is acquired is what we must re-produce. We knowofnoeffectivealgorithmotherthantheonewhichwe observe, butcannot adequately explain, in the development of thehumanchild.Thisalgorithm, anditsmysteries,isthesource of our insight. Theresearch program reported in ChapterFourteenwas not aimed at building a system, so much as increasing our under standing of the processes involved in language learning. The smallprograms are simplyexperiments. Ourbookisinthe end acollection ofinformation and hypotheses aboutlanguage and learning from many different disciplines. Our apology for such a wide range ofinterest is only that we do not know where the vital clue lies. Somewhere in this voyage through the archi pelago of language we hope we have shown other travellers a good place to land. vi Acknowledgements David Powers wishes to acknowledge the benefit ofdiscussion and argument, both personally and in letters, with many colleagues. Amongst these Ken Pike, and colleagues at the Department of Computer Science at the University of New South Wales, and the University of Wales Institute ofScience and Technology stand out. Michael Wise was willing, and available, to discuss many obscure comments. Professor Murray Allen and Graham McMahon have given continued moral support and warm encouragement. Christopher Turk wishes to acknowledge many colleagues, particularly at Yale and Wales. Larry Birnbaum and Roger Shank at Yale, Robin Fawcett and David Young at Wales talkedandarguedaboutthe ideas. MaghiKingofISSCO atthe UniversityofGenevaalsolistenedandargued. JenniLloydand RachaelJoneswere aninvaluablehelpassecretariesduringthe preparation of the manuscript. Catherine Aldred gave me support in the early stages, and an increasing amount of free time to finish the book. Contents 1 Art, Science and Engineering 1 Cognitive Structures; Scientific Method; Languageis Contrastive 2 Metaphor as a Cognitive Process 27 Conjecture and Refutation, Theories and Hypotheses; Specialization andAbstraction, Induction and Generalization; PartiaLAnaLysis andNoise; The Importance ofErrors and Restrictions 3 Psychology and Psycholinguistics 55 The ObservabLe Processes ofAcquisition; ExternaL Influences: Parents, Imitation, and Correction; Expansion and Reduction 4 Language Defects and Corre~tion 77 Rate and OrderofLearning; TeLegraphic Speech; Parents and Teachers: Good and Bad ExampLes; Reinforcement: PunishmentandReward 5 Cognition and Restriction 97 The Cognitive Processes ofLanguageAcquisition; The MagicaL Number Seven; Memory and Capacity Phenomena; AduLtCharacteristics 6 Nativism and Constructivism 129 Representations: Deep Structure and Language Acquisition; Acquisition ModeLs, Chomsky and Piaget; Computer Programs as PsychoLogicaLModeLs 7 Neurology and Neurolinguistics 161 Neuroanatomy: Brains, Neurons, and Synapses; NeurophysioLogy andthe Effects ofPLasticity; NeuraL Communication and Languages ofthe Brain; NeuraL Nets: Connectionistic and Locationistic ModeLs 8 The Nature of Language 187 The QuintessenceofLanguage; Epistemology, Phonology, and Prosody; Culture, Perspectives, and Metaphor 9 The Mechanics of Language 209 Contrastand Similarity: Paradigmatic Learningand Context; The Structures ofLanguage; Models ofGrammar 10 The Ubiquity ofthe Sentence 233 PronounsandAnaphora;RecursionofSyntaxandParsing; Transformational Grammar; Generative Grammar; Learning Process and Idiolect 11 Computer Science and Artificial Intelligence 253 Pattern Recognition, Problem Solving and Heuristic Search; LearningStrategies; Problems and Theoretical Limitations 12 Heuristics and Analytic Intransigence 279 Automata and Formal Languages; Methodologies: Implementation vs Experimentation; Cybernetics and Robotics; Deletionless Strategies; Formalisms; Clauses: Horn and non-Horn, unitand non-unitSystems: LUSH and PROLOG 13 Postulates, Claims and Hypotheses 303 The Bases ofMeaning and Learning; The Organization of Concepts in Learning; The Process ofLearning; The ArtificialSubsumes the Natural 14 Computer Modelling Experiments 327 BatteriesOnetoSeven;A GeneralizedToy WorldPackage; PartialAnalysis ofNLA; Future Systems; Conclusions x Chapter One Art, Science and Engineering Our subject is the acquisition of Natural Language (NL) by computers. NL is not, in our view, a surface expression, or epiphenomenon, of a deeper, underlyingcognitiveprocess inthehumanbrain.Itis ratherfun damental to, and pervasive of, cognition itself. For this reason we think thatlanguage is not the solepreserve oflinguists, butispivotal inall our interactionswith tileworld, inourscience, andinourthought. Language, onthis principle, is at the centre ofevery human intellec tual discipline. It is learned, and so is studied by psychologists. It is a function of our brains, and so is studied by neurologists. It is itself a powerful structure combining simplicity and complexity, and so its co nundrums have challenged our mathematicians and philosophers throughout the centuries. But in this electronic age, language is increas ingly the province of workers in the fields of cognitive psychology and Artificial Intelligence (AI), who record, transmit, simulate and analyze its every form and representation. There are senses inwhich language is both more and less than cognition: more because it is a medium for communicating between cognitive entities; less because it is only a me dium, amechanismoratool, servingotherprocesses. This view oflanguage is the starting point and the focus ofthe ideas we describe in this book. But our attention is mainly directed to the question of how computing machinery can acquire the cognitive struc tures which erupt in NL. Work in this field in the last thirty years has movedfrom aposture ofconfidence about the possibilityofNLprocess ingbymachine, through cleverprogramswhichwere, inthe end, seento be emulations, evenpastiches, ofthe actual processes ofunderstanding, to criticism of the very concept ofmachine understanding (best repre sented by John Searle's discussions of "intentionality"l-Christopher Turk hopes to describe this view inm05e detail in a future book-Natu ralLanguageandArtificialMechanisms ). At this time (1989), we do not, as ascientific community, know quite how this will be done. But we are sure that insight into this process will require very basic thinking about the nature of cognitive processes, and the methods of scientific thought which have tried to purify the re searcher's perceptions ofreality. For this reason our book begins with a fundamental summary of basic views of methodology, and only then moves on to bring together linguistic, scientific, neurological, and re- 2 MachineLearningofNaturalLanguage lated cognitive approaches. We endby describing experimentallearning mechanisms, some ofwhich are simplifications ofthe way.a general NL acquisitionsystemmight bebuilt. We emphasize, however, that itis still earlyin the process ofunderstanding NL and its acquisition by an auto maton. We do not pretend to complete answers, only a prospective trawling of the problems. At this stage we claim that thinking in detail aboutfirstprinciplesisthebestwayforward. The start of this exploration is the realization that the integrity of Science demands that any research should be undertaken inthe context of our entire heritage, irrespective of disciplinary divisions. The re searcher in this field should consider topics lying in the intersection of Psychology, Linguistics and Computer Science, along with the under lyingsphere ofNeurology.The disciplines listed inthis figure under Ar tificialIntelligence and Psycholinguistics are not newtoworkers inthese fields, but the scope ofwhat we include under the heading of Natural LanguageProcessing (NLP) iswider than normaLThe areaknownto AI as "Natural Language" is itselfbroad and nebulous, and it is not always clearwhether NLprogrammingismeant, orNLdatabasequeries, orNL deductionsystems, or questionanswering, or system control, ormachine translation, or what. Our argument is that computers and language go hand in hand, and it is helpful to have the widest perspective, both in terms ofwhich language processing functions can already be performed, and thosewhich cannotyetbe achievedsatisfactorily. The direction for NL research we take here is thus a multidimen sional vector of contributing interdisciplinary insights. The exploration of these insights and their background, discipline by discipline, is the subject ofthe first dozen chapters. The most useful ofthese insights are brought together and summarized in Chapter Thirteen, and are used in Chapter Fourteen to develop a research program and a collection of computationalexperiments. This choice ofabroad approach is not without problems. In the pre face to Modelling Language Behaviour, Narasimhan points out the necessityand the pitfalls ofthis approach: Byits very nature the studyoflanguage behaviourcalls for an interdis ciplinary effort. There are obvious dangers in attempting a monograph on an interdisciplinary subject. For no one individual can be a practic ingspecialist in all thedisciplines ... judgementofwhat is relevant and what is not, which issues are c.entral and which ones are not, and so forth, depends on the overall understanding one has of the field in question. I leaveit to the practicingspecialists to decide towhatextent my judgements ofongoing-work in the various fields dealt with in this bookaresound.3 We feel a similar doubt, and for this reason have always given pre cise references for materiaL Wherever possible we quote the authors'
The list of books you might like
