Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp

949 Pages·1992·37.86 MB·English

by  Peter Norvig

Paradigms of
Artificial Intelligence
Programming:
C A S E S T U D I E S IN C O M M O N LISP
Peter Norvig
M O R G A N K A U F M A N N PUBLISHERS ^ SAN F R A N C I S C O , CALIFORNIA
Paradigms of
Artificial Intelligence
Programming:
C A SE S T U D I ES IN C O M M ON LISP
Peter Norvig
M O R G AN K A U F M A NN PUBLISHERS ^ SAN F R A N C I S C O, CALIFORNIA
Sponsoring Editor Michael B. Morgan
Production Manager Yonie Overton
Cover Designer Sandra Popovich
Text Design/Composition SuperScnpt Typography
Copyeditor Barbara Beidler Kendnck
Proofreaders Lynn Meinhardt, Shanlyn Hovind, Gary Morus
Printer Malloy Lithographing
Morgan Kaufmann Publishers, Inc.
Editorial and Sales Office:
340 Pine Street, Sbcth Floor
San Francisco, CA 94104-3205
USA
Telephone [phone]
Facsimüe 415/982-2665
Internet [email protected]
Web site http://mkp.com
© 1992 Morgan Kaufmann Publishers, Inc.
All rights reserved
Printed in the United States of America
03 02 Ol 8 7 6
No part of this publication may be reproduced, stored in a retrieval system, or
transmitted in any form or by any means-electronic, photocopying, recording, or
otherwise—without the prior written permission of the publisher.
Library of Congress Cataloging-in-Publication Data
Norvig, Peter.
Paradigms of artificial inteUigence programming: case studies in
common Lisp / Peter Norvig.
p. cm.
Includes bibliographical references and index.
ISBN 1-55860-191-0:
1. Electronic digital computers-Programming. 2. COMMON LISP
(Computer program language) 3. Artificial intelligence. I. Title.
QA76.6.N6871991
006.3-dc20 91-39187
CIP
To my family,..
Preface
paradigm η 1 an example or pattern; esp an outstandingly clear or typical example.
-Longman's Dictionary of the EngUsh Language, 1984
This book is concerned with three related topics: the field of artificial intelligence, or AI; the skill
of computer programming; and the programming language Common Lisp. Careful readers of
this book can expect to come away with an appreciation of the major questions and techniques
of AI, an understanding of some important AI programs, and an ability to read, modify, and
create programs using Common Lisp. The examples in this book are designed to be clear
examples of good programming style—paradigms of programming. They are also paradigms
of AI research—historically significant programs that use widely applicable techniques to solve
important problems.
Just as a liberal arts education includes a course in "the great books" of a culture, so this book
is, at one level, a course in "the great programs" that define the AI culture.^
At another level, this book is a highly technical compendium of the knowledge you will need
to progress from being an intermediate Lisp programmer to being an expert. Parts I and II are
designed to help the novice get up to speed, but the complete beginner may have a hard time
even with this material. Fortunately, there are at least five good texts available for the beginner;
see page xiii for my recommendations.
^This does not imply that the programs chosen are the best of all AI programs—just that
they are representative.
νίίί PREFACE
All too often, the teaching of computer programming consists of explaining the
syntax of the chosen language, showing the student a 10-line program, and then
asking the student to write programs. In this book, we take the approach that the
best way to learn to write is to read (and conversely, a good way to improve reading
skills is to write). After the briefest of introductions to Lisp, we start right off with
complex programs and ask the reader to understand and make small modifications
to these programs.
The premise of this book is that you can only write something useful and inter­
esting when you both understand what makes good writing and have something
interesting to say. This holds for writing programs as well as for writing prose. As
Kernighan and Plauger put it on the cover of Software Tools in Pascal:
Good programming is not learned from generalities, but by seeing how signif­
icant programs can be made clean, easy to read, easy to maintain and modify,
human-engineered, efficient, and reliable, by the application of common sense
and good programming practices. Careful study and imitation of good programs
leads to better writing.
The proud craftsman is often tempted to display only the finished work, without
any indication of the false starts and mistakes that are an unfortunate but unavoidable
part of the creative process. Unfortunately, this reluctance to unveil the process is
a barrier to learning; a student of mathematics who sees a beautiful 10-line proof in
a textbook can marvel at its conciseness but does not learn how to construct such a
proof. This book attempts to show the complete programming process, "warts and
all." Each chapter starts with a simple version of a program, one that works on some
examples but fails on others. Each chapter shows how these failures can be analyzed
to build increasingly sophisticated versions of the basic program. Thus, the reader
can not only appreciate the final result but also see how to learn from mistakes and
refine an initially incomplete design. Furthermore, the reader who finds a particular
chapter is becoming too difficult can skip to the next chapter, having gained some
appreciation of the problem area, and without being overwhelmed by the details.
This book presents a body of knowledge loosely known as "AI programming
techniques," but it must be recognized that there are no clear-cut boundaries on this
body of knowledge. To be sure, no one can be a good AI programmer without first
being a good programmer. Thus, this book presents topics (especially in parts III
and V) that are not AI per se, but are essential background for any AI practitioner.
Why Lisp? Why Common Lisp?
Lisp is one of the oldest programming languages still in widespread use today. There
have been many versions of Lisp, each sharing basic features but differing in detail.
In this book we use the version called Common Lisp, which is the most widely
accepted standard. Lisp has been chosen for three reasons.
PREFACE IX
First, Lisp is the most popular language for AI programming, particularly in the
United States. If you're going to learn a language, it might as well be one with a
growing literature, rather than a dead tongue.
Second, Lisp makes it easy to capture relevant generalizations in defining new
objects. In particular. Lisp makes it easy to define new languages especially targeted
to the problem at hand. This is especially handy in AI applications, which often
manipulate complex information that is most easily represented in some novel form.
Lisp is one of the few languages that allows full flexibility in defining and manipu­
lating programs as well as data. All programming languages, by definition, provide
a means of defining programs, but many other languages limit the ways in which a
program can be used, or limit the range of programs that can be defined, or require
the programmer to explicitly state irrelevant details.
Third, Lisp makes it very easy to develop a working program fast. Lisp programs
are concise and are uncluttered by low-level detail. Common Lisp offers an unusually
large number of useful predefined objects, including over 700 functions. The pro­
gramming environment (such as debugging tools, incremental compilers, integrated
editors, and interfaces to window systems) that surround Lisp systems are usually
very good. And the dynamic, interactive nature of Lisp makes it easy to experiment
and change a program while it is being developed.
It must be mentioned that in Europe and Japan, Prolog has been as popular as
Lisp for AI work. Prolog shares most of Lisp's advantages in terms of flexibility and
conciseness. Recently, Lisp has gained popularity worldwide, and Prolog is becom­
ing more well known in the United States. As a result, the average AI worker today is
likely to be bilingual. This book presents the key ideas behind Prolog in chapters 11
and 12, and uses these ideas in subsequent chapters, particularly 20 and 21.
The dialect of Lisp known as Scheme is also gaining in popularity, but primarily
for teaching and experimenting with programming language design and techniques,
and not so much for writing large AI programs. Scheme is presented in chapters 22
and 23. Other dialects of Lisp such as Franz Lisp, MacLisp, InterLisp, ZetaLisp,
and Standard Lisp are now considered obsolete. The only new dialect of Lisp to be
proposed recently is EuLisp, the European Lisp. A few dialects of Lisp live on as
embedded extension languages. For example, the Gnu Emacs text editor uses elisp,
and the AutoCad computer-aided design package uses AutoLisp, a derivative of Xlisp.
In the future, it is likely that Scheme will become a popular extension language, since
it is small but powerful and has an officially sanctioned standard definition.
There is a myth that Lisp (and Prolog) are "special-purpose" languages, while
languages like Pascal and C are "general purpose." Actually, just the reverse is
true. Pascal and C are special-purpose languages for manipulating the registers and
memory of a von Neumann-style computer. The majority of their syntax is devoted
to arithmetic and Boolean expressions, and while they provide some facilities for
forming data structures, they have poor mechanisms for procedural abstraction
or control abstraction. In addition, they are designed for the state-oriented style
PREFACE
of programming: computing a result by changing the value of variables through
assignment statements.
Lisp, on the other hand, has no special syntax for arithmetic. Addition and
multiplication are no more or less basic than list operations like appending, or string
operations like converting to upper case. But Lisp provides all you will need for
programming in general: defining data structures, functions, and the means for
combining them.
The assignment-dominated, state-oriented style of programming is possible in
Lisp, but in addition object-oriented, rule-based, and functional styles are all sup­
ported within Lisp. This flexibihty derives from two key features of Lisp: First, Lisp
has a powerful macro facility, which can be used to extend the basic language. When
new styles of programming were invented, other languages died out; Lisp simply
incorporated the new styles by defining some new macros. The macro facility is
possible because Lisp programs are composed of a simple data structure: the list.
In the early days, when Lisp was interpreted, most manipulation of programs was
done through this data structure. Nowadays, Lisp is more often compiled than in­
terpreted, and programmers rely more on Lisp's second great flexible feature: the
function. Of course, other languages have functions, but Lisp is rare in allowing the
creation of new functions while a program is running.
Lisp's flexibility allows it to adapt as programming styles change, but more impor­
tantly. Lisp can adapt to your particular programming problem. In other languages
you fit your problem to the language; with Lisp you extend the language to fit your
problem.
Because of its flexibility. Lisp has been succesful as a high-level language for rapid
prototyping in areas such as AI, graphics, and user interfaces. Lisp has also been
the dominant language for exploratory programming, where the problems are so
complex that no clear solution is available at the start of the project. Much of AI falls
under this heading.
The size of Common Lisp can be either an advantage or a disadvantage, depending
on your outlook. In David Touretzky's (1989) fine book for beginning programmers,
the emphasis is on simplicity. He chooses to write some programs slightly less
concisely, rather than introduce an esoteric new feature (he cites pushnew as an
example). That approach is entirely appropriate for beginners, but this book goes
well past the level of beginner. This means exposing the reader to new features of
the language whenever they are appropriate. Most of the time, new features are
described as they are introduced, but sometimes explaining the details of a low-
level function would detract from the explanation of the workings of a program.
In accepting the privilege of being treated as an "adult," the reader also accepts a
responsibility—to look up unfamiliar terms in an appropriate reference source.
PREFACE Χ"
Outline of the Book
This book is organized into five parts.
Part I introduces the Common Lisp programming language.
Chapter 1 gives a quick introduction by way of small examples that demonstrate
the novel features of Lisp. It can be safely skipped or skimmed by the experienced
programmer.
Chapter 2 is a more extended example showing how the Lisp primitives can be
put together to form a program. It should be studied carefully by the novice, and
even the experienced programmer will want to look through it to get a feel for my
programming style.
Chapter 3 provides an overview of the Lisp primitives. It can be skimmed on first
reading and used as a reference whenever an unfamiliar function is mentioned in
the text.
Part I has been kept intentionally brief, so that there is more room for presenting
actual AI programs. Unfortunately, that means that another text or reference book
(or online help) may be needed to clarify some of the more esoteric features of the
language. My recommendations for texts are on page xiii.
The reader may also want to refer to chapter 25, which offers some debugging
and troubleshooting hints.
Part II covers four early AI programs that all use rule-based pattern-matching
techniques. By starting with relatively simple versions of the programs and then
improving them and moving on to more complex programs, the reader is able to
gradually acquire increasingly advanced programming skills.
Chapter 4 presents a reconstruction of GPS, the General Problem Solver. The
implementation follows the STRIPS approach.
Chapter 5 describes ELIZA, a program that mimics human dialogue. This is
followed by a chapter that generalizes some of the techniques used in GPS and ELIZA
and makes them available as tools for use in subsequent programs.
Chapter 7 covers STUDENT, a program that solves high-school-level algebra word
problems.
Chapter 8 develops a small subset of the MACSYMA program for doing symbolic
algebra, including differential and integral calculus. It may be skipped by those who
shy away from heavy mathematics.
Part III detours from AI for a moment to present some general tools for more
efficient programming. The reader who masters the material in this part can be
considered an advanced Lisp programmer.
Chapter 9 is a detailed study of efficiency techniques, concentrating on caching,
indexing, compilation, and delaying computation. Chapter 10 covers lower-level effi­
ciency issues such as using declarations, avoiding garbage generation, and choosing
the right data structure.
χίί PREFACE
Chapter 11 presents the Prolog language. The aim is two-fold: to show how to
write an interpreter for another language, and to introduce the important features
of Prolog, so that they can be used where appropriate. Chapter 12 shows how a
compiler for Prolog can be 20 to 200 times faster than the interpreter.
Chapter 13 introduces object-oriented programming in general, then explores the
Common Lisp Object System (CLOS).
Chapter 14 discusses the advantages and limitations of both logic-oriented and
object-oriented programming, and develops a knowledge representation formalism
using all the techniques of part III.
Part IV covers some advanced AI programs.
Chapter 15 uses the techniques of part III to come up with a much more efficient
implementation of MACSYMA. It uses the idea of a canonical form, and replaces the
very general rewrite rule approach with a series of more specific functions.
Chapter 16 covers the EMYCIN expert system shell, a backward chaining rule-
based system based on certainty factors. The MYCIN medical expert system is also
covered briefly.
Chapter 17 covers the Waltz line-labeling algorithm for polyhedra (using Huffman-
Clowes labels). Different approaches to constraint propagation and backtracking
are discussed.
Chapter 18 presents a program that plays an excellent game of Othello. The
technique used, alpha-beta searching, is appropriate to a wide variety of two-person
games.
Chapter 19 is an introduction to natural language processing. It covers context-
free grammar, top-down and bottom-up parsing, chart parsing, and some semantic
interpretation and preferences.
Chapter 20 extends the linguistic coverage of the previous chapter and introduces
logic grammars, using the Prolog compiler developed in chapter 11.
Chapter 21 is a fairly comprehensive grammar of English using the logic grammar
formalism. The problems of going from a simple idea to a realistic, comprehensive
program are discussed.
Part V includes material that is peripheral to AI but important for any serious
Lisp programmer.
Chapter 22 presents the Scheme dialect of Lisp. A simple Scheme interpreter is
developed, then a properly tail-recursive interpreter, then an interpreter that explic­
itly manipulates continuations and supports cal 1 /cc. Chapter 23 presents a Scheme
compiler.
Chapter 24 presents the features that are unique to American National Standards
Institute (ANSI) Common Lisp. This includes the 1 oop macro, as well as error
handling, pretty printing, series and sequences, and the package facility.
Chapter 25 is a guide to troubleshooting and debugging Lisp programs.