Fundamentals of the New Artificial Intelligence: Neural, Evolutionary, Fuzzy and More

266 Pages·2007·3.99 MB·English

by  Toshinori Munakata

TEXTS IN COMPUTER SCIENCE
Editors
David Gries
Fred B. Schneider
TEXTS IN COMPUTER SCIENCE
Apt and Olderog, Verification of Sequential and Concurrent
Programs, Second Edition
Alagar and Periyasamy, Specification of Software Systems
Back and von Wright, Refinement Calculus: A Systematic
Introduction
Beidler, Data Structures and Algorithms: An Object-Oriented
Approach Using Ada 95
Bergin, Data Structures Programming: With the Standard
Template Library in C++
Brooks, C Programming: The Essentials for Engineers and
Scientists
Brooks, Problem Solving with Fortran 90: For Scientists and
Engineers
Dandamudi, Fundamentals of Computer Organization and Design
Dandamudi, Introduction to Assembly Language Programming:
For Pentium and RISC Processors, Second Edition
Dandamudi, Introduction to Assembly Language Programming:
From 8086 to Pentium Processors
Fitting, First-Order Logic and Automated Theorem Proving,
Second Edition
Grillmeyer, Exploring Computer Science with Scheme
Homer and Selman, Computability and Complexity Theory
Immerman, Descriptive Complexity
Jalote, An Integrated Approach to Software Engineering, Third
Edition
(continued after index)
Toshinori Munakata
Fundamentals of the New
Artificial Intelligence
Neural, Evolutionary, Fuzzy and More
Second Edition
Toshinori Munakata
Computer and Information Science Department
Cleveland State University
Cleveland, OH 44115
USA
[e-mail]
ISBN: 978-1-84628-838-8 e-ISBN: 978-1-84628-839-5
DOI: 10.1007/978-1-84628-839-5
British Library Cataloguing in Publication Data
A catalogue record for this book is available from the British Library
Library of Congress Control Number: 2007929732
© Springer-Verlag London Limited 2008
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the
Copyright, Designs and Patents Act of 1988, this publication may only be reproduced, stored or transmitted, in any
form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction
in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction
outside those terms should be sent to the publishers.
The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement,
that such names are exempt from the relevant laws and regulations and therefore free for general use.
The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in
this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made.
Printed on acid-free paper
9 8 7 6 5 4 3 2 1
Springer Science+Business Media
springer.com
Preface

This book was originally titled “Fundamentals of the New Artificial Intelligence:
Beyond Traditional Paradigms.” I have changed the subtitle to better represent the
contents of the book. The basic philosophy of the original version has been kept in
the new edition. That is, the book covers the most essential and widely employed
material in each area, particularly the material important for real-world applications.
Our goal is not to cover every latest progress in the fields, nor to discuss every
detail of various techniques that have been developed. New sections/subsections
added in this edition are: Simulated Annealing (Section 3.7), Boltzmann Machines
(Section 3.8) and Extended Fuzzy if-then Rules Tables (Sub-section 5.5.3). Also,
numerous changes and typographical corrections have been made throughout the
manuscript. The Preface to the first edition follows.

General scope of the book
Artificial intelligence (AI) as a field has undergone rapid growth in diversification
and practicality. For the past few decades, the repertoire of AI techniques has
evolved and expanded. Scores of newer fields have been added to the traditional
symbolic AI. Symbolic AI covers areas such as knowledge-based systems, logical
reasoning, symbolic machine learning, search techniques, and natural language
processing. The newer fields include neural networks, genetic algorithms or
evolutionary computing, fuzzy systems, rough set theory, and chaotic systems.
The traditional symbolic AI has been taught as the standard AI course, and there
are many books that deal with this aspect. The topics in the newer areas are often
taught individually as special courses, that is, one course for neural networks,
another course for fuzzy systems, and so on. Given the importance of these fields
together with the time constraints in most undergraduate and graduate computer
science curricula, a single book covering the areas at an advanced level is desirable.
This book is an answer to that need.

Specific features and target audience
The book covers the most essential and widely employed material in each area, at a
level appropriate for upper undergraduate and graduate students. Fundamentals of
both theoretical and practical aspects are discussed in an easily understandable
vi Preface

fashion. Concise yet clear description of the technical substance, rather than
journalistic fairy tale, is the major focus of this book. Other non-technical
information, such as the history of each area, is kept brief. Also, lists of references
and their citations are kept minimal.
The book may be used as a one-semester or one-quarter textbook for majors in
computer science, artificial intelligence, and other related disciplines, including
electrical, mechanical and industrial engineering, psychology, linguistics, and
medicine. The instructor may add supplementary material from abundant resources,
or the book itself can also be used as a supplement for other AI courses.
The primary target audience is seniors and first- or second-year graduates. The
book is also a valuable reference for researchers in many disciplines, such as
computer science, engineering, the social sciences, management, finance, education,
medicine, and agriculture.

How to read the book
Each chapter is designed to be as independent as possible of the others. This is
because of the independent nature of the subjects covered in the book. The
objective here is to provide an easy and fast acquaintance with any of the topics.
Therefore, after glancing over the brief Chapter 1, Introduction, the reader can start
from any chapter, also proceeding through the remaining chapters in any order
depending on the reader's interests. An exception to this is that Sections 2.1 and
2.2 should precede Chapter 3. In diagram form, the required sequence can be
depicted as follows.

the rest of Chapter 2
Sections 2.1 and 2.2
Chapter 3
Chapter 1 —— Chapter 4
Chapter 5
Chapter 6
Chapter 7

The relationship among topics in different chapters is typically discussed close to
the end of each chapter, whenever appropriate.
The book can be read without writing programs, but coding and experimentation
on a computer is essential for complete understanding these subjects. Running so-
called canned programs or software packages does not provide the target
comprehension level intended for the majority of readers of this book.

Prerequisites
Prerequisites in mathematics. College mathematics at freshman (or possibly at
sophomore) level are required as follows:
Chapters 2 and 3 Neural Networks: Calculus, especially partial differentiation,
concept of vectors and matrices, and
elementary probability.
Preface vii

Chapter 4 Genetic algorithms: Discrete probability.
Chapter 5 Fuzzy Systems: Sets and relations, logic, concept of vectors
and matrices, and integral calculus.
Chapter 6 Rough Sets: Sets and relations. Discrete probability.
Chapter 7 Chaos: Concept of recurrence and ordinary
differential equations, and vectors.
Highlights of necessary mathematics are often discussed very briefly before the
subject material. Instructors may further augment the basics if students are
unprepared. Occasionally some basic mathematics elements are repeated briefly in
relevant chapters for an easy reference and to keep each chapter independent as
possible.
Prerequisites in computer science. Introductory programming in a conventional
high-level language (such as C or Java) and data structures. Knowledge of a
symbolic AI language, such as Lisp or Prolog, is not required.

Toshinori Munakata
Contents

Preface .............................................................................................................. v

1 Introduction ..................................................................................................... 1
1.1 An Overview of the Field of Artificial Intelligence ................................ 1
1.2 An Overview of the Areas Covered in this Book .................................... 3

2 Neural Networks: Fundamentals and the Backpropagation Model ........... 7
2.1 What is a Neural Network? ..................................................................... 7
2.2 A Neuron ................................................................................................. 7
2.3 Basic Idea of the Backpropagation Model .............................................. 8
2.4 Details of the Backpropagation Mode ..................................................... 15
2.5 A Cookbook Recipe to Implement the Backpropagation Model ............. 22
2.6 Additional Technical Remarks on the Backpropagation Model .............. 24
2.7 Simple Perceptrons .................................................................................. 28
2.8 Applications of the Backpropagation Model ........................................... 31
2.9 General Remarks on Neural Networks .................................................... 33

3 Neural Networks: Other Models .................................................................... 37
3.1 Prelude ..................................................................................................... 37
3.2 Associative Memory ................................................................................ 40
3.3 Hopfield Networks .................................................................................. 41
3.4 The Hopfield-Tank Model for Optimization Problems: The Basics ....... 46
3.4.1 One-Dimensional Layout ............................................................. 46
3.4.2 Two-Dimensional Layout ............................................................ 48
3.5 The Hopfield-Tank Model for Optimization Problems: Applications ..... 49
3.5.1 The N-Queen Problem ................................................................. 49
3.5.2 A General Guideline to Apply the Hopfield-Tank Model to
Optimization Problems ................................................................ 54
3.5.3 Traveling Salesman Problem (TSP) ............................................. 55
3.6 The Kohonen Model ................................................................................ 58
3.7 Simulated Annealing ............................................................................... 63
x Contents

3.8 Boltzmann Machines ............................................................................... 69
3.8.1 An Overview ................................................................................ 69
3.8.2 Unsupervised Learning by the Boltzmann Machine: The Basics
Architecture.................................................................................. 70
3.8.3 Unsupervised Learning by the Boltzmann Machine: Algorithms ..... 76
3.8.4 Appendix. Derivation of Delta-Weights ..................................... 81

4 Genetic Algorithms and Evolutionary Computing ....................................... 85
4.1 What are Genetic Algorithms and Evolutionary Computing? ................. 85
4.2 Fundamentals of Genetic Algorithms ...................................................... 87
4.3 A Simple Illustration of Genetic Algorithms .......................................... 90
4.4 A Machine Learning Example: Input-to-Output Mapping ...................... 95
4.5 A Hard Optimization Example: the Traveling Salesman
Problem (TSP) ......................................................................................... 102
4.6 Schemata ................................................................................................. 108
4.6.1 Changes of Schemata Over Generations ...................................... 109
4.6.2 Example of Schema Processing ................................................... 113
4.7 Genetic Programming .............................................................................. 116
4.8 Additional Remarks ................................................................................. 118

5 Fuzzy Systems .................................................................................................. 121
5.1 Introduction ............................................................................................. 121
5.2 Fundamentals of Fuzzy Sets .................................................................... 123
5.2.1 What is a Fuzzy Set? .................................................................... 123
5.2.2 Basic Fuzzy Set Relations ............................................................ 125
5.2.3 Basic Fuzzy Set Operations and Their Properties ........................ 126
5.2.4 Operations Unique to Fuzzy Sets ................................................. 128
5.3 Fuzzy Relations ....................................................................................... 130
5.3.1 Ordinary (Nonfuzzy) Relations .................................................... 130
5.3.2 Fuzzy Relations Defined on Ordinary Sets .................................. 133
5.3.3 Fuzzy Relations Derived from Fuzzy Sets ................................... 138
5.4 Fuzzy Logic ............................................................................................. 138
5.4.1 Ordinary Set Theory and Ordinary Logic .................................... 138
5.4.2 Fuzzy Logic Fundamentals .......................................................... 139
5.5 Fuzzy Control .......................................................................................... 143
5.5.1 Fuzzy Control Basics ................................................................... 143
5.5.2 Case Study: Controlling Temperature with a Variable
Heat Source 150
5.5.3 Extended Fuzzy if-then Rules Tables .......................................... 152
5.5.4 A Note on Fuzzy Control Expert Systems ................................... 155
5.6 Hybrid Systems ....................................................................................... 156
5.7 Fundamental Issues ................................................................................. 157
5.8 Additional Remarks ................................................................................. 158

6 Rough Sets ........................................................................................................ 162
6.1 Introduction ............................................................................................. 162
6.2 Review of Ordinary Sets and Relations ................................................... 165