Evolving Connectionist Systems: Methods and Applications in Bioinformatics, Brain Study and Intelligent Machines

307 Pages·2003·18.936 MB·English

by  Nikola Kasabov PhD, MSc, FRSNZ (auth.)

Perspectives in Neural Computing
Springer-Verlag London Ltd.
Also in this series:
Gustavo Deco and Dragan Obradovic
An Information-Theoretic Approach to Neural Computing
0-387-94666-7
Achilleas Zapranis and Apostolos-Paul Refenes
Principles of Neural Model Identification, Selection and Adequacy
1-85233-139-9
Walter J. Freeman
Neurodynamics: An Exploration in Mesoscopic Brain Dynamics
1-85233-616-1
H. Malmgren, M Borga and L. Niklasson (Eds)
Artificial Neural Networks in Medicine and Biology
1-85233-289-1
Mark Girolami
Advances in Independent Component Analysis
1-85233-263-8
Robert M. French and Jacques P. Sougne (Eds)
Connectionist Models of Learning, Development and Evolution
1-85233-354-5
Roberto Tagliaferri and Maria Marinaro (Eds)
Neural Nets -WIRN Vietri-Ol
1-8S233-505-X
Artur S. d'Avila Garcez, Krysia B. Broda and Dov M. Gabbay
Neural-Symbolic Learning Systems
1-85233-512-2
Jimmy Shadbolt and John G. Taylor (Eds)
Neural Networks and the Financial Markets
1-85233-531-9
Related Title
Robert Hecht-Nielsen and Thomas McKenna (Eds)
Computational Models for Neuroscience: Human Cortical Information
Processing
1-85233-593-9
Nikola Kasabov
Evolving
Con nection ist
Systems
Methods and Applications in Bioinformatics,
Brain Study and Intelligent Machines
t
Springer
Nikola Kasabov, PhD, MSc, FRSNZ
University of Otago, Dunedin, New Zealand
Director of Knowledge Engineering and Discovery Research Institute,
Auckland University of Technology, New Zealand
Series Editor
J.G. Taylor, BA, BSc, MA, PhD, FlnstP
Centre for Neural Networks, Department of Mathematics,
King's College, Strand, London WC2R 2LS, UK
British Library Cataloguing in Publication Data
A catalogue record for this book is available from the British Library
Library of Congress Cataloging-in-Publication Data
Kasabov, Nikola, K.
Evolving connectionist systems : methods and applications in bioinformatics, brain
study and intelligent machi nes 1 Nikola Kasabov.
p. cm. --(Perspectives in neural computing, ISSN 1431-6854)
Includes bibliographical references and index.
ISBN 978-1-85233-400-0 ISBN 978-1-4471-3740-5 (eBook)
DOI 10.1007/978-1-4471-3740-5
1. Neural computers. 2. Brain--Computer simulation. 3. Bioinformatics. 4. Artificial
intelligence. 1. Title. II. Series.
QA76.87 .K39 2002
006.3'2--dc21 2002026863
Apart from any fair dealing for the purposes of research or private study, or criticism or
review, as permitted under the Copyright, Designs and Patents Act 1988, this publication
may only be reproduced, stored or transmitted, in any form or by any means, with the prior
permission in writing of the publishers, or in the case of reprographic reproduction in
accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries
concerning reproduction outside those terms should be sent to the publishers.
Perspectives in Neural Computing ISSN 1431-6854
ISBN 978-1-85233-400-0
http://www.springer.co.uk
© Springer-Verlag London 2003
Originally published by Springer-Verlag London Limited in 2003
Whilst we have made considera bie efforts to contact ali holders of copyright material
contained in this book, we have failed to locate some of these. Should holders wish to contact
the Publisher, we will be happy to come to some arrangement with them.
The use of registered names, trademarks etc. in this publication does not imply, even in the
absence of a specific statement, that such names are exempt from the relevant laws and
regulations and therefore free for general use.
The publisher makes no representation, express or implied, with re gard to the accuracy of
the information contained in this book and cannot accept any legal responsibility or Iiability
for any errors or omissions that may be made.
Typesetting: lan Kingston Editorial Services, Nottingham, UK
34/3830-543210 Printed on acid-free paper SPIN 10786103
For Diana
Acknowledgements
This work was partially supported by the research grant UOOX0016,
"Connectionist-based intelligent information systems", funded by the
New Zealand Foundation for Research, Science, and Technology" and the
New Economy Research Fund. Ia m grateful for the support and encour
agement I received from the editorial team of the series "Perspectives in
Neural Computing" of Springer-Verlag in London, especially from the
editor, Professor John G. Taylor.
There are a number of people whom I would like to thank for their
participation in many of the experiments presented in this book. These
are several colleagues, research associates and postgraduate students I
worked with at the University of Otago in the period from 1998 till 2002:
Da Deng, Qun Song, Brendon Woodford, Mike Watts, Richard Kilgour,
Matthias Futschik, Melanie Middlemiss, Akbar Ghobakhlou, Jaesoo Kim,
Irena Koprinska, Georgi Iliev, Mark Laws, Waleed Abdulla, Carl Leichter,
Dr John Taylor, Dr Mike Paulin, and my daughter Assia Kassabova.
I am very grateful to my assistant Mrs Kirsty Richards, who helped me
greatly with the technical preparation and the proofreading of the book.
My thanks and love go to Diana, my wife, who took several of her holi
days to work with me on the book, and was the best critic I have ever had in
my life.
I have presented some parts of the book at conferences and I appreciate
the discussions I had with a number of colleagues. Among them are
Walter Freeman and Lotfi Zadeh - both from University of California at
Berkeley; Takeshi Yamakawa - Kyushu Institute of Technology; John G.
Taylor - Kings College, London; Shun-ichi Amari, Gen Matsumoto,
Andrjey Cichocki and Ceese van Leuwen - RIKEN; Mario Fedrizzi,
University of Trento; Lee Giles, Pennsylvania University; Tony Reeve,
Department of Biochemistry, University of Otago; John Taylor, Depart
ment of Linguistics, University of Otago; Teuvo Kohonen and Errki Oja
Helsinki University; Vic Calaghan and Graham Clarke - University of
Essex; Paul P.Wang - Duke University; Jaap van den Herik, Eric Postma,
Anton Nijholt and Mannes Pool, University of Maastricht and the Univer
sity of Twente; Michael Arbib - University of Southern California; Mitko
Dimitrov - National Cancer Institute in Frederick, and many more.
I remember a comment by Walter Freeman when I presented the
concept of evolving connectionist systems (ECOS) at the Iizuka'98 confer
ence in Japan: "Throw the "chemicals" and let the system grow, is that
vii
viii Acknowledgements
what you are talking about?". After the same presentation at Iizuka'98,
Robert Hecht-Nielsen made the following comment: "This is a powerful
method! Why don't you apply it to challenging real world problems?".
That is exactly what my research associates, students and I have been
doing since, and that is what I present here. Walter Freeman made another
comment at the ICONIP conference in Shanghai, November, 2001-"Inte
grating genetic level and neuronal level in brain modelling and intelligent
machines is a very important and a promising approach, but how to do
that is the big question".
Contents
Prologue 1
Part I Evolving Connectionist Systems: Methods and Techniques
1 Evolving Processes and Evolving Connectionist Systems 7
1.1 Evolving Processes .......... . ...... 7
1.2 Working Classification Scheme for Learning in
Connectionist Systems ............ 12
1.3 Artificial Intelligence (AI) Versus Emerging
Intelligence (EI) . . .............. 25
1.4 Introduction to Evolving Connectionist Systems 26
1.5 Summary and Open Problems 28
1.6 Further Reading . . . . . . . . . . . . . . . . . . . 29
2 Evolving Connectionist Systems for Unsupervised Learning 31
2.1 On-Line Unsupervised Learning - General
Principles . .......... 31
2.2 ECOS for On-Line Clustering ...... 39
2.3 Self-Organising Maps - SOMs . . . . . . 49
2.4 Evolving Self-Organising Maps (ESOM) 52
2.5 Summary and Open Problems 55
2.6 Further Reading . . . . . . . . . . . . . . 55
3 Evolving Connectionist Systems for Supervised Learning 57
3.1 Principles and Architectures of Connectionist
Systems for On-Line Supervised Learning ... 57
3.2 Evolving Fuzzy Neural Networks - EFuNN .... 65
3.3 Knowledge Manipulation in Evolving Fuzzy Neural
Networks (EFuNNs) - Rule Insertion, Rule
Extraction, Rule Aggregation . . . . . . . . . . 75
3.4 On-Line Evaluation, Feature Modification and
Parameter Adaptation in EFuNNs 85
3.5 Summary and Open Questions 88
3.6 Further Reading . . . . . . . . . . . . . 88
ix
x Contents
4 Recurrent Evolving Systems, Reinforcement Learning and
Evolving Automata . .... . . . ..... .. . .. .. 91
4.1 Recurrent Evolving Connectionist Systems . . 91
4.2 Evolving Connectionist Systems and Evolving
Automata ....... ... . . 95
4.3 Reinforcement Learning in ECOS 96
4.4 Summary and Open Questions 97
4.5 Further Reading . . . . . . . . . . 98
5 Evolving Neuro-Fuzzy Inference Systems 99
5.1 Different Types of Rules and Inference in
Knowledge-Based Neural Networks . . 99
5.2 Hybrid Neuro-Fuzzy Inference Systems - HyFIS 104
5.3 Dynamic Evolving Neuro-Fuzzy Inference
Systems - DENFIS . . . . . . . . . . . 107
5.4 Different Types of Fuzzy Rules in ECOS . . 116
5.5 Type-2 Evolving Connectionist Systems . . 118
5.6 Interval-Based Evolving Connectionist Systems and
Other Ways of Defining Receptive Fields 120
5.7 Summary and Open Problems 123
5.8 Further Reading . . . . . . . . . . . . . . . . 123
6 Evolutionary Computation and Evolving Connectionist
Systems . . ... . .. . ... . .... . ... . . .. .. . 125
6.1 Evolutionary Computation - a Brief Introduction 125
6.2 Evolutionary Computation (EC) for the Optimisation
of Off-Line Learning Connectionist Systems . . .. 131
6.3 Evolutionary Computation for the Optimisation of
On-Line Learning Systems 134
6.4 Summary and Open Problems 140
6.5 Further Reading . . . . . . . . . 140
7 Evolving Connectionist Machines - Framework, Biological
Motivation and Implementation Issues . . ... .. .... 143
7.1 A framework for evolving connectionist machines. 143
7.2 Biological Motivation for ECOS - the "Instinct" for
Information .. ... . .. .. .. ........ 148
7.3 Spatial and Temporal Complexity of ECOS .... 150
7.4 On-Line Feature Selection and Feature Evaluation 151
7.5 An Agent-Based Framework for Evolving
Connectionist Machines 156
7.6 Evolving Hardware . . . . . . . . 158
7.7 Conclusion and Open Questions 160
7.8 Further Reading . . . . . . . . . . 160
Contents xi
Part II Evolving Connectionist Systems: Applications in
Bioinformatics, Brain Study, and Intelligent Systems
8 Data Analysis, Modelling and Knowiedge Discovery in
Bioinformatics ....................... 165
8.1 Bioinformatics - an Area of Information Growth and
Emergence of Knowledge . . . . . . . . . . .. . . . 165
8.2 Dynamic DNA and RNA Sequence Data Analysis and
Knowledge Discovery .. . . . . . . . . . . ... . . 170
8.3 Gene Expression Data Analysis, Rule Extraction and
Disease Profiling . . . . . . . . . . . . . . . . 1.7 4.
8.4 Fuzzy Evolving Clustering of Genes According to
Their Time-Course Expression 184
8.5 Protein Structure Prediction 186
8.6 Dynamic Cell Modelling . . . . 189
8.7 Summary and Open Problems 191
8.8 Further Reading. . . . . . . . . 191
9 Dynamic Modelling of Brain Functions and Cognitive
Processes . . . . . . . . . . . . . . . . . . . . . . . .1 9.3
9.1 Evolving Structure of the Brain and Evolving
Cognition . . . . . . . . . . . . . . . . . . . 193
9.2 Dynamic Modelling of Brain States Based on EEG
Signals . . . . . . . . . . . . . . . . . . . . . 1. 9.7 .
9.3 Dynamic Modelling of Cognitive Processes Based on
Brain Imaging .......... . ....... 200
9.4 Modelling Perception - the Auditory System . . 202
9.5 Dynamic Modelling of Integrated Auditory and
Visual Systems ... . . . . . . . . . . . . 204
9.6 Computational Models of the Entire Brain 206
9.7 Summary and Open Problems 207
9.8 Further Reading . . . . . . . . . . . . . . . . 208
10 Modelling the Emergence of Acoustic Segments (Phonemes)
in Spoken Languages ... . . . . . . . . . . . . . . . . 209
10.1 Introduction to the Issues of Learning Spoken
Languages . . . . . . . . . . . . . . . . . . . 2.0 9
10.2 The Dilemma of "Innateness Versus Learning" or
"Nature Versus Nurture" Revisited . . . . . . . 211
10.3 ECOS for Modelling the Emergence of Phones and
Phonemes . . . . . . . . . . . . . . . 213
lOA Modelling Evolving Bilingual Systems 221
10.5 Summary and Open Problems 225
10.6 Further Reading . . . . . . . . . 227
liOn-Line Adaptive Speech Recognition 229
11.1 Introduction to Adaptive Speech Recognition
Problems .... .. ............ .. 229