The Logic of Adaptive Behavior: Knowledge Representation and Algorithms for Adaptive Sequential Decision Making under Uncertainty in First-Order and Relational Domains

506 Pages·2009·8.94 MB·English

by  M. Van Otterlo

THE LOGIC OF ADAPTIVE BEHAVIOR
Frontiers in Artificial Intelligence and Applications
FAIA covers all aspects of theoretical and applied artificial intelligence research in the
form of monographs, doctoral dissertations, textbooks, handbooks and proceedings
volumes. The FAIA series contains several sub-series, including ”Information Modelling
and Knowledge Bases” and ”Knowledge-Based Intelligent Engineering Systems”. It also
includes the biennial ECAI, the European Conference on Artificial Intelligence,
proceedings volumes, and other ECCAI – the European Coordinating Committee on
Artificial Intelligence – sponsored publications. An editorial panel of internationally
well-known scholars is appointed to provide a high quality selection.
Series Editors:
J. Breuker, R. Dieng-Kuntz, N. Guarino, J.N. Kok, J. Liu,
R. Lo´pez de Ma´ntaras, R. Mizoguchi, M. Musen, S.K. Pal and N. Zhong
Volume 192
Recently published in this series
Vol. 191. J. Piskorski et al. (Eds.), Finite-State Methods and Natural
Language Processing – Post-proceedings of the 7th International
Workshop FSMNLP 2008
Vol. 190. Y. Kiyoki et al. (Eds.), Information Modelling and Knowledge Bases XX
Vol. 189. E. Francesconi et al. (Eds.), Legal Knowledge and Information Systems –
JURIX 2008: The Twenty-First Annual Conference
Vol. 188. J. Breuker et al. (Eds.), Law, Ontologies and the Semantic Web –
Channelling the Legal Information Flood
Vol. 187. H.-M. Haav and A. Kalja (Eds.), Databases and Information Systems V
– Selected Papers from the Eighth International Baltic Conference,
DB&IS 2008
Vol. 186. G. Lambert-Torres et al. (Eds.), Advances in Technological
Applications of Logical and Intelligent Systems – Selected Papers from
the Sixth Congress on Logic Applied to Technology
Vol. 185. A. Biere et al. (Eds.), Handbook of Satisfiability
Vol. 184. T. Alsinet, J. Puyol-Gruart and C. Torras (Eds.), Artificial Intelligence
Research and Development – Proceedings of the 11th International
Conference of the Catalan Association for Artificial Intelligence
Vol. 183. C. Eschenbach and M. Gru¨ninger (Eds.), Formal Ontology in Information
Systems – Proceedings of the Fifth International Conference (FOIS 2008)
Vol. 182. H. Fujita and I. Zualkernan (Eds.), New Trends in Software Methodologies,
Tools and Techniques – Proceedings of the seventh SoMeT 08
ISSN 0922–6389
THE LOGIC OF
ADAPTIVE BEHAVIOR
Knowledge Representation and Algorithms for
Adaptive Sequential Decision Making under Uncertainty
in First-Order and Relational Domains
Martijn van Otterlo
Department of Computer Science
Katholieke Universiteit Leuven, Belgium
• • • •
Amsterdam Berlin Oxford Tokyo Washington, DC
(cid:2)c 2009 The author and IOS Press.
All rights reserved. No part of this book may be reproduced, stored in a retrieval system,
or transmitted, in any form or by any means, without prior written permission from the
publisher.
ISBN 978–1–58603–969–1
Library of Congress Control Number: 2008944033
doi:10.3233/978–1–58603–969–1–i
Publisher
IOS Press
Nieuwe Hemweg 6B
1013 BG Amsterdam
Netherlands
fax: [phone]
e-mail: [e-mail]
Distributor in the UK and Ireland Distributor in the USA and Canada
Gazelle Books Services Ltd. IOS Press, Inc.
White Cross Mills 4502 Rachael Manor Drive
Hightown Fairfax, VA 22032
Lancaster LA1 4XS USA
United Kingdom fax: [phone]
fax: [phone] e-mail: [e-mail]
e-mail: [e-mail]
LEGAL NOTICE
The publisher is not responsible for the use which might be made
of the following information.
PRINTED IN THE NETHERLANDS
Toastmaster
”Gentlemen, pray silence for the President of the Royal Society for Putting
Things on Top of Other Things.”
Sir William
”I thank you, gentlemen. The year has been a good one for the Society (hear,
hear). This year our members have put more things on top of other things than
ever before. But, I should warn you, this is no time for complacency. No, there are
still many things, and I cannot emphasize this too strongly, not on top of other
things. I myself, on my way here this evening, saw a thing that was not on top
of another thing in any way. (shame!) Shame indeed but we must not allow
ourselves to become too despondent. For, we must never forget that if there was
not one thing that was not on top of another thing our society would be nothing
more than a meaningless body of men that had gathered together for no good
purpose. But we flourish. This year our Australasian members and the various
organizations affiliated to our Australasian branches put no fewer than twenty-
two things on top of other things. (applause) Well done all of you. But there
is one cloud on the horizon. In this last year our Staffordshire branch has not
succeeded in putting one thing on top of another (shame!). Therefore I call upon
our Staffordshire delegate to explain this weird behaviour.”
— ”The Royal Society For Putting Things On Top Of Other Things” sketch,
Monty Python’s Flying Circus, programme 18 (1970)
”Er wordt niets nieuws gezegd. Alles wordt nieuw gezegd.” (Bomans)
”Es gibt nichts neues, nur neue Kombinationen.” (Goethe)
”No one really starts anything new, Mrs. Nemur. Everyone builds on other men’s
failures. There is nothing really original in science. What each man contributes to
the sum of knowledge is what counts.” (Flowers for Algernon - D. Keyes, 1966)
This page intentionally left blank
Preface
One of my favorite stories is The Library of Babel (1941) by the Argentinean writer and
librarian Jorge Luis Borges (1899–1986). In this fantastic story, Borges describes an imagi-
nary library containing all possible books of a specific length, containing a specific number
of pages and symbols. The library itself consists of an enormous amount of interconnected
hexagonal rooms. Borges describes that ”[W]hen it was proclaimed that the Library con-
tainedallbooks,thefirstimpressionwasoneofextravaganthappiness. Allmenfeltthemselves
to be the masters of an intact and secret treasure. There was no personal or world problem
whose eloquent solution did not exist in some hexagon.”
Forcomputerscientistsandmathematicians, theconceptoftheuniversallibraryimme-
diately activates connections with combinatorics and permutations. In fact, most computer
scientistswillbeabletoprograminroughlytenminutesanalgorithmto generateallpossi-
ble books (though actually running it to completion will not be possible in their lifetime).
The physical concept of the interconnected hexagons has interesting links to graph theory
and search spaces. By all means, if the universal library would really exist, every writer
could just ’look up’, or search the connection graph for, the book he or she wants to write,
instead of writing it him- or herself. Looking up the right book is, unfortunately, not very
easy given that the library is extremely large. Traveling through even the slightest portion
of corridors and hexagons will take up more than a lifetime for any mere mortal being.
Even stronger, the physical size of the library exceeds the capacity of our universe, as the
mathematician Goldbloom Bloch (2008) writes in his exciting book on the library. The
labyrinth of the library is thus fictional, yet it is far more intuitive (and fun) to imagine
wandering through the physical version. Borges tells us that ”[F]or four centuries now men
have exhausted the hexagons ...There are official searchers, inquisitors. I have seen them
in the performance of their function: they always arrive extremely tired from their journeys;
they speak of a broken stairway which almost killed them; they talk with the librarian of
galleries and stairs; sometimes they pick up the nearest volume and leaf through it, looking
for infamous words. Obviously, no one expects to discover anything.”
Wandering through a library, not expecting to discover anything is one of the pleasant
things in life, I think, although more modern variants exist through the vast network of
interconnected webpages on the internet, where hyperlinks correspond to corridors. Um-
berto Eco was inspired by Borges’ library for his book The Name of the Rose (1980), in
which he describes an exciting hunt through a physical version of a similar library. Here
though, the travelers were looking for something specific, for knowledge, even though it
remains unclear what exactly that is but until the end of the story. Eco acknowledged the
inspiration through the librarian in his story, named Jorge von Burgos. And although the
travelers in Eco’s story were looking for general knowledge, the ultimate form of general
knowledge in the library has many connections with compression, orderings and informa-
tion theory: ”[W]e also know of another superstition of that time: that of the Man of the
Book. On some shelf in some hexagon (men reasoned) there must exist a book which is the
formula and perfect compendium of all the rest: some librarian has gone through it and he is
analogous to a god. In the language of this zone vestiges of this remote functionary’s cult still
persist. Many wandered in search of Him.”
Out of the many interesting ideas in Borges’ story, I find this concept of a book that
provides a perfect compendium to all other books most intriguing. The existence of this
one book that summarizes all other books in whatever ’best’ way is an extraordinary phe-
nomenon. It has to exist though, given that the library is finite. Yet, I find it hard to
imagine what the contents would look like. In a much more modest setting, imagining a
book that summarizes a well-chosen set of texts on a particular topic is much more con-
ceivable. In fact, this is what I have tried to do in this book: providing a compendium of
all work concerning learning sequential decision making under uncertainty in first-order and
relational domains, in whatever ’best’ way possible. I have wandered through a partially
physical, partially electronic, Borgian library of literature, discovered new hexagons and
books myself, and I have drawn a map of interesting hexagons, corridors, galleries and
books that were visited by other travelers.
Researchon thistopic beganin 1998andhas sincethen continuedto grow. Itis nowan
established subfield of AI and results appear at international scientific fora that deal with
topics such as machine learning, knowledge representation, decision making and planning.
Furthermore, it is also an active field, and there is a growing attention for its problems and
results. Among others, this year the ICAPS 2008 workshop on a reality check for planning
and scheduling under uncertainty was organized, as well as the AAAI 2008 workshop on
transferlearningforcomplextasks. Botharemuchrelatedtothecoreproblemsinthisbook.
Furthermore, there was a tutorial on decision-theoretic planning and learning in relational
domains at AAAI 2008, and a tutorial on first-order planning techniques at ICAPS 2008.
In the coming years, there will be many more scientific events where results concerning
sequentialdecisionmakinginfirst-orderdomainswillbepresented. However,theresearch
in the past decade has already developed a thorough understanding of, and a relatively
established core set of techniques for, the Markov decision process setting in first-order
and relational domains. It is exactly this topic that I describe in the book. A similar
setting, butfor non-relational domains, was described a decade agoin the famous book by
Sutton and Barto (1998).
Even though I did my ultimate best to cover every text that was relevant for the topic
in this book, I cannot possibly guarantee that there are no hidden hexagons that I missed
by mistake. In addition, almost all hexagons I describe contain books that are written in
English. Though on my journey, I have encountered several texts written in Japanese and
Chinese texts for example, that seemed to describe matters that were of interest to me.
Unfortunately my knowledge of these languages is limited. Yet, for all English texts I am
confident that I might have discovered them all. This book contains a bibliographic map
that describes all relevant hexagons. As we will see, there are cases where fellow travelers
werenotawareofothertravelershavingdiscoveredinterestinghexagonsbeforethem,and
others had locked themselves inside a particular hexagon without looking for paths that
VIII
connected their hexagon with another that contained similar books. As far as possible,
I have tried to establish links between hexagons, and books, assembling the knowledge
of all travelers that have gone before me, and I have even dug new tunnels connecting
hexagons when relevant or necessary.
This book grew out of my recent dissertation (van Otterlo, 2008a). Writing a book like
this one has been a long, personal journey through a vast, Borgian library. Fortunately,
during this journey, I have met many kind and interesting fellow travelers. Sometimes
our paths crossed several times, sometimes I stumbled across them in some remote, dimly
lit, hidden hexagon, and some people have watched over me while I was traveling from
one hexagon to another. I have had the pleasure to engage in interesting conversations
with many of the fellow travelers that I mention in this book. Many talks in person or by
e-mail have helped me in increasing my understanding of sometimes complicated matters.
Apossible list of names would quickly turn into a Borgian space of its own, and thus here I
take the opportunity to thank you all at once. I also thank all of my coauthors for working
with me on interesting topics. In the recent years, I have had the honor to be a member
of the Dutch discussion group on artificial intelligence, EMERGENTIA, and I would like to
thank all its members and former members for fun and interesting discussions on many
Sunday afternoons.
I owe much to my promotor John-Jules Meyer and assistant-promotor and referent
Marco Wiering. Both have watched over my travels from a small distance and contributed
tothefinalsuccessofmyjourneywiththeirkind,andcomplementary,support. Ialsothank
all other wise men who agreed to take place in my committee and read my dissertation. I
specificallywanttothankLucDeRaedtandJoostKok. Luc,foralltheopportunitieshehas
given me in Dagstuhl, Freiburg and Leuven, and also for showing me how science works,
and Joost, for giving me the kind opportunity to create this book in this form.
The scientific community has much to offer, but friends and family are most important
still. Many thanks go to my parents and my sister and in addition to all friends and people
of all kinds of family. Yet, the one I have to – and gratefully want to – thank the most,
is my dear Marieke. In addition to love, friendship, irresistible humor, and much support
she has given me slightly longer already than the decade I describe in this book, she has
always shown a great understanding of me and my journey. I admire that, especially given
the scope and length of my travels. I’m looking forward to all our travels still to come.
Leuven, November 2008
Martijn van Otterlo
IX