Learning Theory : 17th Annual Conference On Learning Theory, COLT 2004, Banff, Canada, July 1 ...

661 Pages·2005·14.57 MB·English

Lecture Notes in Artificial Intelligence 3120
Edited by J. G. Carbonell and J. Siekmann
Subseries of Lecture Notes in Computer Science
This page intentionally left blank
John Shawe-Taylor Yoram Singer (Eds.)
Learning
Theory
17th Annual Conference on Learning Theory, COLT 2004
Banff, Canada, July 1-4, 2004
Proceedings
Springer
eBook ISBN: 3-540-27819-2
Print ISBN: 3-540-22282-0
©2005 Springer Science + Business Media, Inc.
Print©2004Springer-Verlag
Berlin Heidelberg
All rights reserved
No part of this eBook maybe reproducedor transmitted inanyform or byanymeans,electronic,
mechanical, recording, or otherwise, without written consent from the Publisher
Created in the United States of America
Visit Springer's eBookstore at: http://ebooks.springerlink.com
and the Springer Global Website Online at: http://www.springeronline.com
Preface
This volume contains papers presented at the 17th Annual Conference on Lear-
ning Theory (previously known as the Conference on Computational Learning
Theory) held in Banff, Canada from July 1 to 4, 2004.
The technical program contained 43 papers selected from 107 submissions, 3
open problems selected from among 6 contributed, and 3 invited lectures. The
invited lectures were given by Michael Kearns on ‘Game Theory, Automated
Trading and Social Networks’, Moses Charikar on ‘Algorithmic Aspects of Fi-
nite Metric Spaces’, and Stephen Boyd on ‘Convex Optimization, Semidefinite
Programming, and Recent Applications’. These papers were not included in this
volume.
The Mark Fulk Award is presented annually for the best paper co-authored
by a student. This year the Mark Fulk award was supplemented with two further
awards funded by the Machine Learning Journal and the National Information
Communication Technology Centre, Australia (NICTA). We were therefore able
to select three student papers for prizes. The students selected were Magalie Fro-
mont for the single-author paper “Model Selection by Bootstrap Penalization for
Classification”, Daniel Reidenbach for the single-author paper “On the Learna-
bility of E-Pattern Languages over Small Alphabets”, and Ran Gilad-Bachrach
for the paper “Bayes and Tukey Meet at the Center Point” (co-authored with
Amir Navot and Naftali Tishby).
This year saw an exceptional number of papers submitted to COLT cover-
ing a wider range of topics than has previously been the norm. This exciting
expansion of learning theory analysis to new models and tasks marks an im-
portant development in the growth of the area as well as in the linking with
practical applications. The large number of quality submissions placed a heavy
burden on the program committee of the conference: Shai Ben-David (Cornell
University), Stephane Boucheron (Université Paris-Sud), Olivier Bousquet (Max
Planck Institute), Sanjoy Dasgupta (University of California, San Diego), Vic-
tor Dalmau (Universitat Pompeu Fabra), Andre Elisseeff (IBM Zurich Research
Lab), Thore Graepel (Microsoft Research Labs, Cambridge), Peter Grunwald
(CWI, Amsterdam), Michael Jordan (University of California, Berkeley), Adam
Kalai (Toyota Technological Institute), David McAllester (Toyota Technological
Institute), Manfred Opper (University of Southampton), Alon Orlitsky (Univer-
sity of California, San Diego), Rob Schapire (Princeton University), Matthias
Seeger (University of California, Berkeley), Satinder Singh (University of Michi-
gan), Eiji Takimoto (Tohoku University), Nicolas Vayatis (Université Paris 6),
Bin Yu (University of California, Berkeley) and Thomas Zeugmann (University
at Lübeck). We are extremely grateful for their careful and thorough reviewing
and for the detailed discussions that ensured the very high quality of the final
program. We would like to have mentioned the subreviewers who assisted the
program committee in reaching their assessments, but unfortunately space con-
VI Preface
straints do not permit us to include this long list of names and we must simply
ask them to accept our thanks anonymously.
We particularly thank Rob Holte and Dale Schuurmans, the conference local
chairs, as well as the registration chair Kiri Wagstaff. Together they handled
the conference publicity and all the local arrangements to ensure a successful
event. We would also like to thank Microsoft for providing the software used in
the program committee deliberations, and Ofer Dekel for maintaining this soft-
ware and the conference Web site. Bob Williamson and Jyrki Kivinen assisted
the organization of the conference in their role as consecutive Presidents of the
Association of Computational Learning, and heads of the COLT Steering Com-
mittee. We would also like to thank the ICML organizers for ensuring a smooth
co-location of the two conferences and arranging for a ‘kernel day’ at the overlap
on July 4. The papers appearing as part of this event comprise the last set of 8
full-length papers in this volume.
Finally, we would like to thank the Machine Learning Journal, the Pacific
Institute for the Mathematical Sciences (PIMS), INTEL, SUN, the Informatics
Circle of Research Excellence (iCORE), and the National Information Com-
munication Technology Centre, Australia (NICTA) for their sponsorship of the
conference. This work was also supported in part by the IST Programme of the
European Community, under the PASCAL Network of Excellence, IST-2002-
506778.
April, 2004 John Shawe-Taylor,
Yoram Singer
Program Co-chairs, COLT 2004
Sponsored by:
Table of Contents
Economics and Game Theory
Towards a Characterization of Polynomial Preference Elicitation
with Value Queries in Combinatorial Auctions 1
Paolo Santi, Vincent Conitzer, Tuomas Sandholm
Graphical Economics 17
Sham M. Kakade, Michael Kearns, Luis E. Ortiz
Deterministic Calibration and Nash Equilibrium 33
Sham M. Kakade, Dean P. Foster
Reinforcement Learning for Average Reward Zero-Sum Games 49
Shie Mannor
OnLine Learning
Polynomial Time Prediction Strategy with Almost Optimal
Mistake Probability 64
Nader H. Bshouty
Minimizing Regret with Label Efficient Prediction 77
Nicolò Cesa-Bianchi, Gábor Lugosi, Gilles Stoltz
Regret Bounds for Hierarchical Classification
with Linear-Threshold Functions 93
Nicolò Cesa-Bianchi, Alex Conconi, Claudio Gentile
Online Geometric Optimization in the Bandit Setting Against
an Adaptive Adversary 109
H. Brendan McMahan, Avrim Blum
Inductive Inference
Learning Classes of Probabilistic Automata 124
François Denis, Yann Esposito
On the Learnability of E-pattern Languages over Small Alphabets 140
Daniel Reidenbach
Replacing Limit Learners with Equally Powerful
One-Shot Query Learners 155
Steffen Lange, Sandra Zilles
VIII Table of Contents
Probabilistic Models
Concentration Bounds for Unigrams Language Model 170
Evgeny Drukh, Yishay Mansour
Inferring Mixtures of Markov Chains 186
Sudipto Guha, Sampath Kannan
Boolean Function Learning
PExact = Exact Learning 200
Dmitry Gavinsky, Avi Owshanko
Learning a Hidden Graph Using Queries Per Edge 210
Dana Angluin, Jiang Chen
Toward Attribute Efficient Learning of Decision Lists and Parities 224
Adam R. Klivans, Rocco A. Servedio
Empirical Processes
Learning Over Compact Metric Spaces 239
H. Quang Minh, Thomas Hofmann
A Function Representation for Learning in Banach Spaces 255
Charles A. Micchelli, Massimiliano Pontil
Local Complexities for Empirical Risk Minimization 270
Peter L. Bartlett, Shahar Mendelson, Petra Philips
Model Selection by Bootstrap Penalization for Classification 285
Magalie Fromont
MDL
Convergence of Discrete MDL for Sequential Prediction 300
Jan Poland, Marcus Hutter
On the Convergence of MDL Density Estimation 315
Tong Zhang
Suboptimal Behavior of Bayes and MDL in Classification
Under Misspecification 331
Peter Grünwald, John Langford
Generalisation I
Learning Intersections of Halfspaces with a Margin 348
Adam R. Klivans, Rocco A. Servedio
Table of Contents IX
A General Convergence Theorem for the Decomposition Method 363
Niko List, Hans Ulrich Simon
Generalisation II
Oracle Bounds and Exact Algorithm for Dyadic Classification Trees 378
Gilles Blanchard, Christin Schäfer, Yves Rozenholc
An Improved VC Dimension Bound for Sparse Polynomials 393
Michael Schmitt
A New PAC Bound for Intersection-Closed Concept Classes 408
Peter Auer, Ronald Ortner
Clustering and Distributed Learning
A Framework for Statistical Clustering with a Constant Time
Approximation Algorithms for K-Median Clustering 415
Shai Ben-David
Data Dependent Risk Bounds for Hierarchical Mixture
of Experts Classifiers 427
Arik Azran, Ron Meir
Consistency in Models for Communication Constrained
Distributed Learning 442
J.B. Predd, S.R. Kulkarni, H. V. Poor
On the Convergence of Spectral Clustering on Random Samples:
The Normalized Case 457
Ulrike von Luxburg, Olivier Bousquet, Mikhail Belkin
Boosting
Performance Guarantees for Regularized Maximum Entropy
Density Estimation 472
MiroslavDudík, Steven J. Phillips, Robert E. Schapire
Learning Monotonic Linear Functions 487
Adam Kalai
Boosting Based on a Smooth Margin 502
Cynthia Rudin, Robert E. Schapire, Ingrid Daubechies
Kernels and Probabilities
Bayesian Networks and Inner Product Spaces 518
Atsuyoshi Nakamura, Michael Schmitt, Niels Schmitt,
Hans Ulrich Simon