Conformal Prediction for Reliable Machine Learning Theory, Adaptations and Applications

323 Pages·2014·13.89 MB·English

by  Vineeth Balasubramanian

Conformal Prediction
for Reliable Machine
Learning
This page is intentionally left blank
Conformal Prediction
for Reliable Machine
Learning
Theory, Adaptations
and Applications
Vineeth N. Balasubramanian
Department of Computer Science and Engineering
Indian Institute of Technology, Hyderabad, India
(formerly at Center for Cognitive Ubiquitous Computing
Arizona State University, USA)
Shen-Shyang Ho
School of Computer Engineering
Nanyang Technological University, Singapore
Vladimir Vovk
Department of Computer Science, Royal Holloway
University of London, UK
AMSTERDAM • BOSTON • HEIDELBERG • LONDON
NEW YORK • OXFORD • PARIS • SAN DIEGO
SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO
Morgan Kaufmann is an imprint of Elsevier
Acquiring Editor: Todd Green
Editorial Project Manager: Lindsay Lawrence
Project Manager: Malathi Samayan
Designer: Russell Purdy
Morgan Kaufmann is an imprint of Elsevier
225 Wyman Street, Waltham, MA 02451, USA
Copyright 2014 Elsevier Inc. All rights reserved
No part of this publication may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying, recording, or any information storage and
retrieval system, without permission in writing from the publisher. Details on how to seek
permission, further information about the Publisher’s permissions policies and our arrange-
ments with organizations such as the Copyright Clearance Center and the Copyright Licensing
Agency, can be found at our website: www.elsevier.com/permissions.
This book and the individual contributions contained in it are protected under copyright by the
Publisher (other than as may be noted herein).
Notices
Knowledge and best practice in this field are constantly changing. As new research and experi-
ence broaden our understanding, changes in research methods or professional practices, may
become necessary. Practitioners and researchers must always rely on their own experience and
knowledge in evaluating and using any information or methods described here in. In using
such information or methods they should be mindful of their own safety and the safety of oth-
ers, including parties for whom they have a professional responsibility.
To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors,
assume any liability for any injury and/or damage to persons or property as a matter of prod-
ucts liability, negligence or otherwise, or from any use or operation of any methods, products,
instructions, or ideas contained in the material herein.
Library of Congress Cataloging-in-Publication Data
Application Submitted
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library
ISBN: 978-0-12-398537-8
For information on all MK publications, visit our
website at www.mkp.com or www.elsevierdirect.com
Printed and bound in the United States of America
14 15 16 13 12 11 10 9 8 7 6 5 4 3 2 1
Copyright Permissions
Chapter 3
Figures 3.4, 3.5, 3.6, 3.9 and Tables 3.1, 3.2: Reprinted with permission from Balasubramanian,
Chakraborty and Panchanathan, Generalized Query by Transduction for Online Active Learning,
12th International Conference on Computer Vision Workshops (ICCV Workshops), © 2009 IEEE.
Figures 3.7, 3.8: Reprinted with permission from Makili et al., Active Learning Using Conformal
Predictors: Application to Image Classification, Fusion Science and Technology, Vol 62 No 2 ©
2012 American Nuclear Society, La Grange Park, Illinois.
Chapter 6
Figure 6.4: Reprinted from Artificial Intelligence Applications and Innovations, Meng Yang, Ilia
Nouretdinov, Zhiyuan Luo, Alexander Gammerman, Feature Selection by Conformal Predictor,
439–448, Copyright (2011), with kind permission from Springer Science and Business Media.
Chapter 9
Figures 9.1, 9.2, 9.3, 9.4 and Tables 9.1, 9.2: Reprinted from Intelligent Data Analysis, 13.6,
Smirnov, Evgueni N., Georgi I. Nalbantov, and A. M. Kaptein, Meta-conformity approach to reli-
able classification, 901–915, Copyright (2009), with permission from IOS Press.
Figures 9.5, 9.6, 9.7, 9.8 and Table 9.3: Reprinted from Artificial Intelligence: Methodology,
Systems, and Applications, Smirnov, Evgueni, Nikolay Nikolaev, and Georgi Nalbantov, Single-
stacking conformity approach to reliable classification, 161–170, Copyright (2010), with kind per-
mission from Springer Science and Business Media.
Chapter 10
Figures 10.1 & 10.3: Reprinted with permission from Li and Wechsler, Open Set Face Recognition
Using Transduction, IEEE Transactions on Pattern Analysis and Machine Intelligence, © 2005 IEEE.
Figures 10.5, 10.6 & 10.7: Reprinted with permission from Face Authentication Using Recognition-
by-parts, Boosting and Transduction, Li and Wechsler, International Journal of Pattern Recognition
and Artificial Intelligence, Vol. 23, Issue 3, © 2009 World Scientific.
Figure 10.8: Reprinted with permission from Robust Re-identification Using Randomness and
Statistical Learning: Quo Vadis, Nappi and Wechsler, Pattern Recognition Letters, Vol. 33, Issue 14,
© 2012 Elsevier.
Chapter 11
Figure 11.1: Reprinted with permission from Qualified Predictions from Microarray and Proteomics
Pattern Diagnostics with Confidence Machines, Bellotti, Luo, Gammerman, Van Delft, Saha,
International Journal of Neural Systems, Vol. 15, No 4, pp 247–258, © 2005 World Scientific.
v
This page is intentionally left blank
Contents
Contributing Authorsxiii
Foreword xv
Preface xvii
PART 1 THEORY
CHAPTER 1 The Basic Conformal Prediction Framework ���������������3
1�1 The Basic Setting and Assumptions 3
1�2 Set and Confidence Predictors 4
1�2�1 Validity and Efficiency of Set and
Confidence Predictors 5
1�3 Conformal Prediction 6
1�3�1 The Binary Case 7
1�3�2 The Gaussian Case 7
1�4 Efficiency in the Case of Prediction without Objects 9
1�5 Universality of Conformal Predictors 11
1�6 Structured Case and Classification 13
1�7 Regression 14
1�8 Additional Properties of Validity and Efficiency
in the Online Framework 15
1�8�1 Asymptotically Efficient Conformal Predictors 17
Acknowledgments 19
CHAPTER 2 Beyond the Basic Conformal
Prediction Framework ���������������������������������������������21
2�1 Conditional Validity 22
2�2 Conditional Conformal Predictors 23
2�2�1 Venn’s Dilemma 25
2�3 Inductive Conformal Predictors 25
2�3�1 Conditional Inductive Conformal Predictors 27
2�4 Training Conditional Validity of Inductive
Conformal Predictors 27
2�5 Classical Tolerance Regions 31
2�6 Object Conditional Validity and Efficiency 34
2�6�1 Negative Result 34
2�6�2 Positive Results 36
2�7 Label Conditional Validity and ROC Curves 38
2�8 Venn Predictors 43
vii
viii Contents
2�8�1 Inductive Venn Predictors 44
2�8�2 Venn Prediction without Objects 44
Acknowledgments 46
PART 2 ADAPTATIONS
CHAPTER 3 Active Learning ������������������������������������������������������49
3�1 Introduction 50
3�2 Background and Related Work 51
3�2�1 Pool-based Active Learning with Serial Query 52
3�2�2 Batch Mode Active Learning 55
3�2�3 Online Active Learning56
3�3 Active Learning Using Conformal Prediction 56
3�3�1 Query by Transduction (QBT) 57
3�3�2 Generalized Query by Transduction 59
3�3�3 Multicriteria Extension to QBT 61
3�4 Experimental Results 62
3�4�1 Benchmark Datasets 63
3�4�2 Application to Face Recognition 64
3�4�3 Multicriteria Extension to QBT 66
3�5 Discussion and Conclusions 69
Acknowledgments 70
CHAPTER 4 Anomaly Detection �������������������������������������������������71
4�1 Introduction 72
4�2 Background 73
4�3 Conformal Prediction for Multiclass
Anomaly Detection 74
4�3�1 A Nonconformity Measure for Multiclass
Anomaly Detection 75
4�4 Conformal Anomaly Detection 76
4�4�1 Conformal Anomalies 77
4�4�2 Offline versus Online Conformal Anomaly Detection 77
4�4�3 Unsupervised and Semi-supervised Conformal
Anomaly Detection 78
4�4�4 Classification Performance and Tuning
of the Anomaly Threshold 79
4�5 Inductive Conformal Anomaly Detection 80
4�5�1 Offline and Semi-Offline Inductive Conformal
Anomaly Detection 81
4�5�2 Online Inductive Conformal Anomaly Detection 82
4�6 Nonconformity Measures for Examples Represented
as Sets of Points 83
Contents ix
4�6�1 The Directed Hausdorff Distance 84
4�6�2 The Directed Hausdorff k-Nearest Neighbors
Nonconformity Measure 84
4�7 Sequential Anomaly Detection in Trajectories 86
4�7�1 The Sequential Hausdorff Nearest Neighbors
Conformal Anomaly Detector 86
4�7�2 Empirical Investigations 89
4�8 Conclusions 97
CHAPTER 5 Online Change Detection �����������������������������������������99
5�1 Introduction 99
5�2 Related Work 100
5�3 Background 102
5�4 A Martingale Approach for Change Detection 103
5�5 Experimental Results 105
5�5�1 Simulated Data Stream Using Rotating
Hyperplane 106
5�5�2 Simulated Data Streams Using NDC 106
5�6 Implementation Issues 111
5�6�1 Effect of Various Nonconformity Measures 112
5�6�2 Effect of Parameter ǫ 112
5�7 Conclusions 113
CHAPTER 6 Feature Selection �������������������������������������������������115
6�1 Introduction 116
6�2 Feature Selection Methods 117
6�2�1 Filters 117
6�2�2 Wrappers 118
6�2�3 Embedded Feature Selection 119
6�3 Issues in Feature Selection 119
6�3�1 Selection Bias 119
6�3�2 False Discovery Rate 119
6�3�3 Relevance and Redundancy 120
6�4 Feature Selection for Conformal Predictors 121
6�4�1 Strangeness Minimization Feature Selection 122
6�4�2 Average Confidence Maximization (ACM) 128
6�5 Discussion and Conclusions 129
CHAPTER 7 Model Selection ���������������������������������������������������131
7�1 Introduction 131
7�2 Background 133
7�3 SVM Model Selection Using Nonconformity Measure 134