Statistical Methods for Dynamic Treatment Regimes: Reinforcement Learning, Causal Inference, and Personalized Medicine

220 Pages·2013·2.45 MB·English

by  Bibhas Chakraborty, Erica E.M. Moodie (auth.)

Statistics for Biology and Health
Bibhas Chakraborty
Erica E.M. Moodie
Statistical Methods
for Dynamic
Treatment Regimes
Reinforcement Learning, Causal
Inference, and Personalized Medicine
Statistics for Biology and Health
Series Editors
M. Gail
K. Krickeberg
J. Samet
A. Tsiatis
W. Wong
For further volumes:
http://www.springer.com/series/2848
Bibhas Chakraborty • Erica E.M. Moodie
Statistical Methods
for Dynamic Treatment
Regimes
Reinforcement Learning, Causal Inference,
and Personalized Medicine
123
Bibhas Chakraborty Erica E.M. Moodie
Department of Biostatistics Department of Epidemiology,
Columbia University Biostatistics, and Occupational Health
New York, USA McGill University
Montreal Que´bec
Canada
ISSN 1431-8776
ISBN 978-1-4614-7427-2 ISBN 978-1-4614-7428-9 (eBook)
DOI 10.1007/978-1-4614-7428-9
Springer New York Heidelberg Dordrecht London
Library of Congress Control Number: 2013939595
© Springer Science+Business Media New York 2013
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection
with reviews or scholarly analysis or material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of
this publication or parts thereof is permitted only under the provisions of the Copyright Law of the
Publisher’s location, in its current version, and permission for use must always be obtained from Springer.
Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations
are liable to prosecution under the respective Copyright Law.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
While the advice and information in this book are believed to be true and accurate at the date of pub-
lication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any
errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect
to the material contained herein.
Printed on acid-free paper
Springer is part of Springer Science+Business Media (www.springer.com)
To my parents – Biman and Bani
Chakraborty, and my wife Sanchalika: for all
your love, support, and encouragement.
– Bibhas
To my family. Dave and my lovely boys,
Gordie and Jamie: your unfailing
encouragement and unconditional love keep
me afloat. Mom and Dad: your guidance and
support are invaluable. Zoe: you are my
great listener and friend. I love you all.
– Erica
Preface
This book was written to summarize and describe the state of the art of statistical
methods developed to address questions of estimation and inference for dynamic
treatment regimes, a branch of personalized medicine. The study of dynamic treat-
ment regimes is relatively young, and until now, no single source has aimed to pro-
vide an overview of the methodology and results which are dispersed in journals,
proceedings, and technical reports so as to orient researchers to the field. Our pri-
mary focus is on description of the methods, clear communication of the conceptual
underpinnings, and their illustration via analyses drawn from real applications as
well as results from simulated data. The first chapter serves to set the context for the
statistical reader in the landscape of personalized medicine; we assume a familiarity
with elementary calculus, linear algebra, and basic large-sample theory. Important
theoretical properties of the methods described will be stated when appropriate;
however, the reader will, for the most part, be referred to the primary research arti-
cles for the proofs of the results. By doing so, we hope the book will be accessible to
a wide audience of statisticians, epidemiologists, and medical researchers with some
statistical training, as well as computer scientists (machine/reinforcement learning
researchers) interested in medical applications.
Examples of data analyses from real applications are found throughout the book.
From these, we hope to impart a sense of the power and versatility of the methods
discussed to answer important problems in medical research. Where possible, we
refer readers to available code or packages in different statistical languages to facili-
tate implementation; whether or not such code exists, we aim to describe all analytic
approaches in sufficient detail that any researcher with a reasonable background in
statistical programming could implement the methods from scratch.
We hope that the publication of this book will foster the genuine enthusiasm
that we feel for this important area of research. Indeed, with the demographic shift
of most Western populations to older age, the treatment of chronic conditions will
bring increased pressure to develop evidence-based strategies for care that is tai-
lored to individual changes in health status. The recently proposed methods have not
yet reached a wide audience and consequently are underutilized. We hope that this
vii
viii Preface
text will serve as a useful handbook to those already active in the field of dynamic
regimes and spark a new generation of researchers to turn their attention to this
important and exciting area.
Acknowledgements
Bibhas Chakraborty would like to acknowledge support from the National Insti-
tutes of Health (NIH) grant R01 NS072127-01A1 and the Calderone Research Prize
for Junior Faculty (2011) awarded by the Mailman School of Public Health of the
Columbia University. Erica Moodie is supported by a Natural Sciences and En-
gineering Research Council (NSERC) University Faculty Award and by research
grants from NSERC and the Canadian Institutes of Health Research (CIHR). Finan-
cial support for the writing of this book was provided by the Quebec Population
Health Research Network (QPHRN).
We are indebted to numerous colleagues for lively and insightful discussions.
Our research has been enriched by exchanges with Daniel Almirall, Ken Cheung,
Nema Dean, Eric Laber, Bruce Levin, Susan Murphy, Min Qian, Thomas Richard-
son, Jamie Robins, Susan Shortreed, David Stephens, and Jonathan Wakefield. In
particular, we wish to thank Ashkan Ertefaie, Eric Laber, Min Qian, Olli Saarela,
and Michael Wallace for detailed comments on a first version of the text. Also,
we would like to acknowledge help in software development and creation of some
graphics for this book from Guqian Du, Tianxiao Huang, and Jingyi Xin – students
in the Department of Biostatistics at Columbia University. Jonathan Weinberg, Ben-
jamin Rich, and Yue Ru Sun, students in the Department of Mathematics & Statis-
tics, the Department of Epidemiology, Biostatistics, & Occupational Health, and the
school of Computer Science, respectively, at McGill University, also assisted in the
preparation of some simulation results and graphics.
We wish to thank our many medical and epidemiological collaborators for
thought-provoking discussions and/or the privilege of using their data: Dr. Michael
Kramer (PROBIT), Drs. Merrick Moseley and Catherine Stewart (MOTAS), Dr. Au-
gustus John Rush (STAR*D), and Dr. Victor J. Strecher (Project Quit – Forever
Free). MOTAS was funded by the Guide Dogs for the Blind Association (UK); per-
mission to analyze the data was granted by the MOTAS Cooperative. The follow-up
of the PROBIT study was made possible by a grant from CIHR.
Data used in Sect. 5.2.4 were obtained from the limited access data sets dis-
tributed from the NIMH-supported “Clinical Antipsychotic Trials of Intervention
Effectiveness in Schizophrenia” (CATIE-Sz). This is a multisite, clinical trial of
persons with schizophrenia comparing the effectiveness of randomly assigned med-
ication treatment. The study was supported by NIMH Contract #N01MH90001 to
the University of North Carolina at Chapel Hill. The ClinicalTrials.gov identifier is
NCT00014001. Analyses of the CATIE data presented in the book reflect the views
of the authors and may not reflect the opinions or views of the CATIE-Sz Study
Investigators or the NIH.
Preface ix
Data used in Sect. 8.9 were obtained from the limited access data sets dis-
tributed from the NIMH-supported “Sequenced Treatment Alternatives to Relieve
Depression” (STAR*D) study. The study was supported by NIMH Contract #
N01MH90003 to the University of Texas Southwestern Medical Center. The Clini-
calTrials.gov identifier is NCT00021528. Analyses of the STAR*D data presented
in the book reflect the views of the authors and may not reflect the opinions or views
of the STAR*D Study Investigators or the NIH.
New York, USA Bibhas Chakraborty
Montreal, Canada Erica E.M. Moodie