Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles PDF
Preview Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles
Control Engineering Series 81 RaO Optimal Adaptive Control and enp Optimal Adaptive indt Differential Games by Reinforcement f Dim o Learning Principles rciffal Control and ee A mr ed ena ntp i tat Differential Games This book gives an exposition of recently developed approximate dynamic programming Draguna Vrabie is a Senior Ll iv (ADP) techniques for decision and control in human engineered systems. ADP is a Research Scientist at United eGe rbeioinlofogricceaml aenndt amnaimchainl esy lseteamrnisn. gIt tiesc chonniqnueec ttehda tf rios mm oa ttivhaetoedre tbicya lle paorninint go fm veiecwh awniitshm bso itnh TEeacsht nHoalrotgfoierds , RCeosnenaercchti cCuet.n ter, aram C adaptive control and optimal control methods. The book shows how ADP can be used to Kyriakos G. Vamvoudakis n o by Reinforcement design a family of adaptive optimal control algorithms that converge in real-time to optimal is a Faculty Project Research inen Scientist at the Center for st clocitoofe nnaratturrtootuollr messo yaasltudtiectai mopcnotsivsn, etbbr ouyclt o msndytoesro atnelslmoeutrr ssiyn. iagTenr ladddda oitotapipo tatniimlmaolan aalgl dp ctaeohprneftot irvsroeymls lectaeornsmnc ate rrto.er aOl lejtenwrcs ott ohl edreaiie srosntit.n hoGcenter lmnihneaeern tahidnllo,y ,rdt erisana d lf otithtirimo etn hecaeu lh rdoroeepwnst iitmg tona l CaDCnooednmp ttCrp oooulf,mt eEDrpl eyEuncnttaargmitc.i,oa iUcnl a na(l-niCSvdeyC srDsteiCtmy) ,os ,f g Pri by rol Learning Principles n controllers must be designed offline using full knowledge of the systems dynamics. California, Santa Barbara. c It is also shown how to use ADP methods to solve multi-player differential games online. Frank L. Lewis is the i Moncrief-O'Donnell Endowed p Differential games have been shown to be important in H-infinity robust control for disturbance rejection, and in coordinating activities among multiple agents in networked Chair at the Automation & le Robotics Research Institute, teams. The focus of this book is on continuous-time systems, whose dynamical models can University of Texas at Arlington. s be derived directly from physical principles based on Hamiltonian or Lagrangian dynamics. aVV nd Lewamvourabie, Draguna Vrabie, Kyriakoasn dG .F Vraanmkv Lo.u Ldeawkiiss The Institution of Engineering and Technology isdak www.theiet.org is 978-1-84919-489-1 Optimal Adaptive Control.indd 1 23/10/2012 16:47:04 IET CONTROL ENGINEERING SERIES 81 Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles Other volumes in this series: Volume 8 A history of control engineering, 1800–1930 S. Bennett Volume 18 Applied control theory, 2nd edition J.R. Leigh Volume 20 Design of modern control systems D.J. Bell, P.A. Cook and N. Munro (Editors) Volume 28 Robots and automated manufacture J. Billingsley (Editor) Volume 33 Temperature measurement and control J.R. Leigh Volume 34 Singular perturbation methodology in control systems D.S. Naidu Volume 35 Implementation of self-tuning controllers K. Warwick (Editor) Volume 37 Industrial digital control systems, 2nd edition K. Warwick and D. Rees (Editors) Volume 39 Continuous time controller design R. Balasubramanian Volume 40 Deterministic control of uncertain systems A.S.I. Zinober (Editor) Volume 41 Computer control of real-time processes S. Bennett and G.S. Virk (Editors) Volume 42 Digital signal processing: principles, devices and applications N.B. Jones and J.D.McK. Watson (Editors) Volume 44 Knowledge-based systems for industrial control J. McGhee, M.J. Grimble and A. Mowforth (Editors) Volume 47 A history of control engineering, 1930–1956 S. Bennett Volume 49 Polynomial methods in optimal control and fi ltering K.J. Hunt (Editor) Volume 50 Programming industrial control systems using IEC 1131-3 R.W. Lewis Volume 51 Advanced robotics and intelligent machines J.O. Gray and D.G. Caldwell (Editors) Volume 52 Adaptive prediction and predictive control P.P. Kanjilal Volume 53 Neural network applications in control G.W. Irwin, K. Warwick and K.J. Hunt (Editors) Volume 54 Control engineering solutions: a practical approach P. Albertos, R. Strietzel and N. Mort (Editors) Volume 55 Genetic algorithms in engineering systems A.M.S. Zalzala and P.J. Fleming (Editors) Volume 56 Symbolic methods in control system analysis and design N. Munro (Editor) Volume 57 Flight control systems R.W. Pratt (Editor) Volume 58 Power-plant control and instrumentation D. Lindsley Volume 59 Modelling control systems using IEC 61499 R. Lewis Volume 60 People in control: human factors in control room design J. Noyes and M. Bransby (Editors) Volume 61 Nonlinear predictive control: theory and practice B. Kouvaritakis and M. Cannon (Editors) Volume 62 Active sound and vibration control M.O. Tokhi and S.M. Veres Volume 63 Stepping motors: a guide to theory and practice, 4th edition P.P. Acarnley Volume 64 Control theory, 2nd edition J.R. Leigh Volume 65 Modelling and parameter estimation of dynamic systems J.R. Raol, G. Girija and J. Singh Volume 66 Variable structure systems: from principles to implementation A. Sabanovic, L. Fridman and S. Spurgeon (Editors) Volume 67 Motion vision: design of compact motion sensing solution for autonomous systems J. Kolodko and L. Vlacic Volume 68 Flexible robot manipulators: modelling, simulation and control M.O. Tokhi and A.K.M. Azad (Editors) Volume 69 Advances in unmanned marine vehicles G. Roberts and R. Sutton (Editors) Volume 70 Intelligent control systems using computational intelligence techniques A. Ruano (Editor) Volume 71 Advances in cognitive systems S. Nefti and J. Gray (Editors) Volume 72 Control theory: a guided tour, 3rd edition James Ron Leigh Volume 73 Adaptive sampling with mobile WSN K. Sreenath, M.F. Mysorewala, D.O. Popa and F.L. Lewis Volume 74 Eigenstructure control algorithms: applications to aircraft/rotorcraft handling qualities design S. Srinathkumar Volume 75 Advanced control for constrained processes and systems F. Garelli, R.J. Mantz and H. De Battista Volume 76 Developments in control theory towards glocal control L. Qiu, J. Chen, T. Iwasaki and H. Fujioka (Editors) Volume 77 Further advances in unmanned marine vehicles G.N. Roberts and R. Sutton (Editors) Volume 78 Frequency-domain control design for high-performance systems J. O’Brien Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles Draguna Vrabie, Kyriakos G. Vamvoudakis and Frank L. Lewis The Institution of Engineering and Technology Published by The Institution of Engineering and Technology, London, United Kingdom The Institution of Engineering and Technology is registered as a Charity in England & Wales (no. 211014) and Scotland (no. SC038698). © 2013 The Institution of Engineering and Technology First published 2013 This publication is copyright under the Berne Convention and the Universal Copyright Convention. All rights reserved. Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may be reproduced, stored or transmitted, in any form or by any means, only with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publisher at the undermentioned address: The Institution of Engineering and Technology Michael Faraday House Six Hills Way, Stevenage Herts, SG1 2AY, United Kingdom www.theiet.org While the authors and publisher believe that the information and guidance given in this work are correct, all parties must rely upon their own skill and judgement when making use of them. Neither the authors nor the publisher assumes any liability to anyone for any loss or damage caused by any error or omission in the work, whether such an error or omission is the result of negligence or any other cause. Any and all such liability is disclaimed. The moral rights of the authors to be identifi ed as authors of this work have been asserted by them in accordance with the Copyright, Designs and Patents Act 1988. British Library Cataloguing in Publication Data A catalogue record for this product is available from the British Library ISBN 978-1-84919-489-1 (hardback) ISBN 978-1-84919-490-7 (PDF) Typeset in India by MPS Limited Printed in the UK by CPI Group (UK) Ltd, Croydon, CR0 4YY To Adrian Miron Draguna Vrabie To my parents, George and Evgenia. Your love and support was and will continue to be my biggest inspiration and source of strength Kyriakos Vamvoudakis To Galina, Roma, Chris – who have made every day exciting Frank Lewis Contents Preface xii Acknowledgements xv 1 Introduction to optimal control, adaptive control and reinforcement learning 1 1.1 Optimal control 2 1.1.1 Linear quadratic regulator 2 1.1.2 Linear quadratic zero-sum games 3 1.2 Adaptive control 4 1.3 Reinforcement learning 7 1.4 Optimal adaptive control 8 2 Reinforcement learning and optimal control of discrete-time systems: Using natural decision methods to design optimal adaptive controllers 9 2.1 Markov decision processes 11 2.1.1 Optimal sequential decision problems 12 2.1.2 A backward recursion for the value 14 2.1.3 Dynamic programming 15 2.1.4 Bellman equation and Bellman optimality equation 15 2.2 Policy evaluation and policy improvement 19 2.2.1 Policy iteration 21 2.2.2 Iterative policy iteration 21 2.2.3 Value iteration 22 2.2.4 Generalized policy iteration 25 2.2.5 Q function 26 2.3 Methods for implementing policy iteration and value iteration 29 2.4 Temporal difference learning 30 2.5 Optimal adaptive control for discrete-time systems 32 2.5.1 Policy iteration and value iteration for discrete-time dynamical systems 34 2.5.2 Value function approximation 35 2.5.3 Optimal adaptive control algorithms for discrete-time systems 36 2.5.4 Introduction of a second ‘Actor’ neural network 38 2.5.5 Online solution of Lyapunov and Riccati equations 42 viii Optimal adaptive control and differential games by RL principles 2.5.6 Actor–critic implementation of discrete-time optimal adaptive control 43 2.5.7 Q learning for optimal adaptive control 43 2.6 Reinforcement learning for continuous-time systems 46 PART I Optimal adaptive control using reinforcement learning structures 49 3 Optimal adaptive control using integral reinforcement learning for linear systems 51 3.1 Continuous-time adaptive critic solution for the linear quadratic regulator 53 3.1.1 Policy iteration algorithm using integral reinforcement 54 3.1.2 Proof of convergence 55 3.2 Online implementation of IRL adaptive optimal control 58 3.2.1 Adaptive online implementation of IRL algorithm 58 3.2.2 Structure of the adaptive IRL algorithm 61 3.3 Online IRL load-frequency controller design for a power system 64 3.4 Conclusion 69 4 Integral reinforcement learning (IRL) for non-linear continuous-time systems 71 4.1 Non-linear continuous-time optimal control 72 4.2 Integral reinforcement learning policy iterations 74 4.2.1 Integral reinforcement learning policy iteration algorithm 76 4.2.2 Convergence of IRL policy iteration 78 4.3 Implementation of IRL policy iterations using value function approximation 79 4.3.1 Value function approximation and temporal difference error 79 4.3.2 Convergence of approximate value function to solution of the Bellman equation 81 4.3.3 Convergence of approximate IRL policy iteration to solution of the HJB equation 85 4.4 Online IRL actor–critic algorithm for optimal adaptive control 85 4.4.1 Actor–critic structure for online implementation of adaptive optimal control algorithm 85 4.4.2 Relation of adaptive IRL control structure to learning mechanisms in the mammal brain 88 Contents ix 4.5 Simulation results 89 4.5.1 Non-linear system example 1 89 4.5.2 Non-linear system example 2 90 4.6 Conclusion 92 5 Generalized policy iteration for continuous-time systems 93 5.1 Policy iteration algorithm for optimal control 94 5.1.1 Policy iteration for continuous-time systems 94 5.1.2 Integral reinforcement learning for continuous-time systems 95 5.2 Generalized policy iteration for continuous-time systems 96 5.2.1 Preliminaries: Mathematical operators for policy iteration 97 5.2.2 Contraction maps for policy iteration 98 5.2.3 A new formulation of continuous-time policy iteration: Generalized policy iteration 100 5.2.4 Continuous-time generalized policy iteration 101 5.3 Implementation of generalized policy iteration algorithm 103 5.4 Simulation results 104 5.4.1 Example 1: Linear system 104 5.4.2 Example 2: Non-linear system 105 5.5 Conclusion 107 6 Value iteration for continuous-time systems 109 6.1 Continuous-time heuristic dynamic programming for the LQR problem 110 6.1.1 Continuous-time HDP formulation using integral reinforcement learning 111 6.1.2 Online tuning value iteration algorithm for partially unknown systems 113 6.2 Mathematical formulation of the HDP algorithm 114 6.3 Simulation results for online CT-HDP design 117 6.3.1 System model and motivation 117 6.3.2 Simulation setup and results 118 6.3.3 Comments on the convergence of CT-HDP algorithm 120 6.4 Conclusion 122 PART II Adaptive control structures based on reinforcement learning 123 7 Optimal adaptive control using synchronous online learning 125 7.1 Optimal control and policy iteration 127 7.2 Value function approximation and critic neural network 129 7.3 Tuning and convergence of critic NN 132
The list of books you might like
