Approximate Dynamic Programming with Applications in Multi-Agent Systems

161 Pages·2007·25.21 MB·English

Approximate Dynamic Programming with Applications in
Multi-Agent Systems
by
Mario J. Valenti
Submitted to the Department of Electrical Engineering and Computer Science
in partial fulfillment of the requirements for the degree of
Doctor of Philosophy in Electrical Engineering
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
May 2007
(cid:13)c Massachusetts Institute of Technology 2007. All rights reserved.
Author............................................................................
Department of Electrical Engineering and Computer Science
May 4, 2007
Certified by........................................................................
Daniela Pucci de Farias
Assistant Professor of Mechanical Engineering
Thesis Supervisor
Certified by........................................................................
Jonathan P. How
Associate Professor of Aeronautics and Astronautics
Thesis Supervisor
Accepted by.......................................................................
Arthur C. Smith
Chairman, Department Committee on Graduate Students
2
Approximate Dynamic Programming with Applications in
Multi-Agent Systems
by
Mario J. Valenti
Submitted to the Department of Electrical Engineering and Computer Science
on May 4, 2007, in partial fulfillment of the
requirements for the degree of
Doctor of Philosophy in Electrical Engineering
Abstract
This thesis presents the development and implementation of approximate dynamic
programming methods used to manage multi-agent systems. The purpose of this
thesis is to develop an architectural framework and theoretical methods that enable
an autonomous mission system to manage real-time multi-agent operations. To meet
thisgoal, webeginbydiscussingaspectsofthereal-timemulti-agentmissionproblem.
Next, we formulate this problem as a Markov Decision Process (MDP) and present
a system architecture designed to improve mission-level functional reliability through
system self-awareness and adaptive mission planning. Since most multi-agent mission
problemsarecomputationallydifficulttosolveinreal-time, approximationtechniques
are needed to find policies for these large-scale problems. Thus, we have developed
theoretical methods used to find feasible solutions to large-scale optimization prob-
lems. More specifically, we investigate methods designed to automatically generate
an approximation to the cost-to-go function using basis functions for a given MDP.
Next, these these techniques are used by an autonomous mission system to manage
multi-agent mission scenarios. Simulation results using these methods are provided
for a large-scale mission problem. In addition, this thesis presents the implemen-
tation of techniques used to manage autonomous unmanned aerial vehicles (UAVs)
performing persistent surveillance operations. We present an indoor multi-vehicle
testbed called RAVEN (Real-time indoor Autonomous Vehicle test ENvironment)
that was developed to study long-duration missions in a controlled environment. The
RAVEN’sdesignallowsresearcherstofocusonhigh-leveltasksbyautonomouslyman-
aging the platform’s realistic air and ground vehicles during multi-vehicle operations,
thus promoting the rapid prototyping of UAV technologies by flight testing new ve-
hicle configurations and algorithms without redesigning vehicle hardware. Finally,
using the RAVEN, we present flight test results from autonomous, extended mission
tests using the technologies developed in this thesis. Flight results from a 24 hr,
fully-autonomous air vehicle flight-recharge test and an autonomous, multi-vehicle
extended mission test using small, electric-powered air vehicles are provided.
3
Thesis Supervisor: Daniela Pucci de Farias
Title: Assistant Professor of Mechanical Engineering
Thesis Supervisor: Jonathan P. How
Title: Associate Professor of Aeronautics and Astronautics
4
Acknowledgments
First and foremost, I would like to express gratitude to my thesis supervisors. First,
to Prof. Daniela Pucci de Farias for her assistance, guidance, enthusiasm and patience
with my research work.
Second, to Prof. Jonathan How for his guidance, passion, and positive attitude
toward this research. I appreciated the flexibility he provided me in managing this
research project and I enjoyed the cooperative research environment.
Third, to my committee members Prof. Dimitri Bertsekas, Prof. Daniela Rus, and
Prof. George Verghese for their valuable insight, comments and suggestions.
Fourth, to Dr. John Vian and The Boeing Company for funding this research
under the “Vehicle and Vehicle System Health Management” contract.
Fifth, to the professors at Temple University and MIT that assisted me during my
academic studies. Specifically, to Prof. Brian Butz, Prof. Thomas Ward, Prof. Robert
Yantorno, Prof. John Essignmann and Prof. Eric Feron for their advice during my
collegiate studies.
Sixth, to all of my colleagues and friends at MIT. Specifically, to Brett Bethke,
Keith and Liz Bonawitz, Daniel Dale, and Adrian Frank for fostering a great (and
fun) research environment.
Seventh, to the students and staff of Simmons Hall and members of the Eastgate
Bible Study for their support and friendship.
Eighth, to my family, especially my brother Phillip, who has faithfully served our
country during my stay at MIT, my brother Justin, who always has creative ideas
and a way with words, and my mother, who has comforting thoughts in the best
and worst of times. And, most importantly, to my father, the voice of reason in
challenging situations and a great role model as a father, a leader and an engineer.
He is one of the best engineers I have ever met.
Ninth, to my son Ian for coming to the lab and always brightening up the day for
all of us. I hope his passion for exploring and experiencing the world around him is
as strong today as it was when I wrote this.
5
Finally, I would like to express my deepest gratitude and thanks to my Lord and
savior Jesus Christ, by whom all things are possible. Phil. 3:12-14
6
Dedicated to my loving wife Tricia ...
7
8
Contents
1 Introduction 17
1.1 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.2 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.3 Thesis Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2 The Real-Time Multi-Agent Mission System Problem 29
2.1 Health Management in Real-Time Mission Systems . . . . . . . . . . 32
2.2 The UAV SWARM Mission Management Problem . . . . . . . . . . . 33
2.3 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.4 MDP Model for the UAV SWARM Resource Management Problem . 37
3 Basis Function Generation for Approximate Dynamic Programming
Methods 39
3.1 Approximate Linear Programming Problem Formulation . . . . . . . 41
3.2 Issue: Selecting Basis Functions . . . . . . . . . . . . . . . . . . . . . 42
3.3 Basis Function Generation Algorithm . . . . . . . . . . . . . . . . . . 44
3.3.1 Numerical Complexity of the Basis Function Generation Algo-
rithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.4 Error Bound for a Single Policy, Single Iteration Update . . . . . . . 50
4 The Multi-Vehicle Mission / Task Management Problem 57
4.1 Implementation: MissionManagementforaSimplifiedPersistentSurveil-
lance Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.1.1 Results using Exact Probability Transition Matrices . . . . . . 59
4.1.2 Results using Basis Function Generation Algorithm . . . . . . 61
4.2 Implementation: Large-ScaleMulti-VehicleMissionManagementProb-
lem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.2.1 Action Space Complexity . . . . . . . . . . . . . . . . . . . . . 65
9
4.2.2 Approximate Linear Programming-Based Mission Problem Im-
plementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.2.3 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . 68
4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5 Multi-Agent Mission System Testbed and Health Management 77
5.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.2 System Architecture and Components . . . . . . . . . . . . . . . . . . 79
5.3 Main Testbed Hardware . . . . . . . . . . . . . . . . . . . . . . . . . 81
5.3.1 Task Processing and Operator Interface Components . . . . . 84
5.4 Quadrotor Control Design Model . . . . . . . . . . . . . . . . . . . . 87
5.5 Hovering Airplane Control Design Model . . . . . . . . . . . . . . . . 89
5.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.6.1 Quadrotor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.6.2 Hovering Airplane . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.6.3 Multi-Vehicle Testing using Mission and Tasking System Com-
ponents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6 Multi-Agent Health Management and Extended Mission Testing 109
6.1 Simplified Persistent Surveillance Mission . . . . . . . . . . . . . . . . 110
6.2 Mission Planning and Vehicle Health Monitoring . . . . . . . . . . . . 111
6.3 Battery Health Monitoring . . . . . . . . . . . . . . . . . . . . . . . . 112
6.4 Battery Charging Station . . . . . . . . . . . . . . . . . . . . . . . . 117
6.5 Mission Flight Test Results . . . . . . . . . . . . . . . . . . . . . . . 129
6.5.1 Mission Flight Test Results using the Basis Function Genera-
tion Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7 Conclusions and Future Work 141
7.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
A Basis Function Generation Algorithm Formulation 147
10