Real-Time Supervision for Human Robot Teams in Complex Task Domains

228 Pages·2017·4.97 MB·English

by  Arif Tuna Ozgelen

CCiittyy UUnniivveerrssiittyy ooff NNeeww YYoorrkk ((CCUUNNYY))
CCUUNNYY AAccaaddeemmiicc WWoorrkkss
Dissertations, Theses, and Capstone Projects CUNY Graduate Center
9-2015
RReeaall--TTiimmee SSuuppeerrvviissiioonn ffoorr HHuummaann RRoobboott TTeeaammss iinn CCoommpplleexx TTaasskk
DDoommaaiinnss
Arif Tuna Ozgelen
Graduate Center, City University of New York
How does access to this work benefit you? Let us know!
More information about this work at: https://academicworks.cuny.edu/gc_etds/1084
Discover additional works at: https://academicworks.cuny.edu
This work is made publicly available by the City University of New York (CUNY).
Contact: [email protected]
Real-Time Supervision for Human-Robot Teams in Complex Task Domains
by
Arif Tuna O¨zgelen
A dissertation submitted to the Graduate Faculty in Computer Science in partial fulfillment of
the requirements for the degree of Doctor of Philosophy, The City University of New York
2015
(cid:13)c 2015
Arif Tuna Ozgelen
All Rights Reserved
i
This manuscript has been read and accepted by the Graduate Faculty in Computer Science in
satisfaction of the dissertation requirement for the degree of Doctor of Philosophy.
Professor Elizabeth Sklar
Date Chair of Examining Committee
Professor Robert Haralick
Date Executive Officer
Professor Simon D. Parsons
Professor Jizhong Xiao
Professor Michael Lewis
Supervisory Committee
The City University of New York
ii
Abstract
Real-Time Supervision for Human-Robot Teams in Complex Task Domains
by
Arif Tuna O¨zgelen
Adviser: Proffessor Elizabeth Sklar
Ongoing research on multi-robot teams is focused on methods and systems to be utilized
in dynamic and dangerous environments such as search and rescue missions, often with a human
operator in the loop to supervise the system and make critical decisions. To increase the size of
the team controlled by an operator, and to reduce the operator’s mental workload, the robots will
have to be more autonomous and reliable so that tasks can be issued at a higher level. Typical
in these domains, such high-level tasks are often composed of smaller tasks with dependencies
and constraints. Assigning suitable robot platforms to execute these tasks is a combinatorial
optimization problem. Operations Research and AI techniques can handle large numbers of robot
allocations in real time, however most of these algorithms are opaque to humans; they provide no
explanation or insight about how the solution is produced. Recent studies suggest that interaction
between the human operator and robot team requires human-centric approaches for collaborative
planning and task allocation, since black-box solutions are often too complex to examine under
stressful conditions and are often discarded by experts.
The main contribution of this thesis is a methodology to help operators make decisions
about complex task allocation in real time for high stress missions. First a novel, human-centric
graphicalmodel,TaskAssignmentGraph(TAG),isdescribedtoanalyzeandpredictthecomplexity
of task assignmentandschedulingprobleminstances, taking intoaccount thespatial distributionof
resources and tasks. Then, the TAG model is extended for dynamic environments to the Mission-
level Assignment Problem (MAP) model. Two user studies were conducted, first in static and then
in dynamic environments, in order to identify and empirically verify the key factors, derived from
thegraphical model, which affect thedecision makingof human supervisorsduringtask assignment
for a team of robots. In these userstudies, participants used software tools developed for this work.
iii
One of these software tools allows for two different levels of autonomy for the interaction scheme:
manual control and collaborative control, with an option to invoke an automated assignment tool.
Findings relating to the impact of decision support functionality on the mental workload and the
performance of the supervisor are presented. Finally, steering of the common algorithms utilized
by decision support tools, using the strategies employed by user study participants, related to the
TAG and MAP model parameters, are discussed.
iv
Acknowledgments
First of all I’d like to thank my advisers, Prof. Elizabeth Sklar, who is a mentor to me and kept
me on the path to completion no matter how hard I tried to diverge and Prof. Simon Parsons,
whose invaluable insights and comments shaped the direction of my research. Their continuous
supportand guidance helped me tremendously. I’d also like to thank my committee members Prof.
Jizhong Xiao and Prof. Michael Lewis for their valuable input. Among many faculty members
in The Graduate Center, I’d like to specially thank Prof. Susan Epstein, whom I worked with on
HRTeam project, for her support and all that she have taught me.
During my studies, I’ve met and worked with many people whom I had the pleasure of
calling my friends as well as my colleagues. I’d like to thank all members of the Agents research
group, especially to Sumon Azhar and Artyom Diky whom I worked, traveled and shared many
good memories during this chapter of my life. Also I’d like to thank Michael Kaisers, for always
keeping me in check and for many intellectually stimulating conversations.
In some of my experiments I used HRTeam framework, which is a result of a collaborative
effort. I’d like to thank all the members of HRTeam and Metrobotics projects for their valuable
work. Among many people who participated in these projects, I’d like to specially thank Eric
Schneider, Michael Constantino and Ofear Balas who went out of their way to help me in many
occasions and to Farah Abbasi, who designed the initial prototype of the user interface.
Biggest challenges that I faced during my pursuit of a doctoral degree was motivational.
The moral support that all my extended family have given, gave me strength to continue during
tough times. I’d like to specially thank my cousin, Arda Nural, for constantly pushing me to finish
my dissertation (“Ne zaman bitiyo olm?”). Finally, without the encouragement and unconditional
support of my mother Ufuk O¨zgelen and my wife Burc¸in O¨zgelen, this work would not have been
v
possible. My gratitude for them is beyond words. Dear Burc¸in, you have been with me and beside
me at every step. Please know that this work is a result of your moral and emotional support. I
hope this work to be an example to our son Kagˆan Bora and a motivation to push himself beyond
his limits. Dear Mother, you have given me everything to help me to get to this point and to be
who I am today. This dissertation is dedicated to you.
vi
Contents
Contents vii
List of Figures xi
List of Tables xv
1 Introduction 1
1.1 Motivation and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Research Questions and Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.5 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Background 13
2.1 Human-Robot Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.1 Evaluation of HRI systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Multi-Robot Task Allocation (MRTA) . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.1 Market-based Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.2 Constraint Programming (CP) Techniques . . . . . . . . . . . . . . . . . . . . 27
2.2.3 Algorithm Steerability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.4 Thesis Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3 MRTA Complexity Analysis in Static Environments 43
vii
3.1 Overview of Task Taxonomies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2 Task structure in Multi-Robot Systems. . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3 Task Assignment Graph (TAG) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.4 Task Complexity Assessment Tool (TCAT) Test bed . . . . . . . . . . . . . . . . . . 51
3.5 TCAT User Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.5.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.5.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.5.3 Metrics and Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.5.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.5.5 Post-experiment Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4 MRTA Complexity Analysis in Real-Time and Dynamic Environments 71
4.1 Extending TAG for Real-Time and Dynamic Environments . . . . . . . . . . . . . . 73
4.1.1 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.1.2 MAP adaptation into TASC experiment . . . . . . . . . . . . . . . . . . . . . 79
4.2 Task Assignment Supervisory Control (TASC) Interface and Testbed . . . . . . . . . 80
4.2.1 Manual Allocation Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.2.2 Automated Allocation Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.3 Collaborative Allocation Mode . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.3 HRTeam Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.4 TASC User Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.4.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.4.3 Metrics and Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.4.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5 Mission-level Analysis and Integration of MAP model into MRTA methods 104
5.1 Mission-level analysis of TASC experiment . . . . . . . . . . . . . . . . . . . . . . . . 105
5.1.1 Analysis of Mission-level Objective Metrics . . . . . . . . . . . . . . . . . . . 107
viii