Table Of Content

STRIPE: Remote Driving Using Limited Image Data Jennifer S. Kay January 1997 CMU-CS-97-100 Aapioved tea JTUDüG reiea*a) School of Computer Science Carnegie Mellon University Pittsburgh, PA 15217 Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy aD ,M ° * ™**m . Thesis Committee: —^ms!fImBsa * Chuck Thorpe, Chair Bonnie John Eric Krotkov Larry Matthies, JPL © 1997 by Jennifer S. Kay. All rights reserved. Jennifer Kay was funded by a NASA Graduate Student Researchers Program Fellowship grant number t^VJ NGT-51292. This research was also partly sponsored by DARPA, under contracts 'Technology Enhancements for Unmanned Ground Vehicles," contract number DAAE07-96-C-X075 and "Unmanned Ground Vehicle Sys- tem" contract number DAAE07-90-C-R059 monitored by TACOM; and "Perception for Outdoor Navigation," contract number DACA76-89-C-0014 monitored by the US Army Topographic Center. The views and conclu- sions contained in this document are those of the author and should not be interpreted as representing the official policies, either expressed or implied, of the funding agencies KEYWORDS: Artificial Intelligence, Human Computer Interaction, Mobile Robots, Semi-Autonomous Vehicles, Unmanned Ground Vehicles, Vehicle Teleoperation, Polyhe- dral-Earth Reprojection, Low-Bandwidth Teleoperation, High-Latency Teleoperation, Navlab, STRIPE. egie School of Computer Science fellon DOCTORAL THESIS in the field of Computer Science STRIPE: Remote Driving Using Limited Image Data JENNIFER KAY Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy ACCEPTED: 3 1^7 DATE //g/?-? DEPARTMENT HEAD DATE APPROVED: I^RM lArA? DEAN DATE In Memory of Julia Jacqueline Kay 11 Abstract Driving a vehicle, either directly or remotely, is an inherently visual task. When heavy fog limits visibility, we reduce our car's speed to a slow crawl, even along very familiar roads. In teleoperation systems, an operator's view is limited to images provided by one or more cameras mounted on the remote vehicle. Traditional methods of vehicle teleoperation require that a real time stream of images is transmitted from the vehicle camera to the operator control station, and the operator steers the vehicle accordingly. For this type of teleoperation, the transmission link between the vehicle and operator workstation must be very high bandwidth (because of the high volume of images required) and very low latency (because delayed images can cause operators to steer incorrectly). In many situations, such a high-bandwidth, low-latency communication link is unavailable or even technically impossible to provide. Supervised TeleRobotics using Incremental Polyhedral Earth geometry, or STRIPE, is a teleoperation system for a robot vehicle that allows a human operator to accurately control the remote vehicle across very low bandwidth communication links, and communication links with large delays. In STRIPE, a single image from a camera mounted on the vehicle is transmitted to the operator workstation. The operator uses a mouse to pick a series of "waypoints" in the image that define a path that the vehicle should follow. These 2D waypoints are then transmitted back to the vehicle, where they are used to compute the appropriate steering commands while the next image is being transmitted. STRIPE requires no advance knowledge of the terrain to be traversed, and can be used by novice operators with only minimal training. STRIPE is a unique combination of computer and human control. The computer must determine the 3D world path designated by the 2D waypoints and then accurately control the vehicle over rugged terrain. The human issues involve accurate path selection, and the prevention of disorien- tation, a common problem across all types of teleoperation systems. STRIPE is the only semi- autonomous teleoperation system that can accurately follow paths designated in monocular images on varying terrain. The thesis describes the STRIPE algorithm for tracking points using the incremental geometry model, insight into the design and redesign of the interface, an analysis of the effects of potential errors, details of the user studies, and hints on how to improve both the algorithm and interface for future designs. in IV Acknowledgments It is difficult to reduce to a few paragraphs the thanks that I owe to all of the people who have helped me make it through my graduate student career. So many people have helped and sup- ported me, and made this such a special place. For a start, thanks to my advisor, Chuck Thorpe. Besides his amazing skills in robotics, Chuck can recite poetry faster than a speeding bullet, and approximate the tangent of small angles in a single bound. His encouragement, enthusiasm, and guidance kept me "going for it" despite some seemingly insurmountable hurdles. Thanks also my thesis committee: Bonnie John, Eric Krotkov, and Larry Matthies, for all of their comments, suggestions, and help. An extra thanks to Bonnie, for taking the time to help me in the design and analysis of my user study. Thanks to Sharon Burks and Catherine Copetas for their support, advice, ability to hide all of the bureaucracy that graduate school could be from the students, and for knowing the answers to just about any ques- tion presented to them. Thanks to all the members of the Unmanned Ground Vehicle and Automated Highway Sys- tem projects for all their help. In particular, thanks to Jay Gowdy, Todd Jochem, and Dean Pomer- leau for lots of software that made writing my own software significantly easier. Thanks to Martial Hebert, for support in writing code and text. Thanks to Barry Brumitt, Terry Fong, John Hancock, and Dirk Langer for keeping various parts of the vehicle up and running. Thanks to Jim Frazier, a.k.a. "Mr. Fixit" for keeping the HMMWV running, exterminating wasps, and getting up early lots of mornings just so he could help me lay out traffic cones and then safety driving in both scorching and freezing temperatures. Thanks to Renee Wahl for picking lots of points, moving lots of cones, and enduring the blue goo. Thanks to Kate Fissell, Jim Moody, and Bill Ross for keeping the machines up on and off the vehicle. And thanks to the guys at Gateway Supply, for letting me use their bathroom on those long days spent out at the slag heap. Thanks very much to those insane people who volunteered to read some or all of my thesis just to help out: Maria Ebling, Marie Elm, Michael English, Terry Fong, and Chris Okasaki. Thanks to my officemates of past and present: Anurag Acharya, Justin Boyan, Fabio Cozman, Keith Gremban, John Hancock, Lily Mummert, Carol Novak, Conrad Poelman, David Pugh, Carlo Tomasi, and Todd Williamson. They've helped me with obscure shellscripts, provided com- plimentary rubber slugs and beetles to decorate my desk, put up with my tendency to take over more bookshelves than I should, and been wonderful friends. So many more people deserve my thanks for their help, encouragement, support, and friend- ship: Omead Amidi, Shumeet Baluja, Harry Bovik, Mei Chen, Stewart Clamen, Mary Jo Dowl- ing, Maria Ebling, Mike & Win English, Emily Kingsbury, Jim Kocher, Mark Maimone, Chris Okasaki, Fred Solomon, Mark Stehlik, Jim Stichnoth, Dafna Talmor, Bennet Yee, Amy Moor- mann Zaremski, and more. Finally, thanks to my sister Abby Kay, my parents Gordon and Joyce Kay, and my husband Redmond English, for believing in me, helping me get through the low points, rejoicing with me in the high points, and always being there. VI Table of Contents Chapter 1 Introduction 1 The Need For A Low-Bandwidth High-Delay Teleoperation System 1 How STRIPE Works 2 History 4 Thesis Overview 4 Chapter 2 Previous Work 7 2.1 Classifying Teleoperation Strategies 7 2.2 Continuous and Delay Free 8 2.2.1 Naval Ocean System Center 10 2.2.1.1 Anecdotal Results 10 Operators Get Lost 10 2.2A.2 Empirical Experiments 1] Local vs. Remote Driving 1 ] Direct Driving with Limited Field of View 1 ] Direct Driving with Limited Image Resolution 1 ] 2.2.2 Army Research Laboratory 1 ] 2.2.3 Discussion 12 2.3 Nearly-Continuous or Very-Low-Delay 12 2.3.1 Sandia National Laboratories 13 2.3.1.1 Anecdotal Results 13 Operators Lose Their Way 13 Operators Oversteer 13 vn