MACHINE VISION SYSTEM FOR ROBOTIC APPLE HARVESTING IN FRUITING WALL ...

152 Pages·2016·3.46 MB·English

by  Abhisesh Silwal

MACHINE VISION SYSTEM FOR ROBOTIC APPLE HARVESTING
IN FRUITING WALL ORCHARDS

By
ABHISESH SILWAL

A dissertation submitted in partial fulfillment of
the requirement for the degree of
DOCTOR OF PHILOSOPHY

WASHINGTON STATE UNIVERSITY
Department of Biological Systems Engineering

DECEMBER 2016

© Copyright by ABHISESH SILWAL, 2016
All Rights Reserved
© Copyright by ABHISESH SILWAL, 2016
All Rights Reserved
To the Faculty of Washington State University:
The members of the Committee appointed to examine the dissertation of ABHISESH
SILWAL find it satisfactory and recommend that it be accepted.

________________________________________
Manoj Karkee, Ph.D., Chair

________________________________________
Changki Mo, Ph.D.

________________________________________
Qin Zhang, Ph.D.

ii
ACKNOWLEDGEMENT

I would like to express my sincere gratitude to my advisor Dr. Manoj Karkee for his
patience and continuous support during my Ph.D. Without his motivation, mentorship, and
immense knowledge, this Ph.D. would not have been possible. Beside my advisor, I would like
to thank the rest of my thesis committee Prof. Qin Zhang and Dr. Changki Mo for their insightful
comments, encouragements, and timely feedback in my work. My sincere appreciation also goes
to Linda Root and Patrick Scharf for their administrative and technical support that was essential
during my study at WSU. The mechanical design of the apple harvesting robot is an equally
important component of my dissertation. This work was accomplished by my graduate fellow
Dr. Joe Davidson. It was a pleasure working with him during laboratory mockup’s and field
trials. To my best abilities, I’ve tried to appropriately cite his work on manipulator and end-
effector design. I also thank all of my friends at CPAAS who have provided their critical
feedback during in-house presentations and seminars. Last but not the least, I would like to thank
my wife Pooja, who has constantly motivated me in my purse of this degree. Without her and my
parents’ support, none of this and where I stand in life would have been possible.

iii
MACHINE VISION SYSTEM FOR ROBOTIC APPLE HARVESTING
IN FRUITING WALL ORCHARDS

Abstract

by Abhisesh Silwal, Ph.D.
Washington State University
December 2016

Chair: Manoj Karkee

Mechanization in agriculture is often regarded as one of the greatest human achievements
in the twentieth century. These technological advancements have led to significant reduction in
human effort in the production of bulk agricultural commodities such as corn and wheat.
Specialty crop industry such as fresh fruit market, on the other hand, are still dependent upon
manual labor for various production operations such as training, pruning, and harvesting. Among
these, tree fruit harvesting of high value crops like apples is the most labor intensive and time
sensitive task that requires the right number of farm workers at right time. To increase
productivity and reduce dependency on seasonal labor, researchers have proposed automated
harvesting systems. Because of highly unstructured orchard environment and variable outdoor
conditions, these technologies have achieved only limited successes in the past. No commercial
viability has been achieved and every apple destined for fresh market is still handpicked. The
iv
lack of mechanized harvesting system has the potential to threaten the long-term sustainability of
fresh fruit industry in the United States and around the world.
This dissertation focuses on the study and evaluation of a machine vision and an
integrated robotic system for automated harvesting of apples grown in modern fruiting wall
orchards. The machine vision algorithm designed to work in orchard environment accurately
detected apples growing individually as well as in heavy clusters under variable natural lighting
conditions. A pragmatic approach to harvesting (also called hierarchical approach) showed that
98% of the fruit could be detected with iterative imaging and harvesting of most visible fruits.
The integrated robotic system with global camera and custom built seven degrees of freedom
manipulator successfully picked 84% of attempted fruit with 6 seconds of average harvest time
per fruit. This approach of selective apple harvesting with a global camera system and low-cost
manipulator show a huge potential for cost-effective robotic solutions for harvesting fresh market
apples. However, several limitations still remain to be addressed for commercialization.

v
TABLE OF CONTENTS
Page
ACKNOWLEDGEMENT ............................................................................................................. iii
ABSTRACT ................................................................................................................................... iv
LIST OF TABLES .......................................................................................................................... x
LIST OF FIGURES ....................................................................................................................... xi
CHAPTER 1: INTRODUCTION ................................................................................................... 1
1.1 OVERVIEW ......................................................................................................................... 1
1.2 CHALLENGES IN AUTOMATED FRUIT HARVESTING .............................................. 2
1.2.1 Biologically Driven and Highly Unstructured Environment ......................................... 3
1.2.2 Worker Health and Safety .............................................................................................. 3
1.2.3 Time Constraints ............................................................................................................ 4
1.3 IMPACT OF AUTOMATION IN AGRICULTURE ........................................................... 4
1.4 DESIGN SPECIFICATIONS FOR ROBOTIC HARVESTER ........................................... 5
REFERENCES ........................................................................................................................... 7
CHAPTER 2: LITERATURE REVIEW ...................................................................................... 10
2.1 MACHINE VISION FOR ROBOTIC HARVESTING ..................................................... 10
2.1.1 Image Acquisition ........................................................................................................ 10
2.1.2 Image Segmentation and Feature Extraction ............................................................... 12
2.2 MECHANICAL DESIGN FOR ROBOTIC HARVESTING ............................................ 15
2.3 RESEARCH OBJECTIVES ............................................................................................... 18
2.4 ORGANIZATION OF THE DISSERTATION.................................................................. 18
REFERENCES ......................................................................................................................... 20
vi
CHAPTER 3: IDENTIFICATION OF RED APPLES IN FIELD ENVIRONMENT WITH
OVER THE ROW MACHINE VISION SYSTEM...................................................................... 28
ABSTRACT .............................................................................................................................. 28
3.1 INTRODUCTION .............................................................................................................. 29
3.2 MATERIALS AND METHODS ........................................................................................ 31
3.2.1 Imaging Platform and Data Collection ........................................................................ 31
3.2.2 Image Processing for Apple Identification .................................................................. 33
3.2.2.1 Image Preprocessing ............................................................................................. 33
3.2.2.2 Apple Identification Approach ............................................................................. 35
3.2.3 Object Filtering with Color .......................................................................................... 38
3.3 RESULTS AND DISCUSSION ......................................................................................... 39
3.3.1 Apple Detection using CHT......................................................................................... 39
3.3.2 Apple Detection using Blob Analysis .......................................................................... 40
3.3.3 Accuracy Assessment and Discussion ......................................................................... 41
3.4 CONCLUSION ................................................................................................................... 44
REFERENCES ......................................................................................................................... 46
CHAPTER 4: A HIERARCHICAL APPROACH OF APPLE IDENTIFICATION FOR
ROBOTIC HARVESTING .......................................................................................................... 49
ABSTRACT .............................................................................................................................. 49
4.1 INTRODUCTION .............................................................................................................. 50
4.2 DESIGN SPECIFICATIONS, OBJECTIVES, AND GOALS .......................................... 52
4.3 STUDY SITE ...................................................................................................................... 53
4.4 MATERIALS AND METHODS ........................................................................................ 54
vii
4.4.1 Data acquisition and imaging platform ........................................................................ 54
4.4.2 Hierarchical Fruit Identification .................................................................................. 56
4.5 RESULTS AND DISCUSSION ......................................................................................... 60
4.6 SUMMARY AND CONCLUSION ................................................................................... 66
ACKNOWLEDGEMENTS ...................................................................................................... 67
REFERENCES ......................................................................................................................... 68
CHAPTER 5: DESIGN, INTEGRATION, AND FIELD EVALUATION OF A ROBOTIC
APPLE HARVESTER .................................................................................................................. 71
ABSTRACT .............................................................................................................................. 71
5.1 INTRODUCTION .............................................................................................................. 72
5.2 PRIOR WORK.................................................................................................................... 76
5.2.1 Fruit Identification ....................................................................................................... 76
5.2.2 Manipulation ................................................................................................................ 78
5.3 SYSTEM REQUIREMENTS ............................................................................................. 80
5.4 CROP ENVIRONMENT AND WORKSPACE MODIFICATIONS ................................ 81
5.5 SYSTEM OVERVIEW ...................................................................................................... 85
5.6 MACHINE VISION SYSTEM .......................................................................................... 87
5.6.1 Global Camera System and Apple Identification ........................................................ 88
5.6.2 Exposure Fusion........................................................................................................... 88
5.6.3 Camera Calibration and Hand–Eye Coordination ....................................................... 89
5.6.4 Fruit Localization ......................................................................................................... 93
5.6.5 Apple Prioritization ...................................................................................................... 94
5.6.6 Uniform Background ................................................................................................... 95
viii
5.7 MECHANICAL DESIGN .................................................................................................. 95
5.7.1 Manipulator .................................................................................................................. 96
5.7.2 End–Effector ................................................................................................................ 97
5.8 EXPERIMENTAL SETUP FOR FIELD EVALUATION ................................................. 98
5.9 RESULTS ......................................................................................................................... 101
5.9.1 Machine Vision System ............................................................................................. 101
5.9.2 Manipulation .............................................................................................................. 103
5.10 DISCUSSION ................................................................................................................. 106
5.11 CONCLUSION ............................................................................................................... 110
ACKNOWLEDGEMENTS .................................................................................................... 111
REFERENCES ....................................................................................................................... 112
CHAPTER 6: OVERALL CONCLUSIONS AND RECOMMENDATIONS .......................... 120
6.1 GENERAL CONCLUSIONS ........................................................................................... 120
6.2: FUTURE DIRECTION ................................................................................................... 123
APPENDIX – C++ and MATLAB Vision Codes ...................................................................... 127

ix