OCR with OpenCV, Tesseract, and Python - Intro to OCR

218 Pages·2021·21.535 MB·English

by  Adrian Rosebrock

OCR with OpenCV, Tesseract, and Python
Intro to OCR - 1st Edition (version 1.0)
Adrian Rosebrock, PhD
Abhishek Thanki
Sayak Paul
Jon Haase
py imagesearch
The contents of this book, unless otherwise indicated, are Copyright ©2020 Adrian
Rosebrock, PyImageSearch.com. All rights reserved. Books like this are made possible by
the time invested by the authors. If you received this book and did not purchase it, please
consider making future books possible by buying a copy at https://www.pyimagesearch.com/
books-and-courses/ today.
© 2020 PyImageSearch
Contents
Contents iii
Companion Website xi
Acronyms xiii
1 Introduction 1
1.1 Who This Book Is For . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Do I Need to Know Computer Vision and OpenCV? . . . . . . . . . . . . . . . . 2
1.3 Do I Need to Know Deep Learning? . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Book Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 What Is Optical Character Recognition? 5
2.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 An Introduction to OCR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 A Brief History of OCR . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.2 Applications of OCR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.3 Orientation and Script Detection . . . . . . . . . . . . . . . . . . . . . . 8
2.2.4 The Importance of Pre-Processing and Post-Processing . . . . . . . . . 8
2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3 Tools, Libraries, and Packages for OCR 11
3.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 OCR Tools and Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2.1 Tesseract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2.2 Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2.3 PyTesseract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2.4 OpenCV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
iii
iv CONTENTS
3.2.5 Keras, TensorFlow, and scikit-learn . . . . . . . . . . . . . . . . . . . . . 14
3.2.6 Cloud OCR APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4 Installing OCR Tools, Libraries, and Packages 17
4.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 OCR Development Environment Configuration . . . . . . . . . . . . . . . . . . . 17
4.2.1 A Note on Install Instructions . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2.2 Installing Tesseract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2.2.1 Installing Tesseract on macOS . . . . . . . . . . . . . . . . . . 18
4.2.2.2 Installing Tesseract on Ubuntu . . . . . . . . . . . . . . . . . . 18
4.2.2.3 Installing Tesseract on Windows . . . . . . . . . . . . . . . . . 19
4.2.2.4 Installing Tesseract on Raspberry Pi OS . . . . . . . . . . . . 19
4.2.2.5 Verifying Your Tesseract Install . . . . . . . . . . . . . . . . . . 19
4.2.3 Creating a Python Virtual Environment for OCR . . . . . . . . . . . . . . 20
4.2.4 Installing OpenCV and PyTesseract . . . . . . . . . . . . . . . . . . . . . 20
4.2.5 Installing Other Computer Vision, Deep Learning, and Machine Learn-
ing Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2.6 Pre-Configured VM for OCR . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5 Your First OCR Project with Tesseract 23
5.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5.2 Getting Started with Tesseract . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.2.1 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.2.2 Basic OCR with Tesseract . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.2.3 Tesseract OCR Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
6 Detecting and OCR’ing Digits with Tesseract 31
6.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.2 Digit Detection and Recognition with Tesseract . . . . . . . . . . . . . . . . . . . 32
6.2.1 What Is Digit Detection and Recognition? . . . . . . . . . . . . . . . . . 32
6.2.2 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
6.2.3 OCR’ing Digits with Tesseract and OpenCV . . . . . . . . . . . . . . . . 33
6.2.4 Digit OCR Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
CONTENTS v
7 Whitelisting and Blacklisting Characters with Tesseract 37
7.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
7.2 Whitelisting and Blacklisting Characters for OCR . . . . . . . . . . . . . . . . . . 37
7.2.1 What Are Whitelists and Blacklists? . . . . . . . . . . . . . . . . . . . . . 38
7.2.2 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
7.2.3 Whitelisting and Blacklisting Characters with Tesseract . . . . . . . . . . 38
7.2.4 Whitelisting and Blacklisting with Tesseract Results . . . . . . . . . . . . 40
7.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
8 Correcting Text Orientation with Tesseract 45
8.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
8.2 What Is Orientation and Script Detection? . . . . . . . . . . . . . . . . . . . . . . 46
8.3 Detecting and Correcting Text Orientation with Tesseract . . . . . . . . . . . . . 47
8.3.1 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
8.3.2 Implementing Our Text Orientation and Correction Script . . . . . . . . . 48
8.3.3 Text Orientation and Correction Results . . . . . . . . . . . . . . . . . . 49
8.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
9 Language Translation and OCR with Tesseract 55
9.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
9.2 OCR and Language Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
9.2.1 Translating Text to Different Languages with textblob . . . . . . . . . 56
9.2.2 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
9.2.3 Implementing Our OCR and Language Translation Script. . . . . . . . . 57
9.2.4 OCR Language Translation Results . . . . . . . . . . . . . . . . . . . . . 59
9.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
10 Using Tesseract with Non-English Languages 61
10.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
10.2 Tesseract and Non-English Languages . . . . . . . . . . . . . . . . . . . . . . . 61
10.2.1 Configuring Tesseract for Multiple Languages . . . . . . . . . . . . . . . 62
10.2.2 Adding Language Packs to Tesseract . . . . . . . . . . . . . . . . . . . . 62
10.2.3 The textblob Package’s Relation to This Chapter . . . . . . . . . . . . 63
10.2.4 Verifying Tesseract Support for Non-English Languages . . . . . . . . . 64
10.3 Tesseract, Multiple Languages, and Python . . . . . . . . . . . . . . . . . . . . . 65
10.3.1 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
10.3.2 Implementing Our Tesseract with Non-English Languages Script . . . . 65
vi CONTENTS
10.3.3 Tesseract and Non-English Languages Results . . . . . . . . . . . . . . 67
10.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
11 Improving OCR Results with Tesseract Options 71
11.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
11.2 Tesseract Page Segmentation Modes . . . . . . . . . . . . . . . . . . . . . . . . 72
11.2.1 What Are Page Segmentation Modes? . . . . . . . . . . . . . . . . . . . 72
11.2.2 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
11.2.3 PSM 0. Orientation and Script Detection Only . . . . . . . . . . . . . . . 75
11.2.4 PSM 1. Automatic Page Segmentation with OSD . . . . . . . . . . . . . 77
11.2.5 PSM 2. Automatic Page Segmentation, But No OSD, or OCR . . . . . . 78
11.2.6 PSM 3. Fully Automatic Page Segmentation, But No OSD . . . . . . . . 78
11.2.7 PSM 4. Assume a Single Column of Text of Variable Sizes . . . . . . . . 79
11.2.8 PSM 5. Assume a Single Uniform Block of Vertically Aligned Text . . . . 82
11.2.9 PSM 6. Assume a Single Uniform Block of Text . . . . . . . . . . . . . . 85
11.2.10PSM 7. Treat the Image as a Single Text Line . . . . . . . . . . . . . . . 87
11.2.11PSM 8. Treat the Image as a Single Word . . . . . . . . . . . . . . . . . 88
11.2.12PSM 9. Treat the Image as a Single Word in a Circle . . . . . . . . . . . 89
11.2.13PSM 10. Treat the Image as a Single Character . . . . . . . . . . . . . . 90
11.2.14PSM 11. Sparse Text: Find as Much Text as Possible in No Particular
Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
11.2.15PSM 12. Sparse Text with OSD . . . . . . . . . . . . . . . . . . . . . . . 93
11.2.16PSM 13. Raw Line: Treat the Image as a Single Text Line, Bypassing
Hacks That Are Tesseract-Specific . . . . . . . . . . . . . . . . . . . . . 93
11.2.17Tips, Suggestions, and Best Practices for PSM Modes . . . . . . . . . . 94
11.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
12 Improving OCR Results with Basic Image Processing 97
12.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
12.2 Image Processing and Tesseract OCR . . . . . . . . . . . . . . . . . . . . . . . 98
12.2.1 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
12.2.2 When Tesseract by Itself Fails to OCR an Image . . . . . . . . . . . . . 99
12.2.3 Implementing an Image Processing Pipeline for OCR . . . . . . . . . . . 100
12.2.4 Basic Image Processing and Tesseract OCR Results . . . . . . . . . . . 105
12.2.5 How Should I Improve My Image Processing Pipeline Skills? . . . . . . 105
12.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
CONTENTS vii
13 Improving OCR Results with Spellchecking 107
13.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
13.2 OCR and Spellchecking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
13.2.1 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
13.2.2 Implementing Our OCR Spellchecking Script . . . . . . . . . . . . . . . 108
13.2.3 OCR Spellchecking Results . . . . . . . . . . . . . . . . . . . . . . . . . 110
13.2.4 Limitations and Drawbacks . . . . . . . . . . . . . . . . . . . . . . . . . 111
13.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
14 Finding Text Blobs in an Image with OpenCV 113
14.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
14.2 Finding Text in Images with Image Processing . . . . . . . . . . . . . . . . . . . 114
14.2.1 What Is a Machine-Readable Zone? . . . . . . . . . . . . . . . . . . . . 114
14.2.2 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
14.2.3 Locating MRZs in Passport Images . . . . . . . . . . . . . . . . . . . . . 116
14.2.4 Text Blob Localization Results . . . . . . . . . . . . . . . . . . . . . . . . 123
14.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
15 OCR Using Template Matching 127
15.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
15.2 OCR’ing Text via Template Matching . . . . . . . . . . . . . . . . . . . . . . . . . 128
15.2.1 The OCR-A, OCR-B, and MICR E-13B Fonts . . . . . . . . . . . . . . . 128
15.2.2 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
15.2.3 Credit Card OCR with Template Matching and OpenCV . . . . . . . . . 130
15.2.4 Credit Card OCR Results . . . . . . . . . . . . . . . . . . . . . . . . . . 139
15.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
16 OCR’ing Characters with Basic Image Processing 143
16.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
16.2 OCR’ing 7-Segment Digit Displays with Image Processing . . . . . . . . . . . . 144
16.2.1 The 7-Segment Display . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
16.2.2 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
16.2.3 Implementing Our 7-Segment Visualization Utilities . . . . . . . . . . . . 146
16.2.4 Visualizing Digits on a 7-Segment Display . . . . . . . . . . . . . . . . . 149
16.2.5 Creating Our 7-Segment Digit Recognizer . . . . . . . . . . . . . . . . . 150
16.2.6 Digit OCR with Image Processing and OpenCV . . . . . . . . . . . . . . 152
16.2.7 The 7-Segment Digit OCR Results . . . . . . . . . . . . . . . . . . . . . 155
viii CONTENTS
16.2.8 Suggestions for Your Applications . . . . . . . . . . . . . . . . . . . . . . 156
16.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
17 Text Bounding Box Localization and OCR with Tesseract 159
17.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
17.2 Text Localization and OCR with Tesseract . . . . . . . . . . . . . . . . . . . . . . 160
17.2.1 What Is Text Localization and Detection? . . . . . . . . . . . . . . . . . . 160
17.2.2 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
17.2.3 Drawing OCR’d Text with OpenCV . . . . . . . . . . . . . . . . . . . . . 162
17.2.4 Implementing Text Localization and OCR with Tesseract . . . . . . . . . 162
17.2.5 Text Localization and OCR Results . . . . . . . . . . . . . . . . . . . . . 165
17.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
18 Rotated Text Bounding Box Localization with OpenCV 169
18.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
18.2 Detecting Rotated Text with OpenCV and EAST . . . . . . . . . . . . . . . . . . 170
18.2.1 Why Is Natural Scene Text Detection So Challenging? . . . . . . . . . . 170
18.2.2 EAST Deep Learning Text Detector . . . . . . . . . . . . . . . . . . . . . 172
18.2.3 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
18.2.4 Creating Our Rotated Text Detector Helper Functions . . . . . . . . . . . 174
18.2.5 Implementing the EAST Text Detector with OpenCV . . . . . . . . . . . 177
18.2.6 Rotated Text Bounding Box Results . . . . . . . . . . . . . . . . . . . . . 181
18.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
19 A Complete Text Detection and OCR Pipeline 185
19.1 Chapter Learning Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
19.2 Building a Complete Text Detection and OCR Pipeline . . . . . . . . . . . . . . . 186
19.2.1 How to Combine Text Detection and OCR with OpenCV . . . . . . . . . 186
19.2.2 Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
19.2.3 Implementing Our Text Detection and OCR Pipeline . . . . . . . . . . . 188
19.2.4 Text Detection and OCR with OpenCV Results . . . . . . . . . . . . . . 193
19.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
20 Conclusions 197
Bibliography 199
To my first computer science teacher, Marla Wood.
Thank you for everything you’ve done for me when I was a high school kid.
Rest assured, wherever you are, you’re a big part
and a big reason for the person I am today.