Table Of ContentUndergraduate Topics in Computer
Science
UndergraduateTopicsinComputerScience(UTiCS)delivershigh-qualityinstructionalcontentforun-
dergraduatesstudyinginallareasofcomputingandinformationscience.Fromcorefoundationaland
theoreticalmaterialtofinal-yeartopicsandapplications,UTiCSbookstakeafresh,concise,andmod-
ernapproachandareidealforself-studyorforaone-ortwo-semestercourse.Thetextsareallauthored
byestablishedexpertsintheirfields,reviewedbyaninternationaladvisoryboard,andcontainnumer-
ousexamplesandproblems.Manyincludefullyworkedsolutions.
Forfurthervolumes:
http://www.springer.com/series/7592
Thomas B. Moeslund
Introduction
to Video and
Image Processing
Building Real Systems and Applications
ThomasB.Moeslund
VisualAnalysisofPeopleLaboratory
DepartmentofArchitecture,Design,and
MediaTechnology
AalborgUniversity
Aalborg
Denmark
Serieseditor
IanMackie
Advisoryboard
SamsonAbramsky,UniversityofOxford,Oxford,UK
KarinBreitman,PontificalCatholicUniversityofRiodeJaneiro,RiodeJaneiro,Brazil
ChrisHankin,ImperialCollegeLondon,London,UK
DexterKozen,CornellUniversity,Ithaca,USA
AndrewPitts,UniversityofCambridge,Cambridge,UK
HanneRiisNielson,TechnicalUniversityofDenmark,KongensLyngby,Denmark
StevenSkiena,StonyBrookUniversity,StonyBrook,USA
IainStewart,UniversityofDurham,Durham,UK
ISSN1863-7310 UndergraduateTopicsinComputerScience
ISBN978-1-4471-2502-0 e-ISBN978-1-4471-2503-7
DOI10.1007/978-1-4471-2503-7
SpringerLondonDordrechtHeidelbergNewYork
BritishLibraryCataloguinginPublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
LibraryofCongressControlNumber:2012930996
©Springer-VerlagLondonLimited2012
Apartfromanyfairdealingforthepurposesofresearchorprivatestudy,orcriticismorreview,asper-
mittedundertheCopyright,DesignsandPatentsAct1988,thispublicationmayonlybereproduced,
storedortransmitted,inanyformorbyanymeans,withthepriorpermissioninwritingofthepublish-
ers,orinthecaseofreprographicreproductioninaccordancewiththetermsoflicensesissuedbythe
CopyrightLicensingAgency.Enquiriesconcerningreproductionoutsidethosetermsshouldbesentto
thepublishers.
Theuseofregisterednames,trademarks,etc.,inthispublicationdoesnotimply,evenintheabsenceofa
specificstatement,thatsuchnamesareexemptfromtherelevantlawsandregulationsandthereforefree
forgeneraluse.
Thepublishermakesnorepresentation,expressorimplied,withregardtotheaccuracyoftheinformation
containedinthisbookandcannotacceptanylegalresponsibilityorliabilityforanyerrorsoromissions
thatmaybemade.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
One of the first times I ever encountered video and image processing was in a
semesterprojectatmyfourthyearofstudying.Theaimoftheprojectwastodesign
asystemthatautomaticallylocatedthecenterandsizeofmushroomsinanimage.
Giventhis informationa robotshould pickthemushrooms. I was intriguedby the
notion of a “seeing computer”. Little did I know that this encounter would shape
mostparts(sofar)ofmyprofessionallife.
Idecidedtostudyvideoandimageprocessingindepthandsignedupforamas-
ter’sprogramfocusingonthesetopics.IsoonrealizedthatIhadmadeagoodchoice,
butwaspuzzledbythefactthatthewondersofdigitalvideoandimageprocessing
oftenwerepresentedinastrictmathematicalmanner.Whilethisisfineforhardcore
engineers (including me) and computer scientists, it makes video and image pro-
cessingunnecessarilydifficultforothers.Ireallyfeltthiswasapityanddecidedto
dosomethingaboutit—thatwas15yearsago.
Inthisbooktheconceptsandmethodsaredescribedinalessmathematicalman-
nerandthelanguageisingeneralcasual.Inordertoassistthereaderwiththemath
thatisusedinthebookAppendixBisincluded.Inthisregardsthistextbookisself-
contained.SomeofthekeyalgorithmsareexemplifiedinC-code.Pleasenotethat
the code is neither optimal nor complete and merely serves as an additional input
forcomprehendingthealgorithms.
Anotheraspectthatpuzzledmeasastudentwasthatthetextbookswereallabout
imageprocessing,whileweconstructedsystemsthatworkedwithvideo.Manyof
themethodsdescribedforimageprocessingcanobviouslyalsobeappliedtovideo
data.Butvideodataaddthetemporaldimension,whichisoftenthekeytosuccess
insystemsprocessingvideo.Thisbookthereforeaimsatnotonlyintroducingimage
processing but also video processing. Moreover, the last two chapters of the book
describetheprocessofdesigningandimplementingrealsystemsprocessingvideo
data.Onthewebsiteforthebookyoucanfinddetaileddescriptionsofotherpractical
systemsprocessingvideo:http://www.vip.aau.dk.
Ihavetriedtomakethebookasconciseaspossible.Thishasforcedmetoleave
out details and topics that might be of interest to some readers. As a compromise
eachchapterisendedbya“FurtherInformation”sectionwhereinpointerstoaddi-
tionalconcepts,methodsanddetailsaregiven.
v
vi Preface
For Instructors Each chapter is ended by a number of exercises. The first exer-
ciseaftereachchapteraimsatassessingtowhatdegreethestudentshaveunderstood
themainconcepts.Ifpossible,itisrecommendedthattheseexercisesarediscussed
within small groups. The following exercises have a more practical focus where
concrete problems need to be solved using the different methods/algorithms pre-
sentedintheassociatedchapters.Lastlyoneormoreso-calledadditionalexercises
arepresent.Theseaimattopicsnotdiscusseddirectlyinthechapters.Theideabe-
hind these exercises is that they can serve as self-studies where each student (or
a small group of students) finds the solution by investigating other sources. They
couldthenpresenttheirfindingsforotherstudents.
BesidestheexerciseslistedinthebookIstronglyrecommendtocombinethose
with examples and exercises where real images/videos are processed. Personally
I start with ImageJ for image processing and EyesWeb for video processing. The
main motivation for using these programs is that they are easy to learn and hence
thestudentscanfocusonthevideoandimageprocessingasopposedtoaspecific
programming language, when solving the exercises. However, when it comes to
buildingrealsystemsI recommendusingOpenCVor openFrameworks(EyesWeb
orsimilarcanofcoursealsobeusedtobuildsystems,buttheydonotgeneralizeas
well).Tothisendstudentsofcourseneedtohaveacourseonproceduralprogram-
ming before or in parallel with the image processing course. To make the switch
from ImageJ/Eyesweb to a more low-level environment like OpenCV, I normally
askeachstudenttodoanassignmentwheretheywriteaprogramthatcancapture
animage,makesomeimageprocessinganddisplaytheresult.Whenthestudentcan
dothishehasaframeworkforimplementing“all”otherimageprocessingmethods.
Thetimeallocatedforthisassignmentofcoursedependsontheprogrammingex-
periencesofthestudents.
Acknowledgement Thebookwaswrittenprimarilyatweekendsandlatenights,
and I thank my family for being understanding and supporting during that time!
Iwouldalsoliketothankthefollowingpeople:HansEbertandVolkerKrügerfor
initial discussions on the “book project”. Moritz Störring for providing Fig. 2.3.
Rasmus R. Paulsen for providing Figs. 2.22(a) and 4.5. Rikke Gade for providing
Fig.2.22(b).TobiasThyrrestrupforprovidingFig.2.22(c).DavidMeredith,Rasmus
R.Paulsen,LarsRengandKamalNasrollahiforinsightfuleditorialcomments,and
finally a special thanks to Lars Knudsen and Andreas Møgelmose, who provided
valuableassistancebycreatingmanyoftheillustrationsusedthroughoutthebook.
Enjoy!
Viborg,Denmark ThomasB.Moeslund
Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 TheDifferentFlavorsofVideoandImageProcessing . . . . . . . 2
1.2 GeneralFramework . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 TheChaptersinThisBook . . . . . . . . . . . . . . . . . . . . . 4
1.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 ImageAcquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Energy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1 Illumination . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 TheOpticalSystem . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.1 TheLens . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3 TheImageSensor . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 TheDigitalImage . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.1 TheRegionofInterest(ROI) . . . . . . . . . . . . . . . . 20
2.5 FurtherInformation . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3 ColorImages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1 WhatIsaColor? . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2 RepresentationofanRGBColorImage . . . . . . . . . . . . . . . 27
3.2.1 TheRGBColorSpace . . . . . . . . . . . . . . . . . . . . 30
3.2.2 ConvertingfromRGBtoGray-Scale . . . . . . . . . . . . 30
3.2.3 TheNormalizedRGBColorRepresentation . . . . . . . . 32
3.3 OtherColorRepresentations . . . . . . . . . . . . . . . . . . . . . 34
3.3.1 TheHSIColorRepresentation. . . . . . . . . . . . . . . . 36
3.3.2 TheHSVColorRepresentation . . . . . . . . . . . . . . . 37
3.3.3 TheYUVandYC C ColorRepresentations . . . . . . . . 38
b r
3.4 FurtherInformation . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4 PointProcessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.1 Gray-LevelMapping . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.2 Non-linearGray-LevelMapping . . . . . . . . . . . . . . . . . . 46
4.2.1 GammaMapping . . . . . . . . . . . . . . . . . . . . . . 46
4.2.2 LogarithmicMapping . . . . . . . . . . . . . . . . . . . . 48
vii
viii Contents
4.2.3 ExponentialMapping . . . . . . . . . . . . . . . . . . . . 48
4.3 TheImageHistogram . . . . . . . . . . . . . . . . . . . . . . . . 49
4.3.1 HistogramStretching . . . . . . . . . . . . . . . . . . . . 51
4.3.2 HistogramEqualization . . . . . . . . . . . . . . . . . . . 53
4.4 Thresholding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.4.1 ColorThresholding . . . . . . . . . . . . . . . . . . . . . 57
4.4.2 ThresholdinginVideo . . . . . . . . . . . . . . . . . . . . 59
4.5 LogicOperationsonBinaryImages . . . . . . . . . . . . . . . . . 63
4.6 ImageArithmetic . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.7 ProgrammingPointProcessingOperations . . . . . . . . . . . . . 66
4.8 FurtherInformation . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5 NeighborhoodProcessing . . . . . . . . . . . . . . . . . . . . . . . . 71
5.1 TheMedianFilter . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.1.1 RankFilters . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2 Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2.1 TemplateMatching . . . . . . . . . . . . . . . . . . . . . 78
5.2.2 EdgeDetection. . . . . . . . . . . . . . . . . . . . . . . . 81
5.2.3 ImageSharpening . . . . . . . . . . . . . . . . . . . . . . 85
5.3 FurtherInformation . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6 Morphology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.1 Level1:HitandFit . . . . . . . . . . . . . . . . . . . . . . . . . 92
6.1.1 Hit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
6.1.2 Fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
6.2 Level2:DilationandErosion . . . . . . . . . . . . . . . . . . . . 94
6.2.1 Dilation . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6.2.2 Erosion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
6.3 Level3:CompoundOperations . . . . . . . . . . . . . . . . . . . 96
6.3.1 Closing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
6.3.2 Opening . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
6.3.3 CombiningOpeningandClosing . . . . . . . . . . . . . . 99
6.3.4 BoundaryDetection . . . . . . . . . . . . . . . . . . . . . 99
6.4 FurtherInformation . . . . . . . . . . . . . . . . . . . . . . . . . 100
6.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
7 BLOBAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
7.1 BLOBExtraction . . . . . . . . . . . . . . . . . . . . . . . . . . 103
7.1.1 TheRecursiveGrass-FireAlgorithm . . . . . . . . . . . . 104
7.1.2 TheSequentialGrass-FireAlgorithm . . . . . . . . . . . . 106
7.2 BLOBFeatures . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
7.3 BLOBClassification . . . . . . . . . . . . . . . . . . . . . . . . . 110
7.4 FurtherInformation . . . . . . . . . . . . . . . . . . . . . . . . . 113
7.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Contents ix
8 SegmentationinVideoData . . . . . . . . . . . . . . . . . . . . . . . 117
8.1 VideoAcquisition . . . . . . . . . . . . . . . . . . . . . . . . . . 117
8.2 DetectingChangesintheVideo . . . . . . . . . . . . . . . . . . . 120
8.2.1 TheAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . 120
8.3 BackgroundSubtraction . . . . . . . . . . . . . . . . . . . . . . . 123
8.3.1 DefiningtheThresholdValue . . . . . . . . . . . . . . . . 124
8.4 ImageDifferencing . . . . . . . . . . . . . . . . . . . . . . . . . 125
8.5 FurtherInformation . . . . . . . . . . . . . . . . . . . . . . . . . 126
8.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
9 Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
9.1 Tracking-by-Detection . . . . . . . . . . . . . . . . . . . . . . . . 129
9.2 Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
9.3 TrackingMultipleObjects . . . . . . . . . . . . . . . . . . . . . . 133
9.3.1 GoodFeaturestoTrack . . . . . . . . . . . . . . . . . . . 135
9.4 FurtherInformation . . . . . . . . . . . . . . . . . . . . . . . . . 137
9.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
10 GeometricTransformations . . . . . . . . . . . . . . . . . . . . . . . 141
10.1 AffineTransformations . . . . . . . . . . . . . . . . . . . . . . . 142
10.1.1 Translation . . . . . . . . . . . . . . . . . . . . . . . . . . 142
10.1.2 Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
10.1.3 Rotation . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
10.1.4 Shearing . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
10.1.5 CombiningtheTransformations . . . . . . . . . . . . . . . 144
10.2 MakingItWorkinPractice . . . . . . . . . . . . . . . . . . . . . 145
10.2.1 BackwardMapping . . . . . . . . . . . . . . . . . . . . . 146
10.2.2 Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . 147
10.3 Homography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
10.4 FurtherInformation . . . . . . . . . . . . . . . . . . . . . . . . . 152
10.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
11 VisualEffects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
11.1 VisualEffectsBasedonPixelManipulation. . . . . . . . . . . . . 155
11.1.1 PointProcessing . . . . . . . . . . . . . . . . . . . . . . . 156
11.1.2 NeighborhoodProcessing . . . . . . . . . . . . . . . . . . 157
11.1.3 Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
11.1.4 ReducedColors . . . . . . . . . . . . . . . . . . . . . . . 158
11.1.5 Randomness . . . . . . . . . . . . . . . . . . . . . . . . . 159
11.2 VisualEffectsBasedonGeometricTransformations . . . . . . . . 160
11.2.1 PolarTransformation . . . . . . . . . . . . . . . . . . . . 160
11.2.2 TwirlTransformation . . . . . . . . . . . . . . . . . . . . 162
11.2.3 SphericalTransformation . . . . . . . . . . . . . . . . . . 163
11.2.4 RippleTransformation . . . . . . . . . . . . . . . . . . . . 164
11.2.5 LocalTransformation . . . . . . . . . . . . . . . . . . . . 165
11.3 FurtherInformation . . . . . . . . . . . . . . . . . . . . . . . . . 165
11.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
