Smart Camera Design: Algorithms, Architectures, and Art

234 Pages·2018·7.6 MB·English

by  Marilyn Wolf (auth.)

Marilyn Wolf
Smart
Camera Design
Algorithms, Architectures, and Art
Smart Camera Design
Marilyn Wolf
Smart Camera Design
Algorithms, Architectures, and Art
MarilynWolf
SchoolofECE
GeorgiaInstituteofTechnology
Atlanta,GA,USA
ISBN978-3-319-69522-8 ISBN978-3-319-69523-5 (eBook)
https://doi.org/10.1007/978-3-319-69523-5
LibraryofCongressControlNumber:2017957575
©SpringerInternationalPublishingAG2018
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or
dissimilarmethodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt
fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthis
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, express or implied, with respect to the material contained
hereinor for anyerrors oromissionsthat may havebeenmade. Thepublisher remainsneutralwith
regardtojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
For Dad,
Who set me on this path.
Preface
This book draws from three areas of computing: image processing, computer
vision,andcomputergraphics.Imageprocessingandcomputervisioninparticular
havelongbeenseparatefieldswithoverlappinginterests.Thisispartlyasociolog-
ical phenomenon—image processing comes from electrical engineering, while
computer vision comes from computer science. These two fields blend smoothly
indigitalcameradesign.Themodernimagingchainstartsattraditionalfilteringand
endswithfeatureanalysis.
Parts of this book draw upon my research work with my students at Princeton
and Georgia Tech. Cheng-Yao Chen, Santanu Dutta, Jason Fritts, Se Hun Kim,
Changhong Lin, Chung-Ching Lin, Tiehan Lv, Jason Schlessman, Senem
Velipasalar,JiangXu,HeatherYu,andShengqiYanghaveallworkedonaspects
ofmultimediacomputingandembeddedcomputervision.Iamgratefultothemfor
the opportunity to work with them and learn from them. Burak Ozer was not my
officialstudent,buthehasbeenmyfriendandcollaboratoronsmartcamerasforthe
past15years.
Theinspirationforthisbookcomesfrommyfather,aninventorwhocreatedtwo
differentpanoramiccameras.Thecameraonthededicationpage,whichwasknown
commercially as the CycloPan 360, uses a cylindrical mapping. Here is another
pictureofDadwithhisdonutcamera.
vii
viii Preface
He also created a motion picture version of this camera for use in a flight
simulator. That camera used 500 aerial film. The projector used the same optical
pathasthecamerabutrotatedcontinuouslytosweeptheimage;itwasanimposing
machine.IranDad’scolordarkroomduringthoseyears—Iworkedcheap.
Dad,thisbookisentirelyyourfault.Ihavespentnightafternighttypingaway
becauseyoutaughtmehowtothink.Thisbookismytributetoyou.
Atlanta,GA,USA MarilynWolf
August2016
Contents
1 DigitalPhotography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 PrevisualizationandAutoprevisualization. . . . . . . . . . . . . . . . . 1
1.3 EnhancedImages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 BeyondImagestoAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5 StillandMovingImages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.6 TakingaPicture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.7 HowtoReadthisBook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Light,Optics,andImaging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 ImageFormation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.1 LightandImages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.2 ThePhysicsofLight. . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3 TheHumanVisualSystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 ColorScience. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.1 TheoriesofColorVision. . . . . . . . . . . . . . . . . . . . . . . 19
2.4.2 ColorModels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.5 Lenses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5.1 LensesandImageFormation. . . . . . . . . . . . . . . . . . . . 23
2.5.2 RayOptics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5.3 LensDesign. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.5.4 Panoramas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.5.5 AssessingLenses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.6 GeometryandtheCameraModel. . . . . . . . . . . . . . . . . . . . . . . 41
2.6.1 ProjectiveGeometry. . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.6.2 TheCameraModel. . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.6.3 CameraCalibration. . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.7 ImageDisplay. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.8 PracticalImageCapture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
ix
x Contents
2.8.1 ExposureSettings. . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.8.2 WhichExposureSetting?. . . . . . . . . . . . . . . . . . . . . . . 52
2.8.3 ColorTemperature. . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.8.4 ImageComposition. . . . . . . . . . . . . . . . . . . . . . . . . . . 58
2.8.5 ImageQualityAssessment. . . . . . . . . . . . . . . . . . . . . . 63
2.9 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3 ImageCaptureSystemsandAlgorithms. . . . . . . . . . . . . . . . . . . . . . 67
3.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.2 TheGenericCameraArchitecture. . . . . . . . . . . . . . . . . . . . . . . 68
3.3 TheCameraDesignSpace. . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.3.1 Trade-Offs.. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. 69
3.3.2 UseCasesforCameras. . . . . . . . . . . . . . . . . . . . . . . . 70
3.3.3 FourExamplesofCameraDesigns. . . . . . . . . . . . . . . . 70
3.4 ImageSensors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.4.1 ImageSensorArchitectures. . . . . . . . . . . . . . . . . . . . . 73
3.4.2 Photosensors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.4.3 Charge-CoupledDevices. . . . . . . . . . . . . . . . . . . . . . . 77
3.4.4 APSCMOSImageSensors. . . . . . . . . . . . . . . . . . . . . 78
3.4.5 AdvancedImageSensors. . . . . . . . . . . . . . . . . . . . . . . 82
3.4.6 ImageSensorCharacteristics. . . . . . . . . . . . . . . . . . . . 83
3.4.7 ShuttersandIrises. . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3.5 PreexposureOperations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.5.1 Autofocus. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
3.5.2 Exposure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
3.5.3 ImageStabilization. . . . . . . . . . . . . . . . . . . . . . . . . . . 98
3.5.4 FaceDetectionandTracking. . . . . . . . . . . . . . . . . . . . 99
3.6 PostexposureOperations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
3.6.1 ColorFilterArrayInterpolation. . . . . . . . . . . . . . . . . . 102
3.6.2 WhiteBalance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
3.6.3 Sharpening. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
3.7 ImageandVideoCompression. . . . . . . . . . . . . . . . . . . . . . . . . 108
3.7.1 LossyCompression. . . . . . . . . . . . . . . . . . . . . . . . . . . 108
3.7.2 ImageCodingandJPEG. . . . . . . . . . . . . . . . . . . . . . . 108
3.7.3 VideoCoding,H.264/AVC,andHEVC/H.265. . . . . . . 113
3.7.4 QualityAssessmentofCompressedImages. . . . . . . . . . 120
3.8 ComputingPlatforms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
3.8.1 CamerasasHeterogeneousMultiprocessors. . . . . . . . . 121
3.8.2 Buffering. . . .. . . .. . . .. . . . .. . . .. . . .. . . .. . . . .. 122
3.8.3 InputandOutput. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
3.8.4 FileFormats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
3.8.5 OperatingSystemsandFileSystems. . . . . . . . . . . . . . . 126
3.8.6 Accelerators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Contents xi
3.9 ImageCharacteristicsandImageCapture. . . . .. . . . . .. . . . . .. 130
3.10 StereoandMulticameraSystems. . . . . . . . . . . . . . . . . . . . . . . . 131
3.11 Trade-OffsRevisited. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
4 ImageandVideoEnhancement. . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
4.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
4.2 UsefulAlgorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4.3 TonalMappingandColorGrading. . . . . . . . . . . . . . . . . . . . . . 138
4.4 High-DynamicRangeImages. . . . . . . . . . . . . . . . . . . . . . . . . . 143
4.5 SharpeningandSuperresolution. . . . . . . . . . . . . . . . . . . . . . . . 146
4.6 BokehIntroduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
4.7 LensCorrections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
4.8 FocusStacking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
4.9 KeystoneCorrection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
4.10 MosaicComposition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
4.11 VideoStabilization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
4.11.1 OpticalFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
4.11.2 StabilizationAlgorithms. . . . . .. . . . . . . .. . . . . . . .. . 157
4.12 SoftwareDesignforImageEnhancement. . . . . . . . . . . . . . . . . . 158
4.13 PracticalImageEnhancement. . . . . . . . . . . . . . . . . . . . . . . . . . 160
5 ImageandVideoAnalysis. .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . 163
5.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
5.2 ImageAnalysisAlgorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
5.3 ImageandVideoCharacteristics. . . . . . . . . . . . . . . . . . . . . . . . 167
5.3.1 ImageStatistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
5.3.2 Saliency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
5.3.3 KeyFrameSelection. . . . . . . . . . . . . . . . . . . . . . . . . . 170
5.4 SceneAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
5.4.1 VisualSearch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
5.4.2 FaceDetectionandRecognition. . . . . . . . . . . . . . . . . . 172
5.5 Tracking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
5.5.1 BackgroundElimination. . . .. . . . . . . . .. . . . . . . . .. . 173
5.5.2 TrackingfromaFixedCamera. . . . . . . . . . . . . . . . . . . 175
5.5.3 AppearanceModels. . . . . . . . . . . . . . . . . . . . . . . . . . . 178
5.5.4 ActivityAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
5.5.5 TrackingfromaMovingCamera. . . . . . . . . . . . . . . . . 183
5.6 MulticameraSystems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
5.6.1 MulticameraSystemsasDistributedComputing
Systems. . . . .. . . . . .. . . . . .. . . . . .. . . . .. . . . . .. . 187
5.6.2 MulticameraCalibration. .. . . . .. . . . .. . . . . .. . . . .. 189
5.6.3 MulticameraTracking. . . . . . . . . . . . . . . . . . . . . . . . . 191
5.7 UseCasesandWorkflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197