High Dynamic Range Imaging Sensors and Architectures

140 Pages·2013·17.813 MB·English

by  Arnaud Darmont

SPIE PRESS
Illumination is a crucial element in many applications, matching the luminance of the
scene with the operational range of a camera. When luminance cannot be adequately
controlled, a high dynamic range (HDR) imaging system may be necessary. These
systems are being increasingly used in automotive on-board systems, road traffic
monitoring, and other industrial, security, and military applications. This book provides
readers with an intermediate discussion of HDR image sensors and techniques for
industrial and non-industrial applications. It describes various sensor and pixel
architectures capable of achieving HDR imaging, as well as software approaches to
make high dynamic range images out of lower dynamic range sensors or image sets.
Some methods for automatic control of exposure and dynamic range of image
sensors are also introduced.
P.O. Box 10
Bellingham, WA 98227-0010
ISBN: 9780819488305
SPIE Vol. No.: PM214

Bellingham, Washington USA
LibraryofCongressCataloging-in-PublicationData
Darmont,Arnaud.
Highdynamicrangeimaging:sensorsandarchitectures/ArnaudDarmont.
p.cm.
Includesbibliographicalreferencesandindex.
ISBN978-0-8194-8830-5
1. Image converters. 2. High dynamic range imaging–Equipment and supplies. I.
Title.
TK8316.D372012
621.36(cid:48)7–dc23
2012027698
Publishedby
SPIE—TheInternationalSocietyforOpticalEngineering
P.O.Box10
Bellingham,Washington98227-0010USA
Phone:[phone]
Fax:[phone]
Email:[email protected]
Web:http://spie.org
Copyright(cid:13)c 2012SocietyofPhoto-OpticalInstrumentationEngineers(SPIE)
Allrightsreserved.Nopartofthispublicationmaybereproducedordistributedin
anyformorbyanymeanswithoutwrittenpermissionofthepublisher.
Thecontentofthisbookreflectstheworkandthoughtsoftheauthor(s).
Every effort has been made to publish reliable and accurate information herein,
but the publisher is not responsible for the validity of the information or for any
outcomesresultingfromreliancethereon.
PrintedintheUnitedStatesofAmerica.
Firstprinting

Contents
Acknowledgments.............................................................................................. ix
Preface................................................................................................................. xi
Chapter1 Introduction.................................................................................. 1
1.1 ApplicationsRequiringaHigherDynamicRange............................... 1
1.2 HighDynamicRangePhotography........................................................... 6
1.3 ScientificApplications................................................................................... 8
1.4 High Dynamic Range, Wide Dynamic Range, and Extended
DynamicRange................................................................................................ 8
1.5 OutlineandGoals............................................................................................ 9
1.6 DefiningaCamera........................................................................................... 11
Chapter2 DynamicRange............................................................................ 13
2.1 ImageSensorTheory...................................................................................... 13
2.1.1 Lightsource,scene,pixel,andirradiance............................... 13
2.1.2 Sensingnodeandlight–matterinteraction.............................. 16
2.1.3 Pixel...................................................................................................... 19
2.1.4 Pixelarray.......................................................................................... 19
2.1.5 Readoutcircuits................................................................................ 20
2.1.6 Imageencoding................................................................................ 21
2.2 Low-LightImagingLimitations.................................................................. 21
2.2.1 Noisesourcessummary................................................................. 21
2.2.2 Lowestdetectablelimit.................................................................. 25
2.3 Bright-LightImagingLimitations.............................................................. 26
2.3.1 Saturation............................................................................................ 26
2.3.2 Highestdetectablelevel................................................................. 28
2.4 Signal-to-NoiseRatio..................................................................................... 28
2.5 DynamicRangeGaps..................................................................................... 29
2.5.1 Responsecurve................................................................................. 29
2.5.2 Dynamicrangegaps....................................................................... 30
2.5.3 Presencefunctionofdynamicrangegaps............................... 32
2.6 DynamicRange................................................................................................ 33
2.6.1 Definition............................................................................................ 33
v
vi Contents
2.6.2 Remark................................................................................................ 33
2.6.3 Relativemeasurement.................................................................... 34
2.7 ImageInformation........................................................................................... 35
2.8 HumanVisionSystemandItsDynamicRange..................................... 36
2.8.1 Generalpropertiesofhumanvision.......................................... 37
2.8.2 Dynamicrangeofthehumaneye............................................... 37
2.8.3 Noiseperception.............................................................................. 37
2.8.4 Opticalperformance....................................................................... 38
Chapter3 HardwareMethodstoExtendtheDynamicRange................. 39
3.1 Introduction:IntegratingLinearPixels..................................................... 39
3.1.1 Rolling-shutterpixelarchitecture............................................... 39
3.1.2 Global-shutter-pixelarchitecture................................................ 41
3.1.3 SNRanddynamicrangestudy.................................................... 41
3.2 MultilinearPixels............................................................................................. 44
3.2.1 Principle.............................................................................................. 44
3.2.2 Howcanmultiplesegmentsberealizedpractically?........... 49
3.2.3 Multiple-segmentmethodbasedonwellsizing.................... 51
3.2.4 Dynamiccompression.................................................................... 52
3.2.5 SNRanddynamicrangestudy.................................................... 52
3.3 MultipleSampling........................................................................................... 56
3.4 Multiple-SensingNodes................................................................................ 58
3.5 LogarithmicPixels........................................................................................... 59
3.6 LogarithmicPhotovoltaicPixel................................................................... 61
3.7 TimetoSaturation............................................................................................ 64
3.8 Gradient-BasedImage.................................................................................... 65
3.9 LighttoFrequency........................................................................................... 65
3.10 OtherMethods................................................................................................... 66
3.11 MultipleReadoutGains................................................................................. 67
3.12 Multiple-ExposureWindows....................................................................... 68
3.13 Summary............................................................................................................. 70
3.14 CompandingADCs......................................................................................... 72
3.15 Extended-Dynamic-RangeColorImaging.............................................. 74
3.16 SensorsUsedinApplications...................................................................... 76
Chapter4 SoftwareMethodstoExtendtheDynamicRange.................. 77
4.1 GeneralStructureofaSoftwareApproach............................................. 77
4.2 HighDynamicRangeImageDataMerging............................................ 78
4.2.1 Idealcase............................................................................................ 78
4.2.2 Realsensorsandcameras.............................................................. 79
4.2.3 Debevec’salgorithm....................................................................... 80
4.2.4 Alternatemethod:MannandPicard.......................................... 83
4.2.5 Alternatemethod:MitsunagaandNayar................................. 85
4.2.6 Alternatemethod:Robertsonetal............................................. 85
Contents vii
4.3 NoiseRemoval.................................................................................................. 87
4.3.1 Temporalpixelnoise...................................................................... 87
4.3.2 Ghostsandmisalignments............................................................ 87
4.4 ToneMapping.................................................................................................... 87
4.5 Software Methods Applicable to Certain Image Processing
Applications....................................................................................................... 88
4.6 SensorswithIntegratedProcessing............................................................ 90
4.7 SimulatedHighDynamicRangeImages................................................. 90
Chapter5 OpticalLimitations...................................................................... 93
5.1 LensGlare.......................................................................................................... 93
5.2 ModulationTransferFunction..................................................................... 96
5.3 Conclusions........................................................................................................ 96
Chapter6 AutomaticHighDynamicRangeControl................................. 99
6.1 AutomaticExposureofLinearSensors.................................................... 99
6.1.1 Principle.............................................................................................. 99
6.1.2 Brightnesscalculation.................................................................... 100
6.1.3 Filteringandstabilityformachinevision................................ 100
6.1.4 Filteringandstabilityfordisplay............................................... 100
6.1.5 Guard-band-basedfiltering........................................................... 100
6.2 AutomaticExposureofHighDynamicRangeSensors...................... 101
Chapter7 HighDynamicRangeFileFormats...........................................103
7.1 ColorSpace........................................................................................................ 103
7.1.1 Introduction....................................................................................... 103
7.1.2 Colorspacedefinition.................................................................... 104
7.2 StoringImageDataofExtendedDynamicRangeCameras.............. 105
7.3 Storing Data of Radiance Maps and High Dynamic Range
Software:DirectPixelEncodingMethods.............................................. 105
7.3.1 IEEEsingleprecisionfloatingpoint.......................................... 106
7.3.2 PixarTM logencoding..................................................................... 106
7.3.3 RadianceRGBE............................................................................... 106
7.3.4 SGITM LogLuvTIFF...................................................................... 106
7.3.5 IndustrialLightandMagicTM OpenEXR................................ 106
7.3.6 UnifiedColorTM BEF..................................................................... 107
7.3.7 Microsoft/HPTM scRGB................................................................ 107
7.3.8 JPEGXR............................................................................................ 107
7.3.9 Summaryoffileformats................................................................ 107
7.4 Storing Data of Radiance Maps and High Dynamic Range
Software:Gradient-BasedandFlow-BasedMethods.......................... 107
viii Contents
Chapter 8 Testing High Dynamic Range Sensors, Cameras, and
Systems.............................................................................................................109
8.1 TestingofSoftware-BasedSystems........................................................... 109
8.2 Testing of Non-High Dynamic Range (Linear) Sensors and
Cameras............................................................................................................... 109
8.2.1 TheISOapproach............................................................................ 109
8.2.2 TheEMVA1288approach............................................................ 110
8.3 Testing of High Dynamic Range Sensors and High Dynamic
RangeSensor-BasedCameras..................................................................... 111
8.3.1 TheISOapproach............................................................................ 112
8.3.2 TheEMVA1288approach............................................................ 112
Chapter9 Conclusions.................................................................................113
9.1 ImportantFiguresofMeritofaHighDynamicRangesensor......... 113
9.2 Questions............................................................................................................. 114
Bibliography........................................................................................................115
Index....................................................................................................................123
Acknowledgments
This book uses several images, algorithms, and document reproductions from
various authors. We thank all those authors for granting us the right to reuse their
material.Moredetailsabouttheownershipoftheimagescanbefoundinthefigure
captions.
Special thanks to Martin Wäny, Bart Dierickx, Dirk Hertel, John McCann, and
UweArtmannfortheirhelp,contributions,ormanuscriptreviews.Also,aspecial
thankyoutoallofmyemployees,formeremployees,familymembers,andfriends
whohelpedwritethistextandgatheralloftherequireddocumentation.
ix