Earth Observation Using Python: A Practical Programming Guide (Special Publications)

287 Pages·2021·10.697 MB·English

by  Rebekah B. Esmaili

Earth Observation Using Python
Special Publications 75
EARTH OBSERVATION
USING PYTHON
A Practical Programming Guide
Rebekah B. Esmaili
This Work is aco-publication of
the American Geophysical Union and John Wiley and Sons, Inc.
Thiseditionfirstpublished2021
©2021AmericanGeophysicalUnion
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,
inanyformorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,exceptas
permittedbylaw.Adviceonhowtoobtainpermissiontoreusematerialfromthistitleisavailableat
http://www.wiley.com/go/permissions.
PublishedundertheaegisoftheAGUPublicationsCommittee
BrooksHanson,ExecutiveVicePresident,Science
CarolFrost,Chair,PublicationsCommittee
FordetailsabouttheAmericanGeophysicalUnionvisitusatwww.agu.org.
TherightofRebekahB.Esmailitobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewithlaw.
RegisteredOffice
JohnWiley&Sons,Inc.,111RiverStreet,Hoboken,NJ07030,USA
EditorialOffice
111RiverStreet,Hoboken,NJ07030,USA
Fordetailsofourglobaleditorialoffices,customerservices,andmoreinformationaboutWileyproductsvisitusat
www.wiley.com.
Wileyalsopublishesitsbooksinavarietyofelectronicformatsandbyprint-on-demand.Somecontentthatappears
instandardprintversionsofthisbookmaynotbeavailableinotherformats.
LimitofLiability/DisclaimerofWarranty
Whilethepublisherandauthorshaveusedtheirbesteffortsinpreparingthiswork,theymakenorepresentationsor
warrantieswithrespecttotheaccuracyorcompletenessofthecontentsofthisworkandspecificallydisclaimall
warranties,includingwithoutlimitationanyimpliedwarrantiesofmerchantabilityorfitnessforaparticularpurpose.
Nowarrantymaybecreatedorextendedbysalesrepresentatives,writtensalesmaterialsorpromotionalstatements
forthiswork.Thefactthatanorganization,website,orproductisreferredtointhisworkasacitationand/or
potentialsourceoffurtherinformationdoesnotmeanthatthepublisherandauthorsendorsetheinformationor
servicestheorganization,website,orproductmayprovideorrecommendationsitmaymake.Thisworkissoldwith
theunderstandingthatthepublisherisnotengagedinrenderingprofessionalservices.Theadviceandstrategies
containedhereinmaynotbesuitableforyoursituation.Youshouldconsultwithaspecialistwhereappropriate.
Further,readersshouldbeawarethatwebsiteslistedinthisworkmayhavechangedordisappearedbetweenwhenthis
workwaswrittenandwhenitisread.Neitherthepublishernorauthorsshallbeliableforanylossofprofitorany
othercommercialdamages,includingbutnotlimitedtospecial,incidental,consequential,orotherdamages.
LibraryofCongressCataloging-in-PublicationData
Name:Esmaili,RebekahBradley,author.
Title:EarthobservationusingPython:apracticalprogrammingguide/
RebekahB.Esmaili.
Description:Hoboken,NJ:Wiley,[2021]|Includesbibliographical
referencesandindex.
Identifiers:LCCN2021001631(print)|LCCN2021001632(ebook)|ISBN
9781119606888(hardback)|ISBN9781119606895(adobepdf)|ISBN
9781119606918(epub)
Subjects:LCSH:Earthsciences–Dataprocessing.|Remotesensing–Data
processing.|Python(Computerprogramlanguage)|Information
visualization.|Artificialsatellitesinearthsciences.|Earth
sciences–Methodology.
Classification:LCCQE48.8.E852021 (print)|LCCQE48.8 (ebook)|DDC
550.285/5133–dc23
LCrecordavailableathttps://lccn.loc.gov/2021001631
LCebookrecordavailableathttps://lccn.loc.gov/2021001632
CoverDesign:Wiley
CoverImage:©NASA
Setin10/12ptTimesNewRomanbyStraive,Pondicherry,India
10 9 8 7 6 5 4 3 2 1
CONTENTS
Foreword.........................................................................................................vii
Acknowledgments............................................................................................ix
Introduction......................................................................................................1
Part I: Overview of Satellite Datasets...............................................................5
1 ATour ofCurrentSatellite Missions and Products..................................7
2 Overview of Python................................................................................17
3 ADeep Dive into Scientific Data Sets....................................................25
PartII: PracticalPythonTutorials forRemoteSensing...................................45
4 Practical Python Syntax..........................................................................47
5 Importing Standard Earth Science Datasets...........................................67
6 Plotting and Graphs forAll.....................................................................95
7 Creating Effective and Functional Maps...............................................125
8 Gridding Operations.............................................................................155
9 Meaningful Visuals through Data Combination...................................177
10 Exporting with Ease..............................................................................207
PartIII: Effective Coding Practices...............................................................219
11 Developing aWorkflow.......................................................................221
12 Reproducible and Shareable Science...................................................239
Conclusion.....................................................................................................253
Appendix A: Installing Python.......................................................................255
Appendix B:Jupyter Notebook.....................................................................259
Appendix C: AdditionalLearningResources.................................................267
Appendix D:Tools.........................................................................................269
Appendix E: Finding, Accessing, and Downloading Satellite Datasets.........271
Appendix F: Acronyms..................................................................................279
Index..............................................................................................................283
v
FOREWORD
When I first met theauthor afew years ago, shewaseager to become more
involved in the Joint Polar Satellite System’s Proving Ground. The Proving
Groundbydefinitionassessestheimpactofaproductintheuser’senvironment;
thisintriguedRebekahbecauseasaproductdeveloper,shewantedtounderstand
the user’s perspective. Rebekah worked with the National Weather Service to
demonstrate how satellite-derived atmospheric temperature and water vapor
soundings can be used to describe the atmosphere’s instability to support severe
weatherwarnings.RebekahspentconsiderabletimewithusersattheStormPre-
dictionCenterinNorman,Oklahoma,tounderstandtheirneeds,andshefound
theirthirstfordataandtheneedfordatatobeeasilyvisualizedandunderstand-
able.ThisiswhereRebekahleveragedherexpertskillsinPythontoprovideNWS
withtheinformationtheyfoundtobemostuseful.LittledidIknowatthetimeshe
was writing abook.
As noted in this book, a myriad of Earth-observing satellites collect critical
information of the Earth’s complex and ever-changing environment and land-
scape.However,today,unfortunately,allthatinformationisnoteffectivelybeing
usedforvarious reasons:issues withdataaccess,different dataformats,andthe
needforbettertoolsfordatafusionandvisualization.Ifwewereabletosolvethese
problems,thensuddenlytherewouldbevastimprovementsinprovidingsocieties
withtheinformation neededtosupportdecisions relatedtoweatherandclimate
andtheirimpacts,includinghigh-impactweatherevents,droughts,flooding,wild-
fires,ocean/coastalecosystems,airquality,andmore.Pythonisbecomingtheuni-
versallanguagetobridgethesevariousdatasourcesandtranslatethemintouseful
information.Openandfreeattributes,andthedataandcodesharingmindsetof
thePython communities, make Pythonvery appealing.
Being involved in a number of international collaborations to improve the
integration of Earth observations, I can certainly emphasize the importance
of working together, data sharing, and demonstrating the value of data
fusion.IamveryhonoredtowritethisForeword,sincethisbookfocusesonthese
vii
viii Foreword
issues and provides an excellent guide with relevant examples for the reader to
follow and relate to.
Dr. Mitch Goldberg
Chief Program Scientist
NOAA-National Environmental Satellite, Data,
and InformationService
June 22, 2020
ACKNOWLEDGMENTS
ThisbookevolvedfromaseriesofPythonworkshopsthatIdevelopedwith
thehelpofEviatarBachandKritiBhargavafromtheDepartmentofAtmospheric
and Oceanic Science at the University of Maryland. I am very grateful for their
assistanceprovidingfeedbackfortheexamplesinthisbookandforleadingseveral
ofthese workshops with me.
This book would not exist without their support and contributions from
others, including:
Themanyreviewerswhotookthetimetoreadversionsofthisbook,severalof
whom I have never met in person. Thanks to modern communication systems,
Iwasabletodrawfromtheirexpertise.Theirconstructivefeedbackandinsights
notonlyhelpedtoimprovethisqualityandbreadthofthebookbutalsohelpedme
hone mytechnical writing skills.
RituparnaBose,JennyLunn,LaylaHarden,andtherestoftheteamatAGU
and Wiley for keeping me informed, organized, and on track throughout this
process. They were truly apleasure to work with.
NadiaSmithandChrisBarnet,andmyothercolleaguesatScienceandTech-
nologyCorp.,whoprovidedbothfeedbackandconversationsthathelpedshape
some of theideas and content in this book.
CatherineThomas,ClareFlynn,ErinLynch,andAmyHofortheirendless
encouragement and support.
TracieandFaridEsmaili,myparents,whoencouragedmetoaimhighevenif
they were initially confused when their atmospheric scientist daughter became
interested in “snakes.”
ix
INTRODUCTION
Python is a programming language that is rapidly growing in popularity. The
numberofusersislarge,althoughdifficulttoquantify;infact,Pythoniscurrently
the most tagged language on stackoverflow.com, a coding Q&A website with
approximately 3 million questions a year. Some view this interest as hype, but
there are many reasons to join the movement. Scientists are embracing Python
because it is free, open source, easy to learn, and has thousands of add-on
packages. Many routine tasks in the Earth sciences have already been coded
and stored in off-the-shelf Python libraries. Users can download these libraries
and apply them to their research rather than simply using older, more primitive
functions.ThewidespreadadoptionofPythonmeansscientistsaremovingtoward
acommonprogramminglanguageandsetoftoolsthatwillimprovecodeshare-
ability and research reproducibility.
Amongthewealthofremotesensingdataavailable,satellitedatasetsarepar-
ticularly voluminous and tend to be stored in a variety of binary formats. Some
datasetsconformtoa“standard”structure,suchasnetCDF4.However,because
of uncoordinated efforts across different agencies and countries, such standard
formats bear their own inconsistencies in how data are handled and intended to
be displayed. To address this, many agencies and companies have developed
numerous “quick look” methods. For instance, data can be searched for and
viewed online as Jpeg images, or individual files can be displayed with free,
open-source software tools like Panoply (www.giss.nasa.gov/tools/panoply/)
and HDFView (www.hdfgroup.org/downloads/hdfview/).
Still, scientists who wish to execute more sophisticated visualization techni-
ques will have to learn to code. Coding knowledge is not the only limitation for
users.Notalldataare“analysisready,”i.e.,intheproperinputformatforvisu-
alizationtools.Assuch,manypre-processingstepsarerequiredtomakethedata
usableforscientificanalysis.Thisisparticularlyevidentfordatafusion,wheretwo
datasets with different resolutions must first be mapped to the same grid before
they are compared. Many data users are not satellite scientists or professional
EarthObservationUsingPython:APracticalProgrammingGuide,SpecialPublications75,
FirstEdition.RebekahB.Esmaili.
©2021AmericanGeophysicalUnion.Published2021byJohnWiley&Sons,Inc.
DOI:10.1002/9781119606925.introduction
1
2 EarthObservationUsingPython
programmersbutrathermembersofotherresearchandprofessionalcommunities,
thesebarrierscanbetoogreattoovercome.Eventoatechnicaluser,thenuances
canbefrustrating.Atworst,obstaclesincodinganddatavisualizationcanpoten-
tially lead todata misuse,which can tarnish the work ofan entire community.
Thepurposeofthistextistoprovideanoverviewofthecommonpreparatory
workandvisualizationtechniquesthatareappliedtoenvironmentalsatellitedata
usingthePythonlanguage.Thisbookishighlyexample-driven,andalltheexam-
ples areavailable online. The exercises are primarilybasedon hands-on tutorial
workshops that I have developed. The motivation for producing this book is to
make the contents of the workshops accessible to more Earth scientists, as very
few Python books currently available target the Earth science community.
This book is written to be a practical workbook and not a theoretical text-
book.Forexample,readerswillbeabletointeractivelyrunprewrittencodeinter-
activelyalongsidethetexttoguidethemthroughthecodeexamples.Exercisesin
eachsectionbuildononeanother,withincrementalstepsfoldedin.Readerswith
minimalcodingexperiencecanfolloweach“babystep”togetthemuptobecome
“spunup”quickly,whilemoreexperiencedcodershavetheoptionofworkingwith
thecodedirectlyandspendingmoretimeonbuildingaworkflowasdescribedin
Section III.
The exercises and solutions provided in this book use Jupyter Notebook, a
highly interactive, web-based development environment. Using Jupyter Note-
book,codecanberuninasinglelineorshortblocks,andtheresultsaregenerated
withinaninteractivedocumentedformat.Thisallowsthestudenttoviewboththe
Python commands and comments alongside the expected results. Jupyter Note-
book can also be easily converted to programs or scripts than can be executed
on Linux Machines for high-performance computing. This provides a friendly
work environment to new Python users. Students are also welcome to develop
code in any environment they wish, such asthe Spyder IDE or using iPython.
While the material builds on concepts learned in other chapters, the book
referencesthelocationofearlierdiscussionsofthematerial.Withineachchapter,
theexamplesareprogressive.Thisdesignallowsstudentstobuildontheirunder-
standing knowledge (and learn where to find answers when they need guidance)
ratherthanmemorizingsyntaxora“recipe.”Professionally,Ihaveworkedwith
manydatasetsandIhavefoundthattheskillsandstrategiesthatIapplyonsat-
ellitedataarefairlyuniversal.Theexamplesinthisbookareintendedtohelpread-
ersbecomefamiliarwithsomeofthecharacteristicquirksthattheymayencounter
whenanalyzingvarioussatellitedatasetsintheircareers.Inthisregard,students
are also strongly encouraged to submit requests for improvements in future
editions.
Likemanytechnologicaltexts,thereisariskthatthesolutionspresentedwill
becomeoutdatedasnewtoolsandtechniquesaredeveloped.Thesizableusercom-
munityalreadycontributingtoPythonimpliesitisactivelyadvancing;itisaliving
language in contrast to compiled, more slowly evolving legacy languages like