Table Of Content

May5,2017citywaterpressuredrop ByMattMcBride,October27,2017 Introduction IntheearlyeveningofFriday,May5,2017,NewOrleanseastbankcitywaterpressureexperienceda severedropduetothesimultaneousfailureoftwoofthefourwaterdistributionpumpsoperatingat thetime.Priortothepumplosses,eastbankcitywaterpressureasmeasuredattheHighLiftcontrol roomintheSewerageandWaterBoard'sCarrolltonWaterPlantwas66.5poundspersquareinch (psi).Followingthelossofthetwopumps,itdroppedto23psi.Waterpressurewaspartiallyrestored afterapproximatelytenminutes,andfullyrestoredapproximately80minuteslater. ThepressurelosswasreflectedatnumerousremoteSewerageandWaterBoardsitesacrosstheeast bank.Pressuretransducerslocatedinfiveofthesesitesrecordedpressuresbelow15psi,the state-recommendedthresholdforissuanceofaprecautionaryboilwateradvisory.Transducersat twoothersitesrecordedpressureswithin2psiofthethreshold.Analysisofthetransducerreadings showsthatdataattwoofthefivesiteswhichrecordedsub-15psipressureswaslikelyrepresentative ofwaterpressureinthesurroundingneighborhoods. NoprecautionaryboilwateradvisorywasissuedMay5,2017. AreviewoftheeventsofMay5,2017,aswellasotherlowwaterpressureeventsthrough2017, revealssystemicgapsinthemonitoringofcitywaterpressureandthealarmschemeusedtoalert personneltoalowwaterpressurecondition.Questionsalsoexistregardingthequalityanduseof databyBoarddecisionmakersduringalowwaterpressureevent. ThisreportdocumentstheeventsofMay5,2017basedupontheprimarydocumentsgeneratedthat day.Thesedocumentsincludebothpaperrecordsandelectronicarchives.Inaddition,operational logsandelectronicarchivesforallof2017fromthefollowingfacilitieswerereviewedforinsightinto possiblechallengesfacingtheeastbankwaterdistributionsystem: PowerhouseandHighLift Claibornepumpingstation Panolapumpingstation CentralControl 15drainagepumpingstations Whilenosecondarydocumentssuchasafter-actionreportswerereviewed,itisknownthatatleast onesuchreportexists. Thisreportbuildsoutward,workingfromabarechronologyandexplanationoftheMay5,2017 pressuredropeventparticulars,toadescriptionofhowsucheventsaremonitored,andthentohow theeventwasfeltacrosstheeastbank,includingadetaileddiscussionofthefivelocationswith pressurereadingsbelow15psi.Asurveyofothermajorpressuredropeventsin2017wascompiled forcontext.Finally,conclusionsandrecommendationsarepresented. Pleasenotethatforthepurposesofthisreport,localtimesfromlogsareusedsparinglyandoften withqualifiers.Duetothemultiplicityofclocksatdifferentlocationsacrossthesystemandtheirlack ofsynchronization,eventtimesdonotmatchupacrossvariouslogs.Durationsofeventsand magnitudesofpressuresareconsideredmoretrustworthythanindividualtimestamps. 1 Waterdistributionbasics TheSewerageandWaterBoardoperateseightwaterdistributionpumpsattheCarrolltonWater PlanttoprovidepotablewatertotheeastbankoftheCityofNewOrleans.Thepumpsarelocatedin threedifferentfacilities: Powerhouse:pumpsAandB ClaiborneStation:Claibornepumps1through4 PanolaStation:Panolapumps1and2 ThesixpumpsatClaiborneandPanolaStationsareelectricallydriven.AtClaiborneStation,pumps1 and4runon25cyclepower,whilepumps2and3runon60cyclepower.EachpumpatPanola stationhastwomotors,one25cycleandone60cycle.PumpsAandBatthepowerhousearedriven notbyelectricmotors,butbysteamturbines. PumpsAandBareprimarypumps;in2017,BpumphasrunatleastpartofeverydayandApump hasrunduringeverydaybutone.ClaiborneStationpumpsare"swing"pumps,switchingonandoff withdemandorforotheroperationalormaintenancereasons.ThepumpsatPanolaStationwere traditionallyconsideredbackuppumps,andwerehardlyusedduringthefirsthalfof2017.However, powerandpumpingproblemsinthesecondhalfoftheyearhaveforcedthePanolapumpsinto muchheavierservice. OntheafternoonofMay5,2017,fourpumpswereprovidingcitywaterpressureontheeastbank: pumpsAandBinthepowerhouse,andpumps3and4atClaiborneStation. Chronology:May5,2017 Around5:40PMonMay5,2017,bothwaterdistributionpumpsAandBattheSewerageandWater Boardpowerhouseshutdownunexpectedly.Withthelossofthepumps,citywaterpressurefell from66psitoapproximately23psiasmeasuredatHighLiftontheCarrolltonWaterPlant.City waterpressureisrecorded,amongotherplaces,onachartrecorderintheHighLiftcontrolroom: 2 Thatchartrecordermemorializedthewaterpressuredropinink: HereisanenlargementoftherelevantsectionofpentracefromtheMay5chart: 3 ThelosswasalsonotedonthewrittendatalogsforAandBpumps.Thedatalogsareonadouble sidedsheetofpaper,withApumpononesideandBpumpontheother.ThenotewasontheA pumpside: 4 Soonafterthepressuredrop,theHighLiftoperatorspokewiththeClaiborneStationoperator.Atthe timeofthelossofpumpsAandB,pumps3and4atClaiborneStationwerealsorunningand pumping.Thisstatusisreferredtoas"in."Sincetwopumpsalonearenotenoughtomaintain adequatecitywaterpressure,pump1atClaiborneStationwasput"in"at5:46PM,accordingtothe ClaiborneStationclock.ThisisshowninthewrittenlogbookkeptinClaiborneStation: TheresponsetothepressuredropwasalsocapturedontheClaiborneStationwrittendatalogs.Like theHighLiftpumpA&Bwrittendatalogs,theClaiborneStationlogsarealsotwosided.Pumps2 and4areononeside,whilepumps1and3areontheother: 5 PuttingClaibornepump#1"in"raisedcitywaterpressuretoapproximately51psiasmeasuredat HighLift.Normalcitywaterpressureofapproximately66psiwasrestoredaboutanhourlaterwhen AandBpumpswereputback"in."AccordingtotheHighLiftwrittenlogbook,pumpBwas"in"at 6:46PMandpumpAwas"in"at6:53PM: CentralControlnotedtheseeventsintheirlog,withslightlydifferenttimes: 6 BelowisanenlargementoftherelevantsectionoftheCentralControllog: 7 Forclarity,therelevantentriesareduplicatedhere,withabbreviationsexpandedintofull descriptions: 5:40PM:BoilerRoom(operatorCollins)notifiedCentralControloflowwaterpressure:24psi 5:42PM:HighLift(operatorShellage)notifiedCentralControl[that]HighLiftlostAandBpumps. Started#1pumpatClaiborne. 5:43PM:CentralControlnotifiedEricLabatthewaterpressureatHighLiftis24psi. 5:48PM:Claibornewentinwith#1(25Hz)pump.Waterpressureis32psi,4500kW[load]on#1 turbine. 5:51PM:HighLift(operatorShellage)reportswaterpressureis47psi. 5:57PM:HighLiftreportswaterpressureis51psi. 6:11PM:DamonAdamsstartedBpumpinpumproom. 6:16PM:DamonAdamsstartedApumpinpumproom 6:17PM:DamonAdamsnotifiedCentralControl[thatHighLift]lostACandDCpowertothe controllersforA&Bpumps. 6:34PM:DamonAdamsnotifiedCentralControlA&Bpump[s]and#1pumpatClaibornebackto normal. Basedontheselogsandmanyothers,itappearseastbankcitywaterpressureremainedatitslowest levelsforapproximately8to10minutes. FailureofpumpsAandBexplained ThecabinethousingtheWoodwardmodel505turbinecontrollersforpumpsA&Bandtheir commonhuman-machineinterface(HMI)islocatedintheA&Bpumproom: 8 Accordingtointerviewswithseniorpowerhousestaff,thecabinetispoweredoffa60cycle alternatingcurrentfeedwithbackupdirectcurrentpoweroffthepowerhousebatteries. Unbeknownsttostaff,atsomepointpriortoMay5,2017oneofthosepowersourceshadfailed. Accordingtostaff,thepanelneveralarmedonthisconditionbecauseofaprogrammingerror.When thesecondpowersourcetothecabinetfailed,ittookdowneverythingcontrolledbythecabinet: bothpumpsAandB.Thisiswhatthe6:17PMnoteintheCentralControllogswasreferringto: "6:17PM:DamonAdamsnotifiedCentralControl[thatHighLift]lostACandDCpowertothe controllersforA&Bpumps." Thebatterylogsforthepowerhouseshowadropinvoltageandaspikeincurrentat5:02PMonMay 5,2017,justpriortothelossofpowertothecontroller: 9 Thevoltageonbatterybankonedroppedfromitsnormal136voltsat4PMto114voltsat5PMand thenrosebacktonormalthroughtheevening.Currentat4PMisshownasanormal1.2amps,butat 5PMisshownas"pegged"andisasubstantiallyelevated9ampstherestoftheevening.Anoteat thebottomofthelogwitha5:02PMtimestampsays,"Bankonetoolow,notifiedSamandDamon." SamandDamonareSamLewisandDamonAdams,seniorsupervisorypersonnelinthepowerhouse. Inreviewingallof2017'sbatterylogs,thiswastheonlysubstantialnoteonanyofthem.Inthe absenceofproperalarms,thissuddendrainonthepowerhousebatterieswaslikelythefirst indicationoftroubleonpumpsA&B. Thefewminuteslater,thedwindlingbatteryvoltagewasnotedagainintheHighLiftwritten logbook: Accordingtoseniorpowerhousestaff,thepowersupplyprogrammingerrorwasdiscoveredlaterand rectified;powerlossestothecontrolpanelnowalarmproperly.However,thealarmsignalisstill confinedtothelocalcontrolpanelinthepumproom.Thereisnoconnectivitybetweenthepump 10