Smart Grid using Big Data Analytics. A Random Matrix Theory Approach

607 Pages·2017·20.812 MB·English

by  Robert C. Qiu, Paul Antonik

SmartGridusingBigDataAnalytics:ARandomMatrixTheoryApproach
Smart Grid using Big Data Analytics
ARandomMatrixTheoryApproach
RobertC.QiuandPaulAntonik
Thiseditionfirstpublished2017
©2017JohnWiley&SonsLtd
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,or
transmitted,inanyformorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,
exceptaspermittedbylaw.Adviceonhowtoobtainpermisiontoreusematerialfromthistitleisavailableat
http://www.wiley.com/go/permissions.
TherightofRobertC.QiuandPaulAntoniktobeidentifiedastheauthorsofthisworkhasbeenassertedin
accordancewithlaw.
RegisteredOffices
JohnWiley&Sons,Inc.,111RiverStreet,Hoboken,NJ07030,USA
JohnWiley&SonsLtd,TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UK
EditorialOffice
TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UK
Fordetailsofourglobaleditorialoffices,customerservices,andmoreinformationaboutWileyproducts
visitusatwww.wiley.com.
Wileyalsopublishesitsbooksinavarietyofelectronicformatsandbyprint-on-demand.Somecontentthat
appearsinstandardprintversionsofthisbookmaynotbeavailableinotherformats.
LimitofLiability/DisclaimerofWarranty
Thepublisherandtheauthorsmakenorepresentationsorwarrantieswithrespecttotheaccuracyor
completenessofthecontentsofthisworkandspecificallydisclaimallwarranties,includingwithout
limitationanyimpliedwarrantiesoffitnessforaparticularpurpose.Thisworkissoldwiththe
understandingthatthepublisherisnotengagedinrenderingprofessionalservices.Theadviceandstrategies
containedhereinmaynotbesuitableforeverysituation.Inviewofongoingresearch,equipment
modifications,changesingovernmentalregulations,andtheconstantflowofinformationrelatingtotheuse
ofexperimentalreagents,equipment,anddevices,thereaderisurgedtoreviewandevaluatethe
informationprovidedinthepackageinsertorinstructionsforeachchemical,pieceofequipment,reagent,or
devicefor,amongotherthings,anychangesintheinstructionsorindicationofusageandforadded
warningsandprecautions.Thefactthatanorganizationorwebsiteisreferredtointhisworkasacitation
and/orpotentialsourceoffurtherinformationdoesnotmeanthattheauthororthepublisherendorsesthe
informationtheorganizationorwebsitemayprovideorrecommendationsitmaymake.Further,readers
shouldbeawarethatwebsiteslistedinthisworkmayhavechangedordisappearedbetweenwhenthisworks
waswrittenandwhenitisread.Nowarrantymaybecreatedorextendedbyanypromotionalstatementsfor
thiswork.Neitherthepublishernortheauthorshallbeliableforanydamagesarisingherefrom.
LibraryofCongressCataloging-in-PublicationData
Names:Qiu,RobertC.,1966–author.|Antonik,Paul,author.
Title:Smartgridusingbigdataanalytics/RobertC.Qiu,PaulAntonik.
Description:Chichester,WestSussex,UnitedKingdom:JohnWiley&Sons,Inc.,2017.|
Includesbibliographicalreferencesandindex.
Identifiers:LCCN2016042795|ISBN9781118494059(cloth)|ISBN9781118716793(epub)|
ISBN9781118716809(AdobePDF)
Subjects:LCSH:Smartpowergrids.|Bigdata.
Classification:LCCTK3105.Q252017|DDC621.310285/57–dc23
LCrecordavailableathttps://lccn.loc.gov/2016042795
CoverdesignbyWiley
Coverimage:johnason/loops7/ziggymaj/gettyimages
Setin10/12ptWarnockbySPiGlobal,Pondicherry,India
10 9 8 7 6 5 4 3 2 1
ToLilyL.Li
vii
Contents
Preface xv
Acknowledgments xix
SomeNotation xxi
1 Introduction 1
1.1 BigData:BasicConcepts 1
1.1.1 BigData—BigPicture 1
1.1.2 DARPA’sXDATAProgram 3
1.1.3 NationalScienceFoundation 5
1.1.4 ChallengesandOpportunitieswithBigData 5
1.1.5 SignalProcessingandSystemsEngineeringforBigData 6
1.1.6 LargeRandomMatricesforBigData 8
1.1.7 BigDataAcrosstheUSFederalGovernment 8
1.2 DataMiningwithBigData 9
1.3 AMathematicalIntroductiontoBigData 13
1.4 AMathematicalTheoryofBigData 28
1.4.1 BoltzmannEntropyandH-Theorem 30
1.4.2 ShannonEntropyandClassicalInformationTheory 31
1.4.3 Dan-VirgilVoiculescuandFreeCentralLimitTheorem 31
1.4.4 FreeEntropy 31
1.4.5 JeanGinibreandhisEnsembleofNon-HermitianRandomMatrices 32
1.4.6 CircularLawfortheComplexGinibreEnsemble 33
1.5 SmartGrid 34
1.6 BigDataandSmartGrid 36
1.7 ReadingGuide 37
BibliographicalRemarks 39
PartI FundamentalsofBigData 41
2 TheMathematicalFoundationsofBigDataSystems 43
2.1 BigDataAnalytics 44
2.2 BigData:Sense,Collect,Store,andAnalyze 45
2.2.1 DataCollection 46
viii Contents
2.2.2 DataCleansing 46
2.2.3 DataRepresentationandModeling 47
2.2.4 DataAnalysis 47
2.2.5 DataStorage 48
2.3 IntelligentAlgorithms 48
2.4 SignalProcessingforSmartGrid 48
2.5 MonitoringandOptimizationforPowerGrids 48
2.6 DistributedSensingandMeasurementforPowerGrids 49
2.7 Real-timeAnalysisofStreamingData 50
2.8 SalientFeaturesofBigData 51
2.8.1 SingularValueDecompositionandRandomMatrixTheory 51
2.8.2 Heterogeneity 52
2.8.3 NoiseAccumulation 53
2.8.4 SpuriousCorrelation 53
2.8.5 IncidentalEndogeneity 54
2.8.6 ImpactonComputationalMethods 54
2.9 BigDataforQuantumSystems 54
2.10 BigDataforFinancialSystems 55
2.10.1 Methodology 55
2.10.2 Marchenko–PasturLawforEqualTimeCorrelations 58
2.10.3 SymmetrizedTime-LaggedCorrelationMatrices 59
2.10.4 AsymmetricTime-LaggedCorrelationMatrices 61
2.10.5 NoiseReduction 62
2.10.6 Power-LawTails 63
2.10.7 FreeRandomVariables 65
2.10.8 Cross-CorrelationsbetweenInputandOutputVariables 70
2.11 BigDataforAtmosphericSystems 73
2.12 BigDataforSensingNetworks 74
2.13 BigDataforWirelessNetworks 75
2.13.1 Marchenko–PasturLaw 75
2.13.2 TheSingle“Ring”Law 76
2.13.3 ExperimentalResults 76
2.14 BigDataforTransportation 78
BibliographicalRemarks 78
3 LargeRandomMatrices:AnIntroduction 79
3.1 ModelingofLargeDimensionalDataasRandomMatrices 79
3.2 ABriefofRandomMatrixTheory 81
3.3 ChangePointofViews:FromVectorstoMeasures 85
3.4 TheStieltjesTransformofMeasures 86
3.5 AFundamentalResult:TheMarchenko–PasturEquation 88
3.6 LinearEigenvalueStatisticsandLimitLaws 89
3.7 CentralLimitTheoremforLinearEigenvalueStatistics 99
3.8 CentralLimitTheoremforRandomMatrixS−1T 101
3.9 IndependenceforRandomMatrices 103
3.10 Matrix-ValuedGaussianDistribution 110
Contents ix
3.11 Matrix-ValuedWishartDistribution 112
3.12 MomentMethod 112
3.13 StieltjesTransformMethod 113
3.14 ConcentrationoftheSpectralMeasureforLargeRandomMatrices 114
3.15 FutureDirections 117
BibliographicalRemarks 117
4 LinearSpectralStatisticsoftheSampleCovarianceMatrix 121
4.1 LinearSpectralStatistics 121
4.2 GeneralizedMarchenko–PasturDistributions 122
4.2.1 CentralLimitTheorem 123
4.2.2 SpikedPopulationModels 126
4.2.3 GeneralizedSpikedPopulationModel 126
4.3 EstimationofSpectralDensityFunctions 127
4.3.1 EstimationMethod 128
4.3.2 KernelEstimatoroftheLimitingSpectralDistribution 130
4.3.3 CentralLimitTheoremsforKernelEstimators 140
4.3.4 EstimationofNoiseVariance 143
4.4 LimitingSpectralDistributionofTimeSeries 146
4.4.1 VectorAutoregressiveMovingAverage(VARMA)Models 146
4.4.2 GeneralLinearProcess 147
4.4.3 LargeSampleCovarianceMatricesforLinearProcesses 149
4.4.4 StationaryProcesses 149
4.4.5 SymmetrizedAuto-crossCovarianceMatrix 151
4.4.6 LargeSampleCovarianceMatriceswithHeavyTails 152
BibliographicalRemarks 154
5 LargeHermitianRandomMatricesandFreeRandomVariables 155
5.1 LargeEconomic/FinancialSystems 156
5.2 Matrix-ValuedProbability 157
5.2.1 EigenvalueSpectrafortheCovarianceMatrixanditsEstimator 159
5.3 Wishart-LevyFreeStableRandomMatrices 166
5.4 BasicConceptsforFreeRandomVariables 168
5.5 TheAnalyticalSpectrumoftheWishart–LevyRandomMatrix 172
5.6 BasicPropertiesoftheStieltjesTransform 176
5.7 BasicTheoremsfortheStieltjesTransform 179
5.8 FreeProbabilityforHermitianRandomMatrices 185
5.8.1 RandomMatrixTheory 185
5.8.2 FreeProbabilityTheoryforHermitianRandomMatrices 187
5.8.3 AdditiveFreeConvolution 188
5.8.4 CompressionofRandomMatrix 192
5.8.5 MultiplicativeFreeConvolution 193
5.9 RandomVandermondeMatrix 196
5.10 Non-AsymptoticAnalysisofStateEstimation 200
BibliographicalRemarks 201
x Contents
6 LargeNon-HermitianRandomMatricesandQuatartenionicFree
ProbabilityTheory 203
6.1 QuatartenionicFreeProbabilityTheory 204
6.1.1 StieltjesTransform 205
6.1.2 AdditiveFreeConvolution 206
6.1.3 MultiplicativeFreeConvolution 207
6.1.4 Quaternion-valuedFunctionsforHermitianMatrices 207
6.2 R-diagonalMatrices 209
6.2.1 ClassesofR-diagonalMatrices 209
6.2.2 AdditiveFreeConvolution 210
6.2.3 MultiplicativeFreeConvolution 211
6.2.4 IsotropicRandomMatrices 215
6.3 TheSumofNon-HermitianRandomMatrices 216
6.4 TheProductofNon-HermitianRandomMatrices 220
6.5 SingularValueEquivalentModels 226
6.6 ThePoweroftheNon-HermitianRandomMatrix 234
6.6.1 TheMatrixPower 234
6.6.2 Spectrum 234
6.6.3 TheProduct 236
6.7 PowerSeriesofLargeNon-HermitianRandomMatrices 239
6.7.1 TheGeometricSeries 240
6.7.2 PowerSeries 241
6.8 ProductsofRandomGinibreMatrices 246
6.9 ProductsofRectangularGaussianRandomMatrices 249
6.10 ProductofComplexWishartMatrices 252
6.11 SpectralRelationsbetweenProductsandPowers 254
6.12 ProductsofFinite-SizeI.I.D.GaussianRandomMatrices 258
6.13 LyapunovExponentsforProductsofComplexGaussianRandom
Matrices 260
6.14 EuclideanRandomMatrices 264
6.15 RandomMatriceswithIndependentEntriesandtheCircularLaw 273
6.16 TheCircularLawandOutliers 275
6.17 RandomSVD,SingleRingLaw,andOutliers 285
6.17.1 OutliersforFiniteRankPerturbation:ProofofTheorem6.17.3 292
6.17.2 EigenvaluesInsidetheInnerCircle:ProofofTheorem6.17.4 294
6.18 TheEllipticLawandOutliers 295
BibliographicalRemarks 305
7 TheMathematicalFoundationsofDataCollection 307
7.1 ArchitecturesandApplicationsforBigData 307
7.2 CovarianceMatrixEstimation 308
7.3 SpectralEstimatorsforLargeRandomMatrices 312
7.3.1 SingularValueThresholding 313
7.3.2 Stein’sUnbiasedRiskEstimate(SURE) 314
7.3.3 ExtensionstoSpectralFunctions 316
7.3.4 RegularizedPrincipalComponentAnalysis 318
7.4 AsymptoticFrameworkforMatrixReconstruction 319
Contents xi
7.4.1 MatrixEstimationwithLossFunctions 319
7.4.2 ConnectionwithLargeRandomMatrices 322
7.4.3 AsymptoticMatrixReconstruction 324
7.4.4 EstimationoftheNoiseVariance 325
7.4.5 OptimalHardThresholdforMatrixDenoising 327
7.5 OptimumShrinkage 329
7.6 AShrinkageApproachtoLarge-ScaleCovarianceMatrixEstimation 331
7.7 EigenvectorsofLargeSampleCovarianceMatrixEnsembles 338
7.7.1 StieltjesTransform 338
7.7.2 SampleversusPopulationEigenvectors 341
7.7.3 AsymptoticallyOptimalBiasCorrectionfortheSampleEigenvalues 343
7.7.4 EstimatingPrecisionMatrices 346
7.8 AGeneralClassofRandomMatrices 351
7.8.1 MassiveMIMOSystem 355
BibliographicalRemarks 359
8 MatrixHypothesisTestingusingLargeRandomMatrices 361
8.1 MotivatingExamples 362
8.2 HypothesisTestofTwoAlternativeRandomMatrices 363
8.3 EigenvalueBoundsforExpectationandVariance 364
8.3.1 TheoreticalLocationsofEigenvalues 366
8.3.2 WassersteinDistance 366
8.3.3 SampleCovarianceMatrices—EntrieswithExponentialDecay 367
8.3.4 GaussianCovarianceMatrices 368
8.4 ConcentrationofEmpiricalDistributionFunctions 369
8.4.1 Poincare-TypeInequalities,Tensorization 372
8.4.2 EmpiricalPoincare-TypeInequalities 373
8.4.3 ConcentrationofRandomMatrices 377
8.5 RandomQuadraticForms 381
8.6 Log-DeterminantofRandomMatrices 382
8.7 GeneralMANOVAMatrices 383
8.8 FiniteRankPerturbationsofLargeRandomMatrices 386
8.8.1 Non-asymptoic,Finite-SampleTheory 390
8.9 HypothesisTestsforHigh-DimensionalDatasets 391
8.9.1 MotivationforLikelihoodRatioTest(LRT)andCovarianceMatrix
Tests 392
8.9.2 EstimationofCovarianceMatricesUsingLossFunctions 394
8.9.3 CovarianceMatrixTests 399
8.9.4 OptimalHypothesisTestingforHigh-DimensionalCovariance
Matrices 404
8.9.5 SphericityTest 408
8.9.6 TestingEqualityofMultipleCovarianceMatricesofNormal
Distributions 410
8.9.7 TestingIndependenceofComponentsofNormalDistribution 413
8.9.8 TestofMutualDependence 416
8.9.9 TestofPresenceofSpikeEigenvalues 420
xii Contents
8.9.10 LargeDimensionandSmallSampleSize 422
8.10 Roy’sLargestRootTest 428
8.11 OptimalTestsofHypothesesforLargeRandomMatrices 431
8.12 MatrixEllipticallyContouredDistributions 444
8.13 HypothesisTestingforMatrixEllipticallyContouredDistributions 446
8.13.1 GeneralResults 446
8.13.2 TwoModels 448
8.13.3 TestingCriteria 450
BibliographicalRemarks 452
PartII SmartGrid 455
9 ApplicationsandRequirementsofSmartGrid 457
9.1 History 457
9.2 ConceptsandVision 458
9.3 Today’sElectricGrid 459
9.4 FutureSmartElectricalEnergySystem 464
10 TechnicalChallengesforSmartGrid 471
10.1 TheConceptualFoundationofaSelf-HealingPowerSystem 471
10.2 HowtoMakeanElectricPowerTransmissionSystemSmart 472
10.3 TheElectricPowerSystemasaComplexAdaptiveSystem 473
10.4 MakingthePowerSystemaSelf-HealingNetworkUsingDistributed
ComputerAgents 474
10.5 DistributionGrid 474
10.6 CyberSecurity 476
10.7 SmartMeteringNetwork 477
10.8 CommunicationInfrastructureforSmartGrid 478
10.9 WirelessSensorNetworks 480
BibliographicalRemarks 483
11 BigDataforSmartGrid 485
11.1 PowerinNumbers:BigDataandGridInfrastructure 485
11.2 Energy’sInternet:TheConvergenceofBigDataandtheCloud 486
11.3 EdgeAnalytics:Consumers,ElectricVehicles,andDistributed
Generation 486
11.4 CrosscuttingThemes:BigData 486
11.5 CloudComputingforSmartGrid 488
11.6 DataStorage,DataAccessandDataAnalysis 488
11.7 TheState-of-the-ArtProcessingTechniquesofBigData 488
11.8 BigDataMeetstheSmartElectricalGrid 488
11.9 4VsofBigData:Volume,Variety,ValueandVelocity 489
11.10 CloudComputingforBigData 490