Table Of Content

Cluster Management Guide ImportantNotice ©2010-2021Cloudera,Inc.Allrightsreserved. Cloudera,theClouderalogo,andanyotherproductor servicenamesorsloganscontainedinthisdocumentaretrademarksofClouderaand itssuppliersorlicensors,andmaynotbecopied,imitatedorused,inwholeorinpart, withoutthepriorwrittenpermissionofClouderaortheapplicabletrademarkholder.If thisdocumentationincludescode,includingbutnotlimitedto,codeexamples,Cloudera makesthisavailabletoyouunderthetermsoftheApacheLicense,Version2.0,including anyrequirednotices.AcopyoftheApacheLicenseVersion2.0,includinganynotices, isincludedherein.AcopyoftheApacheLicenseVersion2.0canalsobefoundhere: https://opensource.org/licenses/Apache-2.0 HadoopandtheHadoopelephantlogoaretrademarksoftheApacheSoftware Foundation.Allothertrademarks,registeredtrademarks,productnamesandcompany namesorlogosmentionedinthisdocumentarethepropertyoftheirrespectiveowners. Referencetoanyproducts,services,processesorotherinformation,bytradename, trademark,manufacturer,supplierorotherwisedoesnotconstituteorimply endorsement,sponsorshiporrecommendationthereofbyus. Complyingwithallapplicablecopyrightlawsistheresponsibilityoftheuser.Without limitingtherightsundercopyright,nopartofthisdocumentmaybereproduced,stored inorintroducedintoaretrievalsystem,ortransmittedinanyformorbyanymeans (electronic,mechanical,photocopying,recording,orotherwise),orforanypurpose, withouttheexpresswrittenpermissionofCloudera. Clouderamayhavepatents,patentapplications,trademarks,copyrights,orother intellectualpropertyrightscoveringsubjectmatterinthisdocument.Exceptasexpressly providedinanywrittenlicenseagreementfromCloudera,thefurnishingofthisdocument doesnotgiveyouanylicensetothesepatents,trademarkscopyrights,orother intellectualproperty.ForinformationaboutpatentscoveringClouderaproducts,see http://tiny.cloudera.com/patents. Theinformationinthisdocumentissubjecttochangewithoutnotice.Clouderashall notbeliableforanydamagesresultingfromtechnicalerrorsoromissionswhichmay bepresentinthisdocument,orfromuseofthisdocument. Cloudera,Inc. 395PageMillRoad PaloAlto,CA94306 [email protected] US:1-888-789-1488 Intl:1-650-362-0488 www.cloudera.com ReleaseInformation Version:ClouderaEnterprise6.3.x Date:September30,2021 Table of Contents About Cloudera Administration..............................................................................14 Cloudera Manager.................................................................................................15 Learn about Cloudera Manager ........................................................................................................................17 Installing and Upgrading....................................................................................................................................17 Managing CDH using Cloudera Manager...........................................................................................................17 MonitoringCDH using Cloudera Manager.........................................................................................................17 ManagingCDHusingtheClouderaManagerAPI...............................................................................................17 Cloudera Manager Admin Console....................................................................................................................18 StartingandLoggingintotheAdminConsole......................................................................................................................20 ClouderaManagerAdminConsoleHomePage...................................................................................................................20 DisplayingClouderaManagerDocumentation....................................................................................................................25 Automatic Logout................................................................................................................................................................25 ClouderaManagerFrequentlyAskedQuestions................................................................................................26 GeneralQuestions................................................................................................................................................................26 Cloudera Manager API.......................................................................................................................................28 BackingUpandRestoringtheClouderaManagerConfiguration.......................................................................................30 UsingtheClouderaManagerAPIforClusterAutomation...................................................................................................31 ClouderaManagerAdministration.....................................................................................................................33 Starting,Stopping,andRestartingtheClouderaManagerServer.......................................................................................33 ConfiguringClouderaManagerServerPorts.......................................................................................................................33 MovingtheClouderaManagerServertoaNewHost.........................................................................................................34 MigratingfromtheClouderaManagerEmbeddedPostgreSQLDatabaseServertoanExternalPostgreSQLDatabase .......................................................................................................................................................................................35 MigratingfromtheClouderaManagerExternalPostgreSQLDatabaseServertoaMySQL/OracleDatabaseServer........41 ManagingtheClouderaManagerServerLog......................................................................................................................44 ClouderaManagerAgents...................................................................................................................................................44 ConfiguringNetworkSettings..............................................................................................................................................50 Managing Licenses..............................................................................................................................................................50 SendingUsageandDiagnosticDatatoCloudera................................................................................................................57 ExportingandImportingClouderaManagerConfiguration................................................................................................61 BackingUpClouderaManager............................................................................................................................................61 OtherClouderaManagerTasksandSettings.......................................................................................................................66 ClouderaManagementService............................................................................................................................................67 ExtendingClouderaManager.............................................................................................................................72 Cluster Configuration Overview..............................................................................73 ModifyingConfigurationPropertiesUsingClouderaManager..........................................................................74 ChangingtheConfigurationofaServiceorRoleInstance...................................................................................................74 RestartingServicesandInstancesafterConfigurationChanges..........................................................................................78 SuppressingConfigurationandParameterValidationWarnings........................................................................................78 Autoconfiguration..............................................................................................................................................79 Autoconfiguration................................................................................................................................................................80 Role-HostPlacement............................................................................................................................................................87 Custom Configuration........................................................................................................................................88 Stale Configurations...........................................................................................................................................91 Client Configuration Files...................................................................................................................................93 HowClientConfigurationsareDeployed.............................................................................................................................93 DownloadingClientConfigurationFiles...............................................................................................................................94 ManuallyRedeployingClientConfigurationFiles................................................................................................................94 ViewingandRevertingConfigurationChanges..................................................................................................94 ViewingConfigurationChanges...........................................................................................................................................94 Reverting Configuration Changes........................................................................................................................................95 ExportingandImportingClouderaManagerConfiguration...............................................................................95 ClouderaManagerConfigurationPropertiesReference....................................................................................96 Managing Clusters..................................................................................................97 Adding and Deleting Clusters.............................................................................................................................97 AddingaClusterUsingNewHosts.......................................................................................................................................97 AddingaClusterUsingCurrentlyManagedHosts.............................................................................................................101 DeletingaCluster...............................................................................................................................................................104 Starting,Stopping,Refreshing,andRestartingaCluster..................................................................................104 PausingaClusterinAWS..................................................................................................................................106 ShuttingDownandStartingUptheCluster.......................................................................................................................106 Considerations after Restart..............................................................................................................................................107 Renaming a Cluster..........................................................................................................................................108 Cluster-WideConfiguration..............................................................................................................................108 VirtualPrivateClusters andCloudera SDX.......................................................................................................108 Overview............................................................................................................................................................................109 AdvantagesofSeparatingComputeandDataResources..................................................................................................109 Architecture.......................................................................................................................................................................109 PerformanceTradeOffs.....................................................................................................................................................111 UsingVirtualPrivateClustersinYourApplications............................................................................................................111 AddingaComputeClusterandDataContext....................................................................................................................111 ImprovementsforVirtualPrivateClustersinCDH6.3.......................................................................................................112 CompatibilityConsiderationsforVirtualPrivateClusters.................................................................................................113 Tutorial:UsingImpala,HiveandHuewithVirtualPrivateClusters...................................................................................116 NetworkingConsiderationsforVirtualPrivateClusters.....................................................................................................127 Managing Services...........................................................................................................................................132 ManagingtheHBaseService.............................................................................................................................................132 Managing HDFS.................................................................................................................................................................132 ManagingApacheHiveinCDH..........................................................................................................................................170 Managing Hue...................................................................................................................................................................170 Managing Impala..............................................................................................................................................................174 ManagingKey-ValueStoreIndexer....................................................................................................................................185 Managing Kudu.................................................................................................................................................................186 Managing Solr...................................................................................................................................................................187 ManagingSpark.................................................................................................................................................................192 ManagingtheSqoop1Client.............................................................................................................................................193 ManagingYARN(MRv2)andMapReduce(MRv1).............................................................................................................196 ManagingZooKeeper.........................................................................................................................................................218 ConfiguringServicestoUsetheGPLExtrasParcel.............................................................................................................221 Managing Hosts...................................................................................................223 The Status Tab..................................................................................................................................................223 The Configuration Tab......................................................................................................................................224 TheRolesandDisksOverviewTabs..................................................................................................................224 TheTemplatesTab............................................................................................................................................224 TheParcelsTab.................................................................................................................................................224 Viewing Host Details........................................................................................................................................224 Status.................................................................................................................................................................................225 Processes...........................................................................................................................................................................226 Resources...........................................................................................................................................................................226 Commands.........................................................................................................................................................................227 Configuration.....................................................................................................................................................................227 Components.......................................................................................................................................................................227 Audits.................................................................................................................................................................................227 ChartsLibrary.....................................................................................................................................................................227 UsingtheHostInspector..................................................................................................................................228 RunningtheHostInspector................................................................................................................................................228 ViewingPastHostInspectorResults..................................................................................................................................228 AddingaHosttotheCluster............................................................................................................................228 UsingtheAddHostsWizardtoAddHosts.........................................................................................................................229 AddingaHostbyInstallingthePackagesUsingYourOwnMethod..................................................................................235 SpecifyingRacksforHosts................................................................................................................................235 Host Templates.................................................................................................................................................236 Creating a Host Template..................................................................................................................................................236 EditingaHostTemplate.....................................................................................................................................................236 ApplyingaHostTemplatetoaHost...................................................................................................................................237 PerformingMaintenanceonaClusterHost.....................................................................................................237 Decommissioning Hosts.....................................................................................................................................................237 Recommissioning Hosts.....................................................................................................................................................239 StoppingAlltheRolesonaHost........................................................................................................................................239 StartingAlltheRolesonaHost..........................................................................................................................................239 TuningandTroubleshootingHostDecommissioning.........................................................................................................239 MaintenanceMode............................................................................................................................................................242 ChangingHostnames........................................................................................................................................245 Deleting Hosts..................................................................................................................................................247 Moving a Host Between Clusters.....................................................................................................................248 Managing Services...............................................................................................249 AddingaService...............................................................................................................................................249 ComparingConfigurationsforaServiceBetweenClusters..............................................................................250 Add-onServices................................................................................................................................................251 Custom ServiceDescriptorFiles.........................................................................................................................................251 InstallinganAdd-onService...............................................................................................................................................251 AddinganAdd-onService..................................................................................................................................................253 UninstallinganAdd-onService..........................................................................................................................................253 Starting, Stopping, and Restarting Services.....................................................................................................253 Startingand Stopping Services..........................................................................................................................................253 Restarting a Service...........................................................................................................................................................254 Rolling Restart..................................................................................................................................................254 Aborting a Pending Command.........................................................................................................................257 Deleting Services..............................................................................................................................................257 Renaming a Service..........................................................................................................................................258 ConfiguringMaximumFileDescriptors............................................................................................................258 ExposingHadoopMetricstoGraphite.............................................................................................................258 ConfigureHadoopMetricsforGraphiteUsingClouderaManager....................................................................................259 GraphiteConfigurationSettingsPerDaemon....................................................................................................................260 ExposingHadoopMetricstoGanglia...............................................................................................................261 ConfigureHadoopMetricsforGangliaUsingClouderaManager.....................................................................................261 GangliaConfigurationSettingsPerDaemon.....................................................................................................................263 Managing Roles....................................................................................................265 Role Instances..................................................................................................................................................265 Role Groups......................................................................................................................................................268 CreatingaRoleGroup........................................................................................................................................................268 Managing Role Groups......................................................................................................................................................269 Monitoring and Diagnostics..................................................................................270 Introduction to Cloudera Manager Monitoring...............................................................................................270 TimeLine............................................................................................................................................................................271 HealthTests........................................................................................................................................................................272 ClouderaManagerAdminConsoleHomePage.................................................................................................................274 ViewingChartsforCluster,Service,Role,andHostInstances............................................................................................279 ConfiguringMonitoringSettings........................................................................................................................................280 MonitoringClusters..........................................................................................................................................287 Inspecting Network Performance....................................................................................................................289 RunningtheNetworkPerformanceInspectorFromtheClouderaManagerAdminConsole.............................................289 RunningtheNetworkPerformanceInspectorFromtheClouderaManagerAPI...............................................................291 Monitoring Services.........................................................................................................................................292 MonitoringServiceStatus..................................................................................................................................................292 ViewingServiceStatus.......................................................................................................................................................293 ViewingServiceInstanceDetails........................................................................................................................................296 ViewingRoleInstanceStatus.............................................................................................................................................298 RunningDiagnosticCommandsforRoles..........................................................................................................................299 PeriodicStacksCollection...................................................................................................................................................300 ViewingRunningandRecentCommands..........................................................................................................................301 MonitoringResourceManagement...................................................................................................................................303 Monitoring Hosts.............................................................................................................................................304 HostDetails........................................................................................................................................................................305 Host Inspector....................................................................................................................................................................308 Monitoring Activities........................................................................................................................................309 Monitoring MapReduce Jobs.............................................................................................................................................309 MonitoringImpalaQueries................................................................................................................................................316 MonitoringYARNApplications...........................................................................................................................................327 Monitoring Spark Applications..........................................................................................................................................340 Events...............................................................................................................................................................347 Viewing Events...................................................................................................................................................................348 FilteringEvents...................................................................................................................................................................348 Alerts................................................................................................................................................................349 Managing Alerts................................................................................................................................................................349 Triggers.............................................................................................................................................................356 ClouderaManagerTriggerUseCases................................................................................................................................360 Lifecycle and Security Auditing........................................................................................................................363 Viewing Audit Events.........................................................................................................................................................364 FilteringAuditEvents.........................................................................................................................................................364 DownloadingAuditEvents.................................................................................................................................................365 ChartingTime-SeriesData................................................................................................................................365 Terminology.......................................................................................................................................................................366 BuildingaChartwithTime-SeriesData.............................................................................................................................366 ConfiguringTime-SeriesQueryResults..............................................................................................................................367 UsingContext-SensitiveVariablesinCharts.......................................................................................................................367 Chart Properties.................................................................................................................................................................368 DisplayingChartDetails.....................................................................................................................................................371 Editing a Chart...................................................................................................................................................................373 SavingaChart....................................................................................................................................................................373 ObtainingTime-SeriesDataUsingtheAPI.........................................................................................................................374 Dashboards........................................................................................................................................................................374 tsquery Language..............................................................................................................................................................377 MetricAggregation............................................................................................................................................................386 Logs..................................................................................................................................................................389 Viewing Logs......................................................................................................................................................................389 Logs List.............................................................................................................................................................................389 FilteringLogs......................................................................................................................................................................390 Log Details.........................................................................................................................................................................390 ViewingtheClouderaManagerServerLog.......................................................................................................................391 ViewingtheClouderaManagerAgentLogs......................................................................................................................391 ManagingDiskSpaceforLogFiles....................................................................................................................................392 Reports.............................................................................................................................................................392 DirectoryUsageReport......................................................................................................................................................393 Disk Usage Reports............................................................................................................................................................395 Activity,Application,andQueryReports...........................................................................................................................396 TheFileBrowser.................................................................................................................................................................396 DownloadingHDFSDirectoryAccessPermissionReports..................................................................................................397 TroubleshootingClusterConfigurationandOperation....................................................................................398 SolutionstoCommonProblems.........................................................................................................................................398 Logs and Events.................................................................................................................................................................400 Monitoring Reference......................................................................................................................................400 Performance Management...................................................................................401 OptimizingPerformanceinCDH.......................................................................................................................401 ChoosingandConfiguringDataCompression..................................................................................................405 ConfiguringData Compression..........................................................................................................................................406 Tuningthe Solr Server......................................................................................................................................406 SettingJavaSystemPropertiesforSolr..............................................................................................................................406 TuningtoCompleteDuringSetup......................................................................................................................................406 GeneralTuning...................................................................................................................................................................407 Other Resources.................................................................................................................................................................413 Tuning Apache Spark Applications...................................................................................................................413 TuningSparkShuffleOperations........................................................................................................................................413 ReducingtheSizeofDataStructures.................................................................................................................................419 Choosing Data Formats.....................................................................................................................................................420 Tuning YARN.....................................................................................................................................................420 Overview............................................................................................................................................................................420 Cluster Configuration.........................................................................................................................................................424 YARN Configuration...........................................................................................................................................................425 MapReduce Configuration.................................................................................................................................................426 Step7:MapReduceConfiguration.....................................................................................................................................426 Step7A:MapReduceSanityChecking................................................................................................................................427 ContinuousScheduling.......................................................................................................................................................427 ConfiguringYourClusterInClouderaManager.................................................................................................................428 Tuning JVM Garbage Collection.......................................................................................................................428 Resource Management........................................................................................431 Cloudera Manager Resource Management.....................................................................................................431 StaticServicePools...........................................................................................................................................432 LinuxControlGroups(cgroups)..........................................................................................................................................433 Dynamic Resource Pools..................................................................................................................................437 ManagingDynamicResourcePools...................................................................................................................................437 YARNPoolStatusandConfigurationOptions....................................................................................................................442 Defining ConfigurationSets...............................................................................................................................................444 Scheduling Configuration Sets...........................................................................................................................................445 AssigningApplicationsandQueriestoResourcePools......................................................................................................446 YARN(MRv2)andMapReduce(MRv1)Schedulers..........................................................................................449 ConfiguringtheFairScheduler...........................................................................................................................................450 EnablingandDisablingFairSchedulerPreemption...........................................................................................................453 DataStorageforMonitoringData....................................................................................................................454 ConfiguringServiceMonitorDataStorage........................................................................................................................454 ConfiguringHostMonitorDataStorage............................................................................................................................454 ViewingHostandServiceMonitorDataStorage...............................................................................................................454 DataGranularityandTime-SeriesMetricData..................................................................................................................455 MovingMonitoringDataonanActiveCluster...................................................................................................................455 HostMonitorandServiceMonitorMemoryConfiguration...............................................................................................455 DisablingMetricrollup......................................................................................................................................................457 Cluster Utilization Reports...............................................................................................................................457 ConfiguringtheClusterUtilizationReport.........................................................................................................................458 UsingtheClusterUtilizationReporttoManageResources...............................................................................................460 DownloadingClusterUtilizationReportsUsingtheClouderaManagerAPI......................................................................466 CreatingaCustomClusterUtilizationReport....................................................................................................................466 High Availability...................................................................................................478 HDFS High Availability......................................................................................................................................478 IntroductiontoHDFSHighAvailability...............................................................................................................................479 ConfiguringHardwareforHDFSHA...................................................................................................................................480 EnablingHDFSHA..............................................................................................................................................................481 DisablingandRedeployingHDFSHA..................................................................................................................................485 ConfiguringOtherCDHComponentstoUseHDFSHA.......................................................................................................485 AdministeringanHDFSHighAvailabilityCluster...............................................................................................................486 ChangingaNameserviceNameforHighlyAvailableHDFSUsingClouderaManager......................................................487 MapReduce(MRv1)andYARN(MRv2)HighAvailability..................................................................................488 YARN(MRv2)ResourceManagerHighAvailability.............................................................................................................488 WorkPreservingRecoveryforYARNComponents.............................................................................................................490 MapReduce(MRv1)JobTrackerHighAvailability..............................................................................................................492 ClouderaNavigatorKeyTrusteeServerHighAvailability.................................................................................493 ConfiguringKeyTrusteeServerHighAvailabilityUsingClouderaManager......................................................................493 RecoveringaKeyTrusteeServer........................................................................................................................................494 EnablingKeyTrusteeKMSHighAvailability.....................................................................................................494 EnablingNavigatorHSMKMSHighAvailability................................................................................................496 HSMKMSHighAvailabilityBackupandRecovery.............................................................................................................496 HighAvailabilityforOtherCDHComponents...................................................................................................497 HBaseHighAvailability......................................................................................................................................................497 OozieHighAvailability.......................................................................................................................................................502 Search High Availability.....................................................................................................................................................503 NavigatorDataManagementinaHighAvailabilityEnvironment....................................................................505 ConfiguringClouderaManagerforHighAvailabilityWithaLoadBalancer.....................................................506 IntroductiontoClouderaManagerDeploymentArchitecture...........................................................................................507 PrerequisitesforSettingupClouderaManagerHighAvailability......................................................................................508 ClouderaManagerFailoverProtection..............................................................................................................................509 High-LevelStepstoConfigureClouderaManagerHighAvailability.................................................................................510 DatabaseHighAvailabilityConfiguration..........................................................................................................................536 TLSandKerberosConfigurationforClouderaManagerHighAvailability.........................................................................536 Backup and Disaster Recovery..............................................................................540 PortRequirementsforBackupandDisasterRecovery.....................................................................................540 Data Replication...............................................................................................................................................541 ClouderaLicenseRequirementsforReplication.................................................................................................................541 SupportedandUnsupportedReplicationScenarios...........................................................................................................541 ReplicatingDirectorieswithThousandsofFilesandSubdirectories..................................................................................542 HDFSandHive/ImpalaReplicationToandFromCloudStorage........................................................................................542 SupportedReplicationScenariosforClustersusingIsilonStorage.....................................................................................543 BDR Log Retention.............................................................................................................................................................543 Designating a Replication Source......................................................................................................................................543 HDFS Replication................................................................................................................................................................545 Hive/ImpalaReplication.....................................................................................................................................................559 ReplicatingDatatoImpalaClusters...................................................................................................................................575 UsingSnapshotswithReplication......................................................................................................................................576 EnablingReplicationBetweenClusterswithKerberosAuthentication..............................................................................577 Replication of Encrypted Data...........................................................................................................................................580 HBase Replication..............................................................................................................................................................582 Snapshots.........................................................................................................................................................594 Cloudera Manager Snapshot Policies................................................................................................................................594 Managing HBase Snapshots..............................................................................................................................................598

Description:
or the applicable trademark holder. Hadoop and the Hadoop elephant logo are trademarks of the Apache Software. Foundation. All other trademarks, registered trademarks, product names and company .. Tuning Hive on Spark.
ebook img

Cloudera Administration PDF

551 Pages·2016·9.42 MB·English
by  
#cloudera #administration #apache spark
Save to my drive
Quick download
Download
Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Cloudera Administration

Cluster Management Guide ImportantNotice ©2010-2021Cloudera,Inc.Allrightsreserved. Cloudera,theClouderalogo,andanyotherproductor servicenamesorsloganscontainedinthisdocumentaretrademarksofClouderaand itssuppliersorlicensors,andmaynotbecopied,imitatedorused,inwholeorinpart, withoutthepriorwrittenpermissionofClouderaortheapplicabletrademarkholder.If thisdocumentationincludescode,includingbutnotlimitedto,codeexamples,Cloudera makesthisavailabletoyouunderthetermsoftheApacheLicense,Version2.0,including anyrequirednotices.AcopyoftheApacheLicenseVersion2.0,includinganynotices, isincludedherein.AcopyoftheApacheLicenseVersion2.0canalsobefoundhere: https://opensource.org/licenses/Apache-2.0 HadoopandtheHadoopelephantlogoaretrademarksoftheApacheSoftware Foundation.Allothertrademarks,registeredtrademarks,productnamesandcompany namesorlogosmentionedinthisdocumentarethepropertyoftheirrespectiveowners. Referencetoanyproducts,services,processesorotherinformation,bytradename, trademark,manufacturer,supplierorotherwisedoesnotconstituteorimply endorsement,sponsorshiporrecommendationthereofbyus. Complyingwithallapplicablecopyrightlawsistheresponsibilityoftheuser.Without limitingtherightsundercopyright,nopartofthisdocumentmaybereproduced,stored inorintroducedintoaretrievalsystem,ortransmittedinanyformorbyanymeans (electronic,mechanical,photocopying,recording,orotherwise),orforanypurpose, withouttheexpresswrittenpermissionofCloudera. Clouderamayhavepatents,patentapplications,trademarks,copyrights,orother intellectualpropertyrightscoveringsubjectmatterinthisdocument.Exceptasexpressly providedinanywrittenlicenseagreementfromCloudera,thefurnishingofthisdocument doesnotgiveyouanylicensetothesepatents,trademarkscopyrights,orother intellectualproperty.ForinformationaboutpatentscoveringClouderaproducts,see http://tiny.cloudera.com/patents. Theinformationinthisdocumentissubjecttochangewithoutnotice.Clouderashall notbeliableforanydamagesresultingfromtechnicalerrorsoromissionswhichmay bepresentinthisdocument,orfromuseofthisdocument. Cloudera,Inc. 395PageMillRoad PaloAlto,CA94306 [email protected] US:1-888-789-1488 Intl:1-650-362-0488 www.cloudera.com ReleaseInformation Version:ClouderaEnterprise6.3.x Date:September30,2021 Table of Contents About Cloudera Administration..............................................................................14 Cloudera Manager.................................................................................................15 Learn about Cloudera Manager ........................................................................................................................17 Installing and Upgrading....................................................................................................................................17 Managing CDH using Cloudera Manager...........................................................................................................17 MonitoringCDH using Cloudera Manager.........................................................................................................17 ManagingCDHusingtheClouderaManagerAPI...............................................................................................17 Cloudera Manager Admin Console....................................................................................................................18 StartingandLoggingintotheAdminConsole......................................................................................................................20 ClouderaManagerAdminConsoleHomePage...................................................................................................................20 DisplayingClouderaManagerDocumentation....................................................................................................................25 Automatic Logout................................................................................................................................................................25 ClouderaManagerFrequentlyAskedQuestions................................................................................................26 GeneralQuestions................................................................................................................................................................26 Cloudera Manager API.......................................................................................................................................28 BackingUpandRestoringtheClouderaManagerConfiguration.......................................................................................30 UsingtheClouderaManagerAPIforClusterAutomation...................................................................................................31 ClouderaManagerAdministration.....................................................................................................................33 Starting,Stopping,andRestartingtheClouderaManagerServer.......................................................................................33 ConfiguringClouderaManagerServerPorts.......................................................................................................................33 MovingtheClouderaManagerServertoaNewHost.........................................................................................................34 MigratingfromtheClouderaManagerEmbeddedPostgreSQLDatabaseServertoanExternalPostgreSQLDatabase .......................................................................................................................................................................................35 MigratingfromtheClouderaManagerExternalPostgreSQLDatabaseServertoaMySQL/OracleDatabaseServer........41 ManagingtheClouderaManagerServerLog......................................................................................................................44 ClouderaManagerAgents...................................................................................................................................................44 ConfiguringNetworkSettings..............................................................................................................................................50 Managing Licenses..............................................................................................................................................................50 SendingUsageandDiagnosticDatatoCloudera................................................................................................................57 ExportingandImportingClouderaManagerConfiguration................................................................................................61 BackingUpClouderaManager............................................................................................................................................61 OtherClouderaManagerTasksandSettings.......................................................................................................................66 ClouderaManagementService............................................................................................................................................67 ExtendingClouderaManager.............................................................................................................................72 Cluster Configuration Overview..............................................................................73 ModifyingConfigurationPropertiesUsingClouderaManager..........................................................................74 ChangingtheConfigurationofaServiceorRoleInstance...................................................................................................74 RestartingServicesandInstancesafterConfigurationChanges..........................................................................................78 SuppressingConfigurationandParameterValidationWarnings........................................................................................78 Autoconfiguration..............................................................................................................................................79 Autoconfiguration................................................................................................................................................................80 Role-HostPlacement............................................................................................................................................................87 Custom Configuration........................................................................................................................................88 Stale Configurations...........................................................................................................................................91 Client Configuration Files...................................................................................................................................93 HowClientConfigurationsareDeployed.............................................................................................................................93 DownloadingClientConfigurationFiles...............................................................................................................................94 ManuallyRedeployingClientConfigurationFiles................................................................................................................94 ViewingandRevertingConfigurationChanges..................................................................................................94 ViewingConfigurationChanges...........................................................................................................................................94 Reverting Configuration Changes........................................................................................................................................95 ExportingandImportingClouderaManagerConfiguration...............................................................................95 ClouderaManagerConfigurationPropertiesReference....................................................................................96 Managing Clusters..................................................................................................97 Adding and Deleting Clusters.............................................................................................................................97 AddingaClusterUsingNewHosts.......................................................................................................................................97 AddingaClusterUsingCurrentlyManagedHosts.............................................................................................................101 DeletingaCluster...............................................................................................................................................................104 Starting,Stopping,Refreshing,andRestartingaCluster..................................................................................104 PausingaClusterinAWS..................................................................................................................................106 ShuttingDownandStartingUptheCluster.......................................................................................................................106 Considerations after Restart..............................................................................................................................................107 Renaming a Cluster..........................................................................................................................................108 Cluster-WideConfiguration..............................................................................................................................108 VirtualPrivateClusters andCloudera SDX.......................................................................................................108 Overview............................................................................................................................................................................109 AdvantagesofSeparatingComputeandDataResources..................................................................................................109 Architecture.......................................................................................................................................................................109 PerformanceTradeOffs.....................................................................................................................................................111 UsingVirtualPrivateClustersinYourApplications............................................................................................................111 AddingaComputeClusterandDataContext....................................................................................................................111 ImprovementsforVirtualPrivateClustersinCDH6.3.......................................................................................................112 CompatibilityConsiderationsforVirtualPrivateClusters.................................................................................................113 Tutorial:UsingImpala,HiveandHuewithVirtualPrivateClusters...................................................................................116 NetworkingConsiderationsforVirtualPrivateClusters.....................................................................................................127 Managing Services...........................................................................................................................................132 ManagingtheHBaseService.............................................................................................................................................132 Managing HDFS.................................................................................................................................................................132 ManagingApacheHiveinCDH..........................................................................................................................................170 Managing Hue...................................................................................................................................................................170 Managing Impala..............................................................................................................................................................174 ManagingKey-ValueStoreIndexer....................................................................................................................................185 Managing Kudu.................................................................................................................................................................186 Managing Solr...................................................................................................................................................................187 ManagingSpark.................................................................................................................................................................192 ManagingtheSqoop1Client.............................................................................................................................................193 ManagingYARN(MRv2)andMapReduce(MRv1).............................................................................................................196 ManagingZooKeeper.........................................................................................................................................................218 ConfiguringServicestoUsetheGPLExtrasParcel.............................................................................................................221 Managing Hosts...................................................................................................223 The Status Tab..................................................................................................................................................223 The Configuration Tab......................................................................................................................................224 TheRolesandDisksOverviewTabs..................................................................................................................224 TheTemplatesTab............................................................................................................................................224 TheParcelsTab.................................................................................................................................................224 Viewing Host Details........................................................................................................................................224 Status.................................................................................................................................................................................225 Processes...........................................................................................................................................................................226 Resources...........................................................................................................................................................................226 Commands.........................................................................................................................................................................227 Configuration.....................................................................................................................................................................227 Components.......................................................................................................................................................................227 Audits.................................................................................................................................................................................227 ChartsLibrary.....................................................................................................................................................................227 UsingtheHostInspector..................................................................................................................................228 RunningtheHostInspector................................................................................................................................................228 ViewingPastHostInspectorResults..................................................................................................................................228 AddingaHosttotheCluster............................................................................................................................228 UsingtheAddHostsWizardtoAddHosts.........................................................................................................................229 AddingaHostbyInstallingthePackagesUsingYourOwnMethod..................................................................................235 SpecifyingRacksforHosts................................................................................................................................235 Host Templates.................................................................................................................................................236 Creating a Host Template..................................................................................................................................................236 EditingaHostTemplate.....................................................................................................................................................236 ApplyingaHostTemplatetoaHost...................................................................................................................................237 PerformingMaintenanceonaClusterHost.....................................................................................................237 Decommissioning Hosts.....................................................................................................................................................237 Recommissioning Hosts.....................................................................................................................................................239 StoppingAlltheRolesonaHost........................................................................................................................................239 StartingAlltheRolesonaHost..........................................................................................................................................239 TuningandTroubleshootingHostDecommissioning.........................................................................................................239 MaintenanceMode............................................................................................................................................................242 ChangingHostnames........................................................................................................................................245 Deleting Hosts..................................................................................................................................................247 Moving a Host Between Clusters.....................................................................................................................248 Managing Services...............................................................................................249 AddingaService...............................................................................................................................................249 ComparingConfigurationsforaServiceBetweenClusters..............................................................................250 Add-onServices................................................................................................................................................251 Custom ServiceDescriptorFiles.........................................................................................................................................251 InstallinganAdd-onService...............................................................................................................................................251 AddinganAdd-onService..................................................................................................................................................253 UninstallinganAdd-onService..........................................................................................................................................253 Starting, Stopping, and Restarting Services.....................................................................................................253 Startingand Stopping Services..........................................................................................................................................253 Restarting a Service...........................................................................................................................................................254 Rolling Restart..................................................................................................................................................254 Aborting a Pending Command.........................................................................................................................257 Deleting Services..............................................................................................................................................257 Renaming a Service..........................................................................................................................................258 ConfiguringMaximumFileDescriptors............................................................................................................258 ExposingHadoopMetricstoGraphite.............................................................................................................258 ConfigureHadoopMetricsforGraphiteUsingClouderaManager....................................................................................259 GraphiteConfigurationSettingsPerDaemon....................................................................................................................260 ExposingHadoopMetricstoGanglia...............................................................................................................261 ConfigureHadoopMetricsforGangliaUsingClouderaManager.....................................................................................261 GangliaConfigurationSettingsPerDaemon.....................................................................................................................263 Managing Roles....................................................................................................265 Role Instances..................................................................................................................................................265 Role Groups......................................................................................................................................................268 CreatingaRoleGroup........................................................................................................................................................268 Managing Role Groups......................................................................................................................................................269 Monitoring and Diagnostics..................................................................................270 Introduction to Cloudera Manager Monitoring...............................................................................................270 TimeLine............................................................................................................................................................................271 HealthTests........................................................................................................................................................................272 ClouderaManagerAdminConsoleHomePage.................................................................................................................274 ViewingChartsforCluster,Service,Role,andHostInstances............................................................................................279 ConfiguringMonitoringSettings........................................................................................................................................280 MonitoringClusters..........................................................................................................................................287 Inspecting Network Performance....................................................................................................................289 RunningtheNetworkPerformanceInspectorFromtheClouderaManagerAdminConsole.............................................289 RunningtheNetworkPerformanceInspectorFromtheClouderaManagerAPI...............................................................291 Monitoring Services.........................................................................................................................................292 MonitoringServiceStatus..................................................................................................................................................292 ViewingServiceStatus.......................................................................................................................................................293 ViewingServiceInstanceDetails........................................................................................................................................296 ViewingRoleInstanceStatus.............................................................................................................................................298 RunningDiagnosticCommandsforRoles..........................................................................................................................299 PeriodicStacksCollection...................................................................................................................................................300 ViewingRunningandRecentCommands..........................................................................................................................301 MonitoringResourceManagement...................................................................................................................................303 Monitoring Hosts.............................................................................................................................................304 HostDetails........................................................................................................................................................................305 Host Inspector....................................................................................................................................................................308 Monitoring Activities........................................................................................................................................309 Monitoring MapReduce Jobs.............................................................................................................................................309 MonitoringImpalaQueries................................................................................................................................................316 MonitoringYARNApplications...........................................................................................................................................327 Monitoring Spark Applications..........................................................................................................................................340 Events...............................................................................................................................................................347 Viewing Events...................................................................................................................................................................348 FilteringEvents...................................................................................................................................................................348 Alerts................................................................................................................................................................349 Managing Alerts................................................................................................................................................................349 Triggers.............................................................................................................................................................356 ClouderaManagerTriggerUseCases................................................................................................................................360 Lifecycle and Security Auditing........................................................................................................................363 Viewing Audit Events.........................................................................................................................................................364 FilteringAuditEvents.........................................................................................................................................................364 DownloadingAuditEvents.................................................................................................................................................365 ChartingTime-SeriesData................................................................................................................................365 Terminology.......................................................................................................................................................................366 BuildingaChartwithTime-SeriesData.............................................................................................................................366 ConfiguringTime-SeriesQueryResults..............................................................................................................................367 UsingContext-SensitiveVariablesinCharts.......................................................................................................................367 Chart Properties.................................................................................................................................................................368 DisplayingChartDetails.....................................................................................................................................................371 Editing a Chart...................................................................................................................................................................373 SavingaChart....................................................................................................................................................................373 ObtainingTime-SeriesDataUsingtheAPI.........................................................................................................................374 Dashboards........................................................................................................................................................................374 tsquery Language..............................................................................................................................................................377 MetricAggregation............................................................................................................................................................386 Logs..................................................................................................................................................................389 Viewing Logs......................................................................................................................................................................389 Logs List.............................................................................................................................................................................389 FilteringLogs......................................................................................................................................................................390 Log Details.........................................................................................................................................................................390 ViewingtheClouderaManagerServerLog.......................................................................................................................391 ViewingtheClouderaManagerAgentLogs......................................................................................................................391 ManagingDiskSpaceforLogFiles....................................................................................................................................392 Reports.............................................................................................................................................................392 DirectoryUsageReport......................................................................................................................................................393 Disk Usage Reports............................................................................................................................................................395 Activity,Application,andQueryReports...........................................................................................................................396 TheFileBrowser.................................................................................................................................................................396 DownloadingHDFSDirectoryAccessPermissionReports..................................................................................................397 TroubleshootingClusterConfigurationandOperation....................................................................................398 SolutionstoCommonProblems.........................................................................................................................................398 Logs and Events.................................................................................................................................................................400 Monitoring Reference......................................................................................................................................400 Performance Management...................................................................................401 OptimizingPerformanceinCDH.......................................................................................................................401 ChoosingandConfiguringDataCompression..................................................................................................405 ConfiguringData Compression..........................................................................................................................................406 Tuningthe Solr Server......................................................................................................................................406 SettingJavaSystemPropertiesforSolr..............................................................................................................................406 TuningtoCompleteDuringSetup......................................................................................................................................406 GeneralTuning...................................................................................................................................................................407 Other Resources.................................................................................................................................................................413 Tuning Apache Spark Applications...................................................................................................................413 TuningSparkShuffleOperations........................................................................................................................................413 ReducingtheSizeofDataStructures.................................................................................................................................419 Choosing Data Formats.....................................................................................................................................................420 Tuning YARN.....................................................................................................................................................420 Overview............................................................................................................................................................................420 Cluster Configuration.........................................................................................................................................................424 YARN Configuration...........................................................................................................................................................425 MapReduce Configuration.................................................................................................................................................426 Step7:MapReduceConfiguration.....................................................................................................................................426 Step7A:MapReduceSanityChecking................................................................................................................................427 ContinuousScheduling.......................................................................................................................................................427 ConfiguringYourClusterInClouderaManager.................................................................................................................428 Tuning JVM Garbage Collection.......................................................................................................................428 Resource Management........................................................................................431 Cloudera Manager Resource Management.....................................................................................................431 StaticServicePools...........................................................................................................................................432 LinuxControlGroups(cgroups)..........................................................................................................................................433 Dynamic Resource Pools..................................................................................................................................437 ManagingDynamicResourcePools...................................................................................................................................437 YARNPoolStatusandConfigurationOptions....................................................................................................................442 Defining ConfigurationSets...............................................................................................................................................444 Scheduling Configuration Sets...........................................................................................................................................445 AssigningApplicationsandQueriestoResourcePools......................................................................................................446 YARN(MRv2)andMapReduce(MRv1)Schedulers..........................................................................................449 ConfiguringtheFairScheduler...........................................................................................................................................450 EnablingandDisablingFairSchedulerPreemption...........................................................................................................453 DataStorageforMonitoringData....................................................................................................................454 ConfiguringServiceMonitorDataStorage........................................................................................................................454 ConfiguringHostMonitorDataStorage............................................................................................................................454 ViewingHostandServiceMonitorDataStorage...............................................................................................................454 DataGranularityandTime-SeriesMetricData..................................................................................................................455 MovingMonitoringDataonanActiveCluster...................................................................................................................455 HostMonitorandServiceMonitorMemoryConfiguration...............................................................................................455 DisablingMetricrollup......................................................................................................................................................457 Cluster Utilization Reports...............................................................................................................................457 ConfiguringtheClusterUtilizationReport.........................................................................................................................458 UsingtheClusterUtilizationReporttoManageResources...............................................................................................460 DownloadingClusterUtilizationReportsUsingtheClouderaManagerAPI......................................................................466 CreatingaCustomClusterUtilizationReport....................................................................................................................466 High Availability...................................................................................................478 HDFS High Availability......................................................................................................................................478 IntroductiontoHDFSHighAvailability...............................................................................................................................479 ConfiguringHardwareforHDFSHA...................................................................................................................................480 EnablingHDFSHA..............................................................................................................................................................481 DisablingandRedeployingHDFSHA..................................................................................................................................485 ConfiguringOtherCDHComponentstoUseHDFSHA.......................................................................................................485 AdministeringanHDFSHighAvailabilityCluster...............................................................................................................486 ChangingaNameserviceNameforHighlyAvailableHDFSUsingClouderaManager......................................................487 MapReduce(MRv1)andYARN(MRv2)HighAvailability..................................................................................488 YARN(MRv2)ResourceManagerHighAvailability.............................................................................................................488 WorkPreservingRecoveryforYARNComponents.............................................................................................................490 MapReduce(MRv1)JobTrackerHighAvailability..............................................................................................................492 ClouderaNavigatorKeyTrusteeServerHighAvailability.................................................................................493 ConfiguringKeyTrusteeServerHighAvailabilityUsingClouderaManager......................................................................493 RecoveringaKeyTrusteeServer........................................................................................................................................494 EnablingKeyTrusteeKMSHighAvailability.....................................................................................................494 EnablingNavigatorHSMKMSHighAvailability................................................................................................496 HSMKMSHighAvailabilityBackupandRecovery.............................................................................................................496 HighAvailabilityforOtherCDHComponents...................................................................................................497 HBaseHighAvailability......................................................................................................................................................497 OozieHighAvailability.......................................................................................................................................................502 Search High Availability.....................................................................................................................................................503 NavigatorDataManagementinaHighAvailabilityEnvironment....................................................................505 ConfiguringClouderaManagerforHighAvailabilityWithaLoadBalancer.....................................................506 IntroductiontoClouderaManagerDeploymentArchitecture...........................................................................................507 PrerequisitesforSettingupClouderaManagerHighAvailability......................................................................................508 ClouderaManagerFailoverProtection..............................................................................................................................509 High-LevelStepstoConfigureClouderaManagerHighAvailability.................................................................................510 DatabaseHighAvailabilityConfiguration..........................................................................................................................536 TLSandKerberosConfigurationforClouderaManagerHighAvailability.........................................................................536 Backup and Disaster Recovery..............................................................................540 PortRequirementsforBackupandDisasterRecovery.....................................................................................540 Data Replication...............................................................................................................................................541 ClouderaLicenseRequirementsforReplication.................................................................................................................541 SupportedandUnsupportedReplicationScenarios...........................................................................................................541 ReplicatingDirectorieswithThousandsofFilesandSubdirectories..................................................................................542 HDFSandHive/ImpalaReplicationToandFromCloudStorage........................................................................................542 SupportedReplicationScenariosforClustersusingIsilonStorage.....................................................................................543 BDR Log Retention.............................................................................................................................................................543 Designating a Replication Source......................................................................................................................................543 HDFS Replication................................................................................................................................................................545 Hive/ImpalaReplication.....................................................................................................................................................559 ReplicatingDatatoImpalaClusters...................................................................................................................................575 UsingSnapshotswithReplication......................................................................................................................................576 EnablingReplicationBetweenClusterswithKerberosAuthentication..............................................................................577 Replication of Encrypted Data...........................................................................................................................................580 HBase Replication..............................................................................................................................................................582 Snapshots.........................................................................................................................................................594 Cloudera Manager Snapshot Policies................................................................................................................................594 Managing HBase Snapshots..............................................................................................................................................598

Description:
or the applicable trademark holder. Hadoop and the Hadoop elephant logo are trademarks of the Apache Software. Foundation. All other trademarks, registered trademarks, product names and company .. Tuning Hive on Spark.
See more

The list of books you might like

Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.
DMCA & Copyright: Dear all, most of the website is community built, users are uploading hundred of books everyday, which makes really hard for us to identify copyrighted material, please contact us if you want any material removed.
Copyright @ PDFdrive.to, 2022 | We love our users