Troubleshooting the Cisco APIC-EM

28 Pages·2016·1.61 MB·English

Troubleshooting the Cisco APIC-EM
• CiscoAPIC-EMComponentsandArchitecture, page 1
• TroubleshootinganUnsuccessfulInstallation, page 4
• TroubleshootingtheConfiguration, page 9
• TroubleshootingServices, page 13
• TroubleshootingPasswords, page 18
• TroubleshootingCommands, page 21
• TroubleshootingLogFiles, page 24
Cisco APIC-EM Components and Architecture
ThissectiondescribesthecomponentsandbasicarchitectureoftheCiscoAPIC-EM.Youshouldreviewthis
sectiontobetterunderstandhowthecontrolleranditscomponentsoperatebeforeperforminganyofthe
troubleshootingproceduresdescribedinthischapter.
TheCiscoAPIC-EMconsistsofthefollowingcomponents:
•Hosts
•Linuxcontainers(LXC)
•Grapevine
•Rootsandclients
•Services
•Databases
•Networks(internalandexternal)
•NetworkconnectionsandNICs
Related Topics
Hosts, onpage2
LinuxContainers, onpage2
Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide, Release 1.1.x
1
Troubleshooting the Cisco APIC-EM
Hosts
Grapevine, onpage2
RootandClients, onpage3
Services, onpage3
Databases, onpage3
Networks, onpage3
NetworkConnectionsandNICs, onpage4
Hosts
Ahostcanbeeitheranappliance,serverorvirtualmachinewhereCiscoAPIC-EMhasbeeninstalled.Within
thehostsaretheLinuxcontainersrunninginstancesoftheGrapevineclients.TheGrapevineclientsrunthe
differentservices.TheGrapevinerootrunsasanapplicationonthehostitselfandnotinanyLinuxcontainers.
Youcansetupeitherasinglehostormulti-hostdeploymentforyournetwork.
Related Topics
CiscoAPIC-EMComponentsandArchitecture, onpage1
Linux Containers
Linuxcontainersisavirtualizationtechnologythatprovidesasoftwarevirtualizationsystemforsystems
runningGNU/Linux.Thesecontainersenablemultiplevirtualenvironmentstobesetupandrunsimultaneously.
TheCiscoAPIC-EMcomponents(Grapevinerootsandclients)makeuseoftheUbuntuoperatingsystem
environmentandLinuxcontainers(LXC).TheGrapevinerootrunsasanapplicationonthehostitself.The
GrapevineclientsruninLXCswithinthehost.
Related Topics
CiscoAPIC-EMComponentsandArchitecture, onpage1
Grapevine
TheCiscoAPIC-EMcreatesaPlatformasaService(PaaS)environmentforyournetwork,usingGrapevine
asanElasticServicesplatformtosupportthecontroller'sinfrastructureandservices.TheGrapevinerootand
clientsarekeycomponentsofthisinfrastructure.
Note YoucanuseatoolcalledtheGrapevinedeveloperconsoletotroubleshootrootsandclientsrunning
services.TheGrapevinedeveloperconsolewasbundledwiththedeploymentfilesandinstalledwhenyou
firstdeployedtheCiscoAPIC-EM.YoucanaccesstheGrapevinedeveloperconsoleatthiscontrollerIP
addressandportnumber:https://<controller IP address>:14141
Related Topics
CiscoAPIC-EMComponentsandArchitecture, onpage1
Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide, Release 1.1.x
2
Troubleshooting the Cisco APIC-EM
Root and Clients
Root and Clients
TheGrapevineroothandlesallpolicymanagementinregardstoserviceupdates,aswellastheservicelifecycle
forbothitselfandtheGrapevineclient.TheGrapevineclientiswherethesupportedservicesrun.
Note YoucanremotelylogintotherootusingSSH(SecureShell)totroubleshootanyissues.Adefaultidle
timeoutof1hourhasbeensetforanSSHconsolelogin.Youwillbeautomaticallyloggedoutafter1
hourofinactivityontheSSHconsole.
Related Topics
CiscoAPIC-EMComponentsandArchitecture, onpage1
Services
TheCiscoAPIC-EMcreatesaPlatformasaService(PaaS)environmentforyournetwork.Aserviceinthis
PaaSenvironmentisahorizontallyscalableapplicationthataddsinstancesofitselfwhendemandincreases,
andfreesinstancesofitselfwhendemanddecreases.
Related Topics
CiscoAPIC-EMComponentsandArchitecture, onpage1
Databases
TheCiscoAPIC-EMsupportstwodatabases:applicationandGrapevine.Theapplicationdatabaseisused
fortheapplicationandexternalnetworkingdata.TheGrapevinedatabaseisusedfortheGrapevineandinternal
networkdata.Bothdatabasesarereplicatedinamulti-hostenvironmentforscaleandhighavailability.
Related Topics
CiscoAPIC-EMComponentsandArchitecture, onpage1
Networks
TheCiscoAPIC-EMarchitecturerequiresbothexternalandinternalnetworkstooperate:
•Theexternalnetwork(s)consistsofthenetworkhosts,devices,andNTPservers,aswellasproviding
accesstothenorthboundRESTAPIs.Theexternalnetwork(s)alsoprovidesaccesstothecontroller
GUI.
•TheinternalnetworkconsistsoftheGrapevinerootsandclientsthatareconnectedtoandcommunicate
witheachother(servicetoservice).Forforwardingtoorreceivingtrafficfromthelargerexternalnetwork
(thatconsistsoftheconnecteddevicesandhosts,aswellasNTPservers),allinboundandoutbound
trafficforthisinternalnetworkpassesthroughasubsetofclientsconnectedtotheexternalnetwork.The
internalnetworkisisolatedandnonroutablefromtheexternalnetwork(s),aswellasanyotherinternal
network.
Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide, Release 1.1.x
3
Troubleshooting the Cisco APIC-EM
Network Connections and NICs
Related Topics
CiscoAPIC-EMComponentsandArchitecture, onpage1
Network Connections and NICs
Thenetworkadapters(NICs)onthehost(physicalorvirtual)areconnectedtothefollowingexternalnetworks:
•Internet(networkaccessrequiredforMakeAWishrequests,Telemetry,andtrustpoolupdates)
•NetworkwithNTPserver(s)
•NetworkwithdevicesthataretobemanagedbytheCiscoAPIC-EM
Note TheCiscoAPIC-EMshouldneverbedirectlyconnectedtotheInternet.Itshouldnotbedeployedoutside
ofaNATconfiguredorprotecteddatacenterenvironment.
Related Topics
CiscoAPIC-EMComponentsandArchitecture, onpage1
Troubleshooting an Unsuccessful Installation
Afterperforminganinstallationusingtheconfigurationwizard,youshouldhavereceivedan‘installationwas
successful’message.Ifyoudidnotreceivethismessageandyourinstallationwasunsuccessful,youcan
performthefollowingtroubleshootingprocedures.
Confirming that Core Services are Running
Before You Begin
YoushouldhaveattemptedtodeploytheCiscoAPIC-EMfollowingtheproceduredescribedinthisguide.
Step 1 UsingaSecureShell(SSH)client,logintothehost(physicalorvirtual)withtheIPaddressthatyouspecifiedusingthe
configurationwizard.
Note TheIPaddresstoenterfortheSSHclientistheIPaddressthatyouconfiguredforthenetworkadapter.This
IPaddressconnectsthehosttotheexternalnetwork.
Step 2 Whenprompted,enteryourLinuxusername('grapevine')andpasswordforSSHaccess.
Step 3 Enterthefollowingcommandtodisplaythestatusofthecoreservices:
$ sudo service grapevine status
Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide, Release 1.1.x
4
Troubleshooting the Cisco APIC-EM
Confirming the Multi-Host Cluster Configuration Values
Step 4 Enteryourpasswordasecondtimewhenprompted.
$[sudo] password for grapevine: ********
Commandoutputsimilartothefollowingshouldappear.ThecoreservicesshouldhaveaRUNNINGstatus.
grapevine is running
grapevine_beacon RUNNING pid 30243, uptime 0:11:11
grapevine_capacity_manager RUNNING pid 30549, uptime 0:11:02
grapevine_capacity_manager_lxc_plugin RUNNING pid 30247, uptime 0:11:11
grapevine_cassandra RUNNING pid 30244, uptime 0:11:11
grapevine_root RUNNING pid 30537, uptime 0:11:03status
Step 5 IfanyofthecoreservicesarenotintheRUNNINGstate,entertherootcauseanalysis(rca)command.
$ rca
Thercacommandrunsarootcauseanalysisscriptthatcreatesatarfilethatcontainsthefollowingdata:
•Logfiles
•Configurationfiles
•Commandoutput
Step 6 SendthetarfilecreatedbythercacommandproceduretoCiscosupportforassistanceinresolvingyourissue.
Confirming the Multi-Host Cluster Configuration Values
Before You Begin
YoushouldhaveattemptedtodeploytheCiscoAPIC-EMfollowingtheproceduredescribedinthisguide.
Step 1 UsingaSecureShell(SSH)client,logintothehost(physicalorvirtual)withtheIPaddressthatyouspecifiedusingthe
configurationwizard.
Note TheIPaddresstoenterfortheSSHclientistheIPaddressthatyouconfiguredforthenetworkadapter.This
IPaddressconnectsthehosttotheexternalnetwork.
Step 2 Whenprompted,enteryourLinuxusername('grapevine')andpasswordforSSHaccess.
Step 3 Enterthefollowingcommandtodisplaythemulti-hostconfiguration.
$ grape root display
Commandoutputsimilartothefollowingshouldappear.
ROOT PROPERTY VALUE
Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide, Release 1.1.x
5
Troubleshooting the Cisco APIC-EM
Confirming the Multi-Host Cluster Configuration Values
----------------------------------------------------------------------
4cbe3972-9872-4771-800d-08c89463f1eb hostname root-1
4cbe3972-9872-4771-800d-08c89463f1eb interfaces [{'interface': 'eth0', 'ip':
'209.165.200.10', 'mac': '00:50:56:100:d2:14', 'netmask': '255.255.255.0'}, {'interface': 'eth1',
'ip': '209.165.200.10', 'mac': '00:50:56:95:5c:18', 'net mask': '255.255.255.0'}, {'interface':
'grape-br0', 'ip': '209.165.200.11', 'mac': 'ba:ed:c4:19:0d:77', 'netmask': '255.255.255.0'}]
4cbe3972-9872-4771-800d-08c89463f1eb is_alive True
4cbe3972-9872-4771-800d-08c89463f1eb last_heartbeat Wed Sep 09, 2015 11:02:52 PM (just now)
4cbe3972-9872-4771-800d-08c89463f1eb public_key ssh-rsa
c2EAAAADAQABAAABAQDYlyCfidke3MTjGkzsTAu73MtG+lynFFvxWZ4xVIkDkhGC7KCs6XMhORMaABb6
bU4EX/6osa4qyta4NYaijxjL6GL6kPkSBZiEKcUekHCmk1+H+Ypp5tc0wyvSpe5HtbLvPicLrXHHI/TS
Fsa+gLPqg55TflX8RH3i8dHf1Zwq6v4nHVryjAzMXeFYnFHST9e0P62QnkAGh29ktxUpS3fKua9iCVIE
V44t+VvtFaLurG9+FW/ngZwGrR/N0ZJZl6/MQTN3 grapevine@grapevine-root
4cbe3972-9872-4771-800d-08c89463f1eb root_id 4cbe3972-9872-4771-800d-08c89463f1eb
4cbe3972-9872-4771-800d-08c89463f1eb root_index 0
4cbe3972-9872-4771-800d-08c89463f1eb root_version 0.3.0.958.dev140-gda6a16
4cbe3972-9872-4771-800d-08c89463f1eb vm_password ******
(grapevine)
#
ROOT PROPERTY VALUE
----------------------------------------------------------------------
4cbe3972-9872-4771-800d-08c89463f1eb hostname root-2
4cbe3972-9872-4771-800d-08c89463f1eb interfaces [{'interface': 'eth0', 'ip':
'209.165.200.101, 'mac': '00:50:56:100:d2:14', 'netmask': '255.255.255.0'}, {'interface': 'eth1',
'ip': '209.165.200.11', 'mac': '00:50:56:95:5c:18', 'net mask': '255.255.255.0'}, {'interface':
'grape-br0', 'ip': '209.165.200.11', 'mac': 'ba:ed:c4:19:0d:77', 'netmask': '255.255.255.0'}]
4cbe3972-9872-4771-800d-08c89463f1eb is_alive True
4cbe3972-9872-4771-800d-08c89463f1eb last_heartbeat Wed Sep 09, 2015 11:02:52 PM (just now)
4cbe3972-9872-4771-800d-08c89463f1eb public_key ssh-rsa
c2EAAAADAQABAAABAQDYlyCfidke3MTjGkzsTAu73MtG+lynFFvxWZ4xVIkDkhGC7KCs6XMhORMaABb6
bU4EX/6osa4qyta4NYaijxjL6GL6kPkSBZiEKcUekHCmk1+H+Ypp5tc0wyvSpe5HtbLvPicLrXHHI/TS
Fsa+gLPqg55TflX8RH3i8dHf1Zwq6v4nHVryjAzMXeFYnFHST9e0P62QnkAGh29ktxUpS3fKua9iCVIE
V44t+VvtFaLurG9+FW/ngZwGrR/N0ZJZl6/MQTN3 grapevine@grapevine-root
4cbe3972-9872-4771-800d-08c89463f1eb root_id 4cbe3972-9873-4771-800d-08c89463f1eb
4cbe3972-9872-4771-800d-08c89463f1eb root_index 0
4cbe3972-9872-4771-800d-08c89463f1eb root_version 0.3.0.958.dev140-gda6a16
4cbe3972-9872-4771-800d-08c89463f1eb vm_password ******
(grapevine)
Thefollowingdataisdisplayedbythiscommand:
•hostname—Theconfiguredhostname.
•interfaces—Theconfiguredinterfacevalues,includingEthernetport,IPaddress,andnetmask.
•is_alive—Statusofthehost.Trueindicatesarunninghost,Falseindicatesahostthathasshutdown.
Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide, Release 1.1.x
6
Troubleshooting the Cisco APIC-EM
Resolving Access to the Cisco APIC-EM GUI
•last_heartbeat—Dateandtimeoflastheartbeatmessagesentfromthehost.
•public_key—Publickeyusedbyhost.
•root_id—Individualrootidentificationnumber.
•root_index—Individualrootindexnumber.
•root_version—Softwareversionofroot.
•vm_password—VMwarevSpherepasswordthatismasked.
Step 4 Ifanyofthefieldsinthecommandoutputappearincorrect,entertherootcauseanalysis(rca)command.
$ rca
Thercacommandrunsarootcauseanalysisscriptthatcreatesatarfilethatcontainsthefollowingdata:
•Logfiles
•Configurationfiles
•Commandoutput
Step 5 SendthetarfilecreatedbythercacommandproceduretoCiscosupportforassistanceinresolvingyourissue.
Resolving Access to the Cisco APIC-EM GUI
YouaccesstheCiscoAPIC-EMGUIbyenteringtheIPaddressthatyouconfiguredforthenetworkadapter
usingtheconfigurationwizard.ThisIPaddressconnectstotheexternalnetwork.EntertheIPaddressinyour
browserinthefollowingformat:
https://IPaddress
IfyouareunabletoaccesstheCiscoAPIC-EMGUI,youmustaccesstheGrapevinedeveloperconsoleto
checkforfaultyorfailedservices.
TheGrapevinedeveloperconsoleallowsyoutomonitorthehealthofyourCiscoAPIC-EMdeployment.The
GrapevinedeveloperconsoleispartoftheServiceElasticityPlatform(Grapevine).YouaccesstheGrapevine
developerconsolebyenteringtheIPaddressthatyouconfiguredforthenetworkadapterusingtheconfiguration
wizard,butwithaspecificportnumber(14141).
Foramulti-hostcluster,youdonothavetologintoeachhost.Inamulti-hostcluster,yougetasingle,
consolidatedviewofalloftheservicesrunningonallthreehosts.Multipleinstancesofservicesrunningon
differenthostswillappearintheGrapevinedeveloperconsoleinamulti-hostcluster.
Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide, Release 1.1.x
7
Troubleshooting the Cisco APIC-EM
Resolving Access to the Cisco APIC-EM GUI
Note Adefaultidletimeoutof1hourhasbeensetfortheGrapevinedeveloperconsole.Youwillbeautomatically
loggedoutafter1hourofinactivityontheGrapevinedeveloperconsole.
Figure 1: Grapevine Developer Console
ToaccesstheGrapevinedeveloperconsoletocheckforfaultyorfailedservices,followtheproceduredescribed
below.
Before You Begin
YoushouldhaveattemptedtodeploytheCiscoAPIC-EMfollowingtheproceduredescribedinthisguide.
Step 1 AccesstheGrapevinedeveloperconsolebyopeningabrowserwindowandenteringtheIPaddressthatyouconfigured
forthenetworkadapterusingtheconfigurationwizard.
Note ThisIPaddressconnectstheappliancetotheexternalnetwork.
Forexample,enterthefollowingIPaddresswithrequiredportnumber:
https://externalnetworkIPaddress:14141
Step 2 Enteryouradministrativeusernameandpasswordwhenprompted.
Theadministrativeusernameandpasswordwereconfiguredbyyouusingtheconfigurationwizard.
Afteryouentertheusernameandpassword,theGrapevinedeveloperconsoleappears.Eachinstalledservicewithits
versionnumberappearsintheconsoleinanalphabeticallist.Beloweachserviceisasquareiconthatrepresentsthe
healthoftheservice.Servicesthathavebeeninstalledandareoperationalaregreen.Faultyorfailedservicesarered.
Step 3 Scrolldownthelistandconfirmthatthefollowingservicesareinstalledandrunningforyourdeployment:
•reverse-proxy
•router
•ui
Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide, Release 1.1.x
8
Troubleshooting the Cisco APIC-EM
Troubleshooting the Configuration
Notewhethertheserviceisoperationalorfaulty.
Step 4 ReviewtheconsoleTaskstabbelowthelistofservicesforanyerrormessagesaboutanyfaultyservices.Notethereason
givenforthefaultyorfailedservice.
Step 5 ContactCiscosupportwiththefollowinginformation:
•WhetheranyoftheserviceslistedinStep3areinoperableorfaulty.
•WhetheranyerrorsareintheconsoleTaskstablocatedatthebottomoftheconsole.
Troubleshooting the Configuration
Performthefollowingprocedurestotroubleshootpossibleconfigurationissues.
Updating the Configuration Using the Wizard
YoucantroubleshoottheCiscoAPIC-EMdeploymentbyrunningtheconfigurationwizardasecondtime
andupdatinganyearlierconfigurationentries.Theconfigurationwizardsavesanddisplaysyourprevious
configurationsettings,soyoudonothavetoreenterthem.
Note Whenperformingthisprocedure,controllerdowntimeoccurs.Forthisreason,werecommendthatyou
performthisprocedureduringamaintenancetimeperiod.Forinformationaboutchangingsettingsfora
multi-hostconfiguration,seeChangingtheSettingsinaMulti-HostClusterUsingtheWizard
Before You Begin
YouhavedeployedCiscoAPIC-EMusingtheproceduresdescribedinthisguide.
Step 1 UsingaSecureShell(SSH)client,logintothehost(physicalorvirtual)withtheIPaddressthatyouspecifiedusingthe
configurationwizard.
Note TheIPaddresstoenterfortheSSHclientistheIPaddressthatyouconfiguredforthenetworkadapter.This
IPaddressconnectsthehosttotheexternalnetwork.
Step 2 Whenprompted,enteryourLinuxusername('grapevine')andpasswordforSSHaccess.
Step 3 Restarttheconfigurationwizardusingthefollowingcommand.
$ config_wizard
Step 4 Reviewthecurrentconfigurationvaluesintheconfigurationwizardandclicknext>>,untilyouaccessthespecificstep
whereyouwishtoupdateyourpreviousconfigurationentry.
Forexample,ifyouneedtoenteranewNTPserverIPaddress,clicknext>>untilyougettotheNTPSERVER
SETTINGSscreen.
Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide, Release 1.1.x
9
Troubleshooting the Cisco APIC-EM
Resetting the Cisco APIC-EM
Step 5 Updatethevaluethatwaspreviouslyenteredintheconfigurationwizardandiscurrentlydisplayed.
Forexample,youcanupdatetheNTPserversettingsbyenteringanewIPaddress.
Step 6 Clicknext>>untilthelaststepoftheconfigurationwizardprocess.
Step 7 Clickproceed>>tohavetheconfigurationwizardsaveandapplyyourconfigurationchangestoyourCiscoAPIC-EM
deployment.
Resetting the Cisco APIC-EM
YoucantroubleshootaCiscoAPIC-EMdeploymentbyresettingthecontrollerbacktoconfigurationvalues
thatwereoriginallysetusingtheconfigurationwizardthefirsttime.Aresetofthecontrollerishelpful,when
thecontrollerhasgottenitselfintoanunstablestateandothertroubleshootingactivitieshavenotresolvedthe
situation.
Inamulti-hostenvironment,youneedtoperformthisprocedureononlyasinglehost.Afterperformingthis
procedureonasinglehost,theothertwohostswillbeautomaticallyreset.
Before You Begin
YouhavedeployedtheCiscoAPIC-EMusingtheproceduresdescribedinthisguide.
Step 1 UsingaSecureShell(SSH)client,logintothehost(physicalorvirtual)withtheIPaddressthatyouspecifiedusingthe
configurationwizard.
Note TheIPaddresstoenterfortheSSHclientistheIPaddressthatyouconfiguredforthenetworkadapter.This
IPaddressconnectsthehosttotheexternalnetwork.
Step 2 Whenprompted,enteryourLinuxusername('grapevine')andpasswordforSSHaccess.
Step 3 NavigatetothebindirectoryontheGrapevineroot.Thebindirectorycontainsthegrapevinescripts.
Step 4 Enterthereset_grapevine commandattheprompttoruntheresetgrapevinescript.
$ reset_grapevine
Thereset_grapevinecommandreturnstheconfigurationsettingsbacktovaluesthatyouconfiguredwhenrunningthe
configurationwizardforthefirsttime.Theconfigurationsettingsaresavedtoa.JSONfile.This.JSONfileislocated
at: /etc/grapevine/controller-config.json.Thereset_grapevinecommandusesthedatainthe
controller-config.jsonfiletoreturntotheearlierconfigurationsettings,sodonotdeletethisfile.Ifyoudelete
thisfile,youmustruntheconfigurationwizardagainandreenteryourconfigurationdata.
YouarethenpromptedtoreenteryourGrapevinepassword.
Step 5 EnteryourGrapevinepasswordasecondtime.
[sudo] password for grapevine:********
YouarethenpromptedtodeleteallvirtualdisksThevirtualdisksarewheretheCiscoAPIC-EMdatabaseresides.For
example,dataaboutdevicesthatthecontrollerdiscoveredaresavedonthesevirtualdisks.Ifyouenteryes(y),allof
Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide, Release 1.1.x
10