ACI
Troubleshooting

This section will focus on how to troubleshoot Cisco ACI. It is important to understand that you can troubleshoot ACI using different methods such as:

  • The APIC Graphical User Inteface (GUI)
  • The APIC Command Line Interface (CLI)
  • The Open APIC APIs

Either of the three methods listed above will provide the information needed to diagnose a problem in ACI. In this particular section, we will be providing some important tips on how to troubleshoot ACI and which commands to leverage when someone needs to identify an issue. We will be focusing on the APIC Command Line Interface (CLI).

Step 1 - Connect APIC via SSH

Click on the console ICON on the left to start the connection to the APIC Controller Username: acilabpod04 Password: cisco.123

At this point you should be in the apic prompt.

Step 2 - Fabric Membership

show switch: Provides the information of the entire "switch". Some times we refer to ACI as a "switch" because of the stateless and distributed architecture ACI has, with this command users can identify every single leaf and spine, and their respective information such as:

  • Node IP
  • Pod ID
  • VTEP address
  • In-Band IPv4
  • In-Band IPv6
  • OOB IPv4
  • OOB IPv6
  • Version of Code
  • Flags
  • Serial Number
  • Name

show switch

apic1# show switch
 ID    Pod   Address          In-Band IPv4     In-Band IPv6               OOB IPv4         OOB IPv6                   Version             Flags  Serial Number     Name               
 ----  ----  ---------------  ---------------  -------------------------  ---------------  -------------------------  ------------------  -----  ----------------  ------------------ 
 101   1     10.9.200.129     10.0.236.177     ::                         10.0.226.37      ::                         n9000-16.0(1j)      asiv   FDO221425XC       S1                 
 102   1     10.9.216.160     10.0.236.178     ::                         10.0.226.38      ::                         n9000-16.0(1j)      asiv   FDO221425TL       S2                 
 203   1     10.9.200.130     10.0.236.173     ::                         10.0.226.33      ::                         n9000-16.0(1j)      aliv   FDO21280JKY       L3                 
 204   1     10.9.88.66       10.0.236.174     ::                         10.0.226.34      ::                         n9000-16.0(1j)      aliv   FDO21400SS1       L4                 
 205   1     10.9.200.128     10.0.236.175     ::                         10.0.226.35      ::                         n9000-16.0(1j)      aliv   FDO21360AU2       L5                 
 206   1     10.9.216.161     10.0.236.176     ::                         10.0.226.36      ::                         n9000-16.0(1j)      aliv   FDO21293NG8       L6                 
 207   1     10.9.88.64       0.0.0.0          ::                         10.0.226.110     ::                         n9000-16.0(1j)      aliv   FDO211218CD       L7                 
 208   1     10.9.240.32      0.0.0.0          ::                         10.0.226.111     ::                         n9000-16.0(1j)      aliv   FDO20270CT3       L8                 

Flags - a:Active | l/s:Leaf/Spine | v:Valid Certificate | i:In-Service 

Step 3 - Locating an EndPoint

ACI provides a very easy way to find out where a particular EndPoint is located anywhere in the fabric. The command is called show endpoints which provides valuable information from the EndPoints.

In this particular scenario, we will be troubleshooting POD04-WEB-SRV-02. If you recall from the previous section we left a continuous ping. This was done in order to make sure the EndPoint was still active in the ACI.


show endpoints ip 10.0.144.27

apic1# show endpoints ip 10.0.144.27
Legends:
(P):Primary VLAN
(S):Secondary VLAN


Dynamic Endpoints:
Tenant      : aci_p04_tenant
Application : aci_p04_ap
AEPg        : aci_p04_epg_web

End Point MAC      IP Address                                Source        Node        Interface                       Encap            Multicast Address  Create TS            
-----------------  ----------------------------------------  ------------  ----------  ------------------------------  ---------------  ---------------    -------------------- 
00:50:56:02:01:18  10.0.144.27                              learned,vmm   207 208     vpc aci_p04_intpolg_vpc         vlan-1884        not-applicable     2023-01-26T15:29:44. 
                                                                                                                                                            519+00:00            

Total Dynamic Endpoints: 1
Total Static Endpoints: 0

As you can see POD04-WEB-SRV-02 is behind vPC in leafs 207 208 with the Interface Policy Group aci_p04_intpolg_vpc

The next step is to identify which port(s) are part of this vPC, since the show endpoints ip have already provided the vPC Policy Group aci_p04_intpolg_vpc. Let's identify the ports by doing the following command.


show vpc map aci_p04_intpolg_vpc

apic1# show vpc map aci_p04_intpolg_vpc
Legends:
N/D : Not Deployed

 
 Virtual Port-Channel Name         Domain      Virtual IP        Peer IP           VPC         Leaf Id, Name                     Fex Id      PC Id       Ports                
 --------------------------------  ----------  ----------------  ----------------  ----------  --------------------------------  ----------  ----------  -------------------- 
 aci_p04_intpolg_vpc               207         10.9.104.67/32    10.9.240.32/32    347         208,L8                                        po4         eth1/4              
 aci_p04_intpolg_vpc               207         10.9.104.67/32    10.9.88.64/32     347         207,L7                                        po4         eth1/4

As you can see the port members are eth1/4 in Leaf 207 and eth1/4 on Leaf 208.

Then, you can follow the commands that you have been using on a daily basis such as - show interface, show port-channel, etc. You can execute these Leaf commands directly from the APIC. Here is an sample on how to check the interface eth1/4 on Leaf 207.

You should notice that you will be executing the commands on the leaf from the APIC prompt. There is no need to ssh to Leaf 207 since APIC is able to query Leaf 207.


fabric 207 show interface eth1/4

apic1# fabric 207 show interface eth1/4
----------------------------------------------------------------
 Node 207 (L7)
----------------------------------------------------------------
Ethernet1/4 is up
admin state is up, Dedicated Interface
  Belongs to po4
  Hardware: 100/1000/10000/25000/auto Ethernet, address: 00a3.8ebf.ff00 (bia 00a3.8ebf.ff00)
  MTU 9000 bytes, BW 10000000 Kbit, DLY 1 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast

Step 4 - ACI - VMWare integration commands

In the previous step, we configured ACI - VMWare integration. Because of this valuable integration, ACI has full visibility to the Virtualization environment and can provide information around the Virtual Machines. For example, we can do the following command to understand the characteristics of a VM from where the VM is located to the Guest OS.


show vmware domain name aci_p04_dc3_vds vm name POD04-WEB-SRV-02

apic1# show vmware domain name aci_p04_dc3_vds vm name POD04-WEB-SRV-02
VM Name       : POD04-WEB-SRV-02
DVS           : aci_p04_dc3_vds
vCenter       : 10.0.226.193
Host          : pod04-compute1.ecatsrtpdmz.cisco.com
Guest OS      : CentOS 7 (64-bit)
Configured OS : CentOS 7 (64-bit)
VM OID        : vm-7651
VM GUID       : 500c648e-ae86-9753-4147-45345076d111
Power State   : poweredOn
        
Virtual Nics:
        
Name         : Network adapter 2
Type         : Vmxnet3
MAC          : 00:50:56:02:01:05
IP           : 10.0.144.27
State        : up
Switch       : aci_p04_dc3_vds
Port Group   : aci_p04_tenant|aci_p04_ap|aci_p04_epg_web
Encap        : vlan-551
PrimaryEncap : --
Adjacency    : leafNone aci_p04_intpolg_vpc
        
Name         : Network adapter 1
Type         : Vmxnet3
MAC          : 00:50:56:02:00:05
IP           : 10.0.145.26
State        : up
Switch       : aci_p04_dc3_vds
Port Group   : aci_p04_tenant|aci_p04_ap_mgmt|aci_p04_epg_mgmt
Encap        : vlan-550
PrimaryEncap : --
Adjacency    : leafNone aci_p04_intpolg_vpc

Step 5 - Routing commands

When troubleshooting routing issues, it is important to understant what are you troubleshooting. If you recall, ACI has an Underlay and an Overlay routing table. The Underlay, called Overlay-1, carries the routes of the VTEP IP Addresses. The Overlay carries the Tenant information, in your case it would be aci_p04_tenant.

Check the Underlay (overlay-1) routing table.


fabric 207 show ip route vrf overlay-1

apic1# fabric 207 show ip route vrf overlay-1         
----------------------------------------------------------------
Node 207 (L7)
----------------------------------------------------------------
IP Route Table for VRF "overlay-1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
in via output denotes VRF
        
3.0.0.102/32, ubest/mbest: 1/0
    *via 10.9.216.160, eth1/49.15, [115/2], 01w06d, isis-isis_infra, L1
10.9.0.0/27, ubest/mbest: 1/0, attached, direct
    *via 10.9.0.30, vlan13, [1/0], 02w06d, direct
10.9.0.1/32, ubest/mbest: 2/0
    *via 10.9.216.160, eth1/49.15, [115/12], 02w06d, isis-isis_infra, L1
    *via 10.9.200.129, eth1/50.16, [115/12], 02w05d, isis-isis_infra, L1

Check the routing table for aci_p04_tenant


    fabric 207 show ip route vrf aci_p04_tenant:aci_p04_vrf

apic1# fabric 207 show ip route vrf aci_p04_tenant:aci_p04_vrf
----------------------------------------------------------------
    Node 207 (L7)
----------------------------------------------------------------
IP Route Table for VRF "aci_p4_tenant:aci_p4_vrf"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF 
        
0.0.0.0/0, ubest/mbest: 1/0
    *via 10.9.200.130%overlay-1, [200/1], 08:39:07, bgp-65503, internal, tag 65503
1.1.1.1/32, ubest/mbest: 1/0
    *via 10.9.200.130%overlay-1, [200/13], 08:39:07, bgp-65503, internal, tag 65503
10.0.0.5/32, ubest/mbest: 1/0
    *via 10.9.200.130%overlay-1, [1/0], 08:39:07, bgp-65503, internal, tag 65503
10.0.4.0/31, ubest/mbest: 1/0
    *via 10.9.200.130%overlay-1, [200/12], 08:39:07, bgp-65503, internal, tag 65503
10.0.5.0/31, ubest/mbest: 1/0
    *via 10.9.200.130%overlay-1, [200/0], 08:39:07, bgp-65503, internal, tag 65503
 

Then you can check OSPF as well


fabric 203 show ip ospf neighbors vrf aci_p04_tenant:aci_p04_vrf

apic1# fabric 203 show ip ospf neighbors vrf aci_p04_tenant:aci_p04_vrf 
Total number of neighbors: 1
Neighbor ID     Pri State            Up Time  Address         Interface
10.0.4.1          1 FULL/DR          23:54:20 10.0.4.1        Eth1/4

Step 6 - Troubleshooting Conclusion

As you can see, with ACI, the user can still leverage the same CLI commands that they have been leveraging in the past. More importantly, users can apply the same concepts when it comes to troubleshooting an issue. ACI leverages the same methodology as routing and forwarding as before.

Hopefully this troubleshooting session has provided some clarification on how to troubleshot an issue within the Cisco ACI Fabric. If you still have some specific troubleshooting questions/concerns please contact one of the proctors.