This section will focus on how to troubleshoot Cisco ACI. It is important to understand that you
can troubleshoot ACI using different methods such as:
The APIC Graphical User Inteface (GUI)
The APIC Command Line Interface (CLI)
The Open APIC APIs
Either of the three methods listed above will provide the information needed to diagnose a problem in ACI. In this particular section,
we will be providing some important tips on how to troubleshoot ACI and which commands to leverage when someone needs to identify
an issue. We will be focusing on the APIC Command Line Interface (CLI).
Step 1 - Connect APIC via SSH
Click on the console ICON on the left to start the connection to the APIC Controller Username: acilabpod03 Password: cisco.123
At this point you should be in the apic prompt.
Step 2 - Fabric Membership
show switch: Provides the information of the entire "switch". Some times we refer to ACI
as a "switch" because of the stateless and distributed architecture ACI has, with this command users can identify every
single leaf and spine, and their respective information such as:
ACI provides a very easy way to find out where a particular EndPoint is located anywhere in the fabric. The command is called
show endpoints which provides valuable information from the EndPoints.
In this particular scenario, we will be troubleshooting POD03-WEB-SRV-02. If you recall from the previous section
we left a continuous ping. This was done in order to make sure the EndPoint was still active in the ACI.
show endpoints ip 10.0.144.19
apic1# show endpoints ip 10.0.144.19
Legends:
(P):Primary VLAN
(S):Secondary VLAN
Dynamic Endpoints:
Tenant : aci_p03_tenant
Application : aci_p03_ap
AEPg : aci_p03_epg_web
End Point MAC IP Address Source Node Interface Encap Multicast Address Create TS
----------------- ---------------------------------------- ------------ ---------- ------------------------------ --------------- --------------- --------------------
00:50:56:02:01:18 10.0.144.19 learned,vmm 207 208 vpc aci_p03_intpolg_vpc vlan-1884 not-applicable 2023-01-26T15:29:44.
519+00:00
Total Dynamic Endpoints: 1
Total Static Endpoints: 0
As you can see POD03-WEB-SRV-02 is behind vPC in leafs 207 208
with the Interface Policy Group aci_p03_intpolg_vpc
The next step is to identify which port(s) are part of this vPC, since the show endpoints ip have already provided
the vPC Policy Group aci_p03_intpolg_vpc. Let's identify the ports by doing the following command.
show vpc map aci_p03_intpolg_vpc
apic1# show vpc map aci_p03_intpolg_vpc
Legends:
N/D : Not Deployed
Virtual Port-Channel Name Domain Virtual IP Peer IP VPC Leaf Id, Name Fex Id PC Id Ports
-------------------------------- ---------- ---------------- ---------------- ---------- -------------------------------- ---------- ---------- --------------------
aci_p03_intpolg_vpc 207 10.9.104.67/32 10.9.240.32/32 347 208,L8 po4 eth1/3
aci_p03_intpolg_vpc 207 10.9.104.67/32 10.9.88.64/32 347 207,L7 po4 eth1/3
As you can see the port members are eth1/3 in Leaf 207 and eth1/3
on Leaf 208.
Then, you can follow the commands that you have been using on a daily basis such as - show interface, show port-channel, etc. You can execute these Leaf
commands
directly from the APIC. Here is an sample on how to check the interface eth1/3 on Leaf 207.
You should notice that you will be executing the commands on the leaf from the APIC prompt. There is no need to ssh to Leaf 207 since APIC is able
to query Leaf 207.
fabric 207 show interface eth1/3
apic1# fabric 207 show interface eth1/3
----------------------------------------------------------------
Node 207 (L7)
----------------------------------------------------------------
Ethernet1/3 is up
admin state is up, Dedicated Interface
Belongs to po4
Hardware: 100/1000/10000/25000/auto Ethernet, address: 00a3.8ebf.ff00 (bia 00a3.8ebf.ff00)
MTU 9000 bytes, BW 10000000 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Step 4 - ACI - VMWare integration commands
In the previous step, we configured ACI - VMWare integration. Because of this valuable integration, ACI has full
visibility to the Virtualization environment and can provide information around the Virtual Machines. For example,
we can do the following command to understand the characteristics of a VM from where the VM is located to the Guest
OS.
show vmware domain name aci_p03_dc3_vds vm name POD03-WEB-SRV-02
apic1# show vmware domain name aci_p03_dc3_vds vm name POD03-WEB-SRV-02
VM Name : POD03-WEB-SRV-02
DVS : aci_p03_dc3_vds
vCenter : 10.0.226.193
Host : pod03-compute1.ecatsrtpdmz.cisco.com
Guest OS : CentOS 7 (64-bit)
Configured OS : CentOS 7 (64-bit)
VM OID : vm-7651
VM GUID : 500c648e-ae86-9753-4147-45345076d111
Power State : poweredOn
Virtual Nics:
Name : Network adapter 2
Type : Vmxnet3
MAC : 00:50:56:02:01:05
IP : 10.0.144.19
State : up
Switch : aci_p03_dc3_vds
Port Group : aci_p03_tenant|aci_p03_ap|aci_p03_epg_web
Encap : vlan-551
PrimaryEncap : --
Adjacency : leafNone aci_p03_intpolg_vpc
Name : Network adapter 1
Type : Vmxnet3
MAC : 00:50:56:02:00:05
IP : 10.0.145.18
State : up
Switch : aci_p03_dc3_vds
Port Group : aci_p03_tenant|aci_p03_ap_mgmt|aci_p03_epg_mgmt
Encap : vlan-550
PrimaryEncap : --
Adjacency : leafNone aci_p03_intpolg_vpc
Step 5 - Routing commands
When troubleshooting routing issues, it is important to understant what are you troubleshooting. If you recall,
ACI has an Underlay and an Overlay routing table. The Underlay, called Overlay-1, carries the routes of the VTEP
IP Addresses. The Overlay carries the Tenant information, in your case it would be aci_p03_tenant.
Check the Underlay (overlay-1) routing table.
fabric 207 show ip route vrf overlay-1
apic1# fabric 207 show ip route vrf overlay-1
----------------------------------------------------------------
Node 207 (L7)
----------------------------------------------------------------
IP Route Table for VRF "overlay-1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
in via output denotes VRF
3.0.0.102/32, ubest/mbest: 1/0
*via 10.9.216.160, eth1/49.15, [115/2], 01w06d, isis-isis_infra, L1
10.9.0.0/27, ubest/mbest: 1/0, attached, direct
*via 10.9.0.30, vlan13, [1/0], 02w06d, direct
10.9.0.1/32, ubest/mbest: 2/0
*via 10.9.216.160, eth1/49.15, [115/12], 02w06d, isis-isis_infra, L1
*via 10.9.200.129, eth1/50.16, [115/12], 02w05d, isis-isis_infra, L1
Check the routing table for aci_p03_tenant
fabric 207 show ip route vrf aci_p03_tenant:aci_p03_vrf
apic1# fabric 207 show ip route vrf aci_p03_tenant:aci_p03_vrf
----------------------------------------------------------------
Node 207 (L7)
----------------------------------------------------------------
IP Route Table for VRF "aci_p3_tenant:aci_p3_vrf"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
0.0.0.0/0, ubest/mbest: 1/0
*via 10.9.200.130%overlay-1, [200/1], 08:39:07, bgp-65503, internal, tag 65503
1.1.1.1/32, ubest/mbest: 1/0
*via 10.9.200.130%overlay-1, [200/13], 08:39:07, bgp-65503, internal, tag 65503
10.0.0.5/32, ubest/mbest: 1/0
*via 10.9.200.130%overlay-1, [1/0], 08:39:07, bgp-65503, internal, tag 65503
10.0.4.0/31, ubest/mbest: 1/0
*via 10.9.200.130%overlay-1, [200/12], 08:39:07, bgp-65503, internal, tag 65503
10.0.5.0/31, ubest/mbest: 1/0
*via 10.9.200.130%overlay-1, [200/0], 08:39:07, bgp-65503, internal, tag 65503
Then you can check OSPF as well
fabric 203 show ip ospf neighbors vrf aci_p03_tenant:aci_p03_vrf
apic1# fabric 203 show ip ospf neighbors vrf aci_p03_tenant:aci_p03_vrf
Total number of neighbors: 1
Neighbor ID Pri State Up Time Address Interface
10.0.3.1 1 FULL/DR 23:54:20 10.0.3.1 Eth1/3
Step 6 - Troubleshooting Conclusion
As you can see, with ACI, the user can still leverage the same CLI commands that they have been leveraging
in the past. More importantly, users can apply the same concepts when it comes to troubleshooting an issue. ACI
leverages the same methodology as routing and forwarding as before.
Hopefully this troubleshooting session has provided some clarification on how to troubleshot an issue within the Cisco ACI
Fabric. If you still have some specific troubleshooting questions/concerns please contact one of the proctors.
