Cisco 9300L stack failing to learn MAC addresses dynamically

[Philip D'Ath]

I recently replaced a customer's 3750 stack with a 9300L stack (C9300L-24UXG-4X).  Attached to this stack are two HP blade centres.  About 24 hours after the new stack was in, the customer started experiencing intermittent packet loss to both blade centres.

What I discovered is the 9300s have stopped learning dynamic MAC addresses from both sets of blade centres.  Dynamic MAC learning is working fine everywhere else.

show mac address-table dynamic int ten2/1/1
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----

 This is causing all traffic to the servers to be flooded.

I checked spanning tree, and it is stable and had no topology changes for 12 hours.

There are no errors in the logs.  No loops reported.  Nothing.

I did a packet capture on the port from the switch, and I can see lots of MAC addresses from all the servers.

One blade centre is attached via 10GBase-SR fibre, the other via TwinAx.  Because different ports, cables and cable types are being used, I don't think this is a physical layer issue.

Any idea what else would prevent dynamic MAC address learning on a port?

[adarsing]

What is the version?

[Philip D'Ath]

17.09.04

[adarsing]

seems hitting the bug CSCwh55629

[Philip D'Ath]

The TAC engineer doesn't think it is related because we aren't using 802.1x.  But it is exactly the same problem - so perhaps dot1x is not needed to make this big happen.

[thomastd]

Maybe do a show switch to see if all the members are there particularly the switch that the blades are plugged into?   Show ip int brief to make sure that you have port numbering that matches the switches you have online.

[Philip D'Ath]

Everything shows as online and happy.

show switch 
Switch/Stack Mac Address : c47e.xxxx.xxxx - Local Mac Address
Mac persistency wait time: Indefinite
                                             H/W   Current
Switch#   Role    Mac Address     Priority Version  State 
-------------------------------------------------------------------------------------
*1       Active   c47e.xxxx.xxxx     10     V03     Ready                               
 2       Standby  c47e.xxxx.xxxx     8      V03     Ready                               
 3       Member   c47e.xxxx.xxxx     6      V03     Ready                               

  The port number matches up. 

[balaji.bandi]

One blade centre is attached via 10GBase-SR fibre, the other via TwinAx

did you mixed both to One Port-channel , have removed Twinax cable and shut and no shut and check.

what is the logs show on switch ?  you not learning on MAC address only that port, what is the outcome when you show mac address table 

Did you captured before how many mac Address switch were learning before.

post below output :

#show platform hardware fed switch active fwd-asic resource tcam utilization

# show platform software fed switch active ifm mappings

#show run

[Philip D'Ath]

>did you mixed both to One Port-channel 

No.

> have removed Twinax cable and shut and no shut and check.

Yes.  Also tried swapped out a 10GBase-SR SFP+ just in case.

>what is the logs show on switch

Nothing appears in the log that is related.

>what is the outcome when you show mac address table 

Lots of MAC addresses have been learned on other ports.  Nothing shows up against these ports.

show platform hardware fed switch active fwd-asic resource tcam utilization
Codes: EM - Exact_Match, I - Input, O - Output, IO - Input & Output, NA - Not Applicable

CAM Utilization for ASIC  [0]
 Table                  Subtype      Dir      Max     Used    %Used       V4       V6     MPLS    Other
 ------------------------------------------------------------------------------------------------------
 Mac Address Table      EM           I       32768      290    0.89%        0        0        0      290
 Mac Address Table      TCAM         I        1024       22    2.15%        0        0        0       22
 L3 Multicast           EM           I        8192        0    0.00%        0        0        0        0
 L3 Multicast           TCAM         I         512        9    1.76%        3        6        0        0
 L2 Multicast           EM           I        8192        0    0.00%        0        0        0        0
 L2 Multicast           TCAM         I         512       11    2.15%        3        8        0        0
 IP Route Table         EM           I       24576      449    1.83%      448        0        1        0
 IP Route Table         TCAM         I        8192      159    1.94%      146       10        2        1
 QOS ACL                TCAM         IO       5120       85    1.66%       28       38        0       19
                        TCAM         I                   45    0.88%       15       20        0       10
                        TCAM         O                   40    0.78%       13       18        0        9
 Security ACL           TCAM         IO       5120      167    3.26%       26       96        0       45
                        TCAM         I                  124    2.42%       12       72        0       40
                        TCAM         O                   43    0.84%       14       24        0        5
 Netflow ACL            TCAM         I         256       22    8.59%       10       10        0        2
 PBR ACL                TCAM         I        1024       36    3.52%       30        6        0        0
 Netflow ACL            TCAM         O         768       44    5.73%       16       26        0        2
 Flow SPAN ACL          TCAM         IO       1024       13    1.27%        3        6        0        4
                        TCAM         I                    5    0.49%        1        2        0        2
                        TCAM         O                    8    0.78%        2        4        0        2
 Control Plane          TCAM         I         512      290   56.64%      138      106        0       46
 Tunnel Termination     TCAM         I         512       20    3.91%        8       12        0        0
 Lisp Inst Mapping      TCAM         I        2048        1    0.05%        0        0        0        1
 Security Association   TCAM         I         256        4    1.56%        2        2        0        0
 CTS Cell Matrix/VPN
 Label                  EM           O        8192        0    0.00%        0        0        0        0
 CTS Cell Matrix/VPN
 Label                  TCAM         O         512        1    0.20%        0        0        0        1
 Client Table           EM           I        4096        0    0.00%        0        0        0        0
 Client Table           TCAM         I         256        0    0.00%        0        0        0        0
 Input Group LE         TCAM         I        1024        0    0.00%        0        0        0        0
 Output Group LE        TCAM         O        1024        0    0.00%        0        0        0        0
 Macsec SPD             TCAM         I         256        2    0.78%        0        0        0        2

show platform software fed switch active ifm mappings
Interface                 IF_ID    Inst Asic Core Port SubPort Mac  Cntx LPN  GPN  Type Active
GigabitEthernet1/0/1      0x9        1   0   1    0      0      22   2    1    1    NIF  Y
GigabitEthernet1/0/2      0xa        1   0   1    1      0      7    3    2    2    NIF  Y
GigabitEthernet1/0/3      0xb        1   0   1    2      0      24   4    3    3    NIF  Y
GigabitEthernet1/0/4      0xc        1   0   1    3      0      23   5    4    4    NIF  Y
GigabitEthernet1/0/5      0xd        1   0   1    4      0      26   6    5    5    NIF  Y
GigabitEthernet1/0/6      0xe        1   0   1    5      0      25   7    6    6    NIF  Y
GigabitEthernet1/0/7      0xf        1   0   1    6      0      28   8    7    7    NIF  Y
GigabitEthernet1/0/8      0x10       1   0   1    7      0      27   9    8    8    NIF  Y
GigabitEthernet1/0/9      0x11       1   0   1    8      0      15   10   9    9    NIF  Y
GigabitEthernet1/0/10     0x12       1   0   1    9      0      6    11   10   10   NIF  Y
GigabitEthernet1/0/11     0x13       1   0   1    10     0      17   12   11   11   NIF  Y
GigabitEthernet1/0/12     0x14       1   0   1    11     0      16   13   12   12   NIF  Y
GigabitEthernet1/0/13     0x15       1   0   1    12     0      19   14   13   13   NIF  Y
GigabitEthernet1/0/14     0x16       1   0   1    13     0      18   15   14   14   NIF  Y
GigabitEthernet1/0/15     0x17       1   0   1    14     0      21   16   15   15   NIF  Y
GigabitEthernet1/0/16     0x18       1   0   1    15     0      20   17   16   16   NIF  Y
TenGigabitEthernet1/0/17  0x19       1   0   1    16     0      4    18   17   17   NIF  Y
TenGigabitEthernet1/0/18  0x1a       1   0   1    17     0      5    19   18   18   NIF  Y
TenGigabitEthernet1/0/19  0x1b       1   0   1    18     0      2    20   19   19   NIF  Y
TenGigabitEthernet1/0/20  0x1c       1   0   1    19     0      3    21   20   20   NIF  Y
TenGigabitEthernet1/0/21  0x1d       0   0   0    20     0      6    2    21   21   NIF  Y
TenGigabitEthernet1/0/22  0x1e       0   0   0    21     0      7    3    22   22   NIF  Y
TenGigabitEthernet1/0/23  0x1f       0   0   0    22     0      4    4    23   23   NIF  Y
TenGigabitEthernet1/0/24  0x20       0   0   0    23     0      5    5    24   24   NIF  Y
TenGigabitEthernet1/1/1   0x21       1   0   1    24     0      1    1    25   1545 NIF  Y
TenGigabitEthernet1/1/2   0x22       1   0   1    25     0      0    0    26   26   NIF  Y
TenGigabitEthernet1/1/3   0x23       0   0   0    26     0      1    1    27   1561 NIF  Y
TenGigabitEthernet1/1/4   0x24       0   0   0    27     0      0    0    28   1556 NIF  Y
AppGigabitEthernet1/0/1   0x25       1   0   1    28     0      29   22   29   29   NIF  Y

show platform software fed switch 2  ifm mappings
Interface                 IF_ID    Inst Asic Core Port SubPort Mac  Cntx LPN  GPN  Type Active
GigabitEthernet2/0/1      0x27       1   0   1    0      0      22   2    1    97   NIF  Y
GigabitEthernet2/0/2      0x28       1   0   1    1      0      7    3    2    98   NIF  Y
GigabitEthernet2/0/3      0x29       1   0   1    2      0      24   4    3    99   NIF  Y
GigabitEthernet2/0/4      0x2a       1   0   1    3      0      23   5    4    100  NIF  Y
GigabitEthernet2/0/5      0x2b       1   0   1    4      0      26   6    5    101  NIF  Y
GigabitEthernet2/0/6      0x2c       1   0   1    5      0      25   7    6    102  NIF  Y
GigabitEthernet2/0/7      0x2d       1   0   1    6      0      28   8    7    103  NIF  Y
GigabitEthernet2/0/8      0x2e       1   0   1    7      0      27   9    8    104  NIF  Y
GigabitEthernet2/0/9      0x2f       1   0   1    8      0      15   10   9    105  NIF  Y
GigabitEthernet2/0/10     0x30       1   0   1    9      0      6    11   10   106  NIF  Y
GigabitEthernet2/0/11     0x31       1   0   1    10     0      17   12   11   107  NIF  Y
GigabitEthernet2/0/12     0x32       1   0   1    11     0      16   13   12   108  NIF  Y
GigabitEthernet2/0/13     0x33       1   0   1    12     0      19   14   13   109  NIF  Y
GigabitEthernet2/0/14     0x34       1   0   1    13     0      18   15   14   110  NIF  Y
GigabitEthernet2/0/15     0x35       1   0   1    14     0      21   16   15   111  NIF  Y
GigabitEthernet2/0/16     0x36       1   0   1    15     0      20   17   16   112  NIF  Y
TenGigabitEthernet2/0/17  0x37       1   0   1    16     0      4    18   17   113  NIF  Y
TenGigabitEthernet2/0/18  0x38       1   0   1    17     0      5    19   18   114  NIF  Y
TenGigabitEthernet2/0/19  0x39       1   0   1    18     0      2    20   19   115  NIF  Y
TenGigabitEthernet2/0/20  0x3a       1   0   1    19     0      3    21   20   116  NIF  Y
TenGigabitEthernet2/0/21  0x3b       0   0   0    20     0      6    2    21   117  NIF  Y
TenGigabitEthernet2/0/22  0x3c       0   0   0    21     0      7    3    22   118  NIF  Y
TenGigabitEthernet2/0/23  0x3d       0   0   0    22     0      4    4    23   119  NIF  Y
TenGigabitEthernet2/0/24  0x3e       0   0   0    23     0      5    5    24   120  NIF  Y
TenGigabitEthernet2/1/1   0x3f       1   0   1    24     0      1    1    25   121  NIF  Y
TenGigabitEthernet2/1/2   0x40       1   0   1    25     0      0    0    26   122  NIF  Y
TenGigabitEthernet2/1/3   0x41       0   0   0    26     0      1    1    27   123  NIF  Y
TenGigabitEthernet2/1/4   0x42       0   0   0    27     0      0    0    28   124  NIF  Y
AppGigabitEthernet2/0/1   0x43       1   0   1    28     0      29   22   29   125  NIF  Y

show platform software fed switch 3  ifm mappings
Interface                 IF_ID    Inst Asic Core Port SubPort Mac  Cntx LPN  GPN  Type Active
GigabitEthernet3/0/1      0x45       1   0   1    0      0      22   2    1    193  NIF  Y
GigabitEthernet3/0/2      0x46       1   0   1    1      0      7    3    2    194  NIF  Y
GigabitEthernet3/0/3      0x47       1   0   1    2      0      24   4    3    195  NIF  Y
GigabitEthernet3/0/4      0x48       1   0   1    3      0      23   5    4    196  NIF  Y
GigabitEthernet3/0/5      0x49       1   0   1    4      0      26   6    5    197  NIF  Y
GigabitEthernet3/0/6      0x4a       1   0   1    5      0      25   7    6    198  NIF  Y
GigabitEthernet3/0/7      0x4b       1   0   1    6      0      28   8    7    199  NIF  Y
GigabitEthernet3/0/8      0x4c       1   0   1    7      0      27   9    8    200  NIF  Y
GigabitEthernet3/0/9      0x4d       1   0   1    8      0      15   10   9    201  NIF  Y
GigabitEthernet3/0/10     0x4e       1   0   1    9      0      6    11   10   202  NIF  Y
GigabitEthernet3/0/11     0x4f       1   0   1    10     0      17   12   11   203  NIF  Y
GigabitEthernet3/0/12     0x50       1   0   1    11     0      16   13   12   204  NIF  Y
GigabitEthernet3/0/13     0x51       1   0   1    12     0      19   14   13   205  NIF  Y
GigabitEthernet3/0/14     0x52       1   0   1    13     0      18   15   14   206  NIF  Y
GigabitEthernet3/0/15     0x53       1   0   1    14     0      21   16   15   207  NIF  Y
GigabitEthernet3/0/16     0x54       1   0   1    15     0      20   17   16   208  NIF  Y
TenGigabitEthernet3/0/17  0x55       1   0   1    16     0      4    18   17   209  NIF  Y
TenGigabitEthernet3/0/18  0x56       1   0   1    17     0      5    19   18   210  NIF  Y
TenGigabitEthernet3/0/19  0x57       1   0   1    18     0      2    20   19   211  NIF  Y
TenGigabitEthernet3/0/20  0x58       1   0   1    19     0      3    21   20   212  NIF  Y
TenGigabitEthernet3/0/21  0x59       0   0   0    20     0      6    2    21   213  NIF  Y
TenGigabitEthernet3/0/22  0x5a       0   0   0    21     0      7    3    22   214  NIF  Y
TenGigabitEthernet3/0/23  0x5b       0   0   0    22     0      4    4    23   215  NIF  Y
TenGigabitEthernet3/0/24  0x5c       0   0   0    23     0      5    5    24   216  NIF  Y
TenGigabitEthernet3/1/1   0x5d       1   0   1    24     0      1    1    25   217  NIF  Y
TenGigabitEthernet3/1/2   0x5e       1   0   1    25     0      0    0    26   1546 NIF  Y
TenGigabitEthernet3/1/3   0x5f       0   0   0    26     0      1    1    27   1540 NIF  Y
TenGigabitEthernet3/1/4   0x60       0   0   0    27     0      0    0    28   220  NIF  Y
AppGigabitEthernet3/0/1   0x61       1   0   1    28     0      29   22   29   221  NIF  Y

[Philip D'Ath]

If you take one of the interfaces with the problem, Ten2/1/2, you can see many inbound and outbound packets.

show int tenGigabitEthernet 2/1/2
TenGigabitEthernet2/1/2 is up, line protocol is up (connected) 
  Hardware is Ten Gigabit Ethernet, address is c47e.e06a.aa9a (bia c47e.e06a.aa9a)
  Description: HP C7K VC-2 X1 - Legacy
  MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 26/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
  input flow-control is on, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:02, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/7 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 7454000 bits/sec, 14156 packets/sec
  5 minute output rate 1052015000 bits/sec, 91020 packets/sec
     1223347071 packets input, 446570809672 bytes, 0 no buffer
     Received 326845 broadcasts (178954 multicasts)
     0 runts, 0 giants, 0 throttles 
     30 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 178954 multicast, 0 pause input
     0 input packets with dribble condition detected
     7697748680 packets output, 9534605929148 bytes, 0 underruns
     Output 35022872 broadcasts (184700495 multicasts)
     0 output errors, 0 collisions, 16 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

It should be learning the MAC addresses from the inbound packets.

[Philip D'Ath]

Thank you everyone for your help.

I ended up getting TAC to assist.  While they were looking at the switch, one of the stack members did an un-commanded reboot.

After the reboot, the switch was able to learn dynamic MAC addresses again.

[balaji.bandi]

Oh my god what a tragedy here...glad all good in end.