Recently encountered an 802.1x onboarding issue to an SDA fabric after upgrading IOS's to anything above 16.12.4 on multiple 3K/9K platforms. This one took some time to resolve. Here were the following issues:
The SDA switch config, 802.1x config, supplicant configs all work with the production IOS (16.12.4) across all of our 3k/9k platforms. We also tested 16.12.3S IOS on multiple platforms, which was also successful with the configurations.
To summarize the following worked in regard to 802.1x onboarding:
-Win10 AnyConnect 4.905042 NAM Supplicant with EAP-FAST(EAP-TLS) on 16.12.3S
-Win10 AnyConnect 4.905042 NAM Supplicant with EAP-FAST(EAP-TLS) on 16.12.4
And the following did not work until an MTU change (described later):
16.12.5/16.12.5B/17.3.3 IOS's do not work with our SDA switch configs, 802.1x, & supplicant configs. Below is a list of tests conducted to aide in troubleshooting while working with TAC (all did not work at first):
-Win10 Native Supplicant with PEAP(EAP-TLS) on both 16.12.5/16.12.5B
-Win10 AnyConnect 4.905042 NAM Supplicant with EAP-FAST(EAP-TLS) on all - 16.12.5/16.12.5B/17.3.3
-Win10 AnyConnect 4.10.00093 NAM Supplicant with EAP-FAST(EAP-TLS) on all - 16.12.5/16.12.5B/17.3.3
Performed captures the following ways:
-Spanned client ports from both Cat9k/3k platforms
-Wireshark captures from the clients we were testing onboarding with
-Multiple ISE TCP Dumps
-Checked NAM profile configurations
-Checked switch configs between a known good with 16.12.4 IOS and non-working equipment running the following IOS's 16.12.5/16.12.5b/17.3.3.
-Client PCAP files depicted EAPOL START packets every time
-Switch debugs always showed this across all platforms/non-working IOS's:
%DOT1X-5-FAIL: Switch 1 R0/0: sessmgrd: Authentication failed for client (e4b9.XXXX.a39d) with reason (Timeout) on Interface Tw1/0/3 AuditSessionID 49CA070A00000018480CDAA8 Username: host/XXXX
-ISE Radius Live logs always depicts this:
5440 Endpoint abandoned EAP session and started new
TAC provided us these bugs to test workarounds (Bug Search (cisco.com))(https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvy09607). At first it did not resolve the issue since the MTU change we tested was local to the client on it's respective SVI. Some other internal folks suggested changing the MTU size on the SVI where ISE connects to to reach the fabric underlay (ip mtu 1500). This actually ended up fixing all issues across the board. Note that in this scenario ISE has a underlay backend connection via the EBNs so the change was conducted on them.
Not sure if anyone else has run into this, but wanted to share as a recent IOS upgrade to half the fabric ENs prevented clients from onboarding via 802.1x (no issues with mab). However, modifying the SVI on the ISE side (EBN SVIs in this case) did in fact fix the issues since ISE does not support jumbo. For the record ISE version is 2.7p3. Also, since SDA inception with this customer there have been no similar issues relating to 802.1x onboarding & fabric mtu. Hopefully this saves time for anyone else planning on upgrades or potentially hitting this issue.
