UCS FI 6332 Unable to login.

[elepeshko]

Hello Community!
I have a cluster of FI 6332 System version: 4.0(4n):

A: UP, SUBORDINATE
B: UP, PRIMARY 

And I have an error trying to ssh primary B or vip ip:

Unable to login. You have reached the maximum allowed number of concurrent sessions. Please clear shell sessions from UCSM CLI/GUI. 

The reason is obvious for me, our NMS utilized all 32 allowed shell connections, we cleared them in UCSM GUI. But, somehow, they got stuck on the primary fabric B. I don't see sessions in GUI,  but I do see them on FI:

A: connect local-mgmt B

B(local-mgmt)# show processes | i ciscoprime
403 root 20 0 154m 100m 69m S 0.0 0.3 0:00.59 /isan/bin/ucssh --ucs-mgmt -p ciscoprime
408 root 20 0 154m 100m 69m S 0.0 0.3 0:00.60 /isan/bin/ucssh --ucs-mgmt -p ciscoprime
1046 root 20 0 154m 100m 69m S 0.0 0.3 0:00.58 /isan/bin/ucssh --ucs-mgmt -p ciscoprime
1391 root 20 0 154m 100m 69m S 0.0 0.3 0:00.56 /isan/bin/ucssh --ucs-mgmt -p ciscoprime
1500 root 20 0 154m 100m 69m S 0.0 0.3 0:00.58 /isan/bin/ucssh --ucs-mgmt -p ciscoprime
2189 root 20 0 154m 100m 69m S 0.0 0.3 0:00.60 /isan/bin/ucssh --ucs-mgmt -p ciscoprime
2771 root 20 0 154m 100m 69m S 0.0 0.3 0:00.57 /isan/bin/ucssh --ucs-mgmt -p ciscoprime
2990 root 20 0 154m 100m 69m S 0.0 0.3 0:00.57 /isan/bin/ucssh --ucs-mgmt -p ciscoprime
3300 root 20 0 154m 100m 69m S 0.0 0.3 0:00.58 /isan/bin/ucssh --ucs-mgmt -p ciscoprime
3394 root 20 0 154m 100m 69m S 0.0 0.3 0:00.58 /isan/bin/ucssh --ucs-mgmt -p ciscoprime
3559 root 20 0 154m 100m 69m S 0.0 0.3 0:00.58 /isan/bin/ucssh --ucs-mgmt -p ciscoprime
4312 root 20 0 154m 100m 69m S 0.0 0.3 0:00.57 /isan/bin/ucssh --ucs-mgmt -p ciscoprime ...

I need help to clear all these hung processes to restore ssh access.

I have no option to make management switchover to A (cluster lead a), as I can't access primary FI B.
And I dont see hung sessions from UCSM GUI.  Is there any chance to access shell of UCS FI to kill hung ssh sessions? Appretiate any advice to resolve the issue.

[Steven Tardy · Accepted solution]

This sounds like the opposite of old CDET CSCvm68038 where SSH on the subordinate FI would have too many open SSH sessions as this is on the primary FI.

That CDET details (temporarily) enabling telnet to access the "inaccessible FI" and then accessing a debug shell (which requires TAC) to stop outstanding SSH processes.

But judging by the snippet you posted, it seems ciscoprime is SSH'ing in to the UCS FI. . . .
Can ciscoprime be temporarily disabled to see if the FI recovers SSH ability after a little time.

[elepeshko]

Appreciate your feedback, Steven.
We managed to connect to Primary with console, executed "clusted lead A". Primary is alive now.
I disabled ciscoprime for a while, but FI-B still unavailable, stuck processes still there...

Is there any chanse to clear hung process without TAC? Or my option is to evacuate the FI and reboot it?

[elepeshko]

We Finally rebooted the FI. No way to acceess debug shell without TAC.