Solved: Cisco UCS C3260 Storage Server - firmware upgrade failed, recovery

Vasyl · ‎02-26-2023

Hello All,

Sounds like I have broken/bricked UCS C3260 M4 chassis/nodes with firmware upgrade.

I did not complete full upgrade and mistakenly rebooted the chassis while BMC and Raid controllers were only updated with ucs-c3260-huu-2.0.13n firmware.

Now it is constantly rebooting. Earlier it able to power on the screen/display, now even screen is not coming up, blank.
From the serial debug console, I can see it is loading U-Boot and hangs right before the command prompt, or within a few seconds of giving that prompt.

Any help? I was unable to find any instructions for firmware recovery for this specific chassis.

Highly appreciate your help

========= Serial console output =========

U-Boot 2011.12 (Sep 23 2019 - 11:31:42) Cisco Systems, Build: jenkins-Component-Builds-Rack-Patch Builds-patch_bmc_rel_ironcanyon_hp_mr3_bld67-83

CPU0: P1020E, Version: 1.1, (0x80ec0011)
Core: E500, Version: 5.1, (0x80212051)
Clock Configuration:
CPU0:533.333 MHz, CPU1:533.333 MHz,
CCB:266.667 MHz,
DDR:333.333 MHz (666.667 MT/s data rate) (Asynchronous), LBC:16.667 MHz
L1: D-cache 32 kB enabled
I-cache 32 kB enabled
[MCPSUMR 0x00000000, RSTRSCR 0x00000000, AUTORSTSR 0x0000c000]
I2C buses: divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
ready
DRAM: Configuring DDR for 666.667 MT/s data rate
DDR configuration get done
512 MiB (DDR3, 32-bit, CL=6, ECC on)
U-boot retry count 1Self checking ...
retrieving MAC addresses from EEPROM Internal Area
NIOM's FRU MAC's are
00:FC:BA:3B:DD:54
00:FC:BA:3B:DD:55
Memory test from 0x40000 to 0x1fdfffff
Data line test................ OK
Address line test............. OK
OK
Flash: 288 MiB
L2: 256 KB enabled
*** Warning - bad CRC, using default environment

Set dbglevel to its default value (0x1)
PCIe1: Root Complex of mini PCIe SLOT, no link, regs @ 0xffe0a000
PCIe1: Bus 00 - 00
PCIe2: Root Complex of PCIe SLOT, no link, regs @ 0xffe09000
PCIe2: Bus 01 - 01
divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
VIC card present = 1
Board: COLUSA2IOM_A0
uboot version 2.23
colusa2iom rev: 4
CMC_SLOT_ID: 0
Net: eTSEC2 is in sgmii mode.
eTSEC1, eTSEC2
I2C Clock is 105 KHz
0
Auto-boot image 2
Boot string: "linux2boot"
. it's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBComputed Hash SHA2: 400a5e2916daaca0779f39b261a2202cfe0d8644878bfeeb19ad90301a0e49fcef7744fdd8f4e4ac4a84a42c1a18dfa046ba66fc2fe929953c677b89d2680c5e

Embedded Hash SHA2: 400a5e2916daaca0779f39b261a2202cfe0d8644878bfeeb19ad90301a0e49fcef7744fdd8f4e4ac4a84a42c1a18dfa046ba66fc2fe929953c677b89d2680c5e

Signature verification was successf. it's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIB## Booting kernel from Legacy Image at 01000000 ...
Image Name: Linux-3.18.7
Created: 2019-12-18 11:53:32 UTC
Image Type: PowerPC Linux Kernel Image (gzip compressed)
Data Size: 9603611 Bytes = 9.2 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
## Flattened Device Tree blob at 00e00000
Booting using the fdt blob at 0x00e00000
Uncompressing Kernel Image ... OK
Loading Device Tree to 03ffa000, end 03fff982 ... OK
setup_arch: bootmem
fex_setup_arch()
arch: exit

Mounting application filesystems: [ OK ]
Locating OBFL filesystem: [ OK ]
Mounting OBFL filesystem: [ OK ]
Mounting Config filesystem: [ OK ]
Setting up /etc/network: [ OK ]
Locating system image: [ OK ]
Unpacking Base FS: [ OK ]
Unpacking CMC FS: [ OK ]
/mnt/jffs2/avct_ems_cfg/BMC_Data....Setting up /flash/data0[ OK ]
Creating link /usr/bin/awk ... [ OK ]
Determining password file ...root: security_manager_setup: avctpasswd_userdb.ini is the current user database file
[ OK ]
kernel.core_uses_pid = 1
kernel.core_pattern = |/nuova/bin/zcore /var/cmc/core/%t_%e_core.%p-%s
vm.panic_on_oom = 1
vm.oom_dump_tasks = 1
net.ipv4.conf.all.route_localnet = 1
net.core.wmem_max = 524288
net.core.rmem_max = 524288
Starting internet superserver: inetd [ OK ]
Setting up networking: [ OK ]
CMC slot 0
Starting network interfaces: ifconfig: SIOCSIFHWADDR: Device or resource busy
[ OK ]
NOT starting OBFL logger
Starting PMON: [ OK ]
Running carduil -d
Discovered 1 Cisco cards.
[ OK ]

Please press Enter to activate this console.
CMC Debug Firmware Utility Shell [ root ]
[ help ]#

U-Boot 2011.12 (Sep 23 2019 - 11:31:42) Cisco Systems, Build: jenkins-Component-Builds-Rack-Patch Builds-patch_bmc_rel_ironcanyon_hp_mr3_bld67-83

CPU0: P1020E, Version: 1.1, (0x80ec0011)
Core: E500, Version: 5.1, (0x80212051)
Clock Configuration:
CPU0:533.333 MHz, CPU1:533.333 MHz,
CCB:266.667 MHz,
DDR:333.333 MHz (666.667 MT/s data rate) (Asynchronous), LBC:16.667 MHz
L1: D-cache 32 kB enabled
I-cache 32 kB enabled

<cut, repeating boot process here>

[ help ]# __________________________________________
Debug Firmware Utility
__________________________________________
Command List
__________________________________________

alarms
cartridges
cms
cores
exit
fans
gpio
i2c
images
led
mctools
memory
messages
network
obfl
ohms
post
programmables
psu
tasks
temperatures
thermal
top
update
voltages
version
fru
sel
sensors
cli
i2cstats
help [COMMAND]
__________________________________________
Notes:
"enter Key" will execute last command
"COMMAND ?" will execute help for that command
__________________________________________

U-Boot 2011.12 (Sep 23 2019 - 11:31:42) Cisco Systems, Build: jenkins-Component-Builds-Rack-Patch Builds-patch_bmc_rel_ironcanyon_hp_mr3_bld67-83

CPU0: P1020E, Version: 1.1, (0x80ec0011)
Core: E500, Version: 5.1, (0x80212051)
Clock Configuration:
CPU0:533.333 MHz, CPU1:533.333 MHz,
CCB:266.667 MHz,
DDR:333.333 MHz (666.667 MT/s data rate) (Asynchronous), LBC:16.667 MHz
L1: D-cache 32 kB enabled
I-cache 32 kB enabled
[MCPSUMR 0x00000000, RSTRSCR 0x00000000, AUTORSTSR 0x0000c000]
I2C buses: divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
ready
DRAM: Configuring DDR for 666.667 MT/s data rate
DDR configuration get done
512 MiB (DDR3, 32-bit, CL=6, ECC on)
U-boot retry count 1Self checking ...
retrieving MAC addresses from EEPROM Internal Area
NIOM's FRU MAC's are
00:FC:BA:3B:DD:54
00:FC:BA:3B:DD:55
Memory test from 0x40000 to 0x1fdfffff
Data line test................ OK
Address line test............. OK
OK
Flash: 288 MiB
L2: 256 KB enabled
*** Warning - bad CRC, using default environment

Set dbglevel to its default value (0x1)
PCIe1: Root Complex of mini PCIe SLOT, no link, regs @ 0xffe0a000
PCIe1: Bus 00 - 00
PCIe2: Root Complex of PCIe SLOT, no link, regs @ 0xffe09000
PCIe2: Bus 01 - 01
divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
VIC card present = 1
Board: COLUSA2IOM_A0
uboot version 2.23
colusa2iom rev: 4
CMC_SLOT_ID: 0
Net: eTSEC2 is in sgmii mode.
eTSEC1, eTSEC2
I2C Clock is 105 KHz
0
Auto-boot image 2
Boot string: "linux2boot"
.. it's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBComputed Hash SHA2: 400a5e2916daaca0779f39b261a2202cfe0d8644878bfeeb19ad90301a0e49fcef7744fdd8f4e4ac4a84a42c1a18dfa046ba66fc2fe929953c677b89d2680c5e

Embedded Hash SHA2: 400a5e2916daaca0779f39b261a2202cfe0d8644878bfeeb19ad90301a0e49fcef7744fdd8f4e4ac4a84a42c1a18dfa046ba66fc2fe929953c677b89d2680c5e

Signature verification was successf.. it's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIBit's JFFS2_COMPR_ZLIB## Booting kernel from Legacy Image at 01000000 ...
Image Name: Linux-3.18.7
Created: 2019-12-18 11:53:32 UTC
Image Type: PowerPC Linux Kernel Image (gzip compressed)
Data Size: 9603611 Bytes = 9.2 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
## Flattened Device Tree blob at 00e00000
Booting using the fdt blob at 0x00e00000
Uncompressing Kernel Image ... OK
Loading Device Tree to 03ffa000, end 03fff982 ... OK
setup_arch: bootmem
fex_setup_arch()
arch: exit

Mounting application filesystems: [ OK ]
Locating OBFL filesystem: [ OK ]
Mounting OBFL filesystem: [ OK ]
Mounting Config filesystem: [ OK ]
Setting up /etc/network: [ OK ]
Locating system image: [ OK ]
Unpacking Base FS:

[ OK ]
Unpacking CMC FS: [ OK ]
/mnt/jffs2/avct_ems_cfg/BMC_Data....Setting up /flash/data0[ OK ]
Creating link /usr/bin/awk ... [ OK ]
Determining password file ...root: security_manager_setup: avctpasswd_userdb.ini is the current user database file
[ OK ]
kernel.core_uses_pid = 1
kernel.core_pattern = |/nuova/bin/zcore /var/cmc/core/%t_%e_core.%p-%s
vm.panic_on_oom = 1
vm.oom_dump_tasks = 1
net.ipv4.conf.all.route_localnet = 1
net.core.wmem_max = 524288
net.core.rmem_max = 524288
Starting internet superserver: inetd [ OK ]
Setting up networking: [ OK ]
CMC slot 0
Starting network interfaces: ifconfig: SIOCSIFHWADDR: Device or resource busy
[ OK ]
NOT starting OBFL logger
Starting PMON: [ OK ]
Running carduil -d

Discovered 1 Cisco cards.
[ OK ]

U-Boot 2011.12 (Sep 23 2019 - 11:31:42) Cisco Systems, Build: jenkins-Component-Builds-Rack-Patch Builds-patch_bmc_rel_ironcanyon_hp_mr3_bld67-83

CPU0: P1020E, Version: 1.1, (0x80ec0011)
Core: E500, Version: 5.1, (0x80212051)
Clock Configuration:
CPU0:533.333 MHz, CPU1:533.333 MHz,
CCB:266.667 MHz,
DDR:333.333 MHz (666.667 MT/s data rate) (Asynchronous), LBC:16.667 MHz
L1: D-cache 32 kB enabled
I-cache 32 kB enabled
[MCPSUMR 0x00000000, RSTRSCR 0x00000000, AUTORSTSR 0x0000c000]
I2C buses: divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
ready
DRAM: Configuring DDR for 666.667 MT/s data rate
DDR configuration get done
512 MiB (DDR3, 32-bit, CL=6, ECC on)
U-boot retry count 1Self checking ...
retrieving MAC addresses from EEPROM Internal Area
NIOM's FRU MAC's are
00:FC:BA:3B:DD:54
00:FC:BA:3B:DD:55
Memory test from 0x40000 to 0x1fdfffff
Data line test................ OK
Address line test............. OK
OK
Flash: 288 MiB
L2: 256 KB enabled
*** Warning - bad CRC, using default environment

Set dbglevel to its default value (0x1)
PCIe1: Root Complex of mini PCIe SLOT, no link, regs @ 0xffe0a000
PCIe1: Bus 00 - 00
PCIe2: Root Complex of PCIe SLOT, no link, regs @ 0xffe09000
PCIe2: Bus 01 - 01
divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
divider:1269, est_div:1280, DFSR:6
FDR:0x2c, speed:104166
VIC card present = 1
Board: COLUSA2IOM_A0
uboot version 2.23
colusa2iom rev: 4

<cut>

Vasyl · ‎03-07-2023

The problem solved. The chassis which has no single HDD in the bays yet, with just 4 SSDs and 2 nodes installed, did not want to boot properly on a single power supply and kept rebooting. There was no any indication or warning for requiring more PSU power available or whatever.

Strange and silly.

The question is closed.

View solution in original post

Vasyl · ‎02-27-2023

Does anyone have any idea how to address it? I am exploring ability to replace existing HP SL4540 storage and possible blade chassis going forward... but met roadblock right at the begging. Having so great experience with Cisco's network equipment, hoping to get the same quality and reliability on servers and storages. Maybe some on from Cisco team itself can point me to the right direction?

Thanks

Vasyl · ‎03-07-2023

The problem solved. The chassis which has no single HDD in the bays yet, with just 4 SSDs and 2 nodes installed, did not want to boot properly on a single power supply and kept rebooting. There was no any indication or warning for requiring more PSU power available or whatever.

Strange and silly.

The question is closed.