SRU mismatch between FMC and FTD causing deployment failure

Marius Gunnerud · ‎01-19-2023

Symptoms

Recurring Rule Update Imports is enabled with policy deploy under Updates > Rule Updates. If the SRU update is not successfully pushed from the FMC to the FTDs there will be a version mismatch and deployments start failing.

Screenshot 2023-01-19 at 11.06.44.png

Diagnosis

Deployments are failing. Check Deployment Transcript and Rule Update Log. This can also be checked by running the command sfcli.pl show version on both FMC and FTD in expert mode.

Deployment transcript:
=========SNORT APPLY=========
May 19 21:05:43 Starting Export for ApplicationDetectors
May 19 21:05:44 Finished Export for ApplicationDetectors
May 19 21:05:44 Starting Export for LSP
May 19 21:05:44 Finished Export for LSP
May 19 21:05:44 Starting Export for NGFWPolicy
May 19 21:05:44 Finished Export for NGFWPolicy
May 19 21:05:44 Starting Export for PlatformSettings
May 19 21:05:44 Finished Export for PlatformSettings
May 19 21:05:44 Starting Export for SensorPolicy
May 19 21:05:44 Finished Export for SensorPolicy
May 19 21:05:44 Starting Export for AccessControl
May 19 21:06:08 Finished Export for AccessControl
May 19 21:06:08 Starting Export for ARC
May 19 21:06:08 Finished Export for ARC
May 19 21:06:08 Starting Export for Beaker
May 19 21:06:08 Finished Export for Beaker
May 19 21:06:08 Starting Export for CrashHandler
May 19 21:06:08 Finished Export for CrashHandler
May 19 21:06:08 Starting Export for Geolocation
May 19 21:06:09 Finished Export for Geolocation
May 19 21:06:09 Starting Export for Onbox
May 19 21:06:09 Finished Export for Onbox
May 19 21:06:09 Starting Export for SFDataCorrelator
May 19 21:06:09 Finished Export for SFDataCorrelator
May 19 21:06:09 Starting Export for SFTunnel
May 19 21:06:09 Finished Export for SFTunnel
May 19 21:06:09 Starting Export for SensorClustering
May 19 21:06:09 Finished Export for SensorClustering
May 19 21:06:09 Starting Export for StateSharing
May 19 21:06:09 Finished Export for StateSharing
May 19 21:06:09 Starting Export for Stealthwatch
May 19 21:06:09 Finished Export for Stealthwatch
May 19 21:06:09 Starting Export for Snort

May 19 21:06:09 System (/usr/bin/rsync -a -L -K -O --no-owner --no-group --no-perms --safe-links /ngfw/var/cisco/deploy/sandbox/modulepack-pkg/so_rules/a6b1558037a38f88c6655aa397b3fd88 /ngfw/var/cisco/deploy/sandbox/exported-files/var/sf/detection_engines/526bb32a-fd46-11ae-89ca-66392d0b3c8c/so_rules) Failed

========= CLI APPLY =========

========= INFRASTRUCTURE MESSAGES =========
SSP Config Rollback Successful

===============TRANSACTION INFO===============

Transaction ID: 214749695519
Device UUID: 76920bb2-fb45-11ea-23ab-821c2d0b3b8c
Container UUID: f3f322b6-4060-11eb-fd99-8dc9c6c846d3

Selected policy group list: Access Control Policy, Intrusion Policy, Intrusion Policy

Out-of-date policy group list: Access Control Policy, Intrusion Policy, Intrusion Policy

Deployment Type: Full Deployment

================================================================

Rule Update Log
Screenshot 2023-01-19 at 11.03.08.png

Solution

In this situation we ended up rolling back the FMC as this would be less time consuming than upgrading 16 FTDs that were affected. First enter expert mode and navigate to /var/sf/SRU and issue the command ls -lh. Identify the SRU package that corresponds to the version currently installed on the FTDs. In this case the SRU package that we will be rolling back to is

Cisco_Firepower_SRU-2022-05-16-001-vrt.sh.REL.tar

admin@firesight:/var/sf/SRU# sudo su -

root@firesight:/var/sf/SRU# tar -xvf Cisco_Firepower_SRU-2022-05-16-001-vrt.sh.REL.tar
bundle.tar
bundle.sig

root@firesight:/var/sf/SRU# tar -xvf bundle.tar
METADATA
Cisco_Firepower_SRU-2022-05-16-001-vrt.sh

root@firesight:/var/sf/SRU# sudo sh Cisco_Firepower_SRU-2022-05-16-001-vrt.sh -- --rollback

The package is /var/sf/SRU/Cisco_Firepower_SRU-2022-05-16-001-vrt.sh
Verifying archive integrity... All good.
Uncompressing Cisco Firepower SRU.............................................
[220520 10:23:23] Lock //tmp/sru.lock for revert created successfully
[220520 10:23:23]
Authority UUID =
AQ UUID =
[220520 10:23:23] Using root directory
[220520 10:23:23] #####################
[220520 10:23:23] # REVERT STARTING
[220520 10:23:23] #####################
[220520 10:23:23]
[220520 10:23:23] BEGIN pre/000_start.sh
[220520 10:23:23] COMPLETED pre/000_start.sh
[220520 10:23:23]
[220520 10:23:23] BEGIN pre/010_check_versions.sh
[220520 10:23:23] COMPLETED pre/010_check_versions.sh
[220520 10:23:23]
[220520 10:23:23] BEGIN pre/020_check_space.sh
[220520 10:23:23] COMPLETED pre/020_check_space.sh
[220520 10:23:23]
[220520 10:23:23] BEGIN pre/999_finish.sh
[220520 10:23:23] COMPLETED pre/999_finish.sh
[220520 10:23:23]
[220520 10:23:23] BEGIN installer/000_start.sh
[220520 10:23:23] COMPLETED installer/000_start.sh
[220520 10:23:23]
[220520 10:23:23] BEGIN installer/050_sru_log_start.pl
[220520 10:23:35] COMPLETED installer/050_sru_log_start.pl
[220520 10:23:35]
[220520 10:23:35] BEGIN installer/100_install_files.pl
[220520 10:23:35] COMPLETED installer/100_install_files.pl
[220520 10:23:35]
[220520 10:23:35] BEGIN installer/510_install_policy.pl
[220520 10:23:56] COMPLETED installer/510_install_policy.pl
[220520 10:23:56]
[220520 10:23:56] BEGIN installer/520_install_rules.pl
[220520 10:30:37] COMPLETED installer/520_install_rules.pl
[220520 10:30:37]
[220520 10:30:37] BEGIN installer/521_rule_docs.sh
[220520 10:30:38] COMPLETED installer/521_rule_docs.sh
[220520 10:30:38]
[220520 10:30:38] BEGIN installer/530_install_module_rules.pl
[220520 10:31:03] COMPLETED installer/530_install_module_rules.pl
[220520 10:31:03]
[220520 10:31:03] BEGIN installer/540_install_decoder_rules.pl
[220520 10:31:16] COMPLETED installer/540_install_decoder_rules.pl
[220520 10:31:16]
[220520 10:31:16] BEGIN installer/602_log_package.pl
[220520 10:32:33] COMPLETED installer/602_log_package.pl
[220520 10:32:33]
[220520 10:32:33] BEGIN installer/900_update_version.sh
[220520 10:32:33] COMPLETED installer/900_update_version.sh
[220520 10:32:33]
[220520 10:32:33] BEGIN installer/999_finish.sh
[220520 10:32:33] COMPLETED installer/999_finish.sh
[220520 10:32:33]
[220520 10:32:33] BEGIN post/000_start.sh
[220520 10:32:33] COMPLETED post/000_start.sh
[220520 10:32:33]
[220520 10:32:33] BEGIN post/500_copy_contents.sh
[220520 10:32:33] COMPLETED post/500_copy_contents.sh
[220520 10:32:33]
[220520 10:32:33] BEGIN post/900_iru_log_finish.pl
[220520 10:32:34] COMPLETED post/900_iru_log_finish.pl
[220520 10:32:34]
[220520 10:32:34] BEGIN post/999_finish.sh
[220520 10:32:34] COMPLETED post/999_finish.sh
[220520 10:32:34]
[220520 10:32:34] Lock //tmp/sru.lock for revert removed successfully.
[220520 10:32:34] Pruning logs and old installs
[220520 10:32:34]
[220520 10:32:34] ####################
[220520 10:32:34] # REVERT COMPLETE
[220520 10:32:34] ####################

root@firesight:/var/sf/SRU# sfcli.pl show version
--------[ firesight1.builtontrust.it ]--------
Model : Cisco Firepower Management Center for VMware (66) Version 7.0.1.1 (Build 11)
UUID : 679fd8c6-56bb-11e9-81de-29aabbcc1440
Rules update version : 2022-05-16-001-vrt
LSP version : lsp-rel-20210503-2107
VDB version : 351
----------------------------------------------------

bmerrill · ‎04-04-2023

In this example, you show how to handle it by back revving the FMC. Is there a way to solve this by updating the FTD? We have 8 FTDs, and only 1 of them is out of sync and failing deployments so it seems like it would be more straightforward to just update that 1 instead of back revving the FMC? Thank you!