Easiest way to upgrade a two node deployment (2.4 to 3.0)

[jacksonajax]

ISE upgrades are a rare occurrence in my environment. My first upgrade from 1.2 to 2.4 involved a migration to a new deployment since a direct upgrade wasn't possible. I would like to upgrade 2.4 to 3.0

I have a simple two node deployment (virtual machines). Each node runs Admin, Monitor and Policy personas and it's essentially a single node deployment since the Secondary address is only used when the Primary is down and historically that hasn't happened.

My first instinct was to use backup/restore method since it's no issue to provision two new VMs, and I thought that it would save downtime. However, the overall procedure now seems a bit complex compared to downloading the upgrade bundle and feeding it to the Primary node. I would appreciate any recommendations or opinions.

One thing that I'm struggling to understand with the backup/restore method is the following:

- When I deregister my secondary node and bring up my new VM with the restored backup is the new VM a new deployment or does it get joined back to the existing deployment? 

[marce1000 · Accepted solution]

                              >...My first upgrade from 1.2 to 2.4 involved a migration to a new deployment

 - I would advise to stay with that approach for this case too. ISE is wonderful but 'large and complex' and the multi node model adds to that. Then it usually being business critical poses usually multiple dilemma's when an upgrade is needed. The knowledge-need for everything that can happen usually goes beyond one  person when that single critical knowledge item is needed once you get stuck with a particular problem during the upgrade process. For that outsourcing with a reseller and handing off responsibility is  a good approach too (Hey boss , it's not me , they did it....).  In our case we took both : building new environment (version) with reseller on site. Additional benefits are that authenticators (switches in radius server) , can be changed slowly to new ISE environment (not all at once), having a more easy feeling if problems would be observed with the new deployment. All of that approach gave me much better sleep....

 M.

[Arne Bier · Accepted solution]

Hi @jacksonajax 

---

@jacksonajax wrote:

One thing that I'm struggling to understand with the backup/restore method is the following:

- When I deregister my secondary node and bring up my new VM with the restored backup is the new VM a new deployment or does it get joined back to the existing deployment? 

---

The "upgrade by restoring config on a new node" is a very popular and sometime necessary method. I would recommend it because it give a lot of flexibility and rollback. In your case, spin up to ISE 3.0 VMs, do not run setup! Then deregister the old standby PAN and shutdown the VM (on ISE CLI (1) application stop ise (2) halt).

This allows you to re-use the IP address of that old VM for your first ISE 3.0 node. Run the setup on the ISE 3.0 VM and use the IP address of the old node you just shut down. While this is happening, your old ISE 2.4 PAN is running the production network.

Restore the ISE 2.4 config backup onto the new ISE 3.0 node - when you do this, you will create a new deployment, and once the config has been applied and upgraded to ISE 3.0 config, the node will become functional. Now you have two standalone Primary PAN nodes - they don't talk to one another.

At this point you'd want to test that your devices are happy talking to the new ISE 3.0 node - test test test. You can always revert back to the old ISE 2.4 VM because it's still there (albeit shutdown)

If happy, then proceed to shut the remaining ISE 2.4 node. Run setup on the new ISE 3.0 VM - so what you need with Admin certs etc. and then register this node via the ISE 3.0 Primary PAN.

Run the patch process on both nodes.

I summarised at high level.

The end result is that you have 4 VMs

2 x old ISE 2.4 (shutdown) - delete these once you're happy with ISE 3.0

2 x new ISE 3.0 running.

[cisco.13]

Hello @Arne Bier , @jacksonajax 

let me ask this question, in this step :

"Restore the ISE 2.4 config backup onto the new ISE 3.0 node - when you do this, you will create a new deployment, and once the config has been applied and upgraded to ISE 3.0 config, the node will become functional. Now you have two standalone Primary PAN nodes - they don't talk to one another."

I restore with the configuration of secondary after having deregister it ? "include-adeos" ?

Thank you.

[Arne Bier]

Hi @cisco.13 

Yes you must first de-register the Secondary ISE 2.4 PAN. That node then becomes a STANDALONE node, and the application processes will restart. You don't need to wait for it to restart - just wait until the de-registration has been completed. Then you can power off the VM of the de-registered node. The remaining ISE 2.4 PAN will now operate on its own.

Now you can spin up a new ISE 3.0 VM, using the same IP details of the de-registered node. After the install and setup has been completed, add a repository config on that new node to point to the config backup location. Restore the ISE 2.4 config backup, but NOT restoring the ADE-OS (because the config backup was probably made on the Primary PAN - IP details won't match). You don't gain much from restoring the ADE-OS details - just add the few CLI commands in manually if you have to. Once the restore is complete, you can promote that node to a PRIMARY.

Now you'll have two separate ISE Deployments - one running 2.4, and the other running 3.0.  Once you are happy with the 3.0 deployment, you can kill the remaining 2.4 node and build a fresh ISE 3.0 using the IP details of that node. Then register that node to the PAN 3.0.

[cisco.13]

Hello @Arne Bier

Thank you very much for the detail, one last question please :
"You don't gain much from restoring the ADE-OS details - just add the few CLI commands in manually if you have to."
=> My new VMs are ready with a temporary IP address and hostname.
if I understood correctly it does not pose a problem if I change the IP and the hostname in CLI?
Apparently it's also possible:
reset-config => reset ADE-OS settings (IP, hostname ...)
application reset-config ise => reset cerificat, ... (because temporary hostname).

Thank you.

[Arne Bier]

If you want to change the IP address of an ISE node, you can do it via "conf t" and then change the IP address under the interface. It will restart all the processes. Main thing is, that the node is not registered to any deployment - in other words, only change the IP address if the ISE node is standalone.

[jacksonajax]

Thanks Arne. Your explanation is clear, and I can see the flexibility of the backup/restore method. If I'm testing with my new primary node (the former secondary) I assume that for EAP authentication the device will need to trust that certificate (as it will have the server name). Is there a way for each node to display the same alias? Or do I need to have them behind a load balancer in order to achieve this?

[Arne Bier]

Good point about the ISE EAP certificate. I was a bit vague on that point.

It is recommended best practice to have the ISE EAP certificate signed by a CA - this could be your organisation's own PKI, or it could be a public CA. If you do this, then it means that your endpoints (clients) need to trust the CA that signed the ISE EAP cert. If you use your own PKI, then of course you need to get the CA chain onto your devices (e.g. via Microsoft Group Policy or via an MDM etc.). If the CA is a public one, then most operating systems already have that built in.

If each ISE server has its own EAP cert signed by the same CA, then your clients will have no issue trusting either (or any) of your ISE nodes that have been setup in this way. The Subject Common Name (and SAN) of the certs is less important (unless your clients also check the server name ... this is less common - in that case you might want to use the same EAP cert for all your ISE nodes).

Therefore the answer to your question is (as always) "it depends". If you are not doing as advise above, then you're using the ISE self-signed certificate for EAP - and that would be bad news, because it's a new cert for your endpoints, and they would require this cert to be trusted. Do not use self-signed certs please!

In summary:

Create a CSR and sign your ISE EAP cert using some sort of CA.

Install CA cert chain in ISE as Trusted CA.

Then ensure the clients have that CA cert chain installed too.

You can either create a unique ISE EAP cert per ISE node, or re-use the same cert for all PSN's that need it (in a distributed environment). Most security experts would recommend creating a unique cert per server.

[julianomluz]

I upgraded a 6 node cluster using backup/restore method and was
straightforward. When you remove the secondary node you are creating a new
deployment. You have to register other nodes in this new deployment. Some
concerns:

You have to backup and import the certificates again
You have to rejoin the active directory.


You can choose not restore the operational data. Logs and historical data
won’t be preserved but it saves a lot of time.

[jacksonajax]

According to TAC whenever a client first does EAP authentication it will have to trust the certificate (even if it's issued by a public CA). We usually see this when we renew certificates. We also saw this when we moved to a parallel deployment for our ISE 1.2 to 2.4 "upgrade".

So in this case of either GUI upgrade or backup/restore I assume that our ISE2 node will be handling EAP authentication at some point and since our clients have never 'seen' that node the user will need to trust ISE2 certificate. It shouldn't be a big deal as the upgrade will occur during off-peak hours.

I was just interested if others had found a way to avoid this prompt.

[Arne Bier]

@jacksonajax - if you find out then let us know - as far as I know, only certain (newer) versions of Apple iOS have this characteristic. And I could be wrong, but if the wifi profile was provided via an MDM then this prompt won't happen. In Android and Windows I have never seen this extra prompt because that is how it should be. If the device has the CA cert chain of the RADIUS server and the device has been explicitly asked to validate the RADIUS EAP certificate, then the user will not be bothered with a prompt if everything checks out ok - and why should they?