13-switch IOS-XE upgrade with auto-abort rollback

Sector: Critical Infrastructure · Platform: Cisco Catalyst 9300X / 9500 · Timeline: 6 weeks

The situation

The client had 13 Catalyst 9K switches running an older IOS-XE 17.12 train that the engineering team had been wary of touching for over a year. Past upgrades had been bundle-mode TFTP affairs done at 2 a.m. with verbal rollback plans. They needed a repeatable process before scaling further.

What we built

• An Ansible playbook (install-add.yml / install-activate.yml / install-commit.yml / install-remove.yml) covering the full IOS-XE install-mode lifecycle.
• 360-minute auto-abort window between activate and commit -- if the playbook (or operator) fails to commit, the switch reverts on its own.
• Image transport via HTTP from the FortiGate-fronted IIS vdir, not Ansible SCP push: roughly 5.8 MB/s vs ~140 KB/s, and no ANSIBLE_PERSISTENT_COMMAND_TIMEOUT deleting partial transfers on a flap.
• install_commit.py using paramiko invoke_shell(), because install commit requires enable-mode plus an interactive session under the active AAA exec authorization chain.
• A separate install-remove.yml pass to clean inactive bundles -- intentionally not folded into the upgrade run.

The gotchas we documented

• install activate alone -- not install activate file ... -- once the image is added.
• write memory before install add, not after.
• U state vs C state in show install summary -- the difference between "activated" and "committed".
• AAA exec-chain lockouts when a RADIUS server returns AUTHZ-REJECT and the chain stops on it instead of falling through.
• The 6-hour timer is real -- a long debug session can blow past it and force an unplanned rollback.

Outcome

13 switches upgraded across two maintenance windows, no outages, no rollbacks invoked. The playbook now lives in the client's automation repo and has been re-used for two subsequent point-release upgrades by their own staff.

← Back to case studies