Failover Readiness

Failover Readiness answers one question for every HA pair: if this pair failed over right now, would it actually work? Standby rot — a change never mirrored to the standby, a certificate expired on the standby only, a route the standby never learned, a standby too loaded to absorb the active's traffic — is a leading cause of failed failovers, and nothing measures it until the failover fails for real. Find it under Diagnostics → Failover Readiness (administrator access).

What it checks

An HA pair binds an active and a standby device from your inventory. Each scoring run measures four independent drift dimensions between them.

Dimension	Weight	What is compared	Penalty
Config parity	35%	The latest config snapshot of each member — the raw device config when a text backup exists, otherwise Versus's vendor-agnostic serialisation. Lines present on the active but missing or different on the standby count as drift.	12 points per divergent stanza
Cert parity	20%	Every certificate still valid on the active, matched by partition and name on the standby.	Missing −35 · expired on standby −40 · expiring within 30 days −20
Route parity	25%	Routed prefixes on the active that the standby does not carry. Connected and local routes are per-node and excluded.	18 points per missing prefix
Capacity headroom	20%	Projected post-failover utilisation: the standby's latest CPU and memory sample plus the active's. Full marks up to 60% projected, zero at 100%.	2.5 points per point above 60%

Config parity deliberately ignores differences that are supposed to exist between two HA nodes — hostname, self-IPs, management IP, unit number, device priority, HA role and preempt settings, router-id, and any line naming either node — so a healthy pair scores clean. Stanzas that exist only on the standby are reported as informational, not penalised.

How the score is built

Each dimension yields its own 0–100 score; the readiness score is their weighted average. A dimension whose input is missing — no snapshot on one side, no certificates or routing table on the active, no stats sample on the standby — is marked unknown and dropped from the average with the remaining weights re-normalised, so a partly instrumented estate still gets a meaningful score for what it can measure. A pair with no active or no standby assigned scores 0 outright.

Score	Status
85–100	Ready
60–84	At risk
0–59	Not ready

Every run is persisted with its per-dimension verdicts and findings, so a pair has a history, not just a latest state.

Reading a readiness gap

The page lists every tracked pair with its status, score bar, active → standby members and, once escalation has fired, the incident number. Select a pair for the breakdown: a gauge with the overall score, vendor and last-scored time plus a Score now button; four drift dimension cards with their own score bars and findings, example lines in monospace (active-only: ltm rule /Common/irule_redirect …, missing: 10.99.0.0/16 via 10.10.10.2 (static)); and an All findings list tagged info, warn or crit.

The demo (python manage.py seed_failover) seeds an F5 BIG-IP pair whose standby is missing a pool member and an iRule, lacks two routes, has one certificate missing and another expiring, and has tight memory headroom — it scores 55, Not ready — and a Palo Alto pair with clean parity that scores 96, Ready. --clear removes them.

Tracking and scoring pairs

Discover pairs proposes pairs from the HA-peer links on your devices and creates any not yet tracked. The member whose status reads active becomes the active side; the pair is named from the devices' common prefix, e.g. dc1-ltm HA pair.
Score now re-scores one pair immediately; the page refreshes every 20 seconds.
python manage.py run_failover_scoring scores every enabled pair for every tenant — suitable for cron.
With the feature enabled (below), an in-process scheduler re-scores all pairs every 5 minutes under a database advisory lock, so only one replica scores at a time.

Incident and AutoPilot escalation

Escalation is gated by FAILOVER_READINESS_ENABLED, which ships off: scoring always works, but nothing is opened until you enable it. When it is on and a run lands at Not ready, Versus:

Opens a P2 Versus Desk incident titled Failover readiness LOW for <pair> — score N/100, carrying the four dimension scores, the top findings and a machine-readable context block.
Attaches both pair members as configuration items.
Creates and starts a Diagnostics AutoPilot session scoped to those two devices.

The run records the incident number and session ID; the pair shows an Escalated · INC-… badge with an AutoPilot session → link, and the summary row shows Auto-escalation ON. Escalation is rolled up per pair: while a pair stays Not ready across scheduler ticks, each new run is linked to the incident that is already open rather than opening another, and that incident is resolved automatically the first time the pair scores Ready again. Enable it once your tracked pairs are ones you intend to act on.

Enabling it

Set FAILOVER_READINESS_ENABLED=true (1, true or yes) in the backend environment. The flag turns on incident + AutoPilot escalation for Not-ready runs and starts the 5-minute scheduler on processes that own in-process schedulers (the worker, or any process not marked DISABLE_INPROC_SCHEDULERS). Remove it to return to on-demand scoring only.

The REST API lives under /api/failover/ha-pairs/: overview/ (every pair, estate summary, weights, escalation state), {id}/score/, {id}/runs/ (last 50) and discover/ ({"create": true} persists the proposals); individual runs are readable at /api/failover/runs/.

Diagnostics AutoPilot — the investigation a Not-ready pair triggers.
Velos Chassis Monitoring and Path Monitoring — the same monitor → incident → AutoPilot loop for chassis and delivery paths.
Certificates — the certificate inventory that cert parity reads.
Rewind — when a member's config changed, and by whom, when a parity finding needs explaining.