Diagnostics AutoPilot

Diagnostics AutoPilot turns Versus Diagnostics from a tool you open when something breaks into a service that is already working the incident by the time you look. It watches your incident queues, opens a Diagnostic session for every new incident, gathers evidence across your managed devices and integrations, produces an AI root‑cause diagnosis, and — within the tolerance you define per severity — either hands you a ready‑to‑approve remediation workflow or resolves the incident autonomously.

It lives inside the existing Diagnostics workspace. Every incident it picks up becomes an ordinary Diagnostic session you can open, read, and act on — AutoPilot simply drives that flow for you and adds a governance layer on top.

Ships switched off. AutoPilot is inert until an administrator enables it. The two master controls — the live feed and unattended action — both default to off, so nothing changes in your environment until you deliberately turn it on.

---

The loop

1. Live feed — every ~20 seconds AutoPilot pulls open incidents from Versus Desk and ServiceNow, de‑duplicates them, and creates one Diagnostic session per new incident.
2. AI triage — for each session it resolves the affected device(s), gathers evidence from your inventory, live telemetry, and integrations, and asks the model for a structured root‑cause diagnosis.
3. Governance — it maps the incident's severity to the tolerance you configured, which decides how far it may go.
4. Remediation — depending on that decision it stops at diagnosis, generates a remediation workflow and holds it for your approval, or (for the highest severity, when you've enabled it) executes the workflow autonomously — always rate‑limited and fully audited.

---

What feeds the queue

AutoPilot reads from two incident sources; you can enable either or both.

Each incident is matched to a device where possible — from a Desk incident's linked configuration item, a ServiceNow cmdb_ci, or by matching a hostname/IP mentioned in the incident text against your inventory. That resolved device is what triage investigates.

Incidents are de‑duplicated on source + number, so an incident is only ever picked up once no matter how many times the feed runs. A look‑back window (default 24 hours) bounds how far back a newly‑enabled feed will reach.

---

AI triage — a virtual expert war room

Triage does not ask the model a single question over a fixed snapshot. It runs an iterative investigation: the model reasons as a standing panel of domain experts, forms the most‑likely hypotheses, then confirms or refutes each by calling one read‑only tool at a time and reading the real output — walking the delivery path hop by hop and crossing domains as the evidence leads, exactly the way a NOC/SRE war room works a live incident.

The panel

Every incident is worked by the relevant members of a dozen‑plus expert roster:

• ADC / Load Balancing — F5 BIG‑IP, NetScaler, A10, Avi (virtual servers, pools, monitors, SNAT, persistence, iRules)
• DNS / GSLB — BIG‑IP DNS, Infoblox (wide IPs, GSLB pools, monitors, TTL, resolution path)
• Firewall / Security Policy — Palo Alto, Fortinet, ASA, Check Point (rules, sessions, dataplane CPU, NAT, App‑ID)
• Routing — BGP / OSPF / EIGRP (adjacencies, flaps, route churn, next‑hop)
• Switching / L2 — VLANs, STP, ARP, interface errors, duplex, MTU
• Data‑Center Fabric (ACI) — Cisco ACI / APIC: fabric health, leaf/spine nodes, APIC cluster, tenants/EPGs, faults
• TLS / PKI — validity, chain / intermediate order, OCSP / CRL, SNI, handshake failures
• Cloud WAF / CDN — Cloudflare, F5 XC, Akamai, Imperva (origin reachability, WAF rules, rate limiting, DDoS)
• AWS — EC2, ELB / ALB / NLB, VPC, security groups & NACLs, Route 53, CloudFront, Transit Gateway / Direct Connect, target‑group health, service quotas
• Azure — VMs, Azure Load Balancer / Application Gateway / Front Door, VNet & NSGs, Azure DNS, Azure Firewall, ExpressRoute, backend‑pool health, service limits
• GCP — Compute Engine, Cloud Load Balancing, VPC firewall rules, Cloud DNS, Cloud CDN, Cloud Interconnect, backend‑service / NEG health, quotas
• Application / HTTP — 5xx, latency, keep‑alive mismatch, connection‑pool exhaustion, backend health
• Observability — SIEM correlation, metrics / telemetry baselines
• Threat / EDR — CrowdStrike detections, IOC correlation
• Virtualization — vSphere / vCenter VM and host state
• IPAM / DDI — address / subnet allocation, record correctness
• Change / ITSM — recent ServiceNow changes as the leading cause of a fresh incident

The method — probe, observe, branch

At each step the agent picks the single probe that most reduces uncertainty, runs it, reads the result, and decides the next move — following the delivery path client → DNS/GSLB → firewall → ADC (virtual server → pool → member) → backend, and jumping domains when a hop is healthy (a "DNS" incident is often really an ADC pool or an expired certificate). It checks for a recent change early, because that is the most common root cause. It keeps going until it can name the root cause with supporting evidence, or it exhausts its step budget (configurable). Nothing it does changes state — the investigation is strictly read‑only; acting on the diagnosis is the governed remediation stage below.

The tools it can call

The agent has a read‑only tool for every relevant capability across your managed devices and integrations:

Each probe runs against the device's own native vendor API — F5 iControl (through the site collector), Citrix NetScaler NITRO, Palo Alto PAN‑OS XML‑API, FortiGate FortiOS REST, Check Point Gaia (clish + cphaprob/cpstat) and the Management Web API (show‑gateways‑and‑servers / show‑access‑rulebase / show‑logs log‑server events), Cisco NX‑API / Arista eAPI, Cisco ACI (APIC REST), A10 aXAPI, Avi and NSX‑T REST, Infoblox WAPI, and the cloud‑WAF/CDN APIs (Cloudflare, F5 Distributed Cloud, and others) — with SSH show as the fallback for CLI devices without a REST driver. Every tool is wrapped, so an unconfigured integration or an unreachable device returns a note the agent reasons about, never an error that stops the investigation.

If no delivery path is discovered

It still investigates. With no topology to walk, the agent resolves the target device(s) from the incident's linked CI / cmdb_ci / text and from inventory, and probes from there — device health, logs, integration signals, recent changes — reaching a diagnosis (with an honestly lower confidence when the evidence is thin).

What you get

A structured, evidence‑backed diagnosis: a classification (domain / subsystem / severity), the most‑likely root cause with a confidence score, per‑expert notes (what each domain concluded), ranked findings, recommended actions (each flagged read‑only vs. state‑changing), and verification steps — plus the full investigation transcript (every probe, its arguments, and the real output) shown on the session, so you can see exactly how the panel reasoned. The agent never invents a device, IP, metric, or change it did not observe.

Configurable. The expert‑panel investigator is on by default, with a step budget you set on the settings page. Turn it off to fall back to a faster single‑shot diagnosis over a fixed evidence snapshot.

---

Governance — the tolerance policy

This is the heart of AutoPilot: how far Versus may go is your decision, made per severity, in advance. Each severity (P1 highest → P4 lowest) maps to one of four action levels.

The default policy — and the recommended starting point — is:

Two independent switches guard autonomy. "Autonomous remediation" only ever runs unattended when both the per‑severity policy says so and the global master switch is on. With the master switch off, a P1 that is set to "autonomous" is still fully diagnosed and its workflow fully built — it simply waits for your approval like a P2/P3. This lets you run AutoPilot in a supervised mode indefinitely and flip to autonomous only when you trust it.

Safety limits

• Master switch — a single global control that must be on for any unattended action, anywhere.
• Rate limit — a hard ceiling on autonomous executions per hour. Beyond it, further incidents are held for approval instead of running.
• Audit trail — every governed decision (skipped, held, approved, autonomously executed, blocked, failed) is written to an immutable audit log, visible on the settings page.
• Backout‑aware execution — remediation runs on the standard Versus workflow engine, which routes device changes through the site collector, enforces single‑writer safety per device, and records reversible steps for rollback.

---

Remediation workflows

When the policy calls for action, AutoPilot generates a real, executable workflow on the Versus workflow engine — the same engine your hand‑built workflows run on. The generated workflow is structured in phases:

1. Confirm — read‑only checks that verify the triage root cause.
2. Remediate — the fix (a device operation, a script using the Versus SDK, an Ansible/AWX job, etc.). In approval mode a human‑approval gate sits immediately before this phase.
3. Verify — re‑runs the confirmation checks and branches on whether the incident is resolved.
4. Close — on success, updates/closes the ticket; on failure, escalates (pages on‑call, opens a change).

• In Diagnose + approve mode the run is created but paused — you Approve or Reject it directly from the live feed (or open the session to review the full workflow first).
• In Autonomous remediation mode (master switch on) the run starts immediately, and AutoPilot tracks it to completion, reflecting executing → remediated / failed back onto the session.

Because these are ordinary workflows, you can open, edit, re‑run, or back them out exactly like any other — and they inherit the engine's collector routing, so they work against on‑prem devices behind a customer firewall.

---

The settings page

Open Diagnostics → AutoPilot (administrator access required). The page is organised into cards:

• Master control — the master switch (allow unattended action) and the live‑feed switch. Turning the master switch on shows a clear warning that unattended remediation is now possible.
• Incident feed — enable Versus Desk and/or ServiceNow, set the poll interval and look‑back window, and toggle auto‑create sessions / auto‑run triage. Shows the last poll's status and a Poll now button.
• Investigation method — turn the expert‑panel investigator on or off, and set its step budget (max probes per incident).
• Remediation tolerance — the per‑severity matrix (P1–P4 → action level).
• Safety — the max unattended actions per hour.
• Simulate incidents — inject realistic sample incidents to exercise the whole loop (see below).
• Recent remediation actions — the live audit trail.

All changes are versioned: every save records who changed what, so the tolerance policy has its own change history.

---

Turning it on safely

A recommended, low‑risk rollout:

1. Open Diagnostics → AutoPilot.
2. Leave the master switch off. Enable the live feed and choose your sources (Versus Desk is always available; enable ServiceNow only if an instance is configured and awake).
3. Save. Sessions now auto‑create and auto‑triage; every severity's remediation is generated but held for your approval.
4. Watch for a while. Review the diagnoses and the generated workflows in the feed. Approve the ones that look right; reject the ones that don't.
5. When you're confident, turn the master switch on. From then on, incidents whose policy is "autonomous remediation" (P1 by default) resolve on their own — rate‑limited and audited — while everything else stays approval‑gated.

You can return to supervised mode at any time by turning the master switch back off; the feed keeps running and keeps generating workflows for approval.

---

Exercising it with sample incidents

AutoPilot ships with a library of realistic, research‑grounded incident templates spanning application delivery (F5/ADC), DNS/GSLB, firewalls, routing, certificates, cloud WAF/CDN, application‑layer errors, and compute — expandable into a very large number of distinct, parameterised incidents that bind to your real inventory device names.

• From the UI — on the settings page, use Simulate incidents: choose a count, a target (Versus Desk, ServiceNow, or both), and an optional severity, then Inject. Within one poll cycle the feed fills and triage begins.
• From the CLI — python manage.py seed_incident_samples --count 200 --to both (also accepts --priority and --domains).

This is the fastest way to see the full loop — feed, triage, governance, approval, and (with the master switch on) autonomous remediation — end to end.

---

Operational notes & limits

• Access — the settings page and its API are administrator‑only; the controls govern autonomous action, so the check is enforced on the server, not just hidden in the UI.
• Single‑node vs. HA — the feed poller runs as a single in‑process worker (the same pattern as the platform's other pollers), so on a multi‑replica deployment it runs on the dedicated worker, not on every web replica.
• On‑prem reach via the collector — every live device probe runs through the site collector (a raw byte tunnel, credentials end‑to‑end) whenever the device is assigned a collector site: F5, Cisco IOS/Nexus, Arista, FortiGate, Cisco ACI/APIC, Check Point (gateway + SMS), NetScaler, A10, Avi, NSX‑T, Palo Alto/Panorama, Infoblox/EfficientIP/BlueCat, and Kemp/Alteon. So cloud Versus reaches gear behind the customer firewall with no direct connectivity, exactly like remediation does. (Public cloud‑WAF/CDN APIs — Cloudflare, F5 XC, AWS/Azure — are reached directly, as intended.)
• Live device reads are best‑effort — triage always has inventory and telemetry to work from; a live read is attempted when the device is reachable and degrades gracefully when it isn't.
• ServiceNow — the ServiceNow side of the feed and the "seed to ServiceNow" option require a configured, reachable ServiceNow instance.
• Everything is auditable — sessions, generated workflows, workflow runs, and the remediation‑action log together give you a complete record of what was diagnosed, what was proposed, who approved it, and what ran.

---

Where to go next

• Diagnostics overview — Sessions and PCAP Analysis, the workspace AutoPilot drives.
• Incident management — the Versus Desk incidents that feed AutoPilot.
• ServiceNow integration — connect the ServiceNow incident queue.
• Workflow builder — the engine every remediation runs on, including approval gates and backout.
• Collector control plane — how Versus reaches on‑prem devices for live reads and remediation.