How the Versus control plane enrolls collectors, brokers work to them, and guarantees safe execution — the Enrollment Service, Gateway, and Task Router, plus the signed task envelope, single-writer device leases, and the unknown-outcome reconciliation ladder.
The collector is the part of Versus that runs inside your network; the control plane is the part that runs in Versus and decides what each collector should do. This page is the architecture of that control plane — the three services that turn "run this operation on that device" into a signed, bounded, single-writer task delivered to exactly the right collector, and the guarantees that hold when a collector dies mid-task.
It complements the operator-facing Collector Fleet Management guide and the Collector Security Architecture guide. Where those cover the admin UI and the trust model, this one covers the machinery in between.
One tenet drives the whole design:
The collector is dumb about policy; the control plane is dumb about reachability.
The control plane decides who runs a task — it holds every routing fact (which collectors serve which sites, what drivers they run, how busy they are, which device is already being written to). The collector decides only how to execute a task and how to refuse one. Neither side negotiates with the other about placement, and collectors never talk to each other at all — there is no peer protocol to import distributed-systems failure modes from.
The control plane is three cooperating services:
| Service | Responsibility |
|---|---|
| Enrollment Service | Issues each collector a certificate identity and bootstraps the trust material it needs (CA bundle, dispatch/release verification keys, gateway address). |
| Gateway | Terminates the collector's authenticated connection, maps it to a collector_id, and is the only component that speaks to collectors. |
| Task Router | Turns an operation request from a workflow (or any Versus module) into a leased, signed dispatch aimed at one eligible collector, and handles the result — including the ones a collector can't confirm. |
A collector cannot do anything until it has proven it is allowed to connect. Enrollment is a one-time event, described in depth from the security angle in Collector Security; here is what the control plane does with it.
An administrator mints a one-time (or limited-use) enrollment token scoped to a site. Versus stores only the SHA-256 hash of the token and shows the plaintext exactly once. The collector generates its own keypair locally, builds a CSR carrying its self-chosen collector_id as the Common Name, and posts the token plus CSR to the public enrollment endpoint:
POST /api/collector/v1/enroll
Content-Type: application/json
{
"token": "<opaque one-time token>",
"csr_pem": "-----BEGIN CERTIFICATE REQUEST-----\n…",
"collector": {
"hostname": "…", "os": "linux", "arch": "amd64", "version": "1.4.2",
"fips_requested": false
}
}
The service validates, in order: the token exists, is unrevoked, unexpired, and under its use limit → the CSR parses and its signature proves possession of the private key → the CSR carries a non-empty CN, and that CN doesn't collide with a different already-enrolled collector. A certificate is then issued from a dedicated Collector CA — an internal CA reserved for collector identities, managed through the same certificate-lifecycle machinery as any other CA in Versus, and kept entirely separate from the CAs you use for your own certificates. The leaf is valid for 90 days and carries a structured SPIFFE-style identity SAN binding the collector to its site.
The enrollment response is the trust bootstrap — everything the collector must trust arrives here, over an authenticated channel, rather than from an operator flag:
{
"collector_id": "col-…",
"cert_chain_pem": "…",
"ca_bundle_pem": "…",
"gateway": { "addrs": ["…:443"], "server_name": "…" },
"auth": { "mode": "bearer", "audience": "versus-collector-gateway" },
"dispatch_keys": [ { "kid": "…", "alg": "ed25519", "pub": "…", "not_after": "…" } ],
"release_keys": [ { "kid": "…", "alg": "ed25519", "pub": "…", "not_after": "…" } ],
"site_id": "…",
"policy": { "vault_mode": "B", "heartbeat_interval_s": 15, "max_concurrency": { "*": 8 } }
}
The dispatch keys are how the collector will verify that a task really came from Versus (below), and the release keys are how it will verify an upgrade manifest — two distinct key sets, not singletons, so either can be rotated in-band without a flag day. Both are pinned to the collector's disk alongside its identity, and never taken from whatever the gateway happens to present.
The gateway is the sole component that communicates with collectors, and it authorizes on the signed bearer token, never on anything a request body claims. Identity is read from the verified bearer token (its collector_id), checked against the collector's enrollment key, on every request.
The original design specified a bidirectional gRPC stream. In practice the control path is an outbound long-poll (Amendment 1 to the spec): three bearer-authenticated HTTPS endpoints the collector calls, always dialing out, never listening.
POST /api/collector/v1/claim long-poll for the next leased task
POST /api/collector/v1/report return a task result
POST /api/collector/v1/capabilities publish the driver manifest
The collector claims work; Versus holds the request briefly and answers with a task if one is ready, otherwise 204 and the collector immediately asks again. Because each request stands alone and is individually authenticated, there is no long-lived stream for a compromised intermediary to ride, and nothing for Versus to push into — it can only answer a request the collector already made.
Two aspects are worth keeping separate: how the collector authenticates, and how the exchange is framed on the wire.
Authentication. Every request carries a short-lived ES256 token signed by the collector's enrollment key, sent as an Authorization header (VS-SPEC-COLLECTOR-GW-443-001). Because identity rides a header rather than a client certificate, it survives a TLS-terminating or -inspecting edge — an ALB, Cloudflare, or a corporate MITM proxy that would strip a client cert. The collector verifies the gateway's server certificate against the host's public roots and the Versus internal CA delivered at enrollment, so it works with a public-cert edge (SaaS) or an internal-CA gateway (air-gapped on-prem) with no public PKI. No client certificate is ever presented.
Transport — how the exchange is framed on the wire.
443: a single persistent socket. It carries the claim/report control channel with less per-call overhead, and — the main reason it exists — it carries the byte-tunnel: the streamed device sessions (SSH to a device CLI / F5 tmsh, HTTPS to an API such as iControl or NITRO) that Versus opens through the collector to reach managed gear. A streamed session can't ride discrete request/response calls, and a raw hijacked socket can't cross an L7/443 edge or an HTTP proxy — as WebSocket frames it traverses ordinary 443 like any web traffic. It falls back to per-call HTTPS automatically if the socket drops. Both directions stay strictly outbound and answer-only — Versus never pushes into the collector.There are really two kinds of traffic, and each has a natural shape:
| Traffic | Shape | Transport |
|---|---|---|
| Control — claim / report / capabilities | short request/response messages | WSS when the socket is up (lower overhead), per-call HTTPS as the fallback |
| Byte-tunnel — a device session (SSH to a CLI / F5 tmsh, HTTPS to an API like iControl or NITRO, file transfer) | a long-lived bidirectional stream | WSS only — a stream can't be chopped into discrete requests |
So WSS is required for the byte-tunnel (a stream) and preferred for control; per-call HTTPS stays as the always-works fallback for control, so the collector stays reachable even where a WebSocket can't be established (some proxies pass plain HTTPS but block or mangle WS upgrades). The two are independent WebSocket endpoints (/tunnel and /control), so a deployment can run the tunnel over WSS while control is momentarily on per-call HTTPS, and everything keeps working.
How they fit together. Authentication (the bearer token) is independent of transport (per-call HTTPS or WSS) — both run on 443. Everything is bearer-on-443, with WSS as the preferred framing (control channel + device tunnel) and per-call HTTPS as its fallback.
There is no separate heartbeat. Every /claim doubles as the liveness signal: it upserts a CollectorLiveness row with last_claim_at, the collector's current capacity, and its drain state. A collector is routable if it claimed within the liveness window (180 s); miss it and the router simply stops seeing it as a candidate. On start (and whenever its capability epoch changes), the collector calls /capabilities to publish its driver manifest — the providers it hosts, each provider's operations, their operation classes, timeouts, and per-provider concurrency ceiling. The router filters against exactly this.
A task arrives from the orchestration engine (or CLM, backup, upgrade workflows — any module) naming a provider, operation, op_class, target_ref, params, idempotency_key, and a deadline. The router runs a six-stage pipeline.
provider and the specific operation at the matching op_class; has spare capacity (in_flight < max_concurrency for that provider); and reports FIPS if the task requires it. If the candidate set is empty the task does not dispatch — it stays queued and raises a no eligible collector condition rather than guessing.collector_id so placement is reproducible.read, the router acquires an exclusive DeviceLease — the single-writer guarantee below.leased; the collector's next /claim picks it up under SELECT … FOR UPDATE SKIP LOCKED, so the same task is never handed to two claimers.Every operation declares one of three classes, and the class — not a guess — drives retry policy, serialization, and whether an unconfirmed outcome may be retried at all:
op_class | Meaning | On an unknown outcome |
|---|---|---|
read | No state change | Freely retried |
mutate_reconcilable | Changes state, but the driver declares an idempotent read that answers "did it land?" | Reconciled, then retried up to a bounded number of attempts |
mutate_terminal | Changes state, not safely repeatable | Never automatically retried |
The router validates the caller's declared op_class against the driver's own operation manifest, so a module can't claim read for something the driver knows is a terminal mutation and skip the lease.
The DeviceLease table guarantees at most one mutating dispatch in flight per device across the entire estate, no matter how many workflows, modules, or collectors are involved. Reads bypass the lease; operations explicitly marked parallel-safe bypass it too. A second mutating dispatch for a leased device is refused, not queued behind a race. This is a control-plane invariant — collectors are never asked to coordinate writes among themselves.
Every dispatched task is signed so a collector can prove it came from Versus, is meant for it, and hasn't expired. The signature is Ed25519 over a canonical byte string with a versioned domain separator (VERSUS-TASK-V1). The format uses length-prefixed fields everywhere — no delimiters to inject, and "absent" and "empty" encode identically — and both the control plane and the collector implement it byte-for-byte, arbitrated by a shared set of conformance vectors checked into both repositories.
The signed bytes bind, among other fields:
kid — which dispatch key signed it (keys are a rotating set).task_id — unique per attempt.collector_id — binds the envelope to one collector; it is useless anywhere else.provider, operation, op_class, target_ref.idempotency_key — stable across attempts of one logical operation.params_sha256 — the SHA-256 of the parameters exactly as transmitted. Params ride the wire as opaque bytes and the signature covers their hash, never a re-serialization, which sidesteps every cross-language canonical-JSON disagreement.not_before / expires_at — the envelope is short-lived.On receipt the collector verifies, in order: the kid is in its trusted set and unexpired → the signature checks out → the collector_id is itself (a mismatch is an attack signature, reported, not a routing bug) → the clock is within the validity window → the params hash matches → the task_id is unseen → and finally, if the idempotency_key is already in its journal, it returns the journaled result instead of re-executing. That last rule is what makes retries safe across both collector and gateway failure.
Because validity is time-bounded, clock skew is a correctness dependency: the collector receives server time and refuses (rather than silently correcting) tasks outside the window.
The honest failure mode of any automation platform is a mutation that might have applied. Versus never assumes a task didn't run just because it didn't hear back. When a result comes back unknown — the collector already tried its own local reconciliation and couldn't be sure, or the collector died mid-task and its lease expired — the router walks a deliberate ladder keyed on op_class:
read → simply retry, with a new task_id and the same idempotency_key.mutate_reconcilable → dispatch a reconcile-read and retry, up to a bounded number of attempts (default 3); still unsure after that → mark indeterminate.mutate_terminal → straight to indeterminate, never retried.indeterminate is a first-class, human-facing outcome: the workflow pauses, the device lease is released, and the task surfaces in a dedicated operator queue with its evidence rather than being silently retried into a possibly-double-applied change. Every retry reuses the same idempotency_key, so a retry that reaches a collector which already executed the operation gets the journaled result back instead of running it twice.
A dispatch therefore moves through a well-defined state machine: queued → routed → leased → dispatched → running → success | failed, with unknown → reconciling → indeterminate for the unconfirmed path, plus expired (deadline blown) and cancelled. Expired dispatches are swept automatically so a dropped connection can't pin a collector's concurrency slots forever — a self-healing behavior that keeps the fleet routable without an operator clearing stuck rows by hand.
Some work doesn't fit one request/response operation. The same outbound-only channel carries a few specialized job types the collector picks up on its /claim:
TunnelSession; the collector's next claim offers it, and the collector dials back a dedicated connection the gateway pairs end-to-end into a byte relay — a bearer-authenticated WebSocket on 443, so the byte path crosses the same standard egress as the control channel. Credentials can be injected locally on the collector so they never leave your network. See Collector Vault Modes.The admin surface is RBAC-gated and covered in Collector Fleet Management. The pieces that touch the control plane directly:
draining so the router immediately stops sending it new work while in-flight tasks finish; this also gates a safe agent upgrade. Undrain resumes normal routing.manual, auto, or pinned policy, and the target rides the same /claim response the collector is already polling.The control plane is the platform's outbound-only edge made concrete. Workflows and modules issue operations without knowing or caring which collector runs them or where it sits; the router resolves site, capability, capacity, and tenancy into a placement, and the signing + lease + idempotency machinery makes that placement safe to retry. Because collectors dial out and Versus only ever answers, there is never an inbound path into your network — the reachability lives entirely on your side of the boundary, and the policy lives entirely on ours.