High Availability, DR, and Kubernetes

The Versus demo and single-node appliance run the whole platform as one web process — correct for an evaluation, wrong for a bank's production estate. For production on-premises, Versus ships a Helm chart that runs the same container image across a Kubernetes cluster: the request-serving tier scales horizontally with no single point of failure, background work is pinned to single-writer processes so nothing double-fires, and durability comes from external HA Postgres plus object-store backups.

This page explains how that chart is structured, why the topology looks the way it does, and how you operate it. It assumes the Platform Architecture tour as background.

---

One image, several roles

Every tier runs the same container image. What a pod does is decided by a single environment variable, PROCESS_ROLE, which the image's entrypoint branches on:

Because there is one image, a feature or fix lands in every role at once, and the cloud, appliance, and Kubernetes deployments are the same software arranged differently — only the topology changes.

Why background work has to move

In a single-process install, the schedulers and pollers run as in-process daemon threads inside the serving process — that is fine when there is exactly one process. Run two web replicas and those threads run twice: duplicate device polling, duplicate backup runs, and — dangerously — duplicate key rotation.

The chart solves this by separating request-serving from background work. The decision is made by a small precedence rule in the platform: ENABLE_INPROC_SCHEDULERS forces the threads on, DISABLE_INPROC_SCHEDULERS forces them off, and one-off management commands (migrate, collectstatic, shell, …) never start them. When you enable the worker, the chart sets DISABLE_INPROC_SCHEDULERS=1 on every web pod and runs a single PROCESS_ROLE=worker Deployment with the threads enabled — so append-heavy pollers write one snapshot set per cadence regardless of how many web replicas you run.

---

The Kubernetes topology

A production install (web.replicaCount=3, worker + scheduler enabled) creates these workloads:

versus-web             Deployment   N replicas   API + SPA (stateless)
versus-worker          Deployment   1 replica    in-process pollers (single-writer)
versus-alert-scheduler Deployment   1 replica    Observe alert-evaluation loop
versus-<cronjob>       CronJob      per schedule  periodic one-shot commands
versus-gateway         Deployment   1+ replicas  collector control plane (optional)
versus-migrate         Job          pre-install/upgrade hook
versus-seed            Job          post-install/upgrade hook

web — the stateless tier

The web Deployment is the only tier you scale for throughput. It is purely request-serving (pollers disabled), so you add replicas freely.

• Readiness probes /api/updates/health/ — a DB-checking endpoint — so a pod takes traffic only once it can reach Postgres. Liveness probes /.
• When you run more than one replica, the chart adds a PodDisruptionBudget (minAvailable: 1) so a node drain or rolling upgrade can't take the whole tier down at once.
• Standard Kubernetes affinity, nodeSelector, and tolerations are passthrough values — set pod anti-affinity across nodes and topology-spread across zones to survive a node or rack loss.

Give the app three or more replicas across at least two failure domains and a node loss becomes a non-event: in-flight requests retry against the survivors, and the scheduler reschedules the dead pod.

worker — a single writer, on purpose

The worker Deployment is pinned to replicas: 1 with a Recreate update strategy — the old pod is torn down before the new one starts, so two workers never run at once. On node loss Kubernetes reschedules it; the brief gap in polling is safe because the pollers write append-only snapshots that thin rather than corrupt.

A companion single-replica alert-scheduler Deployment runs the long-lived Observe alert-evaluation loop the same way.

scheduler — CronJobs with a singleton guarantee

Periodic one-shot commands run as Kubernetes CronJobs, each with concurrencyPolicy: Forbid. For the non-idempotent jobs — device backups, vault secret rotation, KEK rotation — Forbid plus a single Job completion is the singleton guarantee: the platform will never run two backup passes or two key rotations concurrently. Every CronJob also sets startingDeadlineSeconds and bounded history limits.

The default schedule set:

You can retune any schedule, add jobs, or disable the whole set (scheduler.enabled=false) through Helm values.

Race-free rollout: migrate and seed Jobs

Database schema and one-time seeds are handled by Helm hook Jobs, not by every pod:

• The migrate Job runs migrate --noinput as a pre-install,pre-upgrade hook — schema changes are applied before any web, worker, or gateway pod rolls out, so replica startup is race-free. (Each pod's entrypoint would also migrate, but by then it's a no-op.)
• The seed Job runs the idempotent built-in seeds (workflow, report-card, and vault templates) once as a post-install,post-upgrade hook. Web pods are told to skip them with RUN_SEEDS=0, so seeds don't run per-pod.

---

Data: bring your own HA Postgres and object store

The chart runs the stateless compute; you provide the durable tiers, so you can use whatever your organization already operates and trusts.

• PostgreSQL is the system of record. The chart connects to an external / managed Postgres through DATABASE_URL — a managed service (RDS, Cloud SQL), a Postgres operator (e.g. CloudNativePG), or your own HA cluster. Run it with a primary plus replicas, automated failover, and continuous WAL archiving so a primary loss is transparent to the app and a recovery point is always available. Multi-tenant deployments can additionally point a NOBYPASSRLS role at the same database for row-level-security isolation.
• Object storage holds backups. Device-configuration backups are written to an S3-compatible store via BACKUP_S3_* (MinIO, S3, or an on-prem NetApp/ECS endpoint), with a local-disk fallback. On Kubernetes you must set this — otherwise a backup written to one pod's ephemeral disk is lost on reschedule. See Certificates and F5 BIG-IP Integration for what those backups protect.
• Identity runs as a dedicated OIDC provider with its own separate database; small or edge sites may instead use local JWT auth. Either way sign-in enforces SSO and MFA — see Security & Compliance.

Two secrets are DR-critical and must be escrowed out-of-band: the Django SECRET_KEY and the FIELD_ENCRYPTION_KEY (a Fernet key). Lose the field-encryption key and the encrypted columns — including vault material — are unrecoverable from a database restore alone. See Secret Vault.

---

The collector plane under HA

Collectors dial outbound-only to the gateway, so HA is mostly the gateway's problem, not yours. Enable it with gateway.enabled=true and it deploys as its own Deployment behind a LoadBalancer Service on 443, where it terminates the collectors' bearer-authenticated HTTPS/WSS.

The gateway runs its own poller-free process (the entrypoint keeps the device pollers off in this role to protect the gateway's connection pool). On pod or node loss, collectors reconnect automatically with backoff to the same public host, and in-flight operations resume — so a gateway restart is a momentary reconnect, not an outage. Collector Security and Collector Fleet cover the edge in depth.

---

Installing

Push the image to a registry your cluster can pull, generate the two app secrets, and install:

helm install versus deploy/helm/versus \
  --namespace versus --create-namespace \
  --set image.repository=<registry>/versus/web \
  --set image.tag=0.2.0 \
  --set secrets.secretKey="$SECRET_KEY" \
  --set secrets.fieldEncryptionKey="$FIELD_KEY" \
  --set secrets.databaseUrl="postgres://user:pass@pg-host:5432/versus" \
  --set web.replicaCount=3 \
  --set ingress.enabled=true \
  --set ingress.host=versus.example.com

Prefer to manage secrets yourself? Point secrets.existingSecret at a Secret holding SECRET_KEY, FIELD_ENCRYPTION_KEY, and DATABASE_URL and the chart uses it verbatim — so sealed-secrets or an external secret manager stays the source of truth. Add --set gateway.enabled=true to enroll remote collectors.

Setting config.deploymentMode=onprem (the chart default) turns on license enforcement. After install, open the UI, sign in as an admin, and apply your .veslic license under System → Licensing; it is node-locked to the install fingerprint shown there and stored as a single database row shared by every replica — there is no per-pod license file to distribute.

In-pod TLS (optional)

For environments where cleartext must never cross the pod network, enable the TLS sidecar: a small Caddy container terminates HTTPS inside the web pod (from a kubernetes.io/tls Secret, e.g. issued by cert-manager) and proxies to the app over loopback. Set web.tlsSidecar.enabled=true with a tlsSecret, and the Service targets the sidecar instead of the app.

---

Disaster recovery

High availability keeps you running through component failures — a lost pod, node, drive, or database primary. Disaster recovery is what protects you from data-level and site-level events: an accidental delete, corruption, or the loss of a whole cluster. The two are complementary, and the split is deliberate: uptime comes from redundancy; durability comes from backups.

The recovery posture rests on four backup streams, all landing in storage you control:

Rebuilding a lost cluster is: restore the escrowed keys, restore Postgres to a recovery point, restore the object store, then helm install at the matching image version and validate. For the most demanding estates, run a warm second site — a second cluster fed by asynchronous Postgres replication and object-store bucket replication, promoted on failover by repointing DNS. Because collectors dial outbound with backoff, they reconnect to the DR gateway automatically once it answers.

Whatever the topology, a backup isn't "done" until a restore has been demonstrated — schedule a periodic restore drill into a staging namespace rather than trusting that the backups would have worked.

---

How production differs from the demo

The same codebase, the same image — the operator and the topology are what change.

---

Related

• Platform Architecture — the layered system this deploys.
• Collector Security — the outbound-only edge and the gateway.
• Collector Fleet — operating collectors at scale.
• Secret Vault — envelope encryption and the DR-critical keys.
• Security & Compliance — the platform's overall security posture.
• Workflow API — driving automation against an HA install.