Secret Vault

Vault Hardening

How the Versus Vault protects secrets at rest and in use — per-secret envelope encryption with AES-256-GCM, a rotatable master key, fail-closed key checks at boot, a tamper-evident hash-chained audit, scoped app-principal access, and dual-control break-glass reveals.

The Versus Vault is where the platform keeps the credentials that make automation possible — device logins, API tokens, certificate keys. Because those secrets are the crown jewels, the vault is hardened along several independent axes: how a value is encrypted at rest, whether the system will even start with a weak key, who can read a secret and how that read is proven after the fact, and how a human breaks glass when a stored value must actually be seen. This page explains each of those, at the level a security reviewer needs.

None of this changes how you use the vault day to day — you still write and read secrets by path. Hardening is what happens underneath.


Envelope encryption at rest

Every secret value is sealed with envelope encryption rather than a single shared key. When you write a version, the vault:

  1. Generates a fresh 256-bit data encryption key (DEK) for that value alone.
  2. Encrypts the plaintext with AES-256-GCM using the DEK and a fresh 96-bit nonce.
  3. Wraps (encrypts) the DEK itself with a master key-encryption key (KEK), again with AES-256-GCM and its own fresh nonce.
  4. Stores the wrapped DEK and the ciphertext together as a single self-contained base64url token.

Because every secret gets its own DEK, compromising one sealed value never exposes another — there is no shared per-record key to unravel the whole store. And because AES-GCM is an authenticated cipher, the seal is tamper-evident: any bit-flip in the stored ciphertext or the wrapped DEK fails authentication on decrypt rather than silently returning garbage.

The sealed token is versioned with a small header (a magic marker plus a kek_ref tag) so its format can evolve without ambiguity:

magic 'VV1'      version marker
kek_ref          which master KEK sealed this blob
dek_wrapped      nonce ‖ AES-256-GCM(KEK, DEK)
value_wrapped    nonce ‖ AES-256-GCM(DEK, plaintext)

Each secret version stores this token in a dedicated sealed column. A legacy single-key column is retained read-only so that values written before envelope encryption can still be decrypted, but every new write goes through the envelope path and never persists plaintext under the old scheme.

The master key (KEK)

The KEK that wraps every DEK is resolved in one of two ways:

SourceWhen it's used
Explicit VAULT_KEKA 256-bit key supplied as base64 in the environment. Preferred for production — escrow it like any root key.
Derived from SECRET_KEYIf no explicit KEK is set, one is derived via PBKDF2-HMAC-SHA256 (200,000 iterations) with a vault-specific salt.

The derivation salt is vault-specific, so the vault KEK is distinct from the platform's field-encryption key and from other subsystems' keys — one derived key is never reused across purposes.

The kek_ref tag inside each sealed blob records which KEK sealed it. That indirection is what makes the master key rotatable: to roll the KEK you re-wrap the DEKs (cheap — only the small wrapped-DEK portion changes) without ever touching or re-encrypting the value ciphertext. It also leaves room to introduce an HSM- or KMS-backed KEK later without a storage-format change.


Fail-closed on weak keys

The committed code ships with development-default values for SECRET_KEY and the field-encryption key so a laptop checkout runs out of the box. Those defaults are dangerous in production — a known SECRET_KEY makes the derived vault KEK predictable, and a known field key makes encrypted fields world-decryptable.

So the platform refuses to boot in any non-debug deployment if either key is still the built-in default:

ImproperlyConfigured: Refusing to start: the following cryptographic key(s)
are still the built-in development default(s): SECRET_KEY. Set them to strong,
unique values (and escrow them).

This is a deliberate fail-closed posture: a misconfigured instance never comes up in a state where secrets could be trivially decrypted. The check can be waived only for throwaway environments, via an explicit VERSUS_ALLOW_DEFAULT_KEYS=1 opt-out — you have to say, in the environment, that this instance is disposable.


Tamper-evident audit

Every vault operation — write, read, delete, lease, rotate, reveal, enroll — is recorded twice: once in the platform's flat audit log, and once in a dedicated hash-chained vault ledger.

In the chained ledger, each event carries a monotonic sequence number and a hash computed over its payload and its predecessor's hash:

hash = SHA-256( seq · prev_hash · { actor, action, mount, path, details, at } )

Because each row commits to the one before it, any edit or deletion of a past row breaks the chain from that point forward. An integrity walk re-derives the chain and reports exactly where, if anywhere, it was broken:

{ "events": 10423, "intact": true, "broken_at": null }

An intact: false with a broken_at sequence number is a signal that historical audit rows were altered or removed after the fact — something a flat log alone cannot prove.


Who can read a secret — capabilities

Access to the vault is capability-based, evaluated per path. A policy grants a role a set of capabilities on a path glob:

CapabilityGrants
readRead a secret's value.
listEnumerate paths under a prefix.
writeCreate or update a version.
deleteSoft-delete a secret.
manageDestroy, force-release, and other administrative actions.
checkoutExclusively check a credential out for a bounded window.
rotateTrigger rotation of a secret.
revealOpen a break-glass reveal request (see below).
approveApprove someone else's reveal request.
sessionOpen a brokered session against a credential without ever seeing it.

The session capability is deliberately disjoint from read — it lets Privileged Session Management inject a credential into an SSH or HTTPS session that the operator never sees, so "use it" and "see it" are separately grantable.

Path policies can also carve out per-user namespaces. A single policy whose glob contains the ${user} token — for example personal/${user}/** — is substituted with the requesting user's username before matching, so one rule gives every user an isolated personal area and never matches on an anonymous or service call.


Machine access — app principals

Automation reads secrets through app principals, not human logins. An app principal authenticates with client credentials and receives a short-lived bearer token:

POST /api/vault/token
Content-Type: application/json

{ "client_id": "svc-collector", "client_secret": "••••••••" }
200 OK
{ "access_token": "…", "token_type": "Bearer", "expires_in": 900 }

A machine then fetches a secret by reference, presenting that bearer token:

GET /api/vault/fetch?ref=vault://root/devices/edge-fw-1/credential&field=password
Authorization: Bearer <app-principal token>

Several hardening rules apply to this path specifically:

  • App-principal tokens only. The fetch endpoint accepts an app-principal bearer token and never a human user's session token — machine and human access are separate doors.
  • Stored hashed. Only the SHA-256 hash of a token (and of the client secret) is persisted; the raw token is returned once at issue and is shared across workers/replicas so it survives restarts without an in-memory store.
  • Rate-limited. Both the token exchange and the fetch endpoint are throttled, so a leaked client_id can't be brute-forced against.
  • Reduced capability set. App principals can never hold checkout, reveal, or approve — the capabilities that model a human in the loop.
  • Hard-denied prefixes. Regardless of any policy grant, app principals are blocked from an inner set of protected path prefixes (device, certificate-key, SNMP, and integration namespaces). This is a hard deny that no policy can override, plus a per-principal allowed_paths glob that scopes each principal to exactly what it needs.

This is the same trust boundary the collectors rely on when they resolve device credentials — see Collector Vault Modes for how a credential can be resolved locally on the collector so it never leaves your network.


Break-glass reveals

Sometimes a human genuinely must see a stored value — a last-resort recovery, an out-of-band login. That path is deliberately high-friction and dual-controlled:

  1. Request. A user with reveal opens a reveal request against the secret, with a reason. This alone reveals nothing.
  2. Approve. A second principal with approve approves it. Self-approval is rejected — the approver cannot be the requester. Approval issues a one-time token valid for five minutes.
  3. Consume. The requester exchanges that token, once, for the value. It can't be consumed twice, and it can't be consumed after it expires.
  4. Rotate after. If the secret's rotation policy says so, the value is rotated immediately after the reveal — so a broken-glass credential is short-lived by construction.

Every step in that sequence is written to the audit chain, so a reveal is always attributable to a named requester and a named approver.


Rotation as a safety valve

Hardening isn't only about protecting a secret in place — it's also about shrinking the window a leaked secret is useful. Rotation is two-phase and verify-first:

  1. Generate a new credential.
  2. If the secret maps to a device, push the new credential to the device.
  3. Verify the device accepts it — and if verification fails, restore the old credential and abort rather than committing.
  4. Only then write the new version into the vault.

A scheduled sweep rotates secrets whose policy interval has elapsed, and defers any secret that is actively checked out until it's checked back in, so rotation never yanks a credential out from under a running job. Rotation-on-reveal and rotation-on-checkin let you tie a rotation to the exact moments a secret was exposed.


How it fits the platform

Vault hardening is one layer of the platform's defense-in-depth, alongside row-level tenant isolation, the outbound-only collector plane, and OIDC SSO with mandatory MFA. Secrets stay envelope-encrypted at rest, unreadable to a machine outside its scoped principal, un-seeable to a human without a second approver, and every touch is provable after the fact. For an end-to-end narrative of a single secret moving through these controls, see A Day in the Life of a Secret.


Related