SLOs and On-Call

Versus Observe is the platform's built-in observability workspace — datasources, dashboards, and a Grafana-compatible alerting engine. This guide covers the alerting side: how you express a service-level objective (SLO) as an alert rule, how the engine decides when that rule is breaching, and how a breach reaches the person carrying the pager.

There is no separate "SLO object" to learn. An SLO in Versus is an alert rule whose query computes your objective (availability, error rate, burn rate, latency budget) and whose condition fires when the objective is at risk. The same engine that watches an F5 CPU gauge watches your error budget — you just write a different query.

Everything here lives under /api/observe/alerting/, and the same resources appear in the Observe UI.

---

How the engine works

Four models make up the evaluation pipeline. Understanding them is enough to reason about any alert you build.

• Eval group — a bucket of rules that share one evaluation interval. All rules in a group are evaluated together on aligned ticks (e.g. every 30s for critical alerts, every 5m for low-priority ones).
• Alert rule — a query-and-expression DAG plus a condition, a state policy, labels, annotations, and a notification target.
• Alert instance — one live series produced by a rule. A rule that queries "CPU by device" produces one instance per device; each instance carries its own state.
• State history — an append-only log of every state transition, per instance, for audit and timelines.

A dedicated scheduler process evaluates each group on its interval. On every tick it finds groups whose interval has elapsed, evaluates their non-paused rules in a thread pool, updates instance state, and routes any notification-worthy transitions. Instance timers are persisted in the database, so the scheduler is restart-safe — a pending alert's clock is not lost when the process restarts.

The rule DAG

A rule's data is an ordered list of nodes. Two kinds exist:

• Datasource nodes (type: "datasource") — a query dispatched to a telemetry provider (Prometheus, Loki, PostgreSQL, and others). Each has a refId and a look-back window.
• Expression nodes — processed server-side, in dependency order:
  • reduce — collapse a time series to one scalar per series (last, first, mean, min, max, sum, count).
  • math — a safe arithmetic expression across other nodes' scalars, referenced as $A, $B, …. Only arithmetic and comparison operators are allowed — no function calls, no attribute access — so an expression can never execute arbitrary code.
  • threshold — emit 1.0 (firing) or 0.0 (not firing) per series by testing a value against an evaluator (gt, gte, lt, lte, eq, ne, within_range, outside_range).

condition_refid names the node whose output drives the state machine. A value of 1.0 (truthy) from that node means the condition is firing for that series.

---

Defining an SLO alert

Because you have a math node and multiple datasource nodes, you can compute an SLO directly in the rule. The idiom is: query the numerator and denominator, divide them in a math node, then threshold the result.

Here is a rule that fires when a service's fast-window error rate exceeds 2% of an error budget — a single-window burn-rate check:

{
  "title": "Checkout SLO — error budget burning",
  "eval_group": "grp-critical",
  "condition_refid": "D",
  "pending_period_seconds": 120,
  "no_data_state": "NoData",
  "exec_err_state": "Alerting",
  "labels": { "severity": "critical", "team": "platform", "slo": "checkout-availability" },
  "annotations": {
    "summary": "Checkout error budget burning fast",
    "description": "5m error ratio is above the 2% burn threshold for the 99.9% availability SLO.",
    "runbook_url": "https://wiki.example.com/runbooks/checkout-slo"
  },
  "notification": { "mode": "direct", "contact_point_uid": "cp-pagerduty" },
  "data": [
    { "refId": "A", "type": "datasource",
      "datasource": { "uid": "prom-prod", "type": "prometheus" },
      "relativeTimeRange": { "from": 300, "to": 0 },
      "expr": "sum(rate(http_requests_total{job=\"checkout\",code=~\"5..\"}[5m]))" },

    { "refId": "B", "type": "datasource",
      "datasource": { "uid": "prom-prod", "type": "prometheus" },
      "relativeTimeRange": { "from": 300, "to": 0 },
      "expr": "sum(rate(http_requests_total{job=\"checkout\"}[5m]))" },

    { "refId": "RA", "type": "reduce", "expression": "A", "reducer": "last" },
    { "refId": "RB", "type": "reduce", "expression": "B", "reducer": "last" },
    { "refId": "C",  "type": "math",  "expression": "$RA / $RB" },
    { "refId": "D",  "type": "threshold", "expression": "C",
      "conditions": [ { "evaluator": { "type": "gt", "params": [0.02] } } ] }
  ]
}

The engine queries A and B, reduces each to a scalar, divides them in C, and thresholds the ratio in D. Because condition_refid is D, the state machine keys off that comparison.

Multi-window burn rate — the pattern SRE teams use to page fast on severe burns and slow on gentle ones — is the same shape with more nodes: add a long-window datasource pair, reduce and divide them, and require both windows to breach by combining the two threshold outputs in a final math node ($D * $H, thresholded > 0). Give the fast and slow variants different severity labels and route them to different contact points.

For a latency SLO, query your latency quantile, reduce it, and threshold against your objective. For a raw availability SLO, threshold the ratio directly rather than a burn rate. The engine does not care what the number means — only whether the condition node says 1.0.

Labels and annotations

• labels are static key/value pairs attached to every instance the rule produces (merged with any labels the query returns). They are your routing and identification keys — severity, team, slo, service. When a rule routes to Versus Desk, severity (critical/high/medium/low) maps to incident priority.
• annotations carry human context — summary, description, and runbook_url — and are included in the notification body so the responder has the runbook link in hand.

---

The alert lifecycle

Each instance runs an independent state machine. The states:

The normal path is Normal → Pending → Alerting → Normal. pending_period_seconds is your dwell time: set it to 0 for instant firing (a BGP peer dropping), or to a couple of minutes to ride out transient spikes (an SLO burn that must be sustained). When the condition clears, an Alerting instance transitions back to Normal — a resolve — and a resolve notification is sent.

no_data_state and exec_err_state decide what a gap or a failure means for your signal. Options are NoData/Error, Alerting, Normal, and KeepLast (hold the current state). For an availability SLO where "no data" means the scrape target is down, no_data_state: "Alerting" is often the right call; for a noisy log-rate rule, Normal avoids paging on quiet periods.

Only transitions into or out of Alerting are routable, so Pending churn never pages anyone. A NotificationLog row is written before each send and keyed by (fingerprint, contact_point, state), so a scheduler restart or a second worker cannot double-page for the same event.

---

Routing to on-call

A rule's notification field points at a contact point — the delivery target. Set it to {"mode": "direct", "contact_point_uid": "<uid>"}. Four contact-point types ship today:

The webhook type is the bridge to dedicated paging providers. Because it speaks the Alertmanager wire format, you point it at a PagerDuty, Opsgenie, or Splunk On-Call inbound integration URL and let that provider run the rotation and escalation policy. Store the provider's routing key or GenieKey as the webhook's authorization secret.

The servicenow and versus_itsm types are the escalation path when the SLO breach should become a tracked incident, not just a page. Versus Desk intake is idempotent per alert fingerprint: a re-firing alert appends a comment to the existing open incident rather than opening a duplicate, and a resolve comments on the ticket rather than silently closing it (an operator may be mid-investigation).

Contact-point secrets — SMTP passwords, webhook authorization headers, ServiceNow credentials — are held in an encrypted blob, never returned by the API, and set write-only.

Native on-call / paging providers

For workflow-driven paging (as opposed to alert-rule routing), Versus ships a vendor-neutral on-call client covering PagerDuty, Opsgenie, Splunk On-Call (VictorOps), xMatters, and OnPage. You register each as a Pager / On-Call source under Integrations, with its routing key and (optionally) a REST API key, then use it from a workflow's script step:

pager = versus.integrations.pager("Primary On-Call")   # or pager() for the first enabled source

key = pager.critical(
    "Checkout SLO exhausted",
    body="Error budget for 99.9% availability is spent; page primary on-call.",
    component="checkout", group="platform",
    dedup_key="slo-checkout-availability",
)
# later, when the workflow verifies recovery:
pager.resolve(key)

The client normalizes trigger / acknowledge / resolve across all five vendors. dedup_key is the idempotency handle — a repeat trigger with the same key updates the existing incident instead of opening a new one, and it is what you pass to resolve() or acknowledge(). This lets a self-healing workflow page, attempt a fix, and clear its own page — see Workflow Builder and Running Workflows via the API.

---

Operating alerts

Test before you trust

Dry-run a rule without touching state — it runs the queries and the expression DAG and returns the condition results:

POST /api/observe/alerting/rules/{uid}/test/

Send a probe to a contact point to confirm delivery and credentials:

POST /api/observe/alerting/contact-points/{uid}/test/

Silence and pause

• Silence an instance — force a single firing series back to Normal with a reason (recorded in history):

  POST /api/observe/alerting/instances/{id}/silence/
  { "reason": "Known maintenance window" }
  

• Pause a rule — toggle a whole rule off (it is skipped on every tick until re-enabled):

  POST /api/observe/alerting/rules/{uid}/pause/
  

Watch what's firing

List instances, filtered by rule or state, and inspect an instance's transition history:

GET /api/observe/alerting/instances/?rule={uid}&state=Alerting
GET /api/observe/alerting/instances/{id}/history/

Run the evaluator

The scheduler is a long-running management command. In a self-hosted install, run it as its own supervised process alongside the API:

python manage.py run_alert_scheduler --log-level INFO

It loops in the foreground, evaluating each group on its configured interval, and shuts down cleanly on SIGTERM/SIGINT.

---

API reference

All routes are under /api/observe/alerting/ and require an authenticated session.

Pager/on-call sources are managed separately under /api/integrations/pager-sources/.

---

Security and isolation

• Egress protection. Every outbound notification URL — webhook, ServiceNow — is validated before the request. Requests to private (RFC 1918) address space are blocked by default to prevent SSRF; an on-prem target must be explicitly allowed with allow_private_egress on the datasource or contact point.
• Encrypted secrets. Datasource and contact-point secrets are stored in an encrypted field, the same pattern as the rest of the platform. See Secret Vault.
• Tenant isolation. Every alerting model carries a tenant_id and is isolated per tenant, consistent with the platform's row-level security posture described in Platform Architecture.
• Auditability. State history, the notification dedup log, and (for escalations) the incident record give you an end-to-end trail from a breaching SLO to the page and the ticket.

---

Related

• Platform Architecture — where Observe sits in the stack.
• Workflow Builder — build self-healing workflows that page and remediate.
• Running Workflows via the API — trigger remediation from your own tooling.
• Secret Vault — where datasource and contact-point credentials are held.
• Security & Compliance — the platform's security posture.