Monitoring and self-healing

Probes and metrics

NI_AUTH_METRICS_TOKEN must be at least 32 bytes (openssl rand -base64 48). Scrape with it as a bearer token.

The probes, /metrics and /metrics/selfguard are exempt from admission control, so they answer while the node is overloaded.

Selfguard

Selfguard keeps each Govern process healthy on its own. It watches the process and the dependencies Govern needs (PostgreSQL first), detects anomalies, sheds load under pressure, repairs what it safely can, verifies that the repair worked, and escalates to operators when it did not. It is about Govern itself, not tenant workloads.

• Air-gap first. No external calls. Alerts go to the log and, optionally, to a webhook you run.
• Audited. Every automatic or operator action and every finding transition is written to the audit ledger in the same transaction as the change.
• Database outages are the main case. Detection, healing and alerting run from memory; history is written to PostgreSQL in the background, buffered (up to 10,000 records) while it is down.
• Safe by construction. Actions run under a mode, a cluster-wide kill switch, cooldowns, an hourly budget (12 per node), prechecks and post-action verification. Disruptive actions need an operator's approval.

It also runs in the data plane (keystone-bridge), the compliance plane (keystone-comply), the provisioner and the sandbox daemons, from memory and without history or approvals.

What it watches

Set GOMEMLIMIT or a container memory limit; without either, memory pressure cannot be measured.

Detectors and playbooks

The full list, with thresholds, is in the operator runbook shipped with each release. Baselines warm up first and never fire before that.

Admission control

The outermost HTTP handler limits concurrency adaptively (starting at 256, between 32 and 4096) and classifies requests:

Once in-flight requests reach half the limit, one organisation may hold at most a quarter of it (tenant bulkheads). A rejected request gets 503 (node overloaded) or 429 (tenant share exceeded) with Retry-After and X-Keystone-Shed. Long-lived streams give their slot back once they start. NI_AUTH_SELFGUARD_ADMISSION=measure measures without rejecting.

Operate selfguard

All routes are for super admins. Bodies are JSON; every change needs a reason or note, recorded in the audit event.

export KS=https://<your Govern host>
alias ks='curl -sS -H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json"'

ks $KS/api/v1/admin/selfguard/nodes
ks "$KS/api/v1/admin/selfguard/findings?state=open"
ks "$KS/api/v1/admin/selfguard/actions?limit=20"

If the API answers 401 during an incident, the database is probably down and sessions cannot be checked. Use the break-glass status with the metrics token:

curl -sS -H "Authorization: Bearer $NI_AUTH_METRICS_TOKEN" $KS/metrics/selfguard

The kill switch reaches every node within one tick (10 s). Detection and alerting continue while actions are paused. A drill raises a finding whose message starts with [DRILL]; its playbook runs as a dry run unless "live": true. Run a drill after enabling selfguard, after changing the alert webhook, and at on-call handover.

Disruptive actions:

• drain_node makes /readyz answer 503 for 3 minutes, so load balancers route around the node. It is never proposed for the only active node or while another node drains. Approve when the node is clearly sick and the others have headroom.
• terminate_idle_transactions ends Govern's own sessions idle in a transaction for over 10 minutes. Before approving, look at what they hold (pg_stat_activity); reject when it is a migration or a maintenance session. Then set idle_in_transaction_session_timeout on Govern's database role.

Diagnostic bundles (goroutines.txt, heap.pprof, status.json) are written under NI_AUTH_SELFGUARD_DIAGNOSTICS_DIR; mount a volume there to keep them across restarts.

Alerts

Alerts go to the structured log and, when NI_AUTH_SELFGUARD_ALERT_WEBHOOK_URL is set, are posted as JSON with kinds finding.opened, finding.escalated, finding.resolved, action.approval_required, action.failed, action.unverified and remediation.exhausted. Each request carries X-Keystone-Timestamp and X-Keystone-Signature: sha256=<hex HMAC-SHA256(secret, timestamp + "." + body)>; reject timestamps older than a few minutes. Delivery is retried up to 5 times.

The rollout gate

curl --fail -sS -H "Authorization: Bearer $NI_AUTH_METRICS_TOKEN" \
  "$KS/metrics/selfguard/rollout-gate?release=$TAG&min_soak=10m&strict=true"

It answers {"release", "proceed", "reasons", ...} and, with strict=true, 412 while: no node reports the release; a node running it is not ready or has an open critical finding; it has run for less than min_soak; it regresses against the previous release (error ratio more than twice, or p99 more than 1.5 times, the baseline's, with at least 200 requests on each side); or a critical cluster-wide finding is open. Use it as an Argo Rollouts analysis or a Helm post-upgrade hook. Govern does not roll back by itself; the gate lets your tooling do it. The Helm chart sets NI_AUTH_SELFGUARD_RELEASE to the image tag.

Settings

All optional. An invalid value stops the start with a message naming it.

Give server pods a terminationGracePeriodSeconds of at least 30 (the chart uses 60), the bridge at least 50 and the compliance plane at least 45, so their last flushes finish on shutdown.

Metrics to alert on

Not covered yet

• No console page for selfguard beyond the platform area's self-healing page; the API is complete.
• No automatic rollback of a bad release: the rollout gate's 412 lets your tooling do it.
• No circuit breaker per model provider in the model gateway yet.