Upgrades and migrations

The short version

1. Read the release notes for steps the release requires before it starts.
2. Back up the database (Backup and restore).
3. Run the role bootstrap and the migrations as the schema owner, keystone_owner.
4. Start the new version, and check /healthz and /readyz.

Docker Compose and the Helm chart do step 3 for you. Take a backup and restore it rather than migrating down in production.

Which database role runs migrations

Migrations always run as keystone_owner, through DATABASE_OWNER_URL. Never run them through DATABASE_URL.

DATABASE_URL cannot run migrations: the runtime role has no DDL rights by design, and the server refuses to start on a role that could (a superuser, a BYPASSRLS role, or one that owns objects). The server's own migrate command reads only DATABASE_OWNER_URL and stops with "DATABASE_OWNER_URL is required" without it.

Older upgrade notes show migrate -database "$DATABASE_URL" up. That predates the separation of roles and is wrong for current releases: use DATABASE_OWNER_URL.

Docker Compose

docker compose pull          # or build the new release
docker compose up -d

On every up, the db-roles service runs deploy/postgres/bootstrap-roles.sql as the PostgreSQL superuser (idempotent), then the migrate service applies pending migrations as keystone_owner and exits 0. The server starts only after both completed. To run the migrations alone:

docker compose run --rm migrate

Kubernetes

helm upgrade runs pre-upgrade hooks in order: the role bootstrap (databaseRoles.bootstrap), the migrations as keystone_owner from the keystone-migrate image of the same release, then ni-auth cell init. Only the hooks mount the owner's credentials (secrets.databaseAdmin). Database migrations are not rolled back by helm rollback.

Anything else

# as a PostgreSQL superuser, with the three role passwords in the environment
KEYSTONE_OWNER_PASSWORD=... KEYSTONE_APP_PASSWORD=... KEYSTONE_PLATFORM_PASSWORD=... \
  psql -v ON_ERROR_STOP=1 -h <host> -U <superuser> -d <database> -f deploy/postgres/bootstrap-roles.sql

# as the schema owner
DATABASE_OWNER_URL=postgres://keystone_owner:...@<host>/<database> ni-auth migrate up

The migration runner records who ran it from KEYSTONE_MIGRATE_USER (default keystone-cli).

After the migrations

The server checks at start, and refuses to start when any check fails:

• DATABASE_URL is not a superuser, has no BYPASSRLS, owns nothing and cannot create roles or databases.
• DATABASE_PLATFORM_URL is set, differs from DATABASE_URL, has BYPASSRLS, is not a superuser and owns nothing.
• keystone_tenancy_problems() returns no rows: every tenant table is isolated as its decision says.
• Outside development, neither URL uses a password shipped in the repository.

While it runs, the server repeats the tenancy check every five minutes (the job tenancy.drift); a problem raises a critical selfguard finding before the next restart would refuse to start.

Verify by hand if you want:

-- as the superuser: must return no rows
SELECT * FROM keystone_tenancy_problems();

Releases with special steps

Read each release's notes. Examples of steps releases have required:

• Database-enforced tenant isolation. The first release with the three roles needs a short maintenance window on an existing installation: stop the planes, run the role bootstrap (it transfers ownership of every object to keystone_owner), migrate as the owner, verify keystone_tenancy_problems(), switch DATABASE_URL to keystone_app, add DATABASE_PLATFORM_URL, remove the superuser's credentials from the planes, start, and watch the logs for no tenant for this statement (SQLSTATE 42501), which marks a code path without a tenant: report it, do not grant more.
• No rolling upgrade. A release that changes how stored values are encrypted may require every old replica to stop before the new one starts. On Kubernetes, scale the server to 0, wait until no old pod runs, then helm upgrade; with autoscaling, scale back to the minimum afterwards.
• Per-organisation audit chains. Run the migrations before starting the new servers; old servers fail their audited requests from the switch until they are replaced (fail closed). That release's down migration leaves entries written after the switch unverifiable, so restore a backup instead of migrating down.
• New required keys. Releases have made NI_OBSERVABILITY_SIGNING_KEY and KMS_MASTER_KEY required. The server refuses to start without them, and keeping them stable matters more than the upgrade (see Deploy and operate).
• PostgreSQL 15 or later. Migrations stop with "Keystone requires PostgreSQL 15 or later" on an older server, before changing anything. Upgrade PostgreSQL first.

Some migrations build indexes concurrently or validate constraints without the runner's statement timeout, so they can take a while on a large audit ledger without blocking writes. A follow-up migration checks the index is valid and stops the upgrade (dirty) if a build was interrupted; that migration's file has the repair steps.

Rolling out with the selfguard gate

Between rollout steps, ask the rollout gate whether the new release is healthy on live traffic:

curl --fail -sS -H "Authorization: Bearer $NI_AUTH_METRICS_TOKEN" \
  "https://<your Govern host>/metrics/selfguard/rollout-gate?release=$TAG&min_soak=10m&strict=true"

With strict=true it answers 412 while the release has not soaked, a node running it is not ready or has a critical finding, or the release regresses against the previous one. See Monitoring and self-healing.

When something goes wrong

Roll back

1. Stop the new version.
2. Restore the backup you took before the upgrade (preferred), or migrate down as the owner the number of migrations the upgrade added: ni-auth migrate down <n>. Data migrations that delete data cannot be rolled back.
3. Start the old version.

Dirty migration state

If a migration was interrupted or failed, the database is marked dirty at that version and the runner refuses to continue ("database is dirty at version N — manual intervention required"). Read the failed migration's file for repair steps (several have them) and fix the cause. Then, as the owner, clear the flag to the last good version and apply the failed migration again with the migration tool (the Docker Compose migrate service uses migrate/migrate):

migrate -path db/migrations -database "$DATABASE_OWNER_URL" force <failed version - 1>
migrate -path db/migrations -database "$DATABASE_OWNER_URL" up 1

Run the role bootstrap again first if the failure was a privilege error.