Spaces and sources

Spaces

A space holds the sources of one node of your organisation. Govern keeps one space for every live organisation, department, team, group and project, through a reconciliation job that runs every 10 minutes and once at start. Space names follow renames.

• When a node is deleted, its space is archived, never deleted: its sources and chunks are kept, but it is no longer served anywhere (lists, retrieval, lenses, snapshots of live content). A space whose node comes back is restored. Administrators list archived spaces with GET /api/v1/context/spaces?archived=true.
• A project registered moments ago gets its spaces at the next reconciliation, or at once with POST /api/v1/context/project/space and {"projectId": "..."} (needs context:space:write on the project space; idempotent).
• Personal spaces are created when a member first promotes a document (below). Only the owner can use them, in every rollout mode and for super admins too.

Sources

Other kinds (docs or wiki, conversations, regulatory, custom, Linear issues) are refused at creation: no connector exists for them.

Create a source

POST /api/v1/context/spaces/{spaceId}/sources
Content-Type: application/json

{
  "kind": "codebase",
  "name": "payments-api",
  "config": {
    "repoUrl": "https://github.com/acme/payments-api",
    "branch": "main",
    "includePaths": ["src/**"],
    "excludePaths": ["src/generated/**"]
  }
}

The request needs context:space:write. It answers 201 and queues indexing in the same transaction. An issues source takes {"repoUrl", "installationId"?, "includeCiFailures"?}. A source may also carry classification and dlpAction, which can only tighten what the space inherits (Classification).

installationId is optional. When it is omitted, Govern asks GitHub which installation of the organisation's App covers the repository. Errors: 422 context.github_app.not_configured (no App), 422 context.source.installation_not_found (the App is not installed on the repository), 503 (GitHub unreachable).

Routes

Connect GitHub

Govern reads private repositories only through a GitHub App that your organisation owns. Personal access tokens are not accepted.

1. Create a GitHub App with repository permissions Contents: read and Metadata: read, and the push webhook event. Nothing else and no write permission. For issues sources also grant Issues: read and Pull requests: read, and Actions: read for CI failures; without them those sources fail with GitHub's permission error.

2. Mount a vault engine. The App's key is kept in the organisation's Keystone Vault, in a KV version 2 engine mounted at secret/ (the path is secret/data/keystone/context/github-app unless the operator changed KEYSTONE_CONTEXT_GITHUB_APP_SECRET_PATH). Without it, saving the App answers 409 context.github_app.vault_not_ready.

3. Save the App (context:space:admin on the organisation's root space):

   PUT /api/v1/context/github-app
   Content-Type: application/json
   
   {"appId": 12345, "privateKey": "-----BEGIN RSA PRIVATE KEY-----...", "webhookSecret": "..."}

   The key must be an RSA key of at least 2048 bits. Responses never contain it: they carry appId, keyFingerprint (as GitHub displays it), enabled, version and webhookConfigured. Saving again rotates the key; a rotation without webhookSecret keeps the stored secret. GET and DELETE on the same path read and remove the configuration.

4. Set the webhook URL of the App to https://<your Govern host>/api/v1/context/github/webhook/<orgId> with the same secret.

5. Install the App on the repositories you want indexed.

Each fetch reads the key from the vault, signs a short-lived App token and mints an installation token restricted to the one repository and to read access. Tokens are cached in memory until 5 minutes before expiry and never written to the database, logs or errors.

Push webhook

POST /api/v1/context/github/webhook/{orgId} carries no Govern credential. It is authenticated only by X-Hub-Signature-256 (HMAC-SHA256 with the organisation's webhook secret), checked before any other work, and rate limited per source address and per organisation.

• A push to a branch re-indexes every codebase source of the organisation with the same installation, repository and branch. Tags, deleted branches and other events are ignored.
• A delivery id already accepted within 72 hours answers 202 {"duplicate": true} and re-indexes nothing.
• An unknown organisation, a missing secret and a bad signature all answer the same 401.
• GET /api/v1/context/github-app/deliveries lists the 50 newest accepted push deliveries (kept 72 hours).
• Missed deliveries are caught by the maintenance scan.

Only hosts in KEYSTONE_CONTEXT_GITHUB_HOSTS (default github.com) are accepted, at creation and before every fetch, because the connector sends the GitHub token to that host. Add your GitHub Enterprise host there.

Indexing

Sources are indexed by a background job. Retrieval keeps serving the previous version until a new one commits.

• Versioned commits. A run records the source's version before it fetches and commits only if no newer version was written meanwhile. A run that loses that race is re-queued, not failed. A shutdown during a run leaves the source as it was.
• Failures. Permanent failures (no credentials, 401, 403, 404, invalid configuration, too large, nothing indexable) set the source to error without retry. Transient failures retry with backoff, then go to the dead-letter list; a dead job is run again with POST .../reindex. Any successful run clears error.
• Idempotent. When the content and the embedding settings equal the current version, nothing is re-embedded and no version is written.
• Maintenance scan every 15 minutes: sources indexed with another embedding model are marked stale and re-indexed; sources stuck in pending or indexing for 15 minutes with no job are re-queued.

Chunking

A file that cannot be parsed falls back to line windows. Lines longer than the chunk size are split. Files with a NUL byte or invalid UTF-8 are binary and skipped; symbolic links and path-traversal entries are skipped.

Codebase sources skip by default .git/, vendor/, node_modules/, dist/, build/ and testdata/, minified bundles, source maps, *.pb.go, *_generated.go and lockfiles. KEYSTONE_CONTEXT_DEFAULT_EXCLUDES replaces that list (none turns it off); a source's excludePaths apply on top.

Environment, secret, key and credential files are never fetched: the reader skips them by path before any content is read, so they are never chunked, embedded or stored. The rule list is the one described in Agent instruction files.

Repository configuration file

A .keystone.yaml or .keystone.yml file in any directory is always read and never indexed. The strictest classification: it declares raises the source's classification. An unknown value fails the run, and the previous version keeps serving.

Code facts

For codebase sources the index job also records, per chunk: owners from CODEOWNERS; for Go, the qualified symbol, its direct callers, callees and tests; a summary of each go.mod; and the last commit that changed each file (best effort). Retrieval uses the call graph to append up to six related chunks after the top results (see Retrieval).

Run report

GET /api/v1/context/sources/{sourceId}/status returns the queued or running job (job.state, attempts, nextRunAt) and lastRun: outcome (indexed, unchanged, failed), index version, revision, duration, files seen, indexed and skipped, up to 50 skipped files with their reason, chunks, chunks reused, whether chunks were embedded and why not (dlp_action, external_endpoint, policy_unresolved).

Personal documents

A member adds a document to their own personal space:

POST /api/v1/context/me/sources
Content-Type: application/json

{"name": "my notes", "path": "notes.md", "content": "..."}

• At most 1 MiB of UTF-8. Promoting the same name again replaces the content.
• Personal sources are private: only the owner retrieves them, they are never inherited by other spaces, they cannot be added to lenses, and snapshots skip them.
• Environment, secret and key files are refused with 400.
• GET /api/v1/context/me/sources returns the personal space id and its sources.
• On the hosted service the number of personal sources is a counted quota (default 100). A self-hosted installation does not limit them.

From the command line

The Keystone command-line tool shipped with the release calls the same API with KEYSTONE_URL and KEYSTONE_TOKEN; the server enforces every permission.

# Register the current repository as a project, its spaces and a codebase source
ni keystone init --team=platform --write-config

# Add a document to your personal context
ni keystone context promote --source=notes.md

# Register every repository of a GitHub organisation (GITHUB_TOKEN or GH_TOKEN)
ni keystone admin projects import --github-org=acme --team-map=teams.txt --dry-run

The project is detected from the nearest .keystone.yaml project_id, then the git origin remote. --write-config adds project_id to .keystone.yaml and never overwrites a different one. In the import team map, each line is <repository or glob>: <team> and the first match wins; archived, disabled and forked repositories are skipped by default.