Compass Model Roles + stable-name provider routing (RIG-2845)
Ledger-impact: none (platform surface is ungoverned by the design ledger — no DECISIONS.md delta).
Status: draft for freeze (red-teamed + folded — the resolver-seam claim and
the container-registry gap the critique caught are corrected below; the
load-bearing forks are batched into Open Questions for Matt). Composes with
the frozen RIG-1715 gateway record
(docs/designs/platform/compass-server-llm-gateway/design.md) and the
config-passthrough record
(docs/designs/product/compass-agent-config-passthrough/design.md); consumes
the RIG-2562 model-eval suite (in design in the internal fleet-tooling repo)
as evidence, not as a blocker.
Problem / Intent
Section titled “Problem / Intent”Compass runs agents in named roles, but today each agent’s model is a
hard-coded selector: the Runner exports one opaque COMPASS_MODEL string per
agent and the policy of which role gets which model lives nowhere. This record
defines the policy layer the gateway record explicitly scoped out
(compass-server-llm-gateway/design.md:407-413): a first-class Compass
role→model policy (user- and agent-editable) and stable Compass model names
(e.g. claude-opus-4-8) that resolve to a preferred backend ORDER across the
providers a user holds — decoupling which-model (role policy) from
which-backend-serves-it (credential availability).
Global Constraints
Section titled “Global Constraints”Inherited from the gateway record’s Global Constraints
(compass-server-llm-gateway/design.md:678-710), which BIND this record:
- All-Go server, one ratified Bun exception: “the LLM gateway is the
adopted OMP TS auth-gateway and stays TS/Bun for the medium term …
compass-side gateway code (ingestion, UsageService, token minting, stack
supervision) is Go under
go/internal/” (compass-server-llm-gateway/design.md:680-686). The stable-name resolver this record adds is compass-side wiring INSIDE the already-ratified Bun gateway boot entrypoint — no new non-Go runtime. - Fork changes are seam-shaped and upstreamable: “compass touches the
gateway only at injection points (auth verifier,
AuthStorageimplementation, usage hook) — never the routing core” (compass-server-llm-gateway/design.md:687-690). This record adds exactly ONE small, upstreamable seam widening (OQ-1): the injected resolver boot option goes fromresolveModel: (modelId) => Model<Api> | undefined(synchronous, tenant-blind —forks/oh-my-pi/packages/ai/src/auth-gateway/ server.ts:55,65) to(modelId, identity) => Promise<Model<Api> | undefined>, the direct sibling of the RIG-1715 T3authorize(req) → identityboot-option change (compass-server-llm-gateway/design.md:270-281). It stays at the injection point — the routing core is untouched — under the recommended hard-down failover (OQ-3); the pre-first-byte alternative is the one path that WOULD touch the core, which is exactly why OQ-3 raises it as a separate fork-seam question rather than folding it silently. The earlier draft’s “no seam change” claim was wrong: the tenant-blind sync signature cannot carry per-caller candidate selection, which needs the request identity and an async pool query. compass.v1is the sole UI↔server door (design.md:691-694): the role-policy read/write API is a proto change with regenerated clients.- Agents never hold upstream provider credentials post-T5
(
design.md:695-696); role policy therefore NEVER materializes provider keys — only model selectors. - Don’t fork the SDK settings schema: the passthrough record’s ruling —
a curated Compass-owned settings schema “creates a second schema that
must chase the SDK’s
SETTINGS_SCHEMA(5k+ lines,settings-schema.ts:383-5450) on every fork bump — a permanent maintenance tax” (compass-agent-config-passthrough/design.md:593-596). Role policy is Compass-owned server-side data materialized INTO the existing OMP surfaces (COMPASS_MODEL, fleetsettings/config.ymlmodelRoles), never a parallel settings schema. - Markdownlint-clean record; Conventional Commits;
Co-authored-by: Matt Wilkinson <matt@rigel.build>on the record commit (driver-owned). - Ledger: platform is ungoverned →
Ledger-impact: none; no DECISIONS.md delta ships with this record.
Approach
Section titled “Approach”(a) Role taxonomy: two axes, reconciled — Compass persona roles ABOVE, OMP tier roles BELOW
Section titled “(a) Role taxonomy: two axes, reconciled — Compass persona roles ABOVE, OMP tier roles BELOW”OMP’s built-in model roles are model tiers within one session, not agent
personas. ModelRole is the 10-member union
(forks/oh-my-pi/packages/coding-agent/src/config/model-roles.ts:22-32):
export type ModelRole = | "default" | "smol" | "slow" | "vision" | "plan" | "designer" | "commit" | "tiny" | "task" | "advisor";with UI metadata in MODEL_ROLES (model-roles.ts:42-53 — e.g.
smol: { tag: "SMOL", name: "Fast", … }, slow: { tag: "SLOW", name: "Thinking", … }), the canonical id list MODEL_ROLE_IDS
(model-roles.ts:55-66), and an alias selector system
(MODEL_ROLE_ALIAS_PREFIX = "@" at :9, legacy pi/ at :12, default alias
"*" at :15). Custom roles beyond the 10 are already supported: settings
cycleOrder, modelRoles, and modelTags entries introduce them
(getKnownRoleIds, model-roles.ts:86-88).
Compass agent roles are a DIFFERENT axis: the persona/duty a whole container
runs as. The Runner already carries them as env — “COMPASS_PERSONA is the
identity overlay appended to the system prompt, COMPASS_ROLE is the
operator-set block-0 selector delivered as the container’s
customSystemPrompt” (go/internal/runner/agent_exec.go:38-40) — and the
fleet’s working set is the persona roster (Manager/Supervisor, designer,
design-critic, implement, implement-hard, task/scout, review, …).
Decision: Compass’s role set is Compass-native and persona-keyed; OMP’s 10 tier roles stay unchanged underneath. Compass role→model policy is a map
CompassRole → { model: StableName, thinking?: Level, tierOverrides?: Record<OmpModelRole, StableName> }- The
modelfield is the container’s MAIN model, materialized asCOMPASS_MODEL— the existing, tested seam: the Runner exports it (agent_exec.go:83-85spec.Env["COMPASS_MODEL"] = e.Model) and the agent entrypoint forwards it opaquely (“Returned as an opaque pattern string forcreateAgentSessionto resolve against its own model registry”,packages/compass-agent/src/cli.ts:138-139, impl:147-150). tierOverrides(optional, usually empty) rides the passthrough seam as the fleet settings document’smodelRolesrecord (settings-schema.ts:564modelRoles: { type: "record", default: EMPTY_STRING_RECORD }, typedRecord<string, string>atsettings-schema.ts:5743). Inside the container OMP’s own resolver (resolveConfiguredRolePattern,model-resolver.ts:1011-1045; built-in chainspriority.json:2-59) keeps handling tier resolution — compaction onsmol, subagents ontask, commit messages oncommit— with no Compass code in that loop.
This preserves the passthrough record’s already-ruled precedence untouched:
“the Runner’s COMPASS_MODEL → modelPattern stays authoritative for the
main model; the fleet settings’ modelRoles govern the role-resolved
models (compaction, subagent, etc.)”
(compass-agent-config-passthrough/design.md:264-268; the record’s inline
cli.ts:397 anchor has since drifted — resolveModelSelector is defined at
cli.ts:145 and produces modelPattern at cli.ts:865). RIG-2845 does not
re-litigate that seam; it fills in WHO decides the values flowing through it.
Adopting the OMP 10 as Compass’s role set was rejected (see Alternatives):
commit/tiny/vision are per-session utility tiers with no per-agent
policy meaning, and the fleet’s real dial — “the reviewer runs Opus, the
scout runs a fast model” — is persona-keyed.
Name-collision caveat: designer is BOTH an OMP tier role
(model-roles.ts:28, MODEL_ROLES.designer “Designer”) and a Compass
persona. The two axes are structurally separate — a tierOverrides.designer
key sets the tier model inside any container, whereas role policy for the
persona designer sets that container’s MAIN model — but the shared word is
a foreseeable operator confusion; the UI (P4) labels the axis explicitly
(persona role vs tier override).
(b) Stable model names over direct backend routing (the core fork)
Section titled “(b) Stable model names over direct backend routing (the core fork)”Decision: stable-name routing. Compass exposes stable model names
(claude-opus-4-8, gpt-5-5, gemini-3-1-pro, …) as the ONLY vocabulary
role policy speaks. Each stable name maps to an ordered backend-candidate
chain — e.g. for claude-opus-4-8:
anthropicvia own subscription (OAuth),anthropicvia own API key,- alternates the user holds (
openrouter/anthropic/claude-opus-4.8,amazon-bedrock/...), in a per-name declared order.
Rationale, grounded in the current routing:
- The status quo hard-couples policy to provider inventory. Today
COMPASS_MODELcarries a concreteprovider/idselector (e.g. tests pin"anthropic/claude-opus-4-5",cli.test.ts:955), so one role config only works for users holding exactly that provider. A stable name makes one role config valid across users holding different providers — the gateway record’s own framing: “how a stable model name (e.g.claude-opus-4-8) maps to a preferred BACKEND order across the providers a user holds (own subscription, own API key, then alternates like OpenRouter or Bedrock)” (compass-server-llm-gateway/design.md:408-410). - The resolution point already exists and is caller-scoped. Post-T5,
every agent model call egresses through the gateway (
transport: "pi-native"routes “every model under this provider … via the auth-gateway’sPOST /v1/pi/streamendpoint instead of the per-provider SDK”,models-config-schema-bundle.ts:284-287), and the gateway resolves the request’smodelIdthrough the injected resolver boot option (resolveModel,forks/oh-my-pi/packages/ai/src/auth-gateway/server.ts:55, called on both foreign-wire and pi-native paths atserver.ts:377,569). The compass boot entrypoint (gateway record T1) supplies this function — stable names are implemented by REPLACING that lookup with a table-backed resolver. This is NOT zero-fork: the seam is widened to carry the caller identity and to be async (OQ-1, Global Constraints), because per-caller candidate selection needs both — but the widening stays at the injection point, not the routing core. - Per-caller resolution needs a tenant, and only the gateway has one.
Backend order depends on which credentials THIS caller’s pool holds
(own-then-shared,
compass-server-llm-gateway/design.md:375-395). The agent-side registry can’t know that post-T5 (agents hold only a gateway bearer); the gateway’s per-agent token→tenant mapping (RIG-1715 T3) can.
Composition with pool resolution (compose, don’t fork). Resolution is a strict two-stage fold, each stage owned by the record that defined it:
stable name ──(RIG-2845: per-name ordered candidate list)──▶ [(provider, upstreamModelId), …]each candidate ──(RIG-1715: pool membership + precedence)──▶ credential | ∅first candidate with a non-empty pool result winsStage 2 is RIG-1715’s fold verbatim — own-before-shared across sources,
OAuth-before-API-key within a provider (“Own-before-shared … riding OMP’s
existing within-provider type precedence unchanged — a deliberate OAuth/login
credential wins over a stored API key”, compass-server-llm-gateway/ design.md:383-387). RIG-2845 never ACQUIRES or mints credentials and owns no
precedence — it only orders PROVIDERS; the own-sub-then-own-key prefix of
Matt’s example order falls out of stage 2 automatically when the chain lists
the native provider first, and the chain’s remaining entries order the
alternates (OpenRouter before Bedrock, etc.). It does, however, read one
credential-availability bit per candidate: a side-effect-free dry-run peek
(“would this provider yield a usable credential for this tenant”) to choose
the candidate. That is a NEW read surface on the T2 pool resolver (named in
P1), distinct from the routing core’s later stateful acquire
(storage.getApiKey(model.provider, sessionId), server.ts:419); the peek
neither acquires nor sets session stickiness.
Failover trigger (see OQ-3 for Matt’s ruling): a candidate is skipped on
credential absence and on marked usage-limit — both observable in the dry-run
peek before the upstream call (the gateway already derives session identity and
calls markUsageLimitReached on gateway-mediated requests — coding-agent
CHANGELOG.md:4507). The recommendation is v1 hard-down on a provider outage;
admitting a pre-first-byte upstream failure (connect error / immediate 5xx /
429-without-usage-limit) is NOT seam-shaped — the widened resolver returns one
model before the upstream call, so it would need a routing-core loop or a
compass-side wrapper (OQ-3). Mid-stream error failover stays v2. Stage-1
candidate choice is DETERMINISTIC (a pure function of the ordered chain plus
current pool marks), so absent a mark change the same candidate is chosen every
request and prompt-cache continuity holds without the seam carrying sessionId
(derived downstream at server.ts:410/:578).
Direct routing survives as an escape hatch: a provider/id selector that is
not a known stable name passes through to the existing exact-match lookup
unchanged, so tests’ canned providers (e2e/fixture.go:381-384) and
power-user pins keep working.
(c) Config surface: a Compass-owned policy store, materialized through existing seams
Section titled “(c) Config surface: a Compass-owned policy store, materialized through existing seams”Role→model policy and stable-name chains are server-side Compass data in Postgres, not settings-file content. Three layers:
- Policy store (compass-server, Go). Two small tables (folded into the
squashed migration per store discipline,
compass-server-llm-gateway/design.md:704-706):model_role_policy(scope_user_id, compass_role, stable_name, thinking_level, tier_overrides jsonb)andstable_model_names(name, display_name, candidates jsonb /* ordered (provider, upstream_model_id) */). Read/written viacompass.v1RPCs (below) — editable by a user through the UI and by an agent through the same authenticated door. - Materialization (runner). At agent start the Runner resolves the
agent’s Compass role against the policy store and exports the result on
the EXISTING env seam:
COMPASS_MODEL=<stable name>(agent_exec.go:83-85) and, whentierOverridesis non-empty, amodelRoles:block merged into the passthrough bundle’ssettings/config.yml(CP-1 deliver row — the passthrough record already names “model roles (settings.ts:36ModelRole; role storagesettings-schema.ts:72ModelRoleStorage)” among the keys the settings document carries,compass-agent-config-passthrough/design.md:190-192). No new container contract; empty policy degrades to today’s behavior (“Empty Model … is omitted rather” than exported,agent_exec.go:42). - Gateway resolution (Bun entrypoint, compass-side). The compass boot
entrypoint’s
resolveModelimplementation consultsstable_model_names(via the same Server RPC channel the T2AuthStorageadapter uses) and the caller’s pool to pick the winning candidate, returning a concreteModel<Api>to the untouched routing core.listModels(server.ts:66-67) lists the stable names so/v1/modelsshows the Compass vocabulary.
This satisfies the schema-fork constraint by construction: the only settings
content Compass writes is values under the SDK’s own modelRoles record key;
the policy schema lives in Compass proto/SQL where Compass already owns
schema.
(d) Shipped defaults + docs deliverable
Section titled “(d) Shipped defaults + docs deliverable”This record owns the SHAPE and the shipping of per-role defaults (a seeded
model_role_policy scope-default row set + a docs page “recommended model
per role per provider you hold”); the RIG-2562 model-eval suite (per-role
external-first composite scoring, designed in the internal fleet-tooling
repo) supplies the EVIDENCE that picks the values. Until its numbers land,
defaults are seeded from the fleet’s current practice (the same practice
OMP’s priority.json:24-48 slow chain encodes: codex-tier first, then
opus-tier). Consuming RIG-2562 output is a data update, never a schema or
code change.
The fleet model-selection spec (RIG-2573/DL-025, docs/specs/platform/ model-selection.md) is absent in this repo — docs/specs/ contains only
product/ and brand/ (verified by glob this session); it lives in the
internal fleet-tooling repo. Defaults MUST cite its optimization target
conceptually when the docs deliverable is written.
Alternatives considered
Section titled “Alternatives considered”Direct backend routing (status quo) — rejected
Section titled “Direct backend routing (status quo) — rejected”Keep COMPASS_MODEL carrying concrete provider/id selectors and let each
agent’s registry resolve them. Rejected: (a) one role config cannot serve
users with different provider inventories — the selector bakes in the
backend; (b) post-T5 the agent registry cannot even see which providers the
caller holds (agents carry only a gateway bearer,
compass-server-llm-gateway/design.md:695-696), so agent-side fallback
across backends is structurally impossible; (c) the OMP-side priority-chain
machinery (priority.json, rolePriorityDefaults) is pattern-matching over
locally-visible models — the wrong layer once visibility moved server-side.
Survives only as the pass-through escape hatch for unknown names.
Adopt OMP’s 10 tier roles as the Compass role set — rejected
Section titled “Adopt OMP’s 10 tier roles as the Compass role set — rejected”Make Compass policy keyed by default/smol/slow/… directly. Rejected:
the OMP roles are intra-session TIERS (“Fast”, “Thinking”, “Commit” —
MODEL_ROLES, model-roles.ts:42-53), orthogonal to which agent is running.
A “reviewer uses Opus” policy is inexpressible in tier vocabulary without
per-persona settings files — which is exactly the scattered-constants status
quo. The tiers remain fully available underneath via tierOverrides.
Persona-keyed superset inside OMP’s custom-role mechanism — rejected
Section titled “Persona-keyed superset inside OMP’s custom-role mechanism — rejected”OMP already accepts custom role names in modelRoles
(getKnownRoleIds folds settings-defined roles in, model-roles.ts:86-88),
so Compass personas could be shipped as custom OMP roles in the fleet
settings document. Rejected: the container only ever RUNS one persona — a
per-container settings record keyed by all personas is dead weight delivered
everywhere, the main-model seam would shift from the ruled-authoritative
COMPASS_MODEL to modelRoles.<persona> (re-litigating passthrough OQ-3),
and policy edits would require settings-bundle redelivery instead of a
policy-store write.
Stable names resolved agent-side (registry aliases) — rejected
Section titled “Stable names resolved agent-side (registry aliases) — rejected”Ship stable names as models.yml aliases via the passthrough CP-4 channel
(ModelsConfigFile, models-config.ts:105), with the container resolving the
backend. Rejected FOR RESOLUTION: fallback order depends on the caller’s
credential pool, which post-T5 exists only gateway-side, and it would fork
alias semantics per container instead of one authoritative table. Note the
distinction the drafting missed: the stable name STILL must appear agent-side
as a LISTED registry entry (an id with Api/context/cost metadata) or the
container refuses to boot (OQ-2) — listing ≠ resolution. So the CP-4 channel
carries generated stable-name LISTINGS (from the one authoritative table),
while resolution stays gateway-side.
Ordering: P1 → P2 → P3 → P4; P5 (docs/defaults) parallel after P1. P1/P3
depend on RIG-1715 T2/T3 (pool resolver + tenant identity) having landed;
P1 additionally depends on the resolver-seam widening being ratified (OQ-1);
P3 depends on the passthrough record’s CP-1 settings channel for
tierOverrides delivery AND on the P1 container-registry listing (OQ-2) so an
exported stable name resolves in-container.
P1 — Stable-name table + gateway resolver
Section titled “P1 — Stable-name table + gateway resolver”Owner: compass-server (Go store + proto) and the Bun gateway entrypoint (compass-side, within the ratified exception).
Add stable_model_names to the squashed migration:
name text primary key, display_name text, candidates jsonb, metadata jsonb
where candidates is an ordered array of {provider, model_id} and
metadata carries the listing shape (context window, cost, Api type) taken
from the primary candidate — see OQ-2, since candidates map to different
upstream models with different windows.
Implement the compass resolver in the gateway boot entrypoint (the RIG-1715
T1 entrypoint file): look up the request’s modelId in the stable-name table
(cached, invalidated on write); for each candidate in order, dry-run-peek the
T2 pool for a usable (present, non-usage-limited, non-transient-down)
credential for provider; return the first candidate materialized as a
concrete Model<Api>; unknown names fall through to the existing exact-match
model lookup. Extend listModels to emit stable names.
This requires a resolver-seam widening (OQ-1). The current boot option is
ModelResolver = (modelId: string) => Model<Api> | undefined
(forks/oh-my-pi/packages/ai/src/auth-gateway/server.ts:55) — synchronous
and tenant-blind, invoked at server.ts:377/:569 before any credential
work. Candidate selection needs the caller tenant (per-request) and an async
pool query (peekApiKey is async,
forks/oh-my-pi/packages/ai/src/auth-storage.ts:5122).
P1 therefore widens the seam to
(modelId, identity) => Promise<Model<Api> | undefined>, the exact sibling
of the RIG-1715 T3 authorize(req) → identity boot-option change (gateway
design.md:270-281) — a small, upstreamable widening, NOT a routing-core
edit. The candidate dry-run peek is a NEW read on the T2 resolver: a
side-effect-free “would this provider yield a usable credential for this
tenant” that does NOT acquire or set stickiness (distinct from the routing
core’s later stateful storage.getApiKey(model.provider, sessionId) at
server.ts:419). Stage-1 candidate choice is DETERMINISTIC, not a held cache:
it is a pure function of the ordered chain plus the current pool marks, so
absent a mark change the same candidate is chosen on every request of a
conversation and prompt-cache continuity is preserved without the seam
carrying sessionId (which is derived downstream at server.ts:410/:578,
after the seam has returned — OQ-1 sub-point). A mark changes mid-conversation
only on a usage-limit event, exactly when re-resolution is wanted.
Failover trigger (OQ-3): skip a candidate on credential absence and marked
usage-limit — both observable in the dry-run peek before the upstream call, so
both are seam-executable. The recommendation is v1 hard-down on a provider
outage (no pre-first-byte retry): the widened resolver returns one model
before the upstream call happens in completeSimple (server.ts:462/:645)
or streamSimple (server.ts:499/:674), so admitting pre-first-byte
failover would need a
routing-core loop or a compass-side upstream wrapper (OQ-3) — out of P1 scope
pending Matt’s ruling. Mid-stream failover stays v2 regardless.
Container-registry materialization (OQ-2): the same stable_model_names
table generates one gateway-provider (transport: "pi-native") models.yml
entry per stable name (metadata from the row), delivered through the CP-4
bundle, so an exported stable name resolves inside the container and does not
trip the refuse-to-boot belt (packages/compass-agent/src/cli.ts:991-1004).
Ordering: this listing lands with or before P3.
Interfaces:
- Consumes: the WIDENED resolver boot option
(modelId, identity) => Promise<Model<Api> | undefined>+listModels?: () => Iterable<Model<Api>>(forks/oh-my-pi/packages/ai/src/auth-gateway/server.ts:55,65-67, widened per OQ-1); a NEW side-effect-free dry-run peek on the T2 pool resolver (own-then-shared,compass-server-llm-gateway/design.md:375-395); per-agent token→tenant identity (RIG-1715 T3). - Produces:
StableNameResolver(TS, entrypoint-local):resolve(callerTenant: TenantId, modelId: string): Promise<Model<Api> | undefined>; Go storeStableNameStorewithList(ctx) ([]StableName, error)/Put(ctx, StableName) error,StableName{Name, DisplayName string; Candidates []Candidate; Metadata ModelMetadata},Candidate{Provider, ModelID string}; the CP-4 stable-namemodels.ymlgenerator; in-memory reference +pgtestsuite per store discipline.
Test cycle: red — a request for claude-opus-4-8 404s today
(“Unknown model”, server.ts:379); green — it resolves to anthropic-OAuth
for a caller holding a subscription, to openrouter/... for a caller
holding only an OpenRouter key, in the fork test harness pattern of
auth-gateway-model-list.test.ts.
P2 — Role-policy store + compass.v1 API
Section titled “P2 — Role-policy store + compass.v1 API”Owner: compass-server.
Add model_role_policy (scope: a user id now, an org id when the managed
plane’s org entity lands — same forward-shape note as the gateway’s pool
seam, compass-server-llm-gateway/design.md:399-405) and the proto surface
on compass.v1: GetModelPolicy / SetModelRolePolicy /
ListStableModelNames / SetStableModelName. Writes are validated against
the stable-name table (a role may only reference a known stable name or an
explicit provider/id escape-hatch selector), and reject a default key in
tier_overrides (see P3).
Write authority is split by target (OQ-4). Role-policy writes
(SetModelRolePolicy) are open to both the UI and agents — agents are
already authenticated principals on the server door, and per-role edits have
one-role blast radius. Stable-name-chain writes (SetStableModelName) are
user-only: rewriting a chain silently redirects every role referencing
that name (and the container-registry seed) fleet-wide, so it is not an agent
capability. Both write paths are append-auditable — the store keeps prior
rows via a valid_from version column (NOT last-write-wins on the bare
primary key), and each write emits a policy-change activity row so a bad edit
is surfaced, not merely revertable.
Interfaces:
- Consumes:
stable_model_names(P1); compass.v1 service scaffolding + regenerated clients. - Produces: proto messages
ModelRolePolicy{compass_role string, stable_name string, thinking_level string, tier_overrides map<string,string>}; GoModelPolicyStorewithGetForRole(ctx, userID, role) (ModelRolePolicy, error)/Set(ctx, userID, ModelRolePolicy) error/List(ctx, userID) ([]ModelRolePolicy, error); a versioned row schema (valid_from) + apolicy_change_eventaudit row; in-memory reference +pgtest.
P3 — Runner materialization of policy
Section titled “P3 — Runner materialization of policy”Owner: compass-server + runner.
At agent exec, resolve the agent’s Compass role (the Runner already knows it
— it exports COMPASS_ROLE, agent_exec.go:89-91) against
ModelPolicyStore; export the policy’s stable name as the exec’s Model
(flowing through the existing spec.Env["COMPASS_MODEL"] seam,
agent_exec.go:83-85) unless an explicit per-start model override was given
(explicit override > role policy > absent, preserving today’s
“empty leaves each agent on its own default”, runner.go:49-52).
The exported stable name MUST be resolvable inside the container or the agent
refuses to boot: a pinned-but-unresolvable COMPASS_MODEL hard-throws
“refusing to boot model-less” (packages/compass-agent/src/cli.ts:991-1004).
P3 therefore depends on the container-registry materialization named in OQ-2
(the CP-4 models.yml carries one gateway-provider entry per stable name);
listing must land with, or before, policy materialization.
thinking_level is materialized as a :<level> suffix appended to the stable
name in COMPASS_MODEL (the same colon-suffix encoding OMP selectors use —
emitted as ${pattern}:${thinkingLevel} by resolveConfiguredRolePattern,
model-resolver.ts:1044; split back off by splitThinkingSuffix), NOT a
separate env var, so the opaque-pattern seam (cli.ts:145-150) stays
untouched. When tier_overrides is non-empty, merge a modelRoles: mapping
into the CP-1 settings document the passthrough bundle delivers — values only,
under the SDK’s own key (settings-schema.ts:564), no schema fork. P2
validation rejects a default key in tier_overrides (it would fight
COMPASS_MODEL for the main model — the passthrough OQ-3 edge).
Interfaces:
- Consumes:
ModelPolicyStore.GetForRole(P2);AgentEnv.execSpec()env seam (agent_exec.go:36-42,83-91); passthrough CP-1 settings channel (compass-agent-config-passthrough/design.md:190-195); the OQ-2 registry materialization. - Produces: no new container contract; policy-sourced
COMPASS_MODEL(stable name + optional thinking token) + mergedmodelRolesinsettings/config.yml.
Test cycle: red — a role with a configured policy still starts with
COMPASS_MODEL absent; green — table-driven agentenv-style tests
(pattern: agentenv_test.go:49-78) pin policy-present, explicit-override,
and no-policy cases.
P4 — UI read/write surface
Section titled “P4 — UI read/write surface”Owner: compass-obs (UI lane), behind the P2 proto.
A settings view listing Compass roles with their configured model (stable
name + the declared candidate chain for that name), editable per-role. Scope
is the thin CRUD over P2 — no new semantics. The EFFECTIVE backend (which
candidate the caller’s pool currently picks) is deliberately NOT shown in v1:
it depends on live pool state — credential presence + usage-limit marks —
which post-RIG-1715 lives only inside the TS gateway’s per-tenant
AuthStorage (compass-server-llm-gateway/design.md:287-296), not in the Go
server that answers compass.v1. Surfacing it would need a new
Server→gateway pool-state read surface no record designs; if Matt wants the
live display it is a follow-up with that surface as a named, owned interface.
Interfaces:
- Consumes: P2 RPCs via regenerated
compass.v1clients. - Produces: role-policy settings UI.
P5 — Shipped defaults + docs deliverable
Section titled “P5 — Shipped defaults + docs deliverable”Owner: compass-obs.
Seed the default model_role_policy rows (fleet-current practice until
RIG-2562 numbers land) and the day-1 stable_model_names set covering the
gateway record’s day-1 providers (“anthropic (OAuth + api_key),
openai/openai-codex (OAuth + api_key), google (api_key)”,
compass-server-llm-gateway/design.md:719-720) plus OpenRouter/Bedrock
alternate candidates. Write the docs page “recommended model per role per
provider you hold”, forward-referencing RIG-2562 as the evidence source and
citing the fleet model-selection spec’s optimization target (the spec lives
in the internal fleet-tooling repo — cite conceptually, do not link the
private repo from this public one).
Interfaces:
- Consumes: RIG-2562 per-role recommendations (when published); P1/P2 stores.
- Produces: seed data +
docs/page; a data-refresh runbook line (updating defaults is a store write, not a release).
- P1 — Stable-name table + gateway resolver (Owner: compass-server + Bun
entrypoint) — jsonb candidates + metadata, resolver-seam widening
(async + identity, no
sessionId, OQ-1), pool dry-run peek, hard-down failover on absence + usage-limit (pre-first-byte failover deferred pending OQ-3), unknown-name pass-through, CP-4 container-registry listing generator (OQ-2), pgtest + fork-harness green. - P2 — Role-policy store + compass.v1 RPCs (Owner: compass-server) —
Get/Set/List policy, stable-name CRUD, validation (reject
defaultin tier_overrides), write-authority split (agents: role policy only, not stable-name chains — OQ-4), versioned rows +policy_change_eventaudit. - P3 — Runner materialization (Owner: compass-server + runner) —
policy→
COMPASS_MODEL(+ thinking-variant token), tierOverrides→CP-1modelRoles, depends on P1 container listing, precedence pinned by tests. - P4 — UI role-policy view (Owner: compass-obs) — CRUD over P2, stable name + declared chain (no live effective-backend display in v1).
- P5 — Defaults + docs (Owner: compass-obs) — seeded rows, day-1 stable-name set, recommended-models docs page, RIG-2562 refresh path.
Open Questions
Section titled “Open Questions”- (Load-bearing) Resolver-seam shape — the stable-name resolution point.
The drafted P1 says stable-name resolution “rides the existing injected
resolveModelboot option, zero fork changes.” That is not achievable as the seam is currently shaped:ModelResolver = (modelId: string) => Model<Api> | undefined(forks/oh-my-pi/packages/ai/src/auth-gateway/ server.ts:55) is synchronous and tenant-blind, invoked atserver.ts:377(foreign-wire) and:569(pi-native) BEFORE any credential or identity work; the pool query it must make is async (peekApiKeyisasync,forks/oh-my-pi/packages/ai/src/auth-storage.ts:5122) and needs the caller tenant, which is per-request state. P1’s own Produces line (resolve(callerTenant: TenantId, modelId: string)) already assumes a signature the boot option cannot supply. So a seam change is REQUIRED — this record cannot honor the “NO new fork seam” Global Constraint verbatim. Two shapes: (A) widen the resolver to(modelId, identity) => Promise<Model<Api> | undefined>and let it run the candidate loop — a small, upstreamable widening, the exact sibling of the RIG-1715 T3authorize(req) → identityboot-option change already ratified (gatewaydesign.md:270-281); (B) move resolution OUT of the hook — compass-server rewrites the outbound stable name to a concreteprovider/idbefore the gateway call, so the tenant-blindresolveModelstays static. Recommendation: (A) — it keeps one authoritative resolution point gateway-side (where pool + tenant already live post-T5) and rides the T3 wave; the Global Constraints bullet is amended to “one small upstreamable seam widening, sibling of T3’s authorize verifier,” not “no seam.” Matt ratifies the widening (public fork-seam shape). Sub-point (the widening’s exact signature): nosessionId. The seam runs atserver.ts:377/:569;sessionIdis derived DOWNSTREAM atserver.ts:410/:578fromparsed.context(system + tools + first message), which the seam never receives — so the widened signature stays(modelId, identity) => Promise<Model<Api> | undefined>and CANNOT key a per-session stickiness cache. This does not need one: with v1 hard-down failover (OQ-3 rec), stage-1 candidate choice is a PURE function of the ordered chain plus the current pool marks (absence/usage-limit), so absent a mark change the same candidate is selected on every request of a conversation — determinism, not a held cache. A mark DOES change mid-turn only on a usage-limit event, which is exactly when re-resolution is wanted. ThreadingsessionId/contextinto the seam to hold an explicit sticky map would forcederiveSessionIdahead ofresolveModel— a routing-core reorder OQ-1’s (A) is chosen to avoid — and is therefore rejected. (If Matt rules pre-first-byte failover into scope under OQ-3, the transient-down mark it introduces is per-tenant-per-provider pool state, still not per-session, so this signature holds.) - (Load-bearing) Container-side stable-name registry membership. A stable
name exported as
COMPASS_MODELmust ALSO resolve inside the container, or the agent refuses to boot: a pinned-but-unresolvable pattern hard-throws “refusing to boot model-less” (packages/compass-agent/src/cli.ts:991-1004, verified). Gateway-sidelistModelsemitting stable names on/v1/models(P1) does NOT populate the container’s own registry, which post-T5 is seeded from the CP-4models.ymlstatic entries. Nothing in P1–P5 as drafted deliversclaude-opus-4-8as a container-visible model id → every policy-configured agent bricks at boot (the record’s rejected “agent-side aliases” alternative addressed fallback-order visibility, NOT registry membership — resolution ≠ listing). Recommendation: add a P1/P3 deliverable that materializesstable_model_namesinto the container registry — the Runner/CP-4 bundle generates one gateway-provider (transport: "pi-native")models.ymlentry per stable name from the same table. Sub-question (load-bearing): where is a stable name’s model METADATA (context window, cost,Apitype) authored, since candidates map to different upstream models with different windows? Rec: metadata authored on thestable_model_namesrow (from the primary candidate), carried into the generated entry. - (Load-bearing) Failover trigger — the real binary, and its architectural
cost. The seam-executable subset fails over only on credential ABSENCE +
marked-usage-limit — both observable inside the widened resolver’s dry-run
peek, BEFORE the upstream call. State the failure that subset does NOT
cover: a provider OUTAGE (backend in the pool, credential healthy, upstream
5xx) pins every role to that hard-down candidate for the outage — the
single most common reason a multi-candidate chain exists. Admitting the
cheap outage-covering subset (retry the NEXT candidate on a pre-first-byte
failure — connect error / immediate 5xx / 429-without-usage-limit, nothing
streamed yet) is NOT free and NOT seam-shaped. The widened resolver returns
ONE
Model<Api>atserver.ts:377/:569; the upstream call happens downstream incompleteSimple(server.ts:462/:645) orstreamSimple(server.ts:499/:674), AFTER the seam has returned. The gateway’s existing pre-emit retry (server.ts:227-273,:306) swaps the CREDENTIAL within one provider on a usage cap; it does NOT re-invokeresolveModelto pick a different provider. So failing over to the next candidate on a pre-first-byte upstream failure — and SETTING the “transient provider-down” mark that would feed the dry-run peek — requires ONE of: (a) a routing-core retry loop that catches the pre-first-byte failure and re-invokesresolveModel— a core edit, the exact “never the routing core” change OQ-1’s (A) widening was chosen to avoid, so it is a SECOND public fork-seam question, not a free P1 add; or (b) a compass-side upstream wrapper fronting the gateway that catches connect/immediate-5xx and re-drives candidate selection outside the core — a new, un-designed mechanism. Recommendation: v1 hard-down on outage (absence + usage-limit only). It is fully seam-executable through OQ-1’s widening at zero core cost, and it removes the “transient provider-down” mark from v1 entirely (nothing sets it). Matt rules the binary: v1 hard-down (clean, seam-only), or pre-first-byte retry in scope — and if the latter, which of (a) a second fork-seam edit or (b) a compass-side wrapper. Choosing pre-first-byte reopens OQ-1’s “one small widening” framing. - (Load-bearing) Write authority — split by target. P2 puts BOTH role
policy and stable-name-chain writes behind the user+agent door. These have
different blast radii: an agent rewriting one role’s policy touches one
role; an agent rewriting a stable name’s candidate chain silently redirects
EVERY role referencing that name (and the container-registry seed) to a
different upstream — one write to
claude-opus-4-8 → [openrouter/cheap]degrades the whole fleet including the review lane that would catch it. Recommendation: (a) agents MAY write role policy; (b) agents may NOT write stable-name chains (users only). The append-auditable mitigation must be BUILT, not asserted — P2 specs a versioned/audit shape (a valid-from column or a separate audit table; the draftedname text primary key/ last-write-wins tables cannot retain prior rows) plus a policy-change event surface (activity row / log line) so a bad edit is noticed, not just revertable. Matt confirms the (a)/(b) split + that audit is in P2 scope. - (Load-bearing) Policy scope shape. Per-user policy now with the org
scope added when the managed plane lands (mirroring the gateway’s pool
seam note that no org entity exists yet,
compass-server-llm-gateway/ design.md:399-405), or build the org column speculatively today? Recommendation: per-user now, org-ready key shape (nullable scope column) — same posture the gateway record took for pools. - (Non-load-bearing) Stable-name namespace. Bare names
(
claude-opus-4-8, Matt’s example) vs prefixed (compass/...). Recommendation: bare, matching the gateway-record example verbatim; the unknown-name pass-through disambiguates collisions with concreteprovider/idselectors since those always contain a/. - (Non-load-bearing) Interim defaults before RIG-2562. Seed from fleet-current practice (P5) vs wait for eval numbers. Recommendation: seed now; RIG-2562 output is a data update by construction.
- (Non-load-bearing) Model-selection spec location.
docs/specs/platform/model-selection.mddoes not exist in the compass repo (verified:docs/specs/holds onlyproduct/+brand/); it lives in the internal fleet-tooling repo (RIG-2573/DL-025). Recommendation: cite its optimization target conceptually in P5’s docs page; do not link the private repo from this public one.