Mindrails Supervisor

Pick the right AI model and effort before an agent starts working – for Claude Code and Codex.

Open-source tools around Jev, a judgment model that answers fixed questions about a conversation with calibrated probabilities. Everything runs locally; Jev is called through your own Vercel AI Gateway key.

Status (October 2026): finished preview. The project is complete as published and not actively developed further. It is shared as a working reference. Issues and pull requests are welcome, but there is no roadmap.

The Jev app with a sample conversation: Claude Code builds the export, Codex writes the tests, a small rename goes to the cheapest model.

What's inside

Related MCP server: taskguard-mcp

Quick start

Requirements: Node.js 24+ (the plugins run on 20+), and for Jev itself an AI_GATEWAY_API_KEY from Vercel AI Gateway with paid credits (since late September 2026 the gateway no longer serves the Jev model to free-tier credits). Chat excerpts are sent to Jev and may incur small charges (fractions of a cent per check).

Jev plugin – Claude Code

claude plugin marketplace add haskallalk-eng/mindrails-supervisor
claude plugin install jev@mindrails

Jev plugin – Codex

codex plugin marketplace add haskallalk-eng/mindrails-supervisor --ref main
codex plugin add jev@mindrails

Then type #jev an once in a chat (#jev hilfe lists the commands; the plugin speaks German).

Jev app

git clone https://github.com/haskallalk-eng/mindrails-supervisor.git
cd mindrails-supervisor
npm ci --ignore-scripts && npm run build
node dist/jev-app/app.js --cwd /path/to/your/project

It drives the official claude CLI and codex app-server with your own sign-ins (run claude → /login once). Intended for personal use on your own machine.

Supervisor demo (no account, no key)

npm run demo

How Jev chooses

1. New task? Jev answers a yes/no question about the message in the same request. Replies like "ok" or "go on" are left alone. There is no length rule in our code.

2. Kind and difficulty. Coding, agent/terminal, reasoning, research or simple; easy, normal or hard.

3. Model. A fixed policy compares published benchmarks (sources) and a weighted spectrum of 668 practitioner statements (method). It switches only for a clear gain: at least 4 benchmark points, or at least 25 % lower cost per task at equal quality. Claude and Codex models are compared in one cost unit.

4. Effort. The level the evidence names for that model, one level more for hard tasks, never into the range rated "too much".

Details: docs/model-routing.md.

What it cannot do (tested)

• Plugins cannot switch model or effort. In the Claude desktop app (Claude Code 2.1.281) skill model/effort settings never reached the requests, and the app refuses to let a session re-price its own turns. Codex has no such interface at all. So the plugin recommends and waits; you switch. The Jev app can set both because it starts the runs itself.

• The on-screen figure is Windows-only and sits at the window's bottom-right corner, not exactly on the menu.

• Codex: the effort is only known from the conversation history, so a message two or more steps off is held once instead of waiting; sending again lets it through.

• Jev app: it never writes into a chat that is open in the Claude or Codex app; a handover is text, not the other agent's native tool history. The Claude side is tested against a stand-in CLI and the Codex side live.

• No accuracy guarantee. Benchmarks and community evidence are snapshots (September 2026); thresholds are initial policy defaults.

Tests

132 automated tests (policy, hooks, plugins, the app with fake Claude and Codex binaries, real MCP transport) run on Ubuntu and Windows in CI. Live checks against the real services are documented in evidence/ and the changelog.

Supervisor MCP server

A local MCP completion gate for AI agents, with optional Jev judgments and deterministic policies.

An independent open-source project by Mindrails.

Earlier experiments: route before execution. The separate local jev command asks Jev to choose between the available GPT-6 Astra, Sol and Luna models, then starts Codex with the highest-probability model. Run npm run build, npm link, then jev --workspace-write in your project. Each input is routed before execution; subsequent inputs include visible conversation history. New: jev-panel serves a small side panel (open it in Codex's in-app browser) that routes and continues a chosen conversation, with long-history selection, visible probabilities and approvals. This does not intercept the Codex desktop chat composer, and it cannot write to a conversation while the Codex app has it open. See setup, failure behavior and limitations. This preflight argmax policy is separate from the plugin's advisory uncertainty policy below.

Check explicit requirements before an agent stops, and flag repeated steps. The host remains responsible for verification, permissions and execution.

Jev recovery advice: the optional Codex review now asks three additional focused questions in the same request: is the current work advancing, what is the dominant unresolved obstacle, and what next step fits? When the answers agree with sufficient confidence and selected probability, Mindrails presents a prepared follow-up prompt: resolve a prerequisite, ask for a missing decision, change approach, verify the result, or correct a missed requirement. Productive work gets no recovery interruption. Conflicts and uncertainty produce no action prompt; an environment blocker suppresses a contradictory model-upgrade suggestion. Jev classifies; application code selects the fixed wording. Prompts are never dispatched automatically.

See the Jev-specific product comparison and validation plan. This is a development preview; uniqueness, productivity gains and semantic accuracy have not been established.

Uncertainty policy: an unclear intervention preserves the host's normal path (CONTINUE_BASELINE / recovery action: none) without confirming completion. An uncertain downgrade keeps the current model; a plausible but uncertain upgrade prefers stronger-model advice under a documented quality-first rule. Fallback advice is labeled as product policy, not a confident Jev diagnosis. Clearly identified prerequisites override model changes, and clear failures remain visible. The plugin does not automatically interrupt, restart or switch a task.

Try the free offline demo

Requires Node.js 24 or later and npm. No account or API key is needed.

git clone https://github.com/haskallalk-eng/mindrails-supervisor.git
cd mindrails-supervisor
npm ci --ignore-scripts
npm run build
npm run demo

Actual synthetic demo output:

SYNTHETIC MOCK DEMO — no API call, no semantic evaluation
3/5 markers → CONTINUE license, security
5/5 markers → FINISH
3 repeated steps → REVIEW
changed result → CONTINUE

Mock mode uses explicit [done:id] markers and supplied fixture evidence. It demonstrates policy mechanics, not model accuracy. A Jev failure never falls back to mock.

CLI and host loop

node examples/agent-loop.mjs
npm run e2e
npm run evidence

For a direct check, select the provider explicitly. Unix: MINDRAILS_PROVIDER=mock node dist/cli.js check examples/check.json. PowerShell: $env:MINDRAILS_PROVIDER='mock'; node dist/cli.js check examples/check.json. Trace triage is available as MINDRAILS_PROVIDER=mock node dist/cli.js triage examples/trace.json. The stuck command is deterministic and needs no provider.

check <file.json> returns an advisory decision. stuck <file.json> accepts { "steps": [{ "action": "search", "input": "q", "result": "same", "progress": false }] }. Exit status 0 means the evaluation was returned successfully, including continue or review; the host must inspect decision. Invalid input/startup uses exit status 1.

MCP tools

Trace triage asks Jev three fixed-choice questions: whether the task appears complete, whether the user explicitly accepts or rejects the result, and whether the run is healthy, misses expectations, overtly fails, or claims success without support. A deterministic policy maps those labels to a recommendation. Low confidence or uncertainty preserve the baseline without certifying success; clear dissatisfaction or an expectation gap warrant review; a silent failure is prioritized; a clear failure is recommended for issue filing. Caller-reported unpermitted actions bypass Jev and are recommended for on-call review. These labels are initial heuristics, not validated production thresholds. Recommendations are advisory and never run tools, close work, create issues, page anyone, or authorize actions.

The trace, transcript, tool arguments and results are caller supplied and treated as untrusted data. Permission claims are not independently verified. Mock mode uses explicit [mock:...] markers only and is synthetic; it does not assess real traces.

Completion requires every requirement and task signal to reach 0.85, and contradiction signal at most 0.1. By default evidenceMode is artifact: Jev judges whether the requested content appears in the supplied result and does not demand external proof just to confirm the artifact's contents. Set evidenceMode to fact-check when the task requires sources or verification; then global evidence or evidence on every requirement is mandatory, and Jev's evidence score must also reach 0.85. This score is not source authentication. Any declared trustedChecks entry with fail or unknown blocks completion. Evidence mode and trusted-check statuses are caller-supplied, unauthenticated claims. Use trusted host logic for consequential verification.

finish means this policy accepts the supplied information. It does not establish real-world success. Evidence is not fetched or independently verified. Thresholds are initial policy defaults, not calibrated guarantees. Changed results or reported progress interrupt repeat detection; four-step cycles, timestamp-changing errors and other semantic loops may go undetected. Repeat grace is caller supplied and cannot prove that polling is legitimate.

MCP setup

Tested: official @modelcontextprotocol/client 2.0.0 over stdio. The following is a generic host configuration example; individual desktop hosts have not been tested:

{
  "mcpServers": {
    "mindrails-supervisor": {
      "command": "node",
      "args": ["/absolute/path/to/mindrails-supervisor/dist/cli.js", "mcp"],
      "env": { "MINDRAILS_PROVIDER": "mock" }
    }
  }
}

On Windows use your absolute path with JSON-escaped backslashes. MCP results advise the host; this server cannot force it to continue, execute tools or switch its internal model.

Automatic Jev review for Codex (local plugin preview)

The plugins/jev-chat-review package also monitors open Codex chats locally. Ask Jev to open the open_codex_chats overview for session status, elapsed time, repeated identical tool-call warnings, latest request token/context usage, and available Codex rate-limit pressure. Local monitoring stores derived signals only and makes no AI/API calls. At the end of a turn, the optional Jev review sends the bounded visible transcript, tool activity, current model, and usage evidence to Vercel AI Gateway. Jev then gives a confidence-rated model-fit direction when the current model is known: keep it, try a more capable model, try a faster model for comparable low-risk work, or uncertain. This advice does not change the selected model or claim measured cost savings. The feature requires AI_GATEWAY_API_KEY and may incur charges. Local defaults cap use at 20 reviews and 350,000 trace bytes per UTC day, with duplicate invocations skipped. These limits reduce usage but do not guarantee a fixed dollar maximum. A Jev request can delay the end of a turn by up to eight seconds. Results are advisory; Jev never continues or closes work, creates issues, pages anyone, or takes actions. Codex has no supported persistent plugin icon/panel for this use case, and the monitor has no timer that interrupts a still-running tool. Set MINDRAILS_JEV_AUTO_REVIEW=0 or disable the plugin to stop reviews.

The hook reads Codex's local transcript, redacts common credential patterns, and sends the trace to Vercel AI Gateway. This is not a guarantee that every secret or personal detail is detected. Codex's transcript format is not a stable interface. Oversized traces are reported as unreviewed instead of silently truncated. This is a local integration preview, not a published one-click install. See plugins/jev-chat-review/README.md for setup, data handling, and limits.

Optional Jev mode (BYOK)

Set MINDRAILS_PROVIDER=jev and supply TYPESAFE_API_KEY through your host's secret environment. Then use the same commands. The optional official Vercel compatibility route uses MINDRAILS_JEV_ROUTE=vercel-ai-gateway with AI_GATEWAY_API_KEY. Provider selection is mandatory for check, triage and mcp, preventing an accidental synthetic default or billable default. Environment files are not loaded automatically. Do not put keys in committed configuration or command history.

Jev mode sends task, result, requirements and supplied evidence to the selected fixed HTTPS endpoint. Inference may incur charges under the selected account. The Apache license covers this software, not either service. See TypeSafe's API, account terms, data policies, and Vercel's TypeSafe-compatible API. The native adapter pins jev-1.13.0; the gateway adapter pins Vercel's typesafe-ai/jev alias. Completion checks use Noul signals and trace triage uses choice signals. Both verify the returned model. A gateway result does not establish which native Jev version served the alias.

The initial twelve-case Jev run on 2026-09-23 returned continue for every case (7/12, same as an always-continue baseline). After separating artifact completion from optional source-backed fact-checking and lowering the initial signal threshold to 0.85, a follow-up run on the same hand-labeled cases scored 12/12 with no false finishes, false continues or provider errors. This is a tuned synthetic suite, not a production accuracy estimate; validate on held-out cases before relying on the gate. The run used 7,054 input tokens and its catalog-rate estimate was $0.00028216. See the follow-up result, initial result and method.

A separate eight-case set was frozen before being sent to Jev and was not used for tuning: 7/8 correct, no false finishes, one false continue on a valid JSON artifact, and no provider errors. This suggests the policy errs conservatively on some structured outputs. We did not lower thresholds based on this held-out result. See the held-out report.

Limits and privacy

No automatic retries. Failed attempts consume budget. Process restarts reset counters; these caps are not a durable financial spending limit. Each CLI command is a new process. The MCP server preserves counters while running. A caller with unbounded access can restart it; enforce durable quotas in your host.

No telemetry, disk trace storage, action execution, cloud service or model weights. CLI outputs include derived signals and IDs; host logs may store them. Inputs persist in memory during evaluation. TypeSafe's retention is governed separately and is not assumed to be zero. The server is advisory, not an authorization or prompt-injection security boundary.

Development

npm run typecheck
npm test
npm pack

Tests cover policy vetoes, bounds, timeouts, malformed responses, budgets, loop detection, trace triage, CLI and actual MCP transport. npm run e2e prints eleven concrete official-client scenarios. npm run evidence reproduces the fixed completion comparison and deterministic-policy baselines. The datasets are synthetic; no production accuracy, cost savings or performance benchmark is claimed. See evidence, architecture, provenance, security and contributing.

The supervisor server itself intentionally excludes hosted MCP, persistent run state, dashboards and automatic actions (routing lives in the Jev plugin and app). Feedback on incorrect decisions is welcome using synthetic or redacted examples. No npm registry publication is provided; use the repository or GitHub release package.

Deutsch – Kurzfassung

Jev prüft vor jeder neuen Aufgabe, ob Modell und Effort passen (veröffentlichte Benchmarks plus gewichtete Erfahrungswerte). Passt alles: ✓. Eine Stufe daneben: eine Zeile und die Jev-Figur unten rechts. Ab zwei Stufen wartet die Nachricht in der Claude-Desktop-App, bis du im Menü umstellst, und läuft dann von selbst los. Umstellen musst du selbst: Weder Claude noch Codex lassen Plugins Modell oder Effort ändern (live getestet). Die Jev-App startet Claude Code und Codex selbst und kann deshalb Modell und Effort pro Nachricht setzen.

License

New original code: Apache-2.0. Dependencies retain their own licenses; see NOTICE. This project is independent of TypeSafe and is not an official Jev product.