decide

Your agent reasons. Jev sorts.

Bulk decisions for Claude and Codex. Keep the pile out of your agent’s context.

Get started · Workflows & benchmarks · Tool reference · All evaluations

A backlog of issues. Hundreds of product reviews. A wall of logs. Your agent needs to decide what matters before it can do useful work.

Give it a path and a rubric. decide sends each record to Jev, saves the decisions locally, and returns a compact summary plus a review queue. Your agent can inspect the exceptions and use scripts to act on the rest.

You: “Find feature requests in these 500 app reviews.”
                           │
Claude / Codex defines the criteria and passes the file path
                           │
                    decide → Jev
                           │
          ┌────────────────┴────────────────┐
     Accepted decisions              Needs a closer look
     results.jsonl                   review.jsonl
     Process with code               Review with your agent

One MCP tool. One server file. Your rubric, your data. Works with UTF-8 files, log lines, JSONL records, or inline items.

Put it to work

These are sorting jobs. UX synthesis, prioritization, incident diagnosis and code changes still need judgment. decide produces labels; it never executes them.

Related MCP server: Jev MCP

Measured against GPT-4.1 mini

Cut the bill while protecting the quality your task needs. On 1,000 new UX reviews, the selected policy improved all four frozen quality metrics and reduced total inference cost by 48.9%.

Jev handled the whole file. Mini checked 123 uncertain negative answers; valid positive answers stayed with Jev. The cutoff was selected on older data and committed before these 1,000 exact-disjoint reviews were sent to either model. Both agents wrote complete output files and verified every input ID.

New UX records, frozen policy, all mistakes and reproduction →

The cost includes reading data, real MCP, retained history, writing results and the final answer, plus both providers. Both arms discard completed batches. This is a guided workflow with host-enforced completion, not a native Codex/Claude session. Results are observed sample comparisons, not a future quality guarantee or proof of the globally cheapest policy.

Developer workflow: on 300 OpenCV issues, an error-only fallback policy cut tool-loop cost 77.3% and raised accuracy from 68.0% to 74.7%, with higher macro-F1. Feature-label recall fell, so that policy suits a different quality tradeoff. Developer results and class metrics →

Less review is not the goal. The earlier UX policy was cheaper but missed more feature requests. The new policy spends more on review and recovers that recall. The nine-task comparison also includes jobs where mini preserves quality better and where review makes results worse. An agent stopping before the job was done remains published too.

The workflow study covers 3,500 Jev decisions on UX feedback and issues from five projects. The earlier evaluations cover another 8,340 records across ten tasks. Reference labels, predictions, failures and baselines are published throughout.

Review rate is an outcome, not a quota. Choose the greatest savings your task's quality allows; there is no universal “only review 5%” setting.

Get started

You need uv, Git and a TypeSafe API key. No clone required.

Connect Codex or Claude Code / Desktop with this MCP launch command:

uvx --python 3.11 --from git+https://github.com/alsoleg89/decide@v0.1.0 decide-mcp

Set TYPESAFE_API_KEY in the server environment and DECIDE_ROOT to the folder containing your data. The client launches the stdio server; the first launch installs dependencies, then uvx reuses its cache. Local checkout and installation check →

Then ask your agent:

Use decide on feedback.jsonl to find feature requests. Pass the file path directly without reading the whole file first. Use yes/no criteria with a global confidence threshold of 0 and a threshold of 0.9 for no. Read every case in the review queue and save one final decision per input ID. Verify complete coverage before reporting success.

Each JSONL row needs an ID and content:

{"id":"review-1","content":"Please let me export my workout history as CSV."}
{"id":"review-2","content":"The app closes every time I open the activity screen."}

{
  "question": "Does this app review request a new or changed capability?",
  "criteria": {
    "yes": "Requests a new or changed function, content, interface, or capability",
    "no": "Does not request a new or changed capability"
  },
  "source": {"kind": "jsonl", "paths": ["feedback.jsonl"]},
  "confidence_threshold": 0,
  "confidence_thresholds": {"no": 0.9}
}

This reproduces the routing from the new UX validation: accept valid positive answers, review uncertain negatives and errors. Validate it on your own labeled sample; a confidence cutoff does not guarantee accuracy.

Small context. Full audit trail.

The default response contains counts and file paths, with no input previews. Every decision, confidence, probability distribution and provider error stays in results.jsonl. Every case needing review keeps its full original input in review.jsonl. Both live under DECIDE_ROOT/.decide/<run-id>/.

• Read from disk: files and globs, individual log lines, or JSONL records.

• Control escalation: global and per-label confidence cutoffs, plus labels that always need review.

• Keep failures visible: invalid responses and oversized items enter the review queue.

• Stay inspectable: decide.py is the entire production server.

Input content is sent to TypeSafe. Scope source paths accordingly. A confidence score is not a probability of correctness. Limits, retries and data handling →

Try your own workload

The best benchmark is the job you want to delegate. Label a sample and run the real-data evaluator, then compare accepted errors against the bytes and review work saved.

Found a task that works beautifully—or fails badly? Open an issue with the rubric and a small, shareable example. Both belong in the benchmark suite.

Setup & reference · Reproduce the benchmarks · Development

MIT license · Download v0.1.0 · Local release checks