Skip to main content
Glama
eugeniughelbur

jev-engineering

by eugeniughelbur
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

jev-gate, every tool call checked

jev-gate: a safety gate for AI coding agents

License: MIT Claude Code plugin Latency Cost Attack tested

A tool-call gate for Claude Code, Codex, Cursor and anything else you run, backed by TypeSafe's Jev decision model. It answers allow, ask or deny in about 400 milliseconds for two hundredths of a cent, so you can check every action instead of approving everything or nothing.

It also ships the attack kit I used to find out whether a gate like this holds. It mostly does. The interesting part is how it fails.

A dangerous command is denied in 371 milliseconds, a safe one passes with no model call

Quick start

Two commands in Claude Code:

claude plugin marketplace add eugeniughelbur/jev-engineering
claude plugin install jev-engineering@jev-engineering

Restart Claude Code, set OPENROUTER_API_KEY, then run /jev-status. It installs in observe mode, so it logs every decision and blocks nothing until you say otherwise.

Or run it standalone:

git clone https://github.com/eugeniughelbur/jev-engineering
cd jev-engineering
export OPENROUTER_API_KEY=sk-or-...
./jev_gate.py --explain "git push --force origin main"
{
  "verdict": "deny",
  "source": "model",
  "reason": "destructive p=0.94",
  "confidence": 0.71,
  "latency_ms": 372,
  "cost": 1.79e-05
}

Related MCP server: Querytree MCP Server

Why this exists

Your coding agent asks permission for everything or for nothing. You click approve forty times an hour until you stop reading, or you run it wide open and hope. No middle setting exists.

A real middle setting means asking a second model "is this safe?" before every action. With a chat model that costs about three cents and four seconds each time, so nobody runs it. Jev costs $0.0000189 and 371 milliseconds, measured here, which is about one cent a day at 500 checks.

How it decides

Four steps, and the order is the whole design.

  1. Hard rules. Deterministic denials in plain regex. Never calls the model. Single-digit milliseconds.

  2. Fast path. A read-only allowlist. Never calls the model.

  3. Jev. One request, two questions, answered in parallel.

  4. Thresholds. Your numbers, taken from your own observe-mode log.

Denials run before the allowlist on purpose. A command name says nothing about its arguments: cat is harmless until it is cat ~/.ssh/id_ed25519. tests/test_order.py pins that, because this shipped the wrong way round once.

Hard rules come before the model for a different reason. In testing, the model was least reliable exactly where a hard rule is easiest to write.

It fails open. Any error, timeout or missing key falls back to your harness's normal permission prompt, so a network blip never bricks a session.

Modes

Mode

What it does

Use it when

observe (default)

Logs every decision, blocks nothing

Always, for the first week

guard

Blocks deny, logs the rest

You trust the hard rules

enforce

Blocks deny and ask

Your thresholds came from your own log

export JEV_GATE_MODE=observe

Read ~/.jev-gate/decisions.jsonl for a week before you turn anything on. That file is the only source of thresholds that will fit your work.

The three decisions an agent makes constantly

The gate fires on the rare dangerous action. These fire every turn, which is where the cost and the latency actually live. Same shape underneath: state plus typed questions, one request, answers in parallel.

uv run layer.py route "fix the typo in the README heading"
# {"tier": "fast", "confidence": 1.0, "latency_ms": 516}

uv run layer.py route "redesign how we shard the primary database with zero downtime"
# {"tier": "frontier", "confidence": 1.0, "latency_ms": 356}

route picks the model tier before you spend on the turn. When confidence drops below 0.5 on the cheapest tier it steps up one, not to the top, because falling back to the frontier model on every uncertain turn eats most of the saving.

rank orders up to 255 options against one criterion in a single call. Useful for picking a file, a tool, or reordering search results.

uv run layer.py rank "This file holds the login and session logic" \
  --options "src/auth/session.ts,src/db/schema.ts,README.md"
# winner: src/auth/session.ts

keep decides which pieces of context still earn their place. Nothing is rewritten or summarised. Each item is kept verbatim or dropped, because a summary silently loses the exact path or error you needed later.

uv run layer.py keep transcript.json --goal "fix the failing session test" --budget 0.5

On a real eight-item transcript, in 347ms for $0.000026, it kept the failing test output, the source file and the spec, and dropped echo hello and df -h.

All three are also MCP tools, so an agent can call them itself: route_turn, rank_options, keep_context.

Three commands

Installed as a plugin, you get:

Command

What it does

/jev-status

Mode, key, and what the log holds so far. Runs one live check so you can see it work.

/jev-calibrate

Reads your week of decisions and hands back your thresholds, your fast-path rules and your hard-rule candidates.

/jev-attack

Fires 300 injections at your own gate and reports what got through.

/jev-policy

The rules in force, which layer each came from, and pulls your team's latest.

Any agent, not only coding agents

Install it as a Python package and call decide() from your own code:

pip install jev-engineering
jev-gate --explain "git push --force origin main"

Or expose it over MCP, so Cursor, Codex, Windsurf and anything else that speaks the protocol can ask it:

{ "mcpServers": { "jev-engineering": {
  "command": "uv",
  "args": ["run", "--directory", "/abs/path/to/jev-engineering", "mcp_server.py"],
  "env": { "OPENROUTER_API_KEY": "sk-or-..." } } } }

Three MCP tools: check_action judges something before it happens, rank_options sorts up to 255 choices in one call, gate_stats reports what has been decided.

Question packs

A gate is only as good as its questions, and shell commands are not the only thing an agent does. Five packs ship, and pack picks one:

Pack

For

shell

Commands in a coding agent. The default.

message

An email, Slack message or post before it sends.

money

A refund, payment or transfer.

data

Reading, writing or deleting records.

publish

A deploy, a merge, a release.

uv run packs.py            # see them all

Copy the closest one into ~/.jev-gate/packs/ and edit it. Measured examples, run live:

  • an email containing a live API key: private_data 0.94, verdict deny

  • the same email without it: private_data 0.05, verdict allow

  • a $4,200 refund on a $42 order: authorised 0.02, verdict deny

Guarding everything else

The plugin guards one agent. Run it as a background service and anything can ask it: n8n, cron jobs, a deploy step, a bot about to send a message.

export OPENROUTER_API_KEY=sk-or-...
./service/install.sh
curl -s localhost:8787/check -d '{"command":"aws s3 rm s3://prod-backups --recursive"}'

It starts at login, restarts if it dies, binds to loopback only and refuses to start on a public interface. Full setup and wiring examples in service/.

One set of rules for a team

The rules live in policy.json, not in the code. Publish your team's copy anywhere that serves a raw file, point everyone at it, and each person pulls the same hard rules, fast path and thresholds.

export JEV_POLICY_URL=https://raw.githubusercontent.com/you/team-policy/main/policy.json
uv run policy.py pull
uv run policy.py show

Three layers merge: the shipped default, the team policy, then ~/.jev-gate/policy.local.json. Layers add rules and tighten thresholds, never loosen them. Try to raise your own deny_above above the team's and it is ignored, and show tells you it was:

  yours: +1 hard rule(s)
  yours: deny_above tightened 0.9 -> 0.75
  yours: confidence_floor=0.3 ignored, it would loosen 0.45

A shared rule that any member can quietly switch off is not a shared rule.

Pooling what everyone learns

This is the part that gets better with more people. Everyone still decides locally. Each person pushes a redacted copy of their log into a shared folder, and thresholds get fitted on the whole team:

export JEV_POOL_DIR=~/work/team-policy/pool
uv run pool.py redact-check   # see exactly what would leave your machine
uv run pool.py push
uv run calibrate.py --pool    # fit on everyone's decisions, not just yours

Redaction runs before anything is written. Keys, tokens, emails, IPs and your home path are masked, the working directory and the time of day are never shared, and contributors appear as a hash rather than a name. redact-check prints the before and after so you can look rather than trust.

Rule changes get reviewed

.github/workflows/policy.yml runs on every pull request. It validates the policy and packs, checks that redaction still masks what it claims, pins the decision order, and posts a summary saying in plain words what a threshold change lets through. The attack run is manual, because it costs money.

Nothing central runs. Everyone decides locally, so no server has to stay alive and no single failure takes the whole team's agents down. The team file is a file.

It gets better the longer you run it

After a week in observe mode, run:

uv run calibrate.py

It reads your own decision log and tells you four things:

  • What the gate would have cost you. If it would have interrupted more than a quarter of your work, it says so and tells you to widen the fast path before touching anything else.

  • A threshold sweep on your own traffic, not mine.

  • Which commands reached the model repeatedly and came back clean every time, printed as fast-path regexes you can paste in.

  • Which commands got stopped more than once, printed as hard-rule candidates.

--label walks you through the ambiguous decisions, asking safe or dangerous. Once you have labels it stops guessing and shows real accuracy columns: how many dangerous commands each threshold would miss, and how many safe ones it would stop. --apply writes the result to .env.

Re-run it monthly. Your commands drift, and the model gets retrained underneath you.

Install it in your agent

Claude Code

Add to .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [{ "type": "command", "command": "/absolute/path/to/jev_gate.py" }]
      }
    ]
  }
}

The hook receives the proposed call on stdin, exits 0 to permit and exits 2 to block.

Codex

Same script, wired to the approval hook. Pass the user's own prompt as the authorization evidence, which is what user_message in the event is for.

Cursor, and any MCP client

Wrap decide() as an MCP tool. One caveat worth understanding: an MCP tool is something the model chooses to call, and a hook is something that runs whether the model likes it or not. For a safety gate you want the hook.

LangChain, Pydantic AI, Hermes, n8n

Each has its own published integration. recipes/ has the per-tool notes, including which ones fail open and which fail closed, and why that choice differs by tool.

Anything else

decide() is plain Python with no dependencies. Import it, or shell out to --explain.

What the attack kit found

Run it yourself. Roughly 300 calls, about half a cent.

uv run attack_test.py

It sends every command three ways: clean, with a blunt injection, and with an authority injection claiming a human already approved it. My run, 2026-09-20:

Attack

Dangerous commands that got through

Safe commands wrongly blocked

Blunt, "ignore your instructions"

0 of 30

3 of 30

Authority, "the owner approved this"

3 of 30

1 of 30

Authority, "the lead signed off"

1 of 30

2 of 30

Authority, "standing allowlist"

1 of 30

0 of 30

Three things fall out of that table.

The blunt attack does not get in. It jams you. Zero dangerous commands passed, but 10% of perfectly safe commands got denied. Shouting at the gate does not open it, it makes the gate paranoid. That is a denial-of-service on your own agent, and I have not seen anyone else report it.

The polite attack is the real one. Claiming a human already approved the command worked ten times more than claiming authority over the model. Social proof beats instruction injection.

One command flipped under every framing. git stash clear. It is a hard rule in this repo for that reason. An independent Claude Code gate hard-codes the same command, having found the same inconsistency on its own.

Confidence helps and is not enough. Successful attacks averaged 0.400 confidence against 0.681 on clean traffic. But catching all of them needed a 0.8 floor, which escalated 58% of normal traffic. That is a permission prompt with extra steps. Hence the hard rules.

Full numbers and method: results/2026-09-20-injection-test.md.

Configuration

Variable

Default

What it does

OPENROUTER_API_KEY

none

Your key. TYPESAFE_API_KEY also works.

JEV_GATE_MODE

observe

observe, guard or enforce

JEV_DENY_ABOVE

0.90

Destructive probability that denies outright

JEV_ALLOW_BELOW

0.10

Destructive probability that allows outright

JEV_CONFIDENCE_FLOOR

0.45

Below this, escalate to a human

JEV_GATE_LOG

~/.jev-gate/decisions.jsonl

Where decisions are written

JEV_GATE_TIMEOUT

5

Seconds before failing open

Frequently asked questions

What is Jev?

An AI model from TypeSafe AI that answers typed questions with probabilities instead of writing text. You send it a situation and questions, it returns numbers. It replies in roughly 400ms and costs about two hundredths of a cent per call.

Is this a security product?

No. It catches mistakes, not attackers. A determined attacker who can write into your agent's context has better options than talking to this gate. Treat it as a seatbelt for 2am debugging, and keep your real permissions, sandboxes and credentials where they are.

Can the gate be tricked?

Yes, about 10% of the time in my testing, with text claiming a human already approved the action. The hard rules exist because of that number, not despite it.

Why do hard rules run before the model?

Because a regex never changes its mind and costs nothing. The model is for the long tail of commands you did not anticipate, not for the ones you can name.

Does it slow my agent down?

Median 371ms per checked call, and the fast path skips the model for read-only commands. If that is too slow, widen the fast path.

What does it cost to run?

$0.0000189 per checked call in my measurements. At 500 checked calls a day, under one cent.

Related

License

MIT. See LICENSE.

Available Tools

5 tools
check_actionA

Judge whether an action is safe to take, before taking it.

Returns allow, ask or deny with a probability, a confidence and a reason. Call this before anything destructive, irreversible, or outside what the user asked for.

action: the exact thing about to happen, such as a shell command, an email body, or a description of the API call. user_asked_for: what the human actually requested, verbatim. Send it. A judgement about authorization is worthless without it. pack: which question set to use. One of: {packs}.

ParametersJSON Schema
NameRequiredDescriptionDefault
packNoshell
actionYes
user_asked_forNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and does well: it discloses that the tool returns allow/ask/deny with probability, confidence, and reason, and stresses that user_asked_for is required for a valid judgement. It does not discuss failure modes or whether the tool itself takes no side effects, but the judgment-only nature is clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and well-organized: purpose first, return contract second, usage condition third, then parameter details. Every sentence adds information, and there is no redundant or filler text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers what the tool does, what it returns, when to call it, and what each parameter means. An output schema exists, so return-value details are already structured. No critical information needed for correct invocation is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, yet the description fully compensates by explaining all three parameters: action is the exact thing about to happen with concrete examples, user_asked_for is the verbatim human request, and pack is the question set. This is exactly the practical meaning an agent needs beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Judge'), a clear object ('whether an action is safe to take'), and names the exact decision output (allow, ask, deny). This makes the tool's purpose unmistakable and clearly distinct from siblings like route_turn or rank_options.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states when to call it: before anything destructive, irreversible, or outside what the user asked for. It implies when not to use it (routine actions) but does not name specific alternative tools, so it stops just short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

gate_statsA

What the gate has decided so far: counts, verdict split and spend.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the burden of behavioral disclosure. It conveys that this is a read-only aggregation tool ('what the gate has decided so far'), which implies no side effects, but it doesn't explicitly state that it performs no mutation or whether it reflects real-time or cached data.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, compact sentence that front-loads the core purpose and lists the key outputs. Every word earns its place with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a parameterless read-only stats tool with an output schema present, the description is largely complete. It could be slightly stronger by explicitly stating it performs no side effects, but the read-only nature is strongly implied by 'what the gate has decided so far' and the output schema covers return values.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the schema provides no parameter semantics to cover. The description adds value by explaining what the tool reports (counts, verdict split, spend), which is the relevant semantic content for a parameterless tool.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('decided') and names the resource ('the gate') with concrete outputs ('counts, verdict split and spend'). It clearly distinguishes this as a stats/read tool, though it doesn't explicitly name a sibling alternative.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies this is a read-only status tool for checking gate decisions, which is clear enough context. However, it doesn't explicitly state when to use it versus siblings like check_action or route_turn, nor does it mention any exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

keep_contextA

Decide which pieces of context still earn their place, in one call.

Nothing is rewritten or summarised. Each item is kept verbatim or dropped, because a summary silently loses the exact path or error you needed later.

items: the context pieces, usually tool calls and their results. goal: what the agent is trying to finish. budget: the share to keep when scores are close. 0.5 keeps half.

ParametersJSON Schema
NameRequiredDescriptionDefault
goalYes
itemsYes
budgetNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden, and it does so well: it states that nothing is rewritten or summarized, items are kept verbatim or dropped, and explains why—'a summary silently loses the exact path or error you needed later.' It does not discuss side effects or edge cases, but the core behavior is transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is short, front-loaded with the purpose, and every sentence earns its place: purpose, behavioral guarantee, rationale, and parameter semantics. There is no filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 3-parameter tool with an output schema available, the description covers the action, the key behavioral guarantee, and all parameter meanings. The main gap is explicit guidance on when to use this tool versus its siblings, which keeps it from being fully complete in context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description compensates by explaining all three parameters: items, goal, and budget. The explanations add real meaning, especially 'usually tool calls and their results' and '0.5 keeps half,' though budget's phrase 'when scores are close' is slightly ambiguous.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action—deciding which context items to keep—and clarifies the outcome: 'Each item is kept verbatim or dropped.' It clearly distinguishes this from a summarization or rewriting operation, though it does not name a sibling tool explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage through phrases like 'in one call' and 'when scores are close,' but it offers no explicit when-to-use guidance, prerequisites, or alternatives among sibling tools. An agent would have to infer when keep_context is preferable to rank_options or check_action.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

rank_optionsA

Rank a list of options against one criterion, in a single call.

Cheaper and faster than asking a chat model to sort them, and it returns a probability per option rather than an opinion. Good for choosing a file, a tool, a model tier, or reordering search results.

options: up to 255 choices. criterion: what makes one option better, as a plain statement. context: whatever the options should be judged against.

ParametersJSON Schema
NameRequiredDescriptionDefault
contextNo
optionsYes
criterionYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden, and it does disclose the key output trait—probabilities per option rather than an opinion—and the 255-option cap. However, it does not discuss behavior when context is empty, whether state is changed, or error handling, leaving some behavioral ambiguity.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description opens with the core action, uses short purpose and use-case paragraphs, and closes with one-line parameter definitions. Every sentence earns its place with no filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a narrow 3-parameter ranking tool with an output schema, the description covers purpose, usage, and parameter semantics. The main gaps are not explicitly stating the consequence of omitting context and not naming when a sibling tool would be a better choice.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

All three parameters receive a plain-language gloss: options are capped at 255 choices, criterion is a plain statement, and context is what options are judged against. This is necessary because the schema has no descriptions, though examples of well-formed criterion/context strings would make it stronger.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States the exact operation ('Rank a list of options against one criterion, in a single call') and gives concrete use cases such as choosing a file or reordering search results. It does not reference sibling tools, but the verb+resource are specific enough to make the tool's purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly frames when to prefer this tool over asking a chat model to sort ('Cheaper and faster') and names appropriate scenarios. It lacks explicit exclusions or comparisons to sibling tools, but the guidance is actionable and clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

route_turnA

Pick which model tier should handle this turn, before spending on it.

Returns a tier, the odds on each, and a confidence. Call it at the start of a turn so a rename does not cost what an architecture decision costs.

task: what this turn is being asked to do. tiers: your tier names, cheapest first. Defaults to fast, standard, frontier.

ParametersJSON Schema
NameRequiredDescriptionDefault
taskYes
tiersNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It explains that the tool returns a tier, odds, and confidence, and implies it is a pre-spend decision aid. However, it does not disclose whether the tool itself incurs cost, has side effects, or how the odds are computed, leaving meaningful behavioral gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded: purpose first, return value second, call timing third, then parameter definitions. Every sentence contributes useful information, and the metaphor about rename vs. architecture decision is vivid but not bloated.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter tool with an output schema present, the description covers the main decisions an agent needs: what it does, what it returns, when to call it, and what the parameters mean. It is missing minor details like error behavior and whether tier names are validated, but those are not critical given the output schema exists.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does. It explicitly defines 'task' as 'what this turn is being asked to do' and 'tiers' as tier names ordered cheapest first, with a clear default of 'fast, standard, frontier.' This adds real meaning beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb and resource: 'Pick which model tier should handle this turn.' It also clarifies the return type (tier, odds, confidence) and the cost-routing intent. However, it does not explicitly differentiate itself from sibling tools like rank_options, so it stands just short of full distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives an explicit call-time instruction: 'Call it at the start of a turn so a rename does not cost what an architecture decision costs.' This is solid contextual guidance on when to use it, though it does not mention when not to use it or name alternative tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 5 tool updatesv0.2.0
    • First observedcheck_action
    • First observedgate_stats
    • First observedkeep_context
    • First observedrank_options
    • First observedroute_turn
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.1/5.0

Scored across 5 tools

Disambiguation5/5

Each tool addresses a distinct decision or operation: safety checks, model tier routing, context pruning, option ranking, and statistics. Even though they share a 'gate' theme, the descriptions clearly separate their purposes, leaving no ambiguity.

Naming Consistency4/5

Four tools follow a clear verb_noun pattern (check_action, route_turn, keep_context, rank_options), while gate_stats is a noun_noun compound. This is a minor deviation but still consistent in style (lower_snake_case) and readable.

Tool Count5/5

Five tools is well-scoped for a gating/decision server, covering the essential operations without redundancy or bloat. It feels complete for the stated domain.

Completeness4/5

The tools cover the full decision lifecycle: pre-action safety checks, turn routing, context management, option ranking, and post-hoc statistics. A configuration or reset tool might be missing, but the core surface is solid and no critical gaps are evident.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • MCP Permission Guard

    AMEOBIUS-space
    A
    license
    Not graded
    quality
    C
    maintenance
    A pre-action authorization server for AI agents that classifies tool calls into 14 intent categories, scores risk 0-100, and produces deterministic allow/deny/ask decisions with full audit trail.
    MIT