Skip to main content
Glama
kangwa

Penny

by kangwa
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

Penny

Spend frontier-model tokens on planning, not execution

CI License: MIT Python 3.14+ Status: alpha

Frontier planning · Local execution · Deterministic eval gates

Quick start · How it works · Safety model · Development

Penny lets a frontier Claude model inspect a repository and produce a rigorous execution plan, then hands the implementation to a cheaper or local model. A deterministic Python orchestrator controls the workflow, runs the evals, and decides whether each step may proceed.

A run in progress: the worker's reasoning, its tool calls, and the eval verdicts that gate the next step

WARNING

Penny is alpha software. It launches an autonomous coding agent with permission to edit files and run commands inside a repository. Use it only in version-controlled projects, inspect the generated plan, and review the final diff before merging changes.

Why Penny?

Frontier coding models are excellent at understanding unfamiliar systems, making architectural decisions, and decomposing complex work—but long agentic sessions can consume a large number of expensive tokens.

Penny separates the expensive reasoning from the token-heavy implementation loop:

  1. Claude plans once using your Claude subscription.

  2. A cheaper or local model executes small, self-contained steps through Claude Code.

  3. Penny runs the evals itself and advances only when the required checks pass.

The worker never receives the frontier conversation, and the frontier model does not remain in the loop while files are read, edited, tested, and retried.

Related MCP server: cc-agent

How it works

flowchart LR
    U[Developer request] --> P[Claude Code planner skill]
    P --> M[Execution manifest]
    M --> O[Penny orchestrator]
    O --> W[Claude Code worker]
    W --> E[Anthropic-compatible endpoint]
    E --> L[Local or cheaper model]
    L --> R[Repository changes]
    R --> V[Deterministic evals]
    V -->|Pass| N[Next step]
    V -->|Fail| T[Fresh retry session]
    T --> W
    N --> O

The YAML execution manifest is the contract between planning and execution. It contains:

  • the task objective and acceptance criteria;

  • ordered, self-contained worker prompts;

  • allowed and forbidden path scopes;

  • deterministic evals for every step;

  • retry limits and final regression checks.

Frontier Claude
    decides what should change
    and how success will be proven

Worker model
    performs one bounded implementation step

Penny
    owns state, runs evals, enforces scope,
    retries failures, and decides when to advance

A real run

Adding an admin login page to a Python web project: planned by Claude Opus, implemented by qwen/qwen3.5-9b served from LM Studio on a laptop. Penny's own report.md, abridged:

# Admin login page (UI only, no backend auth)

- Status: **completed**
- Duration: 746.7s
- Model: qwen/qwen3.5-9b
- Endpoint: http://localhost:1234

## Steps

- `add-login-view` — passed (1 attempt(s))
  - **PASS** `login-module-exists` — found 1 matching path(s)
  - **PASS** `login-view-markup` — all required strings found
  - **PASS** `login-handler-behavior` — command passed
  - **PASS** `suite-green-after-login-view` — command passed
- `register-login-route` — passed (1 attempt(s))
- `add-login-tests` — passed (1 attempt(s))
- `add-login-styles` — passed (1 attempt(s))

## Final evals

- **PASS** `full-suite` — command passed
- **PASS** `pyproject-untouched` — command passed
- **PASS** `overall-scope` — all changed paths are allowed

Four steps, four files, twelve minutes, 39 eval assertions — and no frontier tokens spent after the plan was written.

Features

  • Claude Code planning skill designed for subscription-backed frontier planning.

  • Fresh worker session for every step and retry.

  • Any sufficiently Anthropic-compatible /v1/messages endpoint.

  • Deterministic command, file, content, and Git-scope evals.

  • Independent detection of out-of-scope changes.

  • Durable run state, logs, transcripts, and reports.

  • MCP integration for plan validation, submission, status, cancellation, and reports.

  • Local web console for observing active and completed runs.

  • Built-in environment doctor and deterministic end-to-end demo.

Requirements

Penny currently requires:

  • macOS or Linux;

  • Python 3.14 or newer;

  • uv;

  • Claude Code on PATH;

  • Git;

  • an Anthropic Messages API-compatible worker endpoint for real executions.

The worker endpoint may be provided by LM Studio, oMLX, a gateway, a remote inference server, or another compatible service. An OpenAI-only /v1/chat/completions endpoint is not enough; the worker path requires Anthropic-style messages, streaming, and tool use.

Quick start

1. Install Penny

git clone https://github.com/kangwa/penny.git
cd penny
uv sync

Optionally install the CLI on your PATH:

uv tool install --editable .

The examples below use uv run penny. After a tool install, you can use penny directly.

2. See it work before configuring anything

uv run penny demo

The demo exercises the complete orchestration loop in throwaway repositories. It validates a manifest, executes a two-step task, intentionally fails one attempt, retries it in a fresh session, and verifies every eval gate — then asserts each guarantee and exits non-zero if any is unmet.

It takes a few seconds, makes no model call, and requires no endpoint.

3. Install the Claude planner skill

Install it globally for your Claude Code user:

uv run penny skill install

Or install it into one repository only:

uv run penny skill install --project /path/to/project

Start a new Claude Code session after installing the skill.

4. Configure the worker endpoint

Create Penny's settings directory and copy the example configuration:

mkdir -p ~/.penny
cp penny.env.example ~/.penny/.env.penny

Edit ~/.penny/.env.penny:

# Root before /v1/messages
PENNY_BASE_URL=http://localhost:1234

# Any model identifier accepted by the endpoint
PENNY_MODEL=openai/gpt-oss-20b

# bearer -> Authorization: Bearer
# api-key -> x-api-key
PENNY_AUTH_TYPE=bearer
PENNY_AUTH_TOKEN=local

PENNY_MAX_TURNS=30
PENNY_AGENT_TIMEOUT=1800

Penny also reads .env.penny from its checkout and accepts ordinary environment variables. Explicit environment variables take precedence over the settings file.

IMPORTANT

Do not exportANTHROPIC_BASE_URL globally. Penny injects the worker endpoint only into the spawned worker process so your normal Claude Code session continues using your Claude subscription.

5. Check the complete worker chain

Run this from a clean Git repository:

uv run --project /path/to/penny penny doctor

doctor verifies:

  • the current Git repository and working tree;

  • the Claude Code executable;

  • endpoint and model configuration;

  • endpoint reachability;

  • Anthropic Messages API compatibility;

  • a real Claude Code read/edit/shell tool-use session in a temporary repository.

Useful variants:

# Fast configuration-only check
uv run --project /path/to/penny penny doctor --skip-smoke

# Machine-readable report
uv run --project /path/to/penny penny doctor --json

Plan and execute through Claude Code

Penny supports two workflows: an integrated MCP path and a manual CLI path.

MCP workflow

Register Penny's MCP server for the repository it is allowed to operate in:

uv run penny mcp install --project /absolute/path/to/project

The default local scope registers the server only for that repository. Restart Claude Code in the target project, then run:

/local-execution-planner Add idempotency support to payment creation

The planner skill will:

flowchart TD
    A[Inspect repository] --> B[Build execution manifest]
    B --> C[Validate through Penny MCP]
    C -->|Invalid| B
    C -->|Valid| D[Show plan and eval summary]
    D --> E{User confirms?}
    E -->|Yes| F[Submit plan]
    E -->|No| G[Stop without execution]
    F --> H[Return run ID and stop]
    H --> I[Penny continues independently]

Use wording such as “plan and execute” when you want the skill to submit the plan without an additional confirmation step.

The MCP server exposes operations for:

  • environment diagnostics;

  • manifest validation;

  • plan submission;

  • status retrieval;

  • cancellation;

  • final reports.

Submitted runs continue in a detached process and do not require the frontier Claude session to remain open.

Manual CLI workflow

You can also save the planner's YAML output and run it directly:

uv run --project /path/to/penny penny validate plan.yaml
uv run --project /path/to/penny penny run plan.yaml --dry-run
uv run --project /path/to/penny penny run plan.yaml

--repo defaults to the current directory. To execute against another repository:

uv run --project /path/to/penny penny run plan.yaml \
  --repo /absolute/path/to/project

Each run writes its artifacts under the target repository:

.local-agent/runs/<run-id>/
├── manifest.yaml
├── state.json
├── report.md
└── logs/
    ├── <step>-attempt-<n>-agent.log
    ├── <step>-attempt-<n>-evals.json
    └── final-evals.json

Read a saved run's state:

uv run penny status .local-agent/runs/<run-id>

Web console

Start the local run console:

uv run penny web --allow-root ~/Work

Then open http://127.0.0.1:8765.

The console following a run as the worker streams its turns

The console can:

  • list active and completed runs;

  • follow current steps and attempts;

  • show worker prompts, reasoning, tool calls, and eval results;

  • display changed files and final reports;

  • cancel an active run;

  • edit worker endpoint settings.

Every run in reach, with its status, progress, and duration:

The runs list

CAUTION

The web console is unauthenticated and binds to loopback by default. Do not expose it publicly. The console is an observability and control layer; it does not execute steps or evals itself.

Safety model

Penny treats the worker model as untrusted.

The worker cannot declare success

The orchestrator runs every eval itself. A worker response saying “all tests pass” has no effect on workflow state.

Supported eval classes include:

  • shell commands with bounded timeouts;

  • file existence checks;

  • file content assertions;

  • Git diff allowlists;

  • final regression suites.

Every attempt is scoped

Before an attempt, Penny records the repository state. After the worker exits, Penny independently checks which files changed and compares them with the step's allowed and forbidden path scopes.

Failures receive bounded retries

When an eval fails, Penny starts a fresh worker session containing only:

  • the original step prompt;

  • the relevant eval failures;

  • the repository state left by the previous attempt.

The default is two attempts. After the limit is exhausted, the run becomes blocked instead of silently proceeding.

Credentials are isolated

The worker subprocess receives only its configured endpoint credentials. Penny strips inherited ANTHROPIC_* credentials so a failed local endpoint cannot silently fall back to the subscription-backed frontier account.

Repository access is explicit

The MCP server accepts only absolute paths inside explicitly configured roots and resolves them to real Git repositories before execution.

Worker endpoint configuration

Penny uses the following settings:

Variable

Purpose

Default

PENNY_BASE_URL

Endpoint root before /v1/messages

—

PENNY_MODEL

Model identifier accepted by the endpoint

—

PENNY_AUTH_TYPE

bearer or api-key

bearer

PENNY_AUTH_TOKEN

Bearer token

—

PENNY_API_KEY

x-api-key credential

—

PENNY_CUSTOM_HEADERS

Optional additional headers

—

PENNY_MAX_TURNS

Maximum Claude Code worker turns

30

PENNY_AGENT_TIMEOUT

Worker timeout in seconds

1800

PENNY_CLAUDE_BIN

Claude Code executable

claude

PENNY_ENV_FILE

Override settings-file location

auto-detected

PENNY_MCP_ALLOWED_ROOTS

Path-separated MCP repository roots

—

PENNY_MCP_STATE_DIR

MCP submission registry

~/.penny

The endpoint must support the parts of the Anthropic Messages API used by Claude Code, including streaming and structured tool-use exchanges. Compatibility with non-Anthropic models depends on both the endpoint implementation and the model's ability to use Claude Code's tools reliably.

See docs/orchestrator.md for endpoint examples, all CLI flags, troubleshooting, MCP security details, and worker invocation behaviour.

Execution manifests

The planner skill emits YAML manifests. The format is documented in:

A simplified example:

task:
  id: payment-idempotency
  title: Add payment idempotency
  objective: Prevent duplicate payment creation for repeated requests.

steps:
  - id: add-model-support
    title: Add database support
    prompt: |
      Inspect the existing payment model and migration conventions.
      Add account-scoped idempotency support without changing current
      behaviour for requests that omit the key.
    allowed_paths:
      - payments/models.py
      - payments/migrations/**
      - payments/tests/**
    forbidden_paths:
      - settings/**
    max_attempts: 2
    evals:
      - id: model-tests
        type: command
        command: pytest payments/tests/test_models.py
        timeout_seconds: 300

final_evals:
  - id: payment-suite
    type: command
    command: pytest payments
    timeout_seconds: 900

A good manifest tells a weaker worker model exactly what to inspect, what behaviour to implement, what not to change, and how Penny will prove completion.

Command reference

Command

Purpose

penny skill install

Install the planner skill globally or in one project.

penny mcp install

Register the MCP server with Claude Code.

penny doctor

Verify the repository, CLI, endpoint, API, and tool-use chain.

penny validate PLAN

Validate a YAML execution manifest.

penny run PLAN

Execute a manifest.

penny status RUN_DIR

Print a saved run's state.

penny web

Start the local run console.

penny demo

Run the deterministic end-to-end demonstration.

Use penny <command> --help for all flags.

Project structure

penny/
├── skill/local-execution-planner/   # Claude Code frontier-planning skill
├── src/penny/                       # Orchestrator, CLI, MCP, doctor, console
├── examples/                        # Example execution manifests
├── docs/orchestrator.md             # Detailed reference documentation
├── tests/                           # Unit and integration tests
├── penny.env.example                # Worker endpoint configuration template
└── pyproject.toml

Development

Clone the repository and install all dependencies:

git clone https://github.com/kangwa/penny.git
cd penny
uv sync

Run the test suite:

uv run pytest

Run the deterministic vertical slice:

uv run penny demo

For skill development, install a symlink instead of copying the skill:

uv run penny skill install --link --force

Build the package:

uv build

Contributing

Penny is early-stage and contributions are welcome, particularly around:

  • Anthropic-compatible endpoint interoperability;

  • manifest and eval design;

  • safer execution isolation;

  • Git worktree support;

  • resumable and parallel workflows;

  • observability and run reporting;

  • Linux and macOS compatibility;

  • documentation and examples.

Before opening a pull request:

  1. keep changes focused;

  2. add or update tests;

  3. run uv run pytest;

  4. run uv run penny demo;

  5. avoid committing .env.penny, .local-agent/, virtual environments, logs, or credentials.

Please open an issue before beginning a large architectural change so the approach can be discussed first.

CONTRIBUTING.md covers the development setup, the testing conventions, and what CI checks. Bug reports are most useful with penny doctor --json output and your endpoint and model — the issue template asks for both. Release notes live in CHANGELOG.md.

Penny runs an autonomous agent with permission to edit files and execute commands, so it has a threat model worth reading before you trust it with a repository: SECURITY.md. Report vulnerabilities privately rather than in a public issue.

Roadmap

Penny's current focus is making the basic planner-to-worker handoff dependable and auditable. Likely next areas include:

  • Git worktree isolation for every run;

  • pause, resume, and explicit step retry controls;

  • improved endpoint compatibility diagnostics;

  • richer manifest schema validation;

  • resource and token accounting;

  • optional frontier escalation when a run becomes blocked;

  • broader platform and Python-version support.

The roadmap is directional rather than a commitment. Issues and pull requests should remain grounded in small, testable improvements.

Status and limits

Penny is alpha, and worth being precise about what that means:

  • Version 0.3. The manifest format and run directory layout may still change. Installation and distribution details may change before a first stable release.

  • Exercised against qwen/qwen3.5-9b on LM Studio. Other Anthropic-compatible servers and models should work, and penny doctor is how you find out. Small models vary enormously in how reliably they hold a Claude Code tool loop, and endpoint implementations vary in how faithfully they reproduce it.

  • Plan quality is the ceiling. Penny makes a weak worker safe, not smart. A vague step guarded by a weak eval produces a confidently wrong change that passes its gate.

  • Costs are shifted, not removed. Planning still spends subscription usage; the worker spends electricity, wall-clock time, and your attention on review.

License

Penny is released under the MIT License.

Available Tools

6 tools
cancel_runB

Request cancellation of an active run.

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
okNo
errorNo
detailNo
run_idNo
statusNo

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure, but it only states 'Request cancellation of an active run.' It does not explain whether cancellation is asynchronous, what state the run enters, whether it is reversible, or what failures may occur. The wording 'Request cancellation' hints at a non-guaranteed result but leaves the outcome undefined.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence with no filler. It earns its place by providing the essential purpose without unnecessary detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a mutating operation with no annotations and no output schema details visible, the description is too sparse. It omits side effects, response/error behavior, and prerequisites. The phrase 'active run' provides some context but does not address what happens for non-active runs or how cancellation results are communicated.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema includes a single required parameter, run_id, and the description adds the qualifier 'active run', implying the run must be in progress. This adds slight semantic meaning beyond the schema. However, the description does not specify the format of run_id or how to obtain it, and schema coverage is effectively 0%.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb phrase, 'Request cancellation', and identifies the target resource, 'active run'. This clearly conveys what the tool does and distinguishes it from sibling tools like get_run and validate_plan. However, it does not elaborate on the scope of cancellation (e.g., partial vs. full).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The intended usage is implied: use when an active run needs to be stopped. However, the description does not provide explicit when-to-use guidance, prerequisites, or alternatives. There is no mention of checking run status first or scenarios where cancellation might not be possible.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

doctorB

Check that the local execution environment is ready for a repository.

ParametersJSON Schema
NameRequiredDescriptionDefault
repositoryYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
okNo
noteNo
errorNo
readyNo
checksNo
configNo
countsNo
repositoryNo

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It confirms a check operation but does not disclose whether it modifies anything, what checks are performed, or what success/failure looks like. The term 'ready' is undefined.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence, direct, and front-loaded with the action. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is too minimal for a tool with an output schema and no annotations. It doesn't explain what readiness means, what the output contains, or any side effects. Sibling tool names provide some context but not enough.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description references 'repository' but adds no semantic detail. The input schema provides only a string type with no description, and the description does not clarify expected values, format, or scope.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: checking local environment readiness for a repository. The verb 'check' and the resource 'local execution environment' are specific, and this distinguishes it from sibling tools that deal with plans and runs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided about when to use this tool versus alternatives like validate_plan or get_report. The description implies a pre-flight check use case but does not state explicit conditions, prerequisites, or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_reportB

Return the execution report for a run.

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
okNo
errorNo
finalNo
reportNo
run_idNo
statusNo

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must disclose behavior itself. It only says 'Return the execution report' which implies a read operation but does not describe whether the report is always available, what it contains beyond execution, or side effects. This is insufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence with no waste. It is direct and appropriately sized.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given there is an output schema, return values are covered. However, the description lacks usage context and alternative differentiation, and with no annotations, the description does not fully stand alone for a simple retrieval tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the description adds minimal meaning: it implies run_id refers to a run but gives no format, source, or relationship to other resources. The param name 'run_id' is self-explanatory but the description doesn't compensate for missing schema docs.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses the specific verb 'Return' and names a distinct resource 'execution report', clearly differentiating from sibling get_run. It states the scope ('for a run') and the action is unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus sibling tools like get_run. There is no mention of prerequisites, alternatives, or contexts where this report is applicable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_runB

Return the current state of a submitted run.

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
okNo
errorNo
stepsNo
run_idNo
statusNo
workerNo
task_idNo
run_errorNo
repositoryNo
final_evalsNo
current_stepNo
submitted_atNo
duration_secondsNo

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of disclosing behavior. It only states that it returns the current state, but it does not mention whether the operation is safe/read-only (implicit from 'get'), error handling for invalid run_ids, or any caveats about staleness. This lack of detail is a notable gap for a tool with zero annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, concise sentence that directly conveys the purpose. It avoids redundant phrasing and is appropriately sized for the tool's simplicity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple (one parameter, output schema provided), and the description is enough for a basic getter. However, it lacks explicit guidance on when to use it relative to siblings and does not mention any prerequisites like 'after submission' beyond the word 'submitted.' Overall, it is minimally adequate but leaves room for more context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description does not elaborate on the run_id parameter. While the parameter name and type are self-explanatory, the description adds no additional meaning about required format, source, or constraints beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'Return the current state of a submitted run' uses a specific verb and resource, clearly indicating a read operation for run status. It distinguishes itself from siblings like cancel_run and submit_plan by focusing on state retrieval, not modification.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage after a run is submitted, but it does not explicitly state when to use this tool versus alternatives like get_report or cancel_run. No exclusions or alternative tool names are mentioned, leaving usage to be inferred from the tool name and context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

submit_planA

Validate a manifest and queue a run. Returns immediately with a run_id.

ParametersJSON Schema
NameRequiredDescriptionDefault
manifestYes
repositoryYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
okNo
errorNo
run_idNo
statusNo
run_dirNo
task_idNo
repositoryNo
manifest_pathNo
existing_run_idNo

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full responsibility for behavioral disclosure. It explicitly notes that the tool returns immediately with a run_id, indicating asynchronous behavior. However, it provides no details about failure modes, permissions, or what happens during validation failure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two compact sentences deliver the essential behavior and return value. The information is front-loaded and every word serves a purpose, with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is relatively simple with only two string parameters and an output schema, so the description does not need to detail return values. However, the lack of parameter clarification and failure behavior leaves some gaps for a critical submission tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must explain the parameters. It only implicitly covers 'manifest' by saying 'Validate a manifest,' leaving 'repository' entirely unspecified. The description adds minimal value over the raw parameter names.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function with a specific verb and resource: 'Validate a manifest and queue a run.' This distinguishes it from siblings like validate_plan (validation only) and get_run (retrieval).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the tool is used to submit a plan for execution, but it does not explicitly state when to prefer this over validate_plan or other siblings. There is no mention of exclusions or prerequisites, so the guidance is only implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

validate_planA

Validate a YAML execution manifest without starting any work.

ParametersJSON Schema
NameRequiredDescriptionDefault
manifestYes
repositoryYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
okNo
errorNo
stepsNo
task_idNo
repositoryNo
step_countNo
task_titleNo
final_eval_commandsNo

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description provides the important behavioral guarantee of no side effects ('without starting any work'), but it does not disclose other characteristics such as return behavior, network/IO dependencies, or how validation errors are reported. Adds some value but remains limited.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, front-loaded sentence that efficiently conveys the core purpose and a key safety property. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has a simple schema and output schema, but with no annotations, the description should provide more guidance. It covers the no-side-effect property but omits repository semantics and any validation behavior. Adequate but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must clarify both parameters. It only clarifies 'manifest' as a YAML execution manifest, while 'repository' is undefined. This leaves a critical gap for correct invocation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (validate), the resource (YAML execution manifest), and the key constraint (without starting any work). This distinguishes it from submit_plan and other siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'without starting any work' implies a use case of pre-flight checking before execution, but it does not explicitly name alternatives or state when to use this rather than submit_plan. Usage context is implied but not explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 6 tool updatesv0.3.0
    • First observedcancel_run
    • First observeddoctor
    • First observedget_report
    • First observedget_run
    • First observedsubmit_plan
    • First observedvalidate_plan
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A3.7/5.0

Scored across 6 tools

Disambiguation5/5

Each tool addresses a distinct step in the execution workflow: environment readiness, plan validation, submission, status retrieval, cancellation, and reporting. There is no overlap or ambiguity between tool purposes.

Naming Consistency4/5

Most tools follow a clear verb_noun pattern (validate_plan, submit_plan, get_run, cancel_run, get_report). The exception is 'doctor', which is a single verb, but its purpose is clear and the deviation is minor.

Tool Count5/5

Six tools provide a focused yet complete coverage of the run lifecycle without unnecessary redundancy. The count feels well-scoped for the server's stated purpose.

Completeness5/5

The toolset covers environment setup, plan validation, run submission, status monitoring, cancellation, and report retrieval—all essential operations for executing and managing repo runs. No critical gaps are apparent.

Maintenance

ActivitySlowing
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • cc-agent

    Gonzih
    A
    license
    A
    quality
    D
    maintenance
    MCP server that spawns autonomous Claude Code agents in GitHub repos, enabling task delegation with persistent state, multi-step workflows, and job monitoring.
    47
    187 npm
    2
    Apache 2.0

  • io.github.ymxlx/polis-protocol

    ymxlx
    A
    license
    Not graded
    quality
    A
    maintenance
    This MCP server provides a local-first control plane for coding agents, enabling task routing, contract management, and shared learning across multiple AI vendors via markdown files.
    5
    MIT