Skip to main content
Glama

devenv-mcp

CI

A Model Context Protocol server that gives an AI agent a safe place to do software-engineering work. The agent can browse tasks, open a sandboxed copy of a broken repository, run commands in it, and diagnose CI failures, all through typed MCP tools.

It is the agent-facing half of rl-env-benchmark and reads the same task format, and it calls ci-doctor for CI log diagnosis.

Tools

Tool

What it does

list_tasks

List tasks with id, title, category and difficulty

get_task

Full prompt for one task

start_session

Create a fresh sandbox holding the task's starting repo; returns session_id

run_in_sandbox

Run a shell command in the session's repo root; returns exit code and output

end_session

Destroy the sandbox

diagnose_ci

Classify failures in a CI log or JUnit XML and suggest a next step

run_grader

Run the hidden grader. Off by default, see below

Every tool declares a zod input schema, returns structuredContent alongside readable text, and reports failures in-band with isError, so the model can read what went wrong and recover instead of the call failing outright.

Related MCP server: nc-tools-mcp

Design decisions

  • Isolation by default. Sessions run in Docker with --network none, 1 CPU, 1 GB of memory and a PID limit. A local mode exists for task authoring, but it runs commands on the host and is clearly labelled as not isolated.

  • Grading is opt-in. If an agent under evaluation can call the grader, it can iterate against hidden tests until they pass, which measures persistence rather than skill. run_grader is only registered when DEVENV_ALLOW_GRADING=1, for task authors and human reviewers.

  • The grader never enters the workspace early. It is copied in only when grading runs, and a test asserts the agent cannot find it beforehand.

  • Bounded output. Command output is collected in a head-plus-tail buffer (the end of a log is usually where the error is), so a runaway process cannot exhaust memory or flood the model's context.

  • Real timeouts. Commands start in their own process group and the whole group is killed on timeout. Killing only sh would leave grandchildren such as pytest holding the pipes open, and the call would hang.

  • Capped, cleaned-up sessions. A session limit prevents runaway container creation, and all sandboxes are destroyed on SIGINT, SIGTERM or client disconnect.

  • stdout is the protocol. All logging goes to stderr.

Install

git clone https://github.com/TyllerDaniel/devenv-mcp
cd devenv-mcp && npm install && npm run build

Use with Claude Desktop or Claude Code

{
  "mcpServers": {
    "devenv": {
      "command": "node",
      "args": ["/path/to/devenv-mcp/dist/index.js"],
      "env": {
        "DEVENV_TASKS_DIR": "/path/to/rl-env-benchmark/tasks",
        "DEVENV_SANDBOX": "docker"
      }
    }
  }
}

With Claude Code: claude mcp add devenv -e DEVENV_TASKS_DIR=/path/to/tasks -- node /path/to/devenv-mcp/dist/index.js

Configuration

Variable

Default

Meaning

DEVENV_TASKS_DIR

./tasks

Directory of tasks (task.toml + repo/ + grader/)

DEVENV_IMAGES_DIR

<tasks>/../images

Base images, images/<name>/Dockerfile

DEVENV_SANDBOX

docker

docker or local (not isolated)

DEVENV_ALLOW_GRADING

0

1 registers run_grader

DEVENV_MAX_SESSIONS

4

Concurrent sandbox limit

CI_DOCTOR_BIN

ci-doctor

Path to the ci-doctor binary

Invalid values fail fast at startup with a message naming the variable.

Example session

From npm run build and a client connected over stdio, against the rl-env-benchmark tasks with grading enabled:

> start_session {"task_id": "004-quadratic-dedupe"}
Started session 72721590 for 004-quadratic-dedupe (local sandbox).

> run_in_sandbox {"session_id": "72721590", "command": "python3 -m pytest -q tests | tail -1"}
[exit code 0]
1 passed in 0.01s

> run_grader {"session_id": "72721590"}
FAILED                    <- visible tests pass, but the hidden 300k-event test times out

> run_in_sandbox {..., "command": "sed -i 's/seen = \\[\\]/seen = set()/; s/seen.append/seen.add/' ingest/dedupe.py"}
> run_grader {"session_id": "72721590"}
PASSED

Development

npm test          # vitest: unit tests plus a real MCP client over an in-memory transport
npm run typecheck
npm run build

The server tests connect an actual MCP Client to the server, so they cover schemas, error handling and the full start, run, grade and end loop rather than calling handlers directly.

License

MIT

Available Tools

6 tools
diagnose_ciDiagnose CI failureA
Read-only

Classify the failures in a CI log or JUnit XML report (flaky test, dependency conflict, timeout, compile error, assertion, ...) and suggest a next step. Uses the ci-doctor CLI.

NameRequiredDescriptionDefault
logYesRaw CI log text or JUnit XML
input_formatNoauto

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already declare readOnlyHint=true and openWorldHint=false, so safety and locality are covered structurally. The description adds one piece of genuine context - it is backed by the ci-doctor CLI - but says nothing about latency, log size limits, or the shape of the classification/suggestion it returns despite there being no output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences with the capability and its output front-loaded and the implementation detail (ci-doctor CLI) trailing. No filler or restatement of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter read-only diagnostic with no output schema, the description covers what goes in and roughly what comes back (a classification plus a suggested next step). The main gap is the 'input_format' semantics and any handling limits for large logs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 50%: the required 'log' parameter is documented, but 'input_format' has only an enum and default with no explanation in either the schema or the description. The description's phrase 'CI log or JUnit XML' hints at the text vs junit distinction but never says when to override the 'auto' default, so the one undocumented parameter stays ambiguous.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb (classify) and resource (failures in a CI log or JUnit XML report), enumerates concrete failure categories, and adds a second capability (suggest a next step). Nothing in the sibling set (list_tasks, get_task, start_session, run_in_sandbox, end_session) overlaps, so an agent can route to it unambiguously.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied by the phrase 'a CI log or JUnit XML report' - there is no explicit statement of when to call this versus other tools, no prerequisites, and no exclusion such as when a log is too large or already parsed. Adequate but leaves the agent to infer the triggering condition.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

end_sessionEnd sessionA
Destructive

Destroy a session's sandbox and free its resources.

NameRequiredDescriptionDefault
session_idYes

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare destructiveHint=true, idempotentHint=false, and openWorldHint=false, so the safety profile is structured. The description adds useful specificity by stating that the sandbox is destroyed and resources are freed, which goes beyond the generic destructive flag. It still does not explain error behavior, whether repeated calls fail, or what happens to associated tasks.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single front-loaded sentence with no filler. Every phrase contributes to the meaning: destroy action, sandbox target, and resource release.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter lifecycle tool with no output schema and strong annotations, the description covers the core effect adequately. The main remaining gaps are where session_id comes from and any precondition around active tasks, but these are relatively minor for this tool's complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% for the single required session_id parameter, so the description must compensate but does not. 'A session's sandbox' only indirectly implies that session_id identifies the target session; no format, source, or validity constraints are provided.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb, 'Destroy,' and names the exact resource affected: a session's sandbox and its resources. It is clearly distinct from lifecycle siblings such as start_session and run_in_sandbox.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied by the lifecycle framing: an agent can infer this should be called when a session is no longer needed. It does not explicitly say when to use this versus leaving a session running, nor does it state preconditions such as whether active tasks must be stopped first.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_taskGet taskA
Read-only

Get the full prompt and metadata for one task.

NameRequiredDescriptionDefault
task_idYesTask id from list_tasks

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

readOnlyHint=true and openWorldHint=false already tell the agent this is a safe, closed-world read. The description adds the useful detail that the response includes the full prompt text plus metadata (helpful since there is no output schema), but says nothing about error behavior for an unknown task_id.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One short sentence with the resource and the returned content front-loaded; every word earns its place and there is no filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter read tool with full schema coverage and annotations covering safety, the description supplies the missing piece: what the call returns (prompt + metadata). Only minor gaps remain around failure modes for an invalid id.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a single required parameter already documented as coming from list_tasks, so the schema carries the load. The description adds no format or validation detail beyond what the schema states; baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Get ... one task') and clarifies the payload ('full prompt and metadata'), which distinguishes it from the sibling list_tasks by scope (single task). It stops short of naming that sibling explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: 'one task' suggests it is the follow-up to a listing call, but the description gives no explicit when-to-use, prerequisites, or alternatives. The routing hint lives in the parameter schema ('Task id from list_tasks') rather than the description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_tasksList tasksA
Read-only

List every available task with its id, title, category and difficulty.

NameRequiredDescriptionDefault

No parameters

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The readOnlyHint and openWorldHint=false annotations already establish that this is a safe, closed-world read. The description adds the return field set, which is genuinely useful with no output schema, but says nothing about ordering, result size, or whether tasks are global or session-scoped.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence that names the action, the scope, and the returned fields with no redundant wording.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a trivial zero-parameter read with annotations and no output schema, the description covers enough to call it correctly. The remaining gap is minor: no mention of result ordering or whether the list is workspace-scoped.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so there is no schema semantics to add beyond confirming that the listing is unconditional and unfiltered. Baseline 4 applies for a no-parameter tool.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

"List every available task" gives a specific verb plus resource, and the description enumerates the returned fields (id, title, category, difficulty). It is clearly distinguishable from get_task by the word "every", though it never names that sibling explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: an agent can infer this is the discovery step before get_task or start_session. There is no explicit statement of when to prefer this over get_task, nor any exclusion about filtering or scope.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_in_sandboxRun command in sandboxA

Run a shell command from the repository root inside a session's sandbox. Returns the exit code and combined stdout/stderr, truncated in the middle if very long. No network access.

NameRequiredDescriptionDefault
commandYesShell command, run with sh -c
timeout_sNoKill the command after this many seconds
session_idYes

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds genuine behavior beyond annotations: the working directory (repository root), that stdout and stderr are combined, that output is truncated in the middle when long, and that there is no network access (consistent with openWorldHint=false). It does not cover what happens on timeout expiry beyond the schema's kill semantics, which keeps it from a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with the action and execution context, then return shape, then the network constraint. No redundant restatement of the title or name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the return-value burden well by naming exit code, combined stdout/stderr, and middle truncation. The main gap is the lifecycle relationship to start_session/end_session, which an agent would need to call this safely.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 67% and the description does not compensate for the undocumented session_id: it never says where the id comes from or whether the session must be active. The 'repository root' phrasing does usefully constrain how the command executes, but the meaning of the remaining parameters comes entirely from the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (run) plus resource (shell command) plus scope (from the repository root, inside a session's sandbox). None of the siblings (list_tasks, get_task, start_session, end_session, diagnose_ci) execute arbitrary commands, so an agent can distinguish it immediately without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: the mention of 'a session's sandbox' hints that a session must already exist, but no explicit prerequisite, no when-to-use framing, and no routing away from siblings like diagnose_ci is given. The agent must infer the workflow context itself.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

start_sessionStart sessionA

Create a fresh sandbox containing the task's starting repository. Returns a session_id for run_in_sandbox. Each session is independent; end it with end_session.

NameRequiredDescriptionDefault
task_idYesTask id from list_tasks

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only supply openWorldHint=false, so the description carries most of the burden and does add real behavioral facts: the return value (session_id), the independence of each session, and the requirement to terminate via end_session. It stops short of saying what happens to an unclosed session or whether the sandbox has a time limit.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences with zero filler; the core action and return value come first, and the lifecycle notes follow. Nothing could be removed without losing information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With one required parameter, no output schema, and thin annotations, the description covers what is needed: what is produced, what is returned, and how the session is closed. A single-parameter creation tool needs little more.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single task_id parameter is fully documented in the schema (100% coverage), and the description only implicitly links it to the starting repository via 'the task's starting repository'. Baseline 3 applies since the schema already does the work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Create a fresh sandbox containing the task's starting repository') and immediately ties it to its siblings by naming run_in_sandbox and end_session, so the agent can place it in the workflow without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear lifecycle guidance: the returned session_id feeds run_in_sandbox and the session must be closed with end_session. It does not explicitly state when to create a new session versus reuse an existing one, but the independence statement covers the main decision.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 6 tool updatesv0.1.0
    • First observeddiagnose_ci
    • First observedend_session
    • First observedget_task
    • First observedlist_tasks
    • First observedrun_in_sandbox
    • First observedstart_session

TDQS

A4/5.0

Scored across 6 tools

Disambiguation4/5

The task-browsing pair (list_tasks/get_task) and the sandbox lifecycle trio (start_session/run_in_sandbox/end_session) are clearly delineated by resource and action. diagnose_ci sits in a slightly separate domain (CI log analysis) but its description makes its purpose distinct from run_in_sandbox.

Naming Consistency5/5

All names use snake_case verb_noun form (list_tasks, get_task, start_session, run_in_sandbox, end_session, diagnose_ci). No mixing of conventions or vague bare verbs.

Tool Count5/5

Six tools is a tight, well-scoped set that covers task discovery, sandbox execution lifecycle, and CI diagnosis without redundancy.

Completeness4/5

The core workflow (find task, start sandbox, run commands, end sandbox, diagnose failures) is fully covered. Minor gaps exist: no session status/list tool and no explicit submit/evaluate step to confirm a solution, though the latter may be out of scope.

Maintenance

ActivityMaintained
ResponsivenessNo recent community issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables coding agents to perform file, search, patch, git, process, test, package, network, and system operations through 60 typed MCP tools with structured inputs/outputs, structured errors, and a full event journal, replacing terminal use with a typed machine API.
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables AI coding agents to retrieve repository context through stdio-based MCP tools, including ranked repo maps, context packs, and live symbol search.
    5 npm
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Enables AI agents to perform software engineering tasks inside an isolated, deterministic sandbox—exploring repositories, reproducing failures, applying patches, running tests, and verifying solutions against hidden suites through MCP tools.
    8
    MIT