devenv-mcp
Provides isolated Docker-based sandbox sessions for running shell commands against task repositories, with network disabled, CPU/memory/PID limits, and automatic cleanup.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@devenv-mcpstart task 004-quadratic-dedupe and run the tests"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
devenv-mcp
A Model Context Protocol server that gives an AI agent a safe place to do software-engineering work. The agent can browse tasks, open a sandboxed copy of a broken repository, run commands in it, and diagnose CI failures, all through typed MCP tools.
It is the agent-facing half of rl-env-benchmark and reads the same task format, and it calls ci-doctor for CI log diagnosis.
Tools
Tool | What it does |
| List tasks with id, title, category and difficulty |
| Full prompt for one task |
| Create a fresh sandbox holding the task's starting repo; returns |
| Run a shell command in the session's repo root; returns exit code and output |
| Destroy the sandbox |
| Classify failures in a CI log or JUnit XML and suggest a next step |
| Run the hidden grader. Off by default, see below |
Every tool declares a zod input schema, returns structuredContent alongside
readable text, and reports failures in-band with isError, so the model can
read what went wrong and recover instead of the call failing outright.
Related MCP server: nc-tools-mcp
Design decisions
Isolation by default. Sessions run in Docker with
--network none, 1 CPU, 1 GB of memory and a PID limit. Alocalmode exists for task authoring, but it runs commands on the host and is clearly labelled as not isolated.Grading is opt-in. If an agent under evaluation can call the grader, it can iterate against hidden tests until they pass, which measures persistence rather than skill.
run_graderis only registered whenDEVENV_ALLOW_GRADING=1, for task authors and human reviewers.The grader never enters the workspace early. It is copied in only when grading runs, and a test asserts the agent cannot find it beforehand.
Bounded output. Command output is collected in a head-plus-tail buffer (the end of a log is usually where the error is), so a runaway process cannot exhaust memory or flood the model's context.
Real timeouts. Commands start in their own process group and the whole group is killed on timeout. Killing only
shwould leave grandchildren such aspytestholding the pipes open, and the call would hang.Capped, cleaned-up sessions. A session limit prevents runaway container creation, and all sandboxes are destroyed on SIGINT, SIGTERM or client disconnect.
stdout is the protocol. All logging goes to stderr.
Install
git clone https://github.com/TyllerDaniel/devenv-mcp
cd devenv-mcp && npm install && npm run buildUse with Claude Desktop or Claude Code
{
"mcpServers": {
"devenv": {
"command": "node",
"args": ["/path/to/devenv-mcp/dist/index.js"],
"env": {
"DEVENV_TASKS_DIR": "/path/to/rl-env-benchmark/tasks",
"DEVENV_SANDBOX": "docker"
}
}
}
}With Claude Code: claude mcp add devenv -e DEVENV_TASKS_DIR=/path/to/tasks -- node /path/to/devenv-mcp/dist/index.js
Configuration
Variable | Default | Meaning |
|
| Directory of tasks ( |
|
| Base images, |
|
|
|
|
|
|
|
| Concurrent sandbox limit |
|
| Path to the ci-doctor binary |
Invalid values fail fast at startup with a message naming the variable.
Example session
From npm run build and a client connected over stdio, against the
rl-env-benchmark tasks with grading enabled:
> start_session {"task_id": "004-quadratic-dedupe"}
Started session 72721590 for 004-quadratic-dedupe (local sandbox).
> run_in_sandbox {"session_id": "72721590", "command": "python3 -m pytest -q tests | tail -1"}
[exit code 0]
1 passed in 0.01s
> run_grader {"session_id": "72721590"}
FAILED <- visible tests pass, but the hidden 300k-event test times out
> run_in_sandbox {..., "command": "sed -i 's/seen = \\[\\]/seen = set()/; s/seen.append/seen.add/' ingest/dedupe.py"}
> run_grader {"session_id": "72721590"}
PASSEDDevelopment
npm test # vitest: unit tests plus a real MCP client over an in-memory transport
npm run typecheck
npm run buildThe server tests connect an actual MCP Client to the server, so they cover
schemas, error handling and the full start, run, grade and end loop rather than
calling handlers directly.
License
MIT
Available Tools
6 toolsdiagnose_ciDiagnose CI failureARead-only
Classify the failures in a CI log or JUnit XML report (flaky test, dependency conflict, timeout, compile error, assertion, ...) and suggest a next step. Uses the ci-doctor CLI.
| Name | Required | Description | Default |
|---|---|---|---|
| log | Yes | Raw CI log text or JUnit XML | |
| input_format | No | auto |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already declare readOnlyHint=true and openWorldHint=false, so safety and locality are covered structurally. The description adds one piece of genuine context - it is backed by the ci-doctor CLI - but says nothing about latency, log size limits, or the shape of the classification/suggestion it returns despite there being no output schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with the capability and its output front-loaded and the implementation detail (ci-doctor CLI) trailing. No filler or restatement of the tool name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter read-only diagnostic with no output schema, the description covers what goes in and roughly what comes back (a classification plus a suggested next step). The main gap is the 'input_format' semantics and any handling limits for large logs.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: the required 'log' parameter is documented, but 'input_format' has only an enum and default with no explanation in either the schema or the description. The description's phrase 'CI log or JUnit XML' hints at the text vs junit distinction but never says when to override the 'auto' default, so the one undocumented parameter stays ambiguous.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb (classify) and resource (failures in a CI log or JUnit XML report), enumerates concrete failure categories, and adds a second capability (suggest a next step). Nothing in the sibling set (list_tasks, get_task, start_session, run_in_sandbox, end_session) overlaps, so an agent can route to it unambiguously.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied by the phrase 'a CI log or JUnit XML report' - there is no explicit statement of when to call this versus other tools, no prerequisites, and no exclusion such as when a log is too large or already parsed. Adequate but leaves the agent to infer the triggering condition.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
end_sessionEnd sessionADestructive
Destroy a session's sandbox and free its resources.
| Name | Required | Description | Default |
|---|---|---|---|
| session_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare destructiveHint=true, idempotentHint=false, and openWorldHint=false, so the safety profile is structured. The description adds useful specificity by stating that the sandbox is destroyed and resources are freed, which goes beyond the generic destructive flag. It still does not explain error behavior, whether repeated calls fail, or what happens to associated tasks.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single front-loaded sentence with no filler. Every phrase contributes to the meaning: destroy action, sandbox target, and resource release.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter lifecycle tool with no output schema and strong annotations, the description covers the core effect adequately. The main remaining gaps are where session_id comes from and any precondition around active tasks, but these are relatively minor for this tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for the single required session_id parameter, so the description must compensate but does not. 'A session's sandbox' only indirectly implies that session_id identifies the target session; no format, source, or validity constraints are provided.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb, 'Destroy,' and names the exact resource affected: a session's sandbox and its resources. It is clearly distinct from lifecycle siblings such as start_session and run_in_sandbox.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied by the lifecycle framing: an agent can infer this should be called when a session is no longer needed. It does not explicitly say when to use this versus leaving a session running, nor does it state preconditions such as whether active tasks must be stopped first.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_taskGet taskARead-only
Get the full prompt and metadata for one task.
| Name | Required | Description | Default |
|---|---|---|---|
| task_id | Yes | Task id from list_tasks |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true and openWorldHint=false already tell the agent this is a safe, closed-world read. The description adds the useful detail that the response includes the full prompt text plus metadata (helpful since there is no output schema), but says nothing about error behavior for an unknown task_id.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One short sentence with the resource and the returned content front-loaded; every word earns its place and there is no filler or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter read tool with full schema coverage and annotations covering safety, the description supplies the missing piece: what the call returns (prompt + metadata). Only minor gaps remain around failure modes for an invalid id.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with a single required parameter already documented as coming from list_tasks, so the schema carries the load. The description adds no format or validation detail beyond what the schema states; baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Get ... one task') and clarifies the payload ('full prompt and metadata'), which distinguishes it from the sibling list_tasks by scope (single task). It stops short of naming that sibling explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: 'one task' suggests it is the follow-up to a listing call, but the description gives no explicit when-to-use, prerequisites, or alternatives. The routing hint lives in the parameter schema ('Task id from list_tasks') rather than the description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_tasksList tasksARead-only
List every available task with its id, title, category and difficulty.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The readOnlyHint and openWorldHint=false annotations already establish that this is a safe, closed-world read. The description adds the return field set, which is genuinely useful with no output schema, but says nothing about ordering, result size, or whether tasks are global or session-scoped.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence that names the action, the scope, and the returned fields with no redundant wording.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a trivial zero-parameter read with annotations and no output schema, the description covers enough to call it correctly. The remaining gap is minor: no mention of result ordering or whether the list is workspace-scoped.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is no schema semantics to add beyond confirming that the listing is unconditional and unfiltered. Baseline 4 applies for a no-parameter tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
"List every available task" gives a specific verb plus resource, and the description enumerates the returned fields (id, title, category, difficulty). It is clearly distinguishable from get_task by the word "every", though it never names that sibling explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: an agent can infer this is the discovery step before get_task or start_session. There is no explicit statement of when to prefer this over get_task, nor any exclusion about filtering or scope.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
run_in_sandboxRun command in sandboxA
Run a shell command from the repository root inside a session's sandbox. Returns the exit code and combined stdout/stderr, truncated in the middle if very long. No network access.
| Name | Required | Description | Default |
|---|---|---|---|
| command | Yes | Shell command, run with sh -c | |
| timeout_s | No | Kill the command after this many seconds | |
| session_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Adds genuine behavior beyond annotations: the working directory (repository root), that stdout and stderr are combined, that output is truncated in the middle when long, and that there is no network access (consistent with openWorldHint=false). It does not cover what happens on timeout expiry beyond the schema's kill semantics, which keeps it from a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the action and execution context, then return shape, then the network constraint. No redundant restatement of the title or name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the return-value burden well by naming exit code, combined stdout/stderr, and middle truncation. The main gap is the lifecycle relationship to start_session/end_session, which an agent would need to call this safely.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 67% and the description does not compensate for the undocumented session_id: it never says where the id comes from or whether the session must be active. The 'repository root' phrasing does usefully constrain how the command executes, but the meaning of the remaining parameters comes entirely from the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (run) plus resource (shell command) plus scope (from the repository root, inside a session's sandbox). None of the siblings (list_tasks, get_task, start_session, end_session, diagnose_ci) execute arbitrary commands, so an agent can distinguish it immediately without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: the mention of 'a session's sandbox' hints that a session must already exist, but no explicit prerequisite, no when-to-use framing, and no routing away from siblings like diagnose_ci is given. The agent must infer the workflow context itself.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
start_sessionStart sessionA
Create a fresh sandbox containing the task's starting repository. Returns a session_id for run_in_sandbox. Each session is independent; end it with end_session.
| Name | Required | Description | Default |
|---|---|---|---|
| task_id | Yes | Task id from list_tasks |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only supply openWorldHint=false, so the description carries most of the burden and does add real behavioral facts: the return value (session_id), the independence of each session, and the requirement to terminate via end_session. It stops short of saying what happens to an unclosed session or whether the sandbox has a time limit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences with zero filler; the core action and return value come first, and the lifecycle notes follow. Nothing could be removed without losing information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With one required parameter, no output schema, and thin annotations, the description covers what is needed: what is produced, what is returned, and how the session is closed. A single-parameter creation tool needs little more.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single task_id parameter is fully documented in the schema (100% coverage), and the description only implicitly links it to the starting repository via 'the task's starting repository'. Baseline 3 applies since the schema already does the work.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Create a fresh sandbox containing the task's starting repository') and immediately ties it to its siblings by naming run_in_sandbox and end_session, so the agent can place it in the workflow without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear lifecycle guidance: the returned session_id feeds run_in_sandbox and the session must be closed with end_session. It does not explicitly state when to create a new session versus reuse an existing one, but the independence statement covers the main decision.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
6 tool updates
v0.1.0- First observed
diagnose_ci - First observed
end_session - First observed
get_task - First observed
list_tasks - First observed
run_in_sandbox - First observed
start_session
TDQS
Scored across 6 tools
The task-browsing pair (list_tasks/get_task) and the sandbox lifecycle trio (start_session/run_in_sandbox/end_session) are clearly delineated by resource and action. diagnose_ci sits in a slightly separate domain (CI log analysis) but its description makes its purpose distinct from run_in_sandbox.
All names use snake_case verb_noun form (list_tasks, get_task, start_session, run_in_sandbox, end_session, diagnose_ci). No mixing of conventions or vague bare verbs.
Six tools is a tight, well-scoped set that covers task discovery, sandbox execution lifecycle, and CI diagnosis without redundancy.
The core workflow (find task, start sandbox, run commands, end sandbox, diagnose failures) is fully covered. Minor gaps exist: no session status/list tool and no explicit submit/evaluate step to confirm a solution, though the latter may be out of scope.
Maintenance
Related MCP Connectors
Your org's AI agents, tasks, runs, search, and brain files as MCP tools and resources.
MCP server for your apps' tools and custom tools, plus hosted AI agents and approval-gated workflows
Task management your AI agents can actually run. One line becomes a context-ready task over MCP.
Nifty's MCP server — exposes tasks, projects, messages, and files as tools for AI agents.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables coding agents to run project-specific checks, replays, simulations, and queries as MCP tools, providing ground-truth feedback on config edits instead of guessing.1MIT
- AlicenseNot gradedqualityBmaintenanceEnables coding agents to perform file, search, patch, git, process, test, package, network, and system operations through 60 typed MCP tools with structured inputs/outputs, structured errors, and a full event journal, replacing terminal use with a typed machine API.MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI coding agents to retrieve repository context through stdio-based MCP tools, including ranked repo maps, context packs, and live symbol search.5 npmMIT
- AlicenseAqualityBmaintenanceEnables AI agents to perform software engineering tasks inside an isolated, deterministic sandbox—exploring repositories, reproducing failures, applying patches, running tests, and verifying solutions against hidden suites through MCP tools.8MIT