Provides integration with Ollama vision models to plan and execute computer actions based on screen observations.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@computeruse click the 'Submit' button on the active window"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
ComputerUse is a local desktop-control agent for Ollama vision models. It captures the screen, asks a selected local model for exactly one structured action, executes that small action through local mouse and keyboard tools, verifies the result with a fresh screenshot, and repeats until the task is done.
It is built for the daily computer-use workflow: observe, plan, act, verify.
Watch the demo: ComputerUse local AI desktop agent
Natural-language task
-> Tauri + React desktop runner
-> Python worker over newline-delimited JSON
-> screenshot + perception
-> Ollama vision planner
-> Pydantic action validation
-> local mouse/keyboard executor
-> verification screenshot
-> session state, timing events, and UI updatesRelated MCP server: openowl
Polished desktop task runner with model selection, screenshot preview, timeline, status, history, and debug timings.
CLI path for development, dry runs, model listing, screenshots, and automation.
Local Ollama integration with known vision-capable models ranked first.
Strict one-action-at-a-time planner contract backed by Pydantic validation.
Screenshot capture with mss, plus planner overlays and optional UI Automation element collection.
Local mouse, keyboard, scroll, drag, wait, screenshot, done, and fail action handlers.
Pause, resume, stop, dry-run, and max-step controls.
Stdio MCP server for external agents that want safe observe -> execute -> verify computer-use tools.
Local-only runtime files for the active session, screenshots, timing logs, and run history.
The current local runtime history has been read from data/computeruse.sqlite3 and logs/debug_timing.jsonl. These numbers are a real desktop-run snapshot from timing records dated 2026-07-05 UTC, not synthetic lab benchmarks.
Metric | Value |
SQLite sessions | 5 |
Completed sessions | 3 |
Cancelled sessions | 1 |
Running session records | 1 |
Tracked SQLite steps | 58 |
Successful tool steps | 56 / 58, 96.6% |
Valid timing log records | 108 |
Timing log models |
|
Most recorded actions were UI-targeted interactions: click_element was the dominant action, followed by type_text, press, done, hotkey, click_target, move, and double_click.
Phase | Samples | Average | P50 | P95 | Notes |
Screenshot capture | 107 | 16.3 ms | 16.0 ms | 25.0 ms | Comfortably under the 100 ms target. |
Screenshot encode | 107 | 83.7 ms | 88.0 ms | 132.0 ms | Usually under the 150 ms target; max observed was 171 ms. |
Planner grid overlay | 107 | 141.0 ms | 150.0 ms | 185.0 ms | Extra cost for coordinate rulers and element markers. |
UI perception | 107 | 2280.8 ms | 2052.0 ms | 4000.0 ms | Largest local overhead; UIA/perception is the main optimization target. |
Ollama/model call | 107 | 9757.1 ms | 7579.0 ms | 23694.0 ms | Dominant end-to-end cost, as expected. |
Tool execution | 107 | 140.1 ms | 5.0 ms | 188.0 ms | Includes one explicit wait outlier at about 5 seconds. |
Verification capture | 107 | 80.0 ms | 87.0 ms | 128.0 ms | Post-action screenshot verification. |
Metrics collection | 108 | 53.0 ms | 51.0 ms | 68.0 ms | CPU/RAM/GPU sampling overhead. |
Derived loop overhead from the same timing log:
Aggregate | Average | P50 | P95 | Interpretation |
Core non-LLM overhead, excluding perception and settle delay | 370.1 ms | 250.0 ms | 568.0 ms | Includes screenshot capture/encode, execution, verification, and metrics. |
Core non-LLM overhead, excluding perception, settle delay, and wait actions | 239.7 ms | 247.0 ms | 337.0 ms | Closer to normal click/type/keypress loop overhead. |
Non-LLM overhead with perception, excluding settle delay | 2769.5 ms | 2419.0 ms | 4569.0 ms | Shows the cost of UI perception on top of screenshot/execute work. |
Non-LLM overhead with perception and settle delay | 3763.0 ms | 3562.5 ms | 5575.0 ms | Reflects the default post-action settle delay for mutating actions. |
session_write_ms is not present in the current timing records, so JSON/session-write overhead is not benchmarked in this snapshot.
Windows desktop session.
Python 3.10 or newer. Python 3.11+ is recommended.
Ollama running locally at http://127.0.0.1:11434.
At least one installed Ollama vision model.
Node.js and npm for the React frontend.
Rust and Cargo for the Tauri desktop shell.
cd E:\ComputerUse
py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e .
npm install --prefix apps\desktopStart the desktop app:
npm run desktop:devThe Tauri app starts the Python worker with:
python -m computeruse.workerFor the smoothest local setup, launch the desktop app from a shell where the intended Python environment is already active.
The first screen is the task runner, not a landing page.
Panel | What it does |
Task runner | Enter a natural-language task, choose dry-run mode, set max steps, and run/pause/resume/stop. |
Model selector | Lists installed Ollama models with known vision models ranked first. |
Screenshot preview | Shows the latest observe or verification capture. |
Last action | Displays the validated JSON action returned by the model. |
Step timeline | Shows action, thought, confidence, result, and inline failures. |
Debug timing | Shows per-phase loop timings when enabled. |
History | Lists recent local tracked sessions and step summaries. |
Run a task like:
Open Chrome, go to YouTube, search for TWICE, and open the first video.ComputerUse will keep taking one small action at a time until the model returns done, returns fail, the task is cancelled, or the max-step limit is reached.
List installed Ollama models:
computeruse modelsCapture the current screen:
computeruse screenshotRun a task:
computeruse run "Open Chrome, go to YouTube, search for TWICE, and open the first video" --model llava:latestRun without executing mouse or keyboard events:
computeruse run "Open Chrome and go to YouTube" --model llava:latest --dry-run --debug-timingStart the MCP server:
computeruse mcpEquivalent module entrypoint:
python -m computeruse.mcp_serverComputerUse queries Ollama through the local HTTP API and displays every installed model. Known vision-capable families are ranked first when present:
LLaVA
BakLLaVA
Moondream
MiniCPM-V
Qwen VL and Qwen2.5-VL
Gemma vision-capable variants
Example model install:
ollama pull llava:latestThe model may return exactly one JSON object per turn. No prose, no Markdown, no multiple actions.
{
"thought": "Use the address bar to navigate directly.",
"action": "hotkey",
"args": {
"keys": ["ctrl", "l"]
},
"done": false,
"confidence": 0.92
}Supported action names:
screenshot
click
double_click
right_click
move
scroll
drag
click_element
double_click_element
move_element
click_target
move_target
type_text
press
hotkey
wait
done
failCoordinate actions use absolute screenshot-pixel coordinates from the latest capture. Element and target actions use the most recent UI Automation/perception data where available.
Use click_element for normal controls: buttons, links, tabs, menus, text fields, checkboxes, browser controls, web results, and app navigation. Use double_click_element only for desktop-style items that conventionally require double-click to open: desktop shortcuts/icons, files, folders, Explorer rows, and open/save dialog file rows. Do not double-click web links, buttons, tabs, YouTube thumbnails, checkboxes, text fields, or menu commands. scroll accepts clicks; negative values scroll down and positive values scroll up. drag performs one atomic click-hold-drag-release with start_x, start_y, end_x, end_y, and optional duration_ms.
The GUI talks to the Python worker with newline-delimited JSON over a managed process. The protocol is intentionally small and explicit.
GUI commands:
{"type":"list_models"}
{"type":"start_task","task":"Open Chrome and go to YouTube","model":"llava:latest","dry_run":false,"max_steps":50}
{"type":"pause"}
{"type":"resume"}
{"type":"stop"}
{"type":"take_screenshot"}
{"type":"list_history","limit":50}
{"type":"get_history_session","session_id":"..."}Worker events:
{"type":"models","models":[{"name":"llava:latest","vision":true}]}
{"type":"session_started","session_id":"..."}
{"type":"screenshot","path":"E:\\ComputerUse\\screenshots\\current.png","width":1920,"height":1080}
{"type":"step_started","step_index":3}
{"type":"model_action","action":{"action":"click","args":{"x":500,"y":300},"confidence":0.82}}
{"type":"tool_result","ok":true,"message":"clicked"}
{"type":"timing","step_index":3,"capture_ms":42,"encode_ms":65,"ollama_ms":1840,"execute_ms":7,"session_write_ms":3}
{"type":"session_done","summary":"The requested page is open."}
{"type":"session_failed","reason":"The browser did not load after repeated attempts."}ComputerUse includes a stdio MCP server so other agents can use local screen observation and one-step execution safely.
Generic MCP configuration:
{
"mcpServers": {
"computeruse": {
"command": "python",
"args": ["-m", "computeruse.mcp_server"],
"cwd": "E:\\ComputerUse"
}
}
}If the MCP client does not inherit your activated shell, point directly at the virtual environment:
{
"mcpServers": {
"computeruse": {
"command": "E:\\ComputerUse\\.venv\\Scripts\\python.exe",
"args": ["-m", "computeruse.mcp_server"],
"cwd": "E:\\ComputerUse"
}
}
}Codex TOML example:
[mcp_servers.computeruse]
command = 'E:\ComputerUse\.venv\Scripts\python.exe'
args = ['-m', 'computeruse.mcp_server']
cwd = 'E:\ComputerUse'
startup_timeout_sec = 120Recommended MCP workflow:
Call computeruse_help.
Call computeruse_start_session with the user task.
Call computeruse_observe.
Call computeruse_execute_step with exactly one validated action.
Inspect the verification observation.
Repeat observe/execute until complete.
Call computeruse_finish_session.
These files are local runtime state and are ignored by Git:
Path | Purpose |
| Current active task state. |
| Latest raw screenshot. |
| Planner screenshot with rulers and element markers. |
| Timing and resource metrics. |
| Local run history and step summaries. |
The runtime does not need cloud storage. Keep these artifacts private unless you deliberately redact and share them.
ComputerUse is designed around narrow, validated actions:
The planner cannot run shell commands or arbitrary Python.
The runtime validates every model action before execution.
Dry-run mode validates model actions without moving the mouse or typing.
Pause and stop are available from the desktop UI.
Passwords, payment details, tokens, destructive changes, purchases, posts, and security-setting changes require explicit user intent before they should be executed.
Web pages are treated as untrusted input.
apps/
desktop/ Tauri 2 + React + TypeScript app
computeruse/
agent/ loop, prompts, Ollama client, session and history logic
schemas/ Pydantic action and session models
tools/ screenshots, screen metadata, mouse, keyboard, windows, executor
cli.py Typer CLI
worker.py newline-delimited JSON worker for the GUI
mcp_server.py stdio MCP server
data/ local SQLite history, ignored by Git
logs/ debug timing logs, ignored by Git
screenshots/ current/planner captures, ignored by Git
sessions/ active session JSON, ignored by GitFrontend typecheck:
npm run desktop:typecheckFrontend/Tauri build:
npm run desktop:buildPython package install in editable mode:
python -m pip install -e .Symptom | Check |
No models appear | Confirm Ollama is running and |
Worker fails to start | Activate |
Screenshot does not update | Check that the app has access to the active Windows desktop session. |
Actions land in the wrong place | Use the latest screenshot and verify DPI/monitor coordinates; avoid stale screenshots. |
Tauri cannot find Rust tooling | Install Rust/Cargo and restart the shell. |
ComputerUse is an MVP local automation tool. Treat real desktop control as powerful and potentially disruptive: start with dry runs, keep tasks specific, and verify the screen after each action.
Execute exactly one ComputerUse action using the latest observation for coordinate and element mapping. Supports dry_run validation and optional verification screenshot. Call observe or inspect verification before the next step.
| Name | Required | Description | Default |
|---|---|---|---|
| action | Yes | ||
| verify | No | Capture a verification screenshot after execution. | |
| dry_run | No | Validate without sending mouse or keyboard events. | |
| settle_ms | No | Optional wait before verification capture. | |
| image_mode | No | planner | |
| include_elements | No | Collect UIA elements in the verification observation. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full transparency burden. It discloses single-action execution, dry_run validation without sending events, and optional verification screenshots. It doesn't detail failure behavior or output, but it significantly clarifies side effects and validation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences with no filler. The core purpose is front-loaded, followed by important validation/verification traits and a sequencing instruction. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the observe-act-observe cycle, dry_run, and verification well. However, it does not mention what the tool returns, how failures are reported, or the terminal significance of done/fail actions. Given no annotations and no output schema, these are meaningful gaps for a complex tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 67%, and the schema already documents dry_run, verify, settle_ms, include_elements, and per-action args in detail. The description only highlights dry_run and verification screenshot without adding meaning beyond the schema, so it meets but doesn't exceed the baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: it executes exactly one ComputerUse action using the latest observation. This distinguishes it from observation and session-management siblings, though it doesn't explicitly name an alternative it is not.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear operational context: use the latest observation for coordinate and element mapping, and call observe or inspect verification before the next step. It lacks explicit when-not-to-use guidance or alternative routing, but the workflow is evident.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Finish the active MCP-tracked session as done, failed, or cancelled with a summary.
| Name | Required | Description | Default |
|---|---|---|---|
| status | Yes | ||
| summary | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It states the terminal action and status values but does not disclose side effects (whether the session is permanently closed, whether subsequent execute_step calls fail, or what happens if no active session exists) or the response format.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, front-loaded sentence communicates the tool's purpose and required outcome in one pass with no filler. This is appropriately concise for a two-parameter terminal action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple 2-parameter tool, the description makes the basic call signature usable, but with no annotations, no output schema, and no side-effect disclosure, the definition is not fully complete. An agent can invoke it but may not know the consequences of finishing a session or how to handle errors.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but the description adds little beyond the schema: 'done, failed, or cancelled' merely restates the enum, and 'with a summary' gives no guidance on what the summary should contain or how to choose a status. The enum values are self-explanatory but not contextualized.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Finish') and resource ('active MCP-tracked session'), and names the three possible terminal statuses plus summary. This clearly distinguishes it from siblings like computeruse_start_session, computeruse_observe, and computeruse_execute_step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'active MCP-tracked session' implies it should be used when a session is open and needs to be ended, but no explicit when/when-not guidance or alternative routing is given. An agent must infer the precondition and that it should not be called before starting or after finishing a session.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Get one tracked run with all recorded step summaries from SQLite history. This is read-only.
| Name | Required | Description | Default |
|---|---|---|---|
| session_id | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the behavioral disclosure burden. It explicitly says 'This is read-only' and mentions 'all recorded step summaries', providing useful trait information ape. However, it does not disclose behavior for missing session IDs, error handling, or output format details.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, with the core purpose front-loaded and the read-only note immediately after. There is no fluff, and every phrase earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter getter with no output schema or annotations, the description covers what is retrieved ('all recorded step summaries'), the storage source (SQLite history), and the safety profile (read-only). It lacks explicit error behavior or return structure, but those gaps are minor given the low complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description does not explain that session_id identifies the target tracked run, how to discover valid IDs, or any format expectations. The parameter name is self-explanatory, but the description adds no meaning beyond the schema's bare type declaration.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description states a specific verb ('Get') and resource ('one tracked run with all recorded step summaries from SQLite history'). It clearly conveys the tool's scope and distinguishes it from list_history by focusing on a single run with step summaries.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies this tool is for retrieving a specific run's step summaries, which contrasts with sibling list_history, but it does not explicitly state when to use this versus alternatives or mention any exclusions. Usage context is implied rather than direct.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Read this first. Explains how external AI agents should use ComputerUse MCP tools, including observe -> one action -> verify workflow, coordinate rules, safety rules, and action schema.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly indicates this is an informational/help tool that explains usage, implying no side effects or mutations. It does not explicitly state that it is read-only or that it returns static guidance, but the nature of a help tool is well conveyed, and the description adds value beyond the tool name.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact sentence that front-loads the most important directive ('Read this first') and then enumerates the key contents (workflow, rules, schema). Every word earns its place with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a help tool with no parameters and no output schema, the description is complete. It tells the agent what the tool explains and implies that it should be used as an introduction. The list of covered topics is sufficient for an agent to understand the tool's role and decide to invoke it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the description does not need to add parameter-level meaning. The baseline for zero-parameter tools is 4, and the description appropriately focuses on the tool's purpose rather than parameter details, which is sufficient.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: it explains how to use ComputerUse MCP tools, including the workflow, rules, and schema. It uses a specific verb ('explains') and names the resource (ComputerUse MCP tools), and it is clearly distinct from the sibling tools which are operational tools (start, observe, execute) rather than a meta-guide.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The opening phrase 'Read this first' provides an explicit usage directive: consult this tool before using others. The description also states what the tool covers (workflow, coordinate rules, safety rules, action schema), giving context on when it is relevant. It does not explicitly state when not to use it or alternatives, but the 'first' instruction implies it is a prerequisite.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List recent tracked ComputerUse runs from SQLite history. This is read-only and does not replay actions.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must carry the behavioral burden. It does so by stating the operation is read-only and does not replay actions, which is a critical safe behavior for an agent to know. It does not disclose ordering or pagination details, but 'recent' and the schema-provided limit add reasonable context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences, front-loaded with the core purpose and immediately followed by the crucial safety qualifier. Every sentence earns its place with no redundant phrasing.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter read-only listing tool, the description covers the source, scope, and safety posture. It does not describe the return format, but the absence of an output schema makes that a minor gap for a straightforward history-listing operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description does not explain the meaning of the 'limit' parameter beyond what the schema's default and constraints imply. The description provides no additional semantics for how many runs are listed or how the limit interacts with recency.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('List'), names the resource ('recent tracked ComputerUse runs'), and identifies the data source ('SQLite history'). The phrase 'does not replay actions' explicitly distinguishes it from execution-oriented siblings like computeruse_execute_step and computeruse_start_session.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly signals a read-only, history-viewing use case and explicitly warns that it does not replay actions, so an agent knows not to invoke it for execution. It does not explicitly name alternatives such as computeruse_get_history_session for session-specific history, but the context is strong enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Capture the current screen and optionally visible UI Automation elements. Returns structured screen dimensions, element IDs, and an MCP image. Use this before each click/type decision.
| Name | Required | Description | Default |
|---|---|---|---|
| image_mode | No | planner returns the ruler/element-overlay image; raw returns the plain screenshot; none returns JSON only. | planner |
| monitor_index | No | ||
| include_elements | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It transparently discloses that the tool captures the screen, optionally extracts UI Automation elements, and returns dimensions, element IDs, and an MCP image. This is sufficient for a read-only observation tool with no hidden mutating behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no wasted words. The action and output are front-loaded, and the usage directive is placed at the end as a natural call to action. Every sentence contributes value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the core action, output shape, and usage context, but there is no output schema and the monitor_index and image_mode behaviors are not explained beyond the input schema. An agent could call the tool with defaults, but would need to inspect the schema for non-default scenarios.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 33%, covering only image_mode. The description partially adds semantics for include_elements via 'optionally visible UI Automation elements', but it never explains image_mode outcomes or monitor_index selection. With low schema coverage, the description needed to compensate more than it does.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Capture') and names the resource ('current screen', 'visible UI Automation elements'), and explicitly frames the tool as the observation step before click/type decisions. This clearly distinguishes it from siblings like computeruse_execute_step and computeruse_start_session.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Use this before each click/type decision' is an explicit usage directive that tells the agent when to call the tool. There is no alternative observation sibling, so the lack of a 'when not to use' clause is acceptable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Start a tracked ComputerUse session for an external agent task. This creates/overwrites sessions/active_session.json and adds a run row to SQLite history.
| Name | Required | Description | Default |
|---|---|---|---|
| task | Yes | Natural-language user task being performed. | |
| agent_name | No | Optional external agent name for history tracking. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does well by explicitly disclosing side effects: creating/overwriting sessions/active_session.json and adding a history row to SQLite. This tells the agent the operation is persistent and state-changing, which is essential for a session-start tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact: two sentences with no filler. The main purpose is stated first, followed by the key side effects, making it easy for an agent to quickly grasp the tool's behavior.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a session-start tool with no output schema, the description covers purpose and critical side effects. It could add a note that this should be called before other session tools, but the tool name and sibling context make that reasonably inferable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so both 'task' and 'agent_name' are already documented in the input schema. The description does not add extra semantics beyond restating that the session is for an external agent task, which is sufficient for a baseline score.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action ('Start') on a specific resource ('tracked ComputerUse session') and explains its core effect. It is clearly distinct from sibling tools like computeruse_finish_session or computeruse_execute_step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description makes the usage context clear: this is for initiating a tracked session for an external agent task. It does not explicitly contrast with alternatives, but the start-vs-finish relationship with siblings is strongly implied by the tool name and wording.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0computeruse_execute_stepcomputeruse_finish_sessioncomputeruse_get_history_sessioncomputeruse_helpcomputeruse_list_historycomputeruse_observecomputeruse_start_sessionScored across 7 tools
Each tool maps to a distinct phase of the ComputerUse workflow: help, session lifecycle, observation, action execution, and history retrieval. Even list_history and get_history_session are clearly separated as list-versus-detail operations.
All tools share a consistent computeruse_ prefix and use snake_case. Most follow a verb_noun pattern like start_session and execute_step, though observe and help are slightly less uniform.
Seven tools is a well-scoped set for a computer-use agent server. Each tool supports a necessary part of the observe-act-verify workflow without redundancy.
The tool surface covers the full agent lifecycle: onboarding help, starting a session, observing state, executing steps, finishing sessions, and reviewing history. There are no obvious dead ends in the core workflow.
Remote MCP server for supportsheep: run AI interviews and manage support content for your blog.
A comprehensive Model Context Protocol (MCP) server that enables AI assistants to interact with yo…
OCR, transcription, file extraction, and image generation for AI agents via MCP.
MCP server for visual regression testing: triage a PR's UI diffs from your coding agent.