ComputerPilot MCP

Supervised computer use and developer automation over the Model Context Protocol.

ComputerPilot MCP gives MCP-capable AI agents a local runtime for files, native document intelligence, terminal and process execution, Git, testing, code intelligence, browser automation, durable jobs, crash recovery, and multi-step workflows. On Windows it also exposes native screenshots, input, and semantic UI Automation.

It is designed for developers who want an agent to do real work on a machine without collapsing everything into one unstructured shell tool. Capabilities are exposed as typed MCP tools, platform-specific features are registered only when supported, long-running work can survive MCP restarts, and ambiguous mutations are never replayed blindly.

Security: this project intentionally has broad local capabilities. Treat access to the MCP endpoint as equivalent to granting a development agent access to the OS account running it. See SECURITY.md.

Why ComputerPilot MCP

• One supervised runtime for coding and computer use. Files, code search, LSP, Git, diagnostics, processes, browser automation, jobs, workflows, recovery, and system inspection share one MCP surface.

• Durable instead of request-bound. Long-running commands can use persistent jobs with idempotency keys, bounded concurrency, cancellation, incremental output, and restart survival.

• Recovery is explicit. Interrupted mutations can become uncertain and require evidence-based reconciliation or operator acknowledgement instead of blind replay.

• Cross-platform core, capability-gated edges. Windows, Linux, and macOS share the portable core; native Windows desktop/UI Automation is only registered on supported hosts.

• Verified remote tunnel bootstrap. Tunnel mode downloads the official OpenAI Secure MCP Tunnel runtime into local state and verifies release metadata, checksums, and binary version before publication.

• Release evidence is reproducible. CI, platform validation, deterministic packaging, checksums, Doctor checks, benchmarks, and release validation live in the repository.

Related MCP server: JARVIS-MCP

Quick Start

The fastest way to try the project without tunnel credentials is loopback-only local HTTP mode.

Windows

git clone https://github.com/dibbed/computerpilot-mcp.git
Set-Location computerpilot-mcp
$env:MCP_START_MODE = "local-http"
.\START_MCP.bat

Linux / macOS

git clone https://github.com/dibbed/computerpilot-mcp.git
cd computerpilot-mcp
MCP_START_MODE=local-http ./start_mcp.sh

The launchers create or reuse .venv, validate Python and dependencies, run cached/full startup checks, and start the supervisor.

• MCP endpoint: http://127.0.0.1:8765/mcp

• Local control panel: http://127.0.0.1:8766/

For Secure Tunnel mode, browser setup, release archives, and platform-specific notes, see Installation.

What You Can Do

See Tooling and workflows for the public tool model and practical examples.

Architecture

flowchart TD
    A[MCP client / AI agent] -->|Local HTTP| B[ComputerPilot MCP]
    A -->|OpenAI Secure MCP Tunnel| T[Managed tunnel runtime]
    T -->|local stdio| B

    B --> D[Typed tool domains]
    D --> F[Filesystem / project / LSP]
    D --> P[Terminal / process / system]
    D --> G[Git / testing]
    D --> R[Jobs / recovery / workflows]
    D --> O[Documents: Excel / DOCX / PDF]
    D --> U[Browser / desktop]

    B --> S[Supervisor + lifecycle]
    S --> H[Health / control panel]
    S --> L[.agent_state]
    R --> L

Key runtime invariants:

• local HTTP and the control panel bind to loopback by default;

• filesystem mutations use scoped locks and guarded edit primitives;

• durable jobs keep persistent state and disk-backed output;

• uncertain side effects are not automatically replayed;

• Windows process ownership uses Job Objects, while POSIX uses sessions/process groups;

• browser sessions use isolated contexts with bounded pool/session lifetimes;

• document mutations use scoped same-file locks, optional SHA-256 preconditions, staged validation, backups, atomic publication, and durable file-hash recovery checkpoints;

• generated runtime state lives under .agent_state/ and is excluded from Git.

A deeper component map is in Architecture.

Platform Support

The release workflow builds five target archives from the same tracked source. Platform claims and limitations are documented in Platform support.

Practical Agent Workflows

Inspect → change → verify

code_context
    ↓
apply_patch / rename_symbol / replace_exact
    ↓
affected_tests
    ↓
verify_changes

Run a long task without tying it to one MCP request

submit_job
    ↓
job_wait
    ↓
job_output
    ↓
cancel_job (when needed)

Recover an interrupted mutation

list_uncertain_operations
    ↓
inspect_uncertain_operation
    ↓
reconcile_operation
    ↓
acknowledge_uncertain_operation (only when evidence cannot resolve it)

Durable multi-step automation

workflow_plan
    ↓
workflow_start
    ↓
workflow_execute
    ↓
workflow_status / workflow_operations
    ↓
workflow_reconcile when a side effect is uncertain

Built-in workflows include implement_and_verify, safe_git_commit, and prepare_release. They do not push, tag, or deploy automatically.

Tool Profiles

The default full profile preserves the complete catalog. Smaller profiles reduce the active domain set for specialized agents:

minimal, coding, git, testing, documents, desktop, browser, operations, full.

discover_tool_domains reports the current profile and platform capabilities. recommend_tools ranks tools that are actually registered in the active profile and enriches recognized tools with shared execution-route metadata. The read-only execution_candidates and execution_recommend diagnostics use that same registered catalog; metadata alone cannot make an unavailable tool routable.

Set a profile with:

MCP_TOOL_PROFILE=coding

Set the variable before startup and restart the runtime to change profiles. coding omits browser, desktop/UIA, and native document tools. Keep full for mixed coding, desktop, and document work; documents includes filesystem, native documents, recovery, and workflows. A smaller catalog saves serialized discovery bytes by removing capabilities; it does not establish a model-token or task-quality improvement.

Profiles filter registration. They do not grant execution permission or implicitly enable extensions; typed tools and external operator policy retain their existing execution checks.

Adaptive Execution Routing

The routing layer introduced in v0.7 is a decision layer, not a universal dispatcher. It receives a structured intent, generates candidates only from tools registered in the active MCP profile, filters platform/capability and safety constraints, and applies the versioned deterministic policy deterministic-v1.

Initial route classes are:

• native: filesystem, code/LSP, Git, process, system, Excel, DOCX, and PDF;

• semantic: browser and Windows UI Automation;

• raw.desktop, only when the request explicitly permits raw desktop fallback and the action is not destructive;

• visual.desktop, reserved for a later visual-grounding phase and unavailable in v0.8.

The public diagnostics are read-only:

execution_candidates
execution_recommend

There is intentionally no execution_run or hidden arbitrary-action dispatcher. Hard constraints run before ranking: unsupported platform/profile routes are excluded, stale semantic references never degrade to coordinates, destructive ambiguous semantic targets fail closed, destructive raw-desktop fallback is forbidden, and fidelity requirements such as Excel macro preservation cannot be overridden by ranking.

A workflow step may optionally persist a bounded execution_intent. Its selected route, router policy version, compact decision metadata, and bounded fallback history are stored before the side effect begins. If an operation becomes uncertain, the existing evidence/reconciliation path remains authoritative; the router does not blindly reroute and replay it.

Router configuration:

MCP_EXECUTION_ROUTER_ENABLED=1
MCP_EXECUTION_ROUTER_POLICY=deterministic-v1
MCP_EXECUTION_ROUTER_EXPLAIN=1

Runtime health and the loopback Control Panel expose bounded route distribution, fallback/no-route counts, native/semantic/raw usage, latency, and recent decisions without storing raw prompts, document contents, screenshots, credentials, or secrets.

Secure Tunnel Mode

Tunnel mode is the default launcher mode. It requires a control-plane API key. A profile can be supplied explicitly or resolved by the runtime configuration.

Recommended secret file:

.secrets/control_plane_api_key.txt

Or set the key in the environment:

$env:CONTROL_PLANE_API_KEY = "<your-key>"
.\START_MCP.bat

The project does not vendor tunnel-client or Cloudflared executables in Git or release archives. The managed updater downloads the matching official openai/tunnel-client runtime, validates the GitHub asset digest and upstream SHA256SUMS.txt, verifies the reported version, and publishes the selected runtime under .agent_state/tunnel-runtime/.

See Binary provenance.

Browser Automation

Browser tools are optional.

python -m pip install -r requirements-browser.txt
python -m playwright install chromium
python -m scripts.doctor --mode local-http --browser

Playwright sessions use isolated contexts. Compatible sessions can share a browser process, and idle sessions/pools are bounded by runtime budgets.

The semantic browser layer prefers accessible meaning over raw selectors or coordinates:

browser_open_page
    ↓
browser_snapshot / browser_query
    ↓
browser_click_semantic / browser_fill_semantic / browser_select
    ↓
browser_wait_for

Snapshot node refs are generation-scoped. Navigation, semantic mutations, or tab changes invalidate old refs, so stale targets fail closed instead of silently acting on a changed page. browser_extract returns bounded text, links, tables, form fields, or a selected subtree. Uploads require an explicit local file, and downloads are saved under controlled runtime storage with filename sanitization and a configured byte limit.

Native Document Intelligence

The portable documents domain works directly with files and does not launch Microsoft Office or another GUI.

Excel tools support .xlsx and .xlsm inspection, sheet/range reads, bounded search, formulas, tables, range writes/clears, and guarded sheet/table mutations. Macro-enabled workbooks are mutated only when the staged output preserves the VBA project bytes exactly; an .xlsx file that unexpectedly contains VBA is rejected instead of risking macro loss.

DOCX tools support structure/metadata inspection, bounded paragraph/table-cell reads and search, conservative text replacement, paragraph insertion, simple table-cell replacement, and simple document creation. .docm, tracked changes, comments, headers/footers, arbitrary WordprocessingML edits, and ambiguous mixed-format replacements are intentionally unsupported.

PDF tools support metadata/page inspection, bounded text extraction with inline or file-backed delivery, and deterministic creation from plain text or a documented Markdown subset. They do not provide arbitrary existing-PDF editing. Encrypted PDFs can be inspected for encryption state, but text extraction requires a decrypted input.

Document mutations share the recovery model: same-file writes are serialized, unrelated files can proceed independently, optional SHA-256 preconditions reject stale writes, output is staged and validated before atomic publication, backups are retained by the existing backup policy, and the expected final file hash is checkpointed durably before replacement.

Local State

Generated machine-local state is kept out of Git.

Never commit .agent_state/, .secrets/, .venv/, logs, local databases, or generated output.

Validation and Quality Gates

The repository's normal CI matrix runs on Windows, Linux, and macOS with Python 3.10 and 3.12. Each job installs development dependencies and runs:

• compileall;

• Ruff;

• mypy;

• MCP startup check;

• MCP health smoke;

• full pytest.

Release preparation also has:

• Platform Release Validation on Windows, Linux, and macOS with Python 3.12;

• Release Packaging Smoke that builds all five archives and validates SHA256SUMS.txt;

• exact-SHA publication gates for release commits.

Release-specific evidence is stored under docs/V*-VALIDATION.md. The current release evidence is v0.8.0 validation; earlier releases remain in the same directory and docs/README.md lists them.

Run the common checks locally:

python -m pytest
python -m ruff check .
python -m mypy core tools scripts tests
python -m scripts.health_check --json

See Contributing for the full development workflow.

Release Artifacts

Published releases contain deterministic source/runtime-controller archives for:

• Windows amd64

• Linux amd64

• Linux arm64

• macOS amd64

• macOS arm64

• SHA256SUMS.txt

Each archive includes RELEASE-MANIFEST.json. Packaging is built from tracked Git blobs and fails if forbidden runtime, secret, virtualenv, test-cache, or generated-output paths are tracked.

Browse the latest release.

Documentation

Historical release notes and validation records remain under docs/. Internal local planning directories are ignored by Git and are not part of the public documentation surface.

Known Limits

• Native desktop screenshot/input and semantic UI Automation are Windows-only.

• Linux and macOS support is still labeled preview.

• Linux arm64 and macOS amd64 are package targets without a dedicated native hosted-runner claim in v0.7.1 / v0.8.0.

• Browser automation requires Playwright and installed browser runtimes.

• Excel document tools support .xlsx and .xlsm; macro mutation is accepted only when VBA bytes can be verified unchanged.

• DOCX mutation is intentionally conservative: .docm, tracked changes, comments, headers/footers, arbitrary XML edits, and ambiguous mixed-format replacements are not supported.

• PDF creation uses built-in PDF fonts and a limited Markdown subset; complex-script shaping and arbitrary editing of existing PDFs are not supported.

• LSP tools require a trusted pyright-langserver --stdio command already available on PATH; the MCP does not install it implicitly.

• The local HTTP endpoint and control panel are intended for loopback use, not direct exposure to untrusted networks.

• Built-in workflows deliberately avoid blind push/tag/deploy behavior.

Contributing

Contributions should be made on focused branches, validated locally, and merged through pull requests with CI passing.

Read CONTRIBUTING.md before changing platform, recovery, release, or tunnel behavior.

License

Apache License 2.0. See LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md.

Author: Ali Khalili