computer-use-mcp

A framework-agnostic computer-use Model Context Protocol (MCP) server. It exposes core desktop operations — screen capture, mouse, keyboard, and file access — as standard MCP tools, so any MCP-compatible agent framework (Claude Desktop, LangChain, AutoGen, a custom client, …) can drive a computer without being tied to a specific agent runtime.

Built on @modelcontextprotocol/sdk, @cfworker/json-schema, and zod, written in TypeScript with full type definitions.

Features

1. Standard MCP interface — speaks MCP over stdio (initialize, tools/list, tools/call), decoupled from any agent framework.

2. Core computer operations as MCP tools — screenshot, mouse_click, mouse_double_click, mouse_move, mouse_drag, mouse_scroll, keyboard_type, key_press, read_file, write_file, list_directory.

3. Zod + @cfworker/json-schema — Zod is the single source of truth for input shapes and coercion; z.toJSONSchema() derives the wire JSON Schema, and @cfworker/json-schema validates incoming arguments before dispatch.

4. TypeScript — complete type definitions throughout (src/core/backend.ts, src/tools/base.ts, …).

5. Extensible — tools and the automation engine are both pluggable behind stable interfaces.

Related MCP server: ControlMCP

Requirements

• Node.js ≥ 18 (or Bun).

• Platform automation tools for screen/input (the default driver uses OS facilities):

  • Windows: PowerShell + .NET (built in).

  • Linux: scrot (screenshot) and xdotool (mouse/keyboard).

  • macOS: screencapture + AppleScript (osascript); cliclick for move/drag.

Install & run

bun install          # or: npm install
bun start            # or: npm start  (runs src/index.ts via tsx)

The server reads JSON-RPC over stdio and is meant to be launched by an MCP client. Example client config:

{
  "mcpServers": {
    "computer-use": {
      "command": "bun",
      "args": ["/path/to/computer-use/src/index.ts"]
    }
  }
}

Architecture

src/
  index.ts                 Entry point: wires the default backend to the server.
  server.ts                ComputerUseServer — MCP protocol handling (tools/list, tools/call).
  schema/json-schema.ts    Zod -> JSON Schema, and a @cfworker/json-schema validator factory.
  core/
    backend.ts             ComputerUseBackend — the operation contract (the extension seam).
    native-driver.ts       NativeDriver — low-level screen/input primitives.
    local-backend.ts       LocalComputerUseBackend — filesystem + injected NativeDriver.
    system-driver.ts       Default per-platform NativeDriver (Windows / Linux / macOS).
    pi-backend.ts          PiComputerUseBackend — bridges the pi-computer-use native helper.
  engines/
    pi-types.ts            Structural subset of the pi-computer-use backend contract.
    pi-helper.ts           PiHelperDriver — bridges pi-helper observe/act to NativeDriver.
    robotjs.ts             RobotJsDriver — bridges the robotjs package.
    playwright.ts          PlaywrightDriver — bridges a Playwright Page.
    index.ts               createAutomationEngine — selects an engine.
  tools/
    base.ts                ToolDefinition + ToolResult types.
    screenshot.ts, mouse.ts, keyboard.ts, file.ts
    window.ts              get_window_title — foreground window title + owning process.
    pi.ts                  pi_observe / pi_act / pi_list_apps / pi_list_roots / pi_get_frontmost.
    index.ts               defaultTools (the built-in tool set) + piTools.

Request flow for a tool call

1. The client sends tools/call with arguments.

2. The server validates arguments against the tool's JSON Schema using @cfworker/json-schema (defense-in-depth wire contract). Invalid input returns a clear isError message.

3. Zod parses/coerces the arguments into a typed object (applies defaults, int coercion, …).

4. The tool handler runs against the injected ComputerUseBackend.

Extending

Add a custom tool

Create a ToolDefinition and register it:

import { z } from "zod";
import type { ToolDefinition } from "./tools/base.ts";

const myTool: ToolDefinition<{ url: string }> = {
  name: "open_url",
  description: "Open a URL in the default browser.",
  inputSchema: z.object({ url: z.string().url() }),
  async handler({ url }, { backend }) {
    // ...use backend or any side-effect...
    return { content: [{ type: "text", text: `Opened ${url}` }] };
  },
};

// pass `tools: [...defaultTools, myTool]` to ComputerUseServer, or call server.register(myTool).

Swap the automation engine

The plugin ships a pluggable automation engine layer. Pick one via the COMPUTER_USE_ENGINE environment variable, or construct it programmatically with createAutomationEngine — no tool or server code changes:

COMPUTER_USE_ENGINE=pi-helper bun start    # use the pi-computer-use native helper
COMPUTER_USE_ENGINE=robotjs   bun start    # use robotjs for input

import { createAutomationEngine } from "./engines/index.ts";
import { ComputerUseServer } from "./server.ts";

// programmatic selection (e.g. to drive a Playwright page):
const { backend, tools } = await createAutomationEngine({
  engine: "playwright",
  playwrightPage: page, // a real Playwright Page
});
const server = new ComputerUseServer({ backend, tools });

Engines degrade gracefully: if an optional package (e.g. robotjs, @injaneity/pi-computer-use) is not installed, the engine throws a clear UnsupportedOperationError describing how to enable it.

Bridge to pi-computer-use

The pi-helper engine maps our operations onto the pi-computer-use ComputerUsePlatformBackend:

• Screen capture → observe({ includeImage: true }); mouse/keyboard → act(...) with coordinate targets and the baseline lookId.

• On top of the 11 built-ins it exposes the helper's richer semantic tools: pi_observe (screen + accessibility outline), pi_list_apps, pi_list_roots, pi_get_frontmost, and pi_act (perform a click/typeText/keypress/scroll/ drag/moveMouse targeted by an element ref from pi_observe or by coordinates).

The bridge depends only on a structural subset of the pi backend (src/engines/pi-types.ts), so it does not pull in the pi package's native build or peer dependencies. Provide a custom loader if your pi backend lives elsewhere:

createAutomationEngine({
  engine: "pi-helper",
  piBackendLoader: () => myPiPlatformBackend,
});

Add a custom tool

Create a ToolDefinition and register it:

import { z } from "zod";
import type { ToolDefinition } from "./tools/base.ts";

const myTool: ToolDefinition<{ url: string }> = {
  name: "open_url",
  description: "Open a URL in the default browser.",
  inputSchema: z.object({ url: z.string().url() }),
  async handler({ url }, { backend }) {
    // ...use backend or any side-effect...
    return { content: [{ type: "text", text: `Opened ${url}` }] };
  },
};

// pass `tools: [...defaultTools, myTool]` to ComputerUseServer, or call server.register(myTool).

Notes & limitations

• The default system driver is a dependency-light, best-effort reference implementation. For production input automation on macOS (scroll/middle-click) or locked-down environments, provide a custom NativeDriver.

• Screen capture and input require the OS to grant the process screen-recording / accessibility permissions where applicable.