# computer-use-mcp

A **framework-agnostic** computer-use [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server.
It exposes core desktop operations — screen capture, mouse, keyboard, and file access — as standard MCP
tools, so any MCP-compatible agent framework (Claude Desktop, LangChain, AutoGen, a custom client, …) can
drive a computer without being tied to a specific agent runtime.

Built on [`@modelcontextprotocol/sdk`](https://www.npmjs.com/package/@modelcontextprotocol/sdk),
[`@cfworker/json-schema`](https://www.npmjs.com/package/@cfworker/json-schema), and
[`zod`](https://www.npmjs.com/package/zod), written in TypeScript with full type definitions.

## Features

1. **Standard MCP interface** — speaks MCP over stdio (`initialize`, `tools/list`, `tools/call`), decoupled
   from any agent framework.
2. **Core computer operations as MCP tools** — `screenshot`, `mouse_click`, `mouse_double_click`,
   `mouse_move`, `mouse_drag`, `mouse_scroll`, `keyboard_type`, `key_press`, `read_file`, `write_file`,
   `list_directory`.
3. **Zod + `@cfworker/json-schema`** — Zod is the single source of truth for input shapes and coercion;
   `z.toJSONSchema()` derives the wire JSON Schema, and `@cfworker/json-schema` validates incoming
   arguments before dispatch.
4. **TypeScript** — complete type definitions throughout (`src/core/backend.ts`, `src/tools/base.ts`, …).
5. **Extensible** — tools and the automation engine are both pluggable behind stable interfaces.

## Requirements

- Node.js ≥ 18 (or [Bun](https://bun.sh)).
- Platform automation tools for screen/input (the default driver uses OS facilities):
  - **Windows**: PowerShell + .NET (built in).
  - **Linux**: `scrot` (screenshot) and `xdotool` (mouse/keyboard).
  - **macOS**: `screencapture` + AppleScript (`osascript`); `cliclick` for move/drag.

## Install & run

```bash
bun install          # or: npm install
bun start            # or: npm start  (runs src/index.ts via tsx)
```

The server reads JSON-RPC over stdio and is meant to be launched by an MCP client. Example client config:

```json
{
  "mcpServers": {
    "computer-use": {
      "command": "bun",
      "args": ["/path/to/computer-use/src/index.ts"]
    }
  }
}
```

## Architecture

```
src/
  index.ts                 Entry point: wires the default backend to the server.
  server.ts                ComputerUseServer — MCP protocol handling (tools/list, tools/call).
  schema/json-schema.ts    Zod -> JSON Schema, and a @cfworker/json-schema validator factory.
  core/
    backend.ts             ComputerUseBackend — the operation contract (the extension seam).
    native-driver.ts       NativeDriver — low-level screen/input primitives.
    local-backend.ts       LocalComputerUseBackend — filesystem + injected NativeDriver.
    system-driver.ts       Default per-platform NativeDriver (Windows / Linux / macOS).
    pi-backend.ts          PiComputerUseBackend — bridges the pi-computer-use native helper.
  engines/
    pi-types.ts            Structural subset of the pi-computer-use backend contract.
    pi-helper.ts           PiHelperDriver — bridges pi-helper observe/act to NativeDriver.
    robotjs.ts             RobotJsDriver — bridges the robotjs package.
    playwright.ts          PlaywrightDriver — bridges a Playwright Page.
    index.ts               createAutomationEngine — selects an engine.
  tools/
    base.ts                ToolDefinition + ToolResult types.
    screenshot.ts, mouse.ts, keyboard.ts, file.ts
    window.ts              get_window_title — foreground window title + owning process.
    pi.ts                  pi_observe / pi_act / pi_list_apps / pi_list_roots / pi_get_frontmost.
    index.ts               defaultTools (the built-in tool set) + piTools.
```

### Request flow for a tool call

1. The client sends `tools/call` with `arguments`.
2. The server validates `arguments` against the tool's JSON Schema using `@cfworker/json-schema`
   (defense-in-depth wire contract). Invalid input returns a clear `isError` message.
3. Zod parses/coerces the arguments into a typed object (applies defaults, int coercion, …).
4. The tool handler runs against the injected `ComputerUseBackend`.

## Extending

### Add a custom tool

Create a `ToolDefinition` and register it:

```ts
import { z } from "zod";
import type { ToolDefinition } from "./tools/base.ts";

const myTool: ToolDefinition<{ url: string }> = {
  name: "open_url",
  description: "Open a URL in the default browser.",
  inputSchema: z.object({ url: z.string().url() }),
  async handler({ url }, { backend }) {
    // ...use backend or any side-effect...
    return { content: [{ type: "text", text: `Opened ${url}` }] };
  },
};

// pass `tools: [...defaultTools, myTool]` to ComputerUseServer, or call server.register(myTool).
```

### Swap the automation engine

The plugin ships a pluggable **automation engine** layer. Pick one via the
`COMPUTER_USE_ENGINE` environment variable, or construct it programmatically with
`createAutomationEngine` — no tool or server code changes:

| Engine        | Backed by                                              | Extra tools                          |
|---------------|--------------------------------------------------------|--------------------------------------|
| `system`      | OS facilities (PowerShell/.NET, `scrot`+`xdotool`, …)  | the 11 built-in tools                |
| `pi-helper`   | [`pi-computer-use`](https://github.com/injaneity/pi-computer-use) native helper | built-ins **+** `pi_*` semantic tools |
| `robotjs`     | the `robotjs` npm package (mouse/keyboard)             | the 11 built-in tools                |
| `playwright`  | a Playwright `Page` (browser automation)               | the 11 built-in tools                |

```bash
COMPUTER_USE_ENGINE=pi-helper bun start    # use the pi-computer-use native helper
COMPUTER_USE_ENGINE=robotjs   bun start    # use robotjs for input
```

```ts
import { createAutomationEngine } from "./engines/index.ts";
import { ComputerUseServer } from "./server.ts";

// programmatic selection (e.g. to drive a Playwright page):
const { backend, tools } = await createAutomationEngine({
  engine: "playwright",
  playwrightPage: page, // a real Playwright Page
});
const server = new ComputerUseServer({ backend, tools });
```

Engines degrade gracefully: if an optional package (e.g. `robotjs`,
`@injaneity/pi-computer-use`) is not installed, the engine throws a clear
`UnsupportedOperationError` describing how to enable it.

### Bridge to pi-computer-use

The `pi-helper` engine maps our operations onto the pi-computer-use
`ComputerUsePlatformBackend`:

- Screen capture → `observe({ includeImage: true })`; mouse/keyboard → `act(...)`
  with coordinate targets and the baseline `lookId`.
- On top of the 11 built-ins it exposes the helper's richer semantic tools:
  `pi_observe` (screen + accessibility outline), `pi_list_apps`, `pi_list_roots`,
  `pi_get_frontmost`, and `pi_act` (perform a `click`/`typeText`/`keypress`/`scroll`/
  `drag`/`moveMouse` targeted by an element **ref** from `pi_observe` or by coordinates).

The bridge depends only on a *structural subset* of the pi backend
(`src/engines/pi-types.ts`), so it does not pull in the pi package's native build
or peer dependencies. Provide a custom loader if your pi backend lives elsewhere:

```ts
createAutomationEngine({
  engine: "pi-helper",
  piBackendLoader: () => myPiPlatformBackend,
});
```

### Add a custom tool

Create a `ToolDefinition` and register it:

```ts
import { z } from "zod";
import type { ToolDefinition } from "./tools/base.ts";

const myTool: ToolDefinition<{ url: string }> = {
  name: "open_url",
  description: "Open a URL in the default browser.",
  inputSchema: z.object({ url: z.string().url() }),
  async handler({ url }, { backend }) {
    // ...use backend or any side-effect...
    return { content: [{ type: "text", text: `Opened ${url}` }] };
  },
};

// pass `tools: [...defaultTools, myTool]` to ComputerUseServer, or call server.register(myTool).
```

## Notes & limitations

- The default system driver is a dependency-light, best-effort reference implementation. For production
  input automation on macOS (scroll/middle-click) or locked-down environments, provide a custom
  `NativeDriver`.
- Screen capture and input require the OS to grant the process screen-recording / accessibility
  permissions where applicable.