# local-agent-mcp

Human-like local computer control for Windows, exposed to [Claude Code](https://claude.com/claude-code) as an MCP plugin.

No screenshots in the hot path, no local LLM required. Claude perceives the active window through the UIA (UI Automation) accessibility tree — plain text, milliseconds — and acts through real `SendInput` mouse/keyboard events with human-like motion (cubic Bezier cursor curves, hardware scan codes, randomized timing), not raw `pyautogui`-style teleport clicks or synthetic UIA `Invoke` calls.

## Why

Most computer-use setups drive everything through periodic screenshots fed to a vision model. That's slow, expensive, and unnecessary for ordinary desktop apps — UI Automation already exposes every button, field, and label as structured text. This plugin gives Claude Code that fast path, plus real (not synthetic) input events, so it can drive Windows apps the way a person with a mouse and keyboard would.

An earlier version of this project routed decisions through a locally-hosted vision-language model (Qwen-VL via LM Studio) so the "brain" could run fully offline. We dropped that: it meant installing and keeping a separate model running, a brittle hand-rolled JSON tool-calling protocol (small local models don't reliably emit structured tool calls), unbounded context growth, and it blocked the MCP server's asyncio event loop. Claude itself is a perfectly good — and much more capable — brain for this; the plugin's job is just perception and hands.

## What it gives Claude

| Tool | Does |
|---|---|
| `perceive_active_window` | Reads the UIA tree of the current foreground window — title + clickable elements (id, type, name, x, y). No image involved. |
| `human_click` | Human-like cursor move (Bezier curve, eased, adaptive duration) to (x, y), then a real `SendInput` mouse click. |
| `human_move` | Just the cursor move, no click. |
| `human_type` | Types text via hardware scan codes for the current keyboard layout, falling back to `KEYEVENTF_UNICODE` per-character for anything that doesn't map (e.g. Cyrillic on a US layout). |
| `key` | Presses a key or combo (`"enter"`, `"ctrl+a"`, `"ctrl+shift+t"`) via hardware scan codes. |
| `open_app` | Launches an app (`notepad`, `calc`, `explorer`, `ms-settings:bluetooth`, ...) and forces its window to the foreground, working around Windows' foreground-lock. |

Claude calls these one at a time, in a loop it drives itself: perceive → decide → act → perceive again.

## Safety

`human_click` / `human_type` / `key` refuse to act if the foreground window belongs to Claude Desktop itself (`claude.exe`). This exists because, during development, a bug briefly let a local automation loop type into the developer's own Claude Desktop chat instead of the intended target app. The check is enforced in the tool layer, not left to whatever is deciding actions upstream.

This is a floor, not a complete sandbox — the plugin can genuinely click, type, and launch programs on your machine. Treat it like any other computer-use tool: point it at things deliberately.

## Requirements

- Windows
- Python 3.10+ on `PATH`, with:
  ```
  pip install -r requirements.txt
  ```
  (`pywinauto`, `pywin32`, `mcp[cli]<2`)

## Install

```
/plugin marketplace add OlehHavrilko/local-agent-mcp
/plugin install local-agent@local-agent-marketplace
```

Restart Claude Code (or start a new session) so it picks up the MCP server.

## Known limitations

- One foreground window at a time; no multi-window orchestration.
- `open_app` only reaches things Windows can resolve via `ShellExecute` (installed apps, `ms-settings:` URIs, etc.).
- Not built for games or anything needing real-time reflexes — UIA perception assumes a standard accessible UI tree, which canvas-rendered content (games, some custom-drawn apps) doesn't provide. That needs a different approach (an object detector over frames), which this project intentionally does not attempt.
- `human_click`/`human_type`/`key` check the *process name* of the foreground window, not window identity beyond that — this blocks the specific incident it was built for, not a general sandbox.

## License

MIT — see [LICENSE](LICENSE).