local-agent-mcp

Human-like local computer control for Windows, exposed to Claude Code as an MCP plugin.

No screenshots in the hot path, no local LLM required. Claude perceives the active window through the UIA (UI Automation) accessibility tree — plain text, milliseconds — and acts through real SendInput mouse/keyboard events with human-like motion (cubic Bezier cursor curves, hardware scan codes, randomized timing), not raw pyautogui-style teleport clicks or synthetic UIA Invoke calls.

Why

Most computer-use setups drive everything through periodic screenshots fed to a vision model. That's slow, expensive, and unnecessary for ordinary desktop apps — UI Automation already exposes every button, field, and label as structured text. This plugin gives Claude Code that fast path, plus real (not synthetic) input events, so it can drive Windows apps the way a person with a mouse and keyboard would.

An earlier version of this project routed decisions through a locally-hosted vision-language model (Qwen-VL via LM Studio) so the "brain" could run fully offline. We dropped that: it meant installing and keeping a separate model running, a brittle hand-rolled JSON tool-calling protocol (small local models don't reliably emit structured tool calls), unbounded context growth, and it blocked the MCP server's asyncio event loop. Claude itself is a perfectly good — and much more capable — brain for this; the plugin's job is just perception and hands.

Related MCP server: Windows Computer Control MCP Server

What it gives Claude

Claude calls these one at a time, in a loop it drives itself: perceive → decide → act → perceive again.

Safety

human_click / human_type / key refuse to act if the foreground window belongs to Claude Desktop itself (claude.exe). This exists because, during development, a bug briefly let a local automation loop type into the developer's own Claude Desktop chat instead of the intended target app. The check is enforced in the tool layer, not left to whatever is deciding actions upstream.

This is a floor, not a complete sandbox — the plugin can genuinely click, type, and launch programs on your machine. Treat it like any other computer-use tool: point it at things deliberately.

Requirements

• Windows

• Python 3.10+ on PATH, with:

  pip install -r requirements.txt

  (pywinauto, pywin32, mcp[cli]<2)

Install

/plugin marketplace add OlehHavrilko/local-agent-mcp
/plugin install local-agent@local-agent-marketplace

Restart Claude Code (or start a new session) so it picks up the MCP server.

Known limitations

• One foreground window at a time; no multi-window orchestration.

• open_app only reaches things Windows can resolve via ShellExecute (installed apps, ms-settings: URIs, etc.).

• Not built for games or anything needing real-time reflexes — UIA perception assumes a standard accessible UI tree, which canvas-rendered content (games, some custom-drawn apps) doesn't provide. That needs a different approach (an object detector over frames), which this project intentionally does not attempt.

• human_click/human_type/key check the process name of the foreground window, not window identity beyond that — this blocks the specific incident it was built for, not a general sandbox.

License

MIT — see LICENSE.