🖐️ Hands

A macOS computer-use MCP server for autonomous AI agents.

Give an LLM eyes, a mouse, and a keyboard — with a real security model in between.

Quick Start • Tools • Security Model • Plugins • Docs

Most "computer use" demos give a model raw pixel coordinates and hope for the best. Hands is built differently: every action goes through a typed 7-phase dispatch pipeline — validate → rate-limit → authorize → lock → execute-with-retry → observe → audit — so that giving an agent control of your desktop doesn't mean giving up any of your say in what it's allowed to do.

import anyio
from hands import Container, HandsConfig

config = HandsConfig()
config.driver = "fake"          # no macOS needed — great for a first look
container = Container.build(config)

async def main():
    shot = await container.dispatcher.dispatch("screenshot", {})
    print(shot["ok"], shot["bounds_pt"])

anyio.run(main)

Why Hands

• A real policy engine, not a rubber stamp. Rule-based profiles (strict / default / trusted), an app deny-list (Passwords, 1Password, System Settings — blocked out of the box), secret-pattern detection on typed text, per-tool confirmation, and a sliding-window rate limiter. See Security Model.

• Tamper-evident audit log. Every action is appended to a SHA-256 hash-chained JSONL log; deleting or editing any line downstream breaks the chain, and hands audit verify proves it.

• Redaction by construction. Clipboard content and typed text never enter application state, logs, audit records, or metrics — only their length and a SHA-256 hash ever do.

• A fake driver, not just mocks. FakeDriver is a full in-memory virtual desktop (windows, apps, clipboard, AX tree, OCR boxes) that every tool runs against identically to the real macOS driver. The entire test suite — 215 tests — runs on Linux CI with zero macOS hardware, and it's the fastest way to try Hands out or build a plugin without touching your real screen.

• Perception, not just action. OCR-grounded text search (find_text), an accessibility-tree fallback (get_ui_tree), and a verify/wait condition language so an agent can confirm an action worked instead of flying blind.

• A kill switch that actually works. touch ~/.hands/KILL halts the server immediately — checked before every single dispatch, including mid-way through a batched execute_sequence.

• Extensible without forking. Third-party plugins register tools through the exact same ToolSpec machinery as the 22 built-ins, loaded via Python entry points; a plugin that raises during setup is logged and skipped, not a server crash. (Plugins run in-process with access to the same services — this is crash isolation, not a security sandbox; don't load untrusted plugin code any more than you would an untrusted import.)

Related MCP server: macinput

Requirements

• macOS 13+ (Ventura or later) for real desktop control

• Python 3.12+

• Screen Recording and Accessibility permissions (System Settings → Privacy & Security) — hands permissions tells you exactly what's missing

You can install and explore the library on any OS using the fake driver (HANDS_DRIVER=fake) — only real desktop control requires macOS.

Installation

pip install hands-mcp

With the macOS driver and Vision OCR (needed for real desktop control):

pip install "hands-mcp[macos]"

The import name is still hands regardless of the install name above:

import hands
print(hands.__version__)

For development:

git clone https://github.com/yuvitbatra/hands-mcp.git
cd hands-mcp
uv sync --group dev
uv run pytest -q

Quick Start

As an MCP server (for AI agents)

Add to your MCP client configuration (e.g. Claude Desktop's claude_desktop_config.json):

{
  "mcpServers": {
    "hands": {
      "command": "hands",
      "args": ["serve"]
    }
  }
}

Or run it directly over stdio:

hands serve

As a Python library

import anyio
from hands import Container, HandsConfig

# The fake driver is a full in-memory virtual desktop — no macOS needed.
config = HandsConfig()
config.driver = "fake"
container = Container.build(config)

async def main():
    res = await container.dispatcher.dispatch("screenshot", {})
    print(res["ok"])  # True

anyio.run(main)

# Or run the full MCP server over stdio from your own process:
import anyio
from hands import run_server, load_config

anyio.run(run_server, load_config())

Diagnostics

hands doctor              # resolved config, driver, displays, registered tools
hands doctor --metrics    # ...plus a metrics snapshot (counters + latency histograms)
hands permissions         # TCC grant status (Screen Recording, Accessibility) + fix links
hands audit verify        # verify the audit log's hash chain end to end

Tools (22)

Full argument/return reference: docs/TOOLS.md.

Security Model

An agent that can move your mouse and type on your behalf is a genuinely different risk profile from a chatbot, so the policy layer isn't an afterthought — it's a fixed phase of every dispatch, before the tool ever runs:

validate args → rate limit → authorize (PermissionEngine) → acquire action lock
    → execute with retry → record state/screen-dirty → audit + metrics

Profiles (HANDS_SECURITY__PROFILE, default default):

Evaluation order for every action: (1) is the frontmost app on the deny-list? → denied outright, even under trusted. (2) does a configured rule match? → first match wins. (3) does typed/pasted text match a secret_patterns regex? → confirmation required regardless of profile. (4) fall through to the profile's class default above.

App deny-list (HANDS_SECURITY__DENY_APPS, on by default): blocks acting tools — reads still work — against com.apple.systempreferences*, com.apple.Passwords*, com.apple.keychainaccess, 1Password (both bundle id families). This applies uniformly whether the tool is called directly or from inside execute_sequence.

Rate limiting (HANDS_SECURITY__MAX_ACTIONS_PER_S, default 10.0): a 1-second sliding window over acting tools; bursts past the limit are denied, not queued.

Secure input refusal: keyboard_type and clipboard_get are refused outright while macOS reports secure text entry active (a password field has focus) — independent of profile.

Confirmation hooks: under dialog mode (default, macOS only) a denied-by-default sensitive action pops a real confirmation dialog via osascript; under deny mode (or off-macOS) it's auto-denied. Plug in your own hook by subclassing the PermissionEngine's confirm callback if you need a different UX (e.g. routing through your agent framework's own approval flow).

Redaction invariant: clipboard content and typed text are used only in-memory for policy matching (e.g. secret-pattern checks) — they never reach state, the audit log, or metrics in raw form. Only length and a SHA-256 hash ever leave those boundaries.

Kill switch: touch ~/.hands/KILL halts the server immediately — checked at the top of every dispatch, including mid-sequence inside execute_sequence. Remove the file to resume.

Audit log: every action is appended to ~/.hands/audit.jsonl as {"event", "prev_hash", "hash"}, where hash = sha256(prev_hash + canonical_json(event)). Tampering with or deleting any line downstream breaks the chain — verify with hands audit verify.

Configuration

All settings can be overridden via HANDS_* environment variables (nested fields use __, e.g. HANDS_SECURITY__PROFILE). Full schema: src/hands/config.py.

Plugin System

Hands supports external plugins that register custom tools through the exact same ToolSpec machinery as the built-ins — no side door around the dispatcher, so plugin tools get the same policy, retry, and audit treatment.

from hands.plugins.api import HandsPlugin, PluginContext

class MyPlugin:
    name = "my-plugin"
    version = "1.0.0"

    def setup(self, ctx: PluginContext) -> None:
        from pydantic import BaseModel
        from hands.registry import ToolSpec
        from hands.retry import RetryPolicy

        class Args(BaseModel, extra="forbid"):
            query: str

        async def my_tool(args: Args, ctx_) -> dict:
            return {"result": f"searched for {args.query}"}

        ctx.registry.register(ToolSpec(
            "my_search", "Custom search tool",
            Args, my_tool, "read", RetryPolicy.read(), idempotent=True))

    def teardown(self) -> None:
        pass

Register it via the hands.plugins entry-point group in your own package's pyproject.toml:

[project.entry-points."hands.plugins"]
my-plugin = "my_package.plugin:MyPlugin"

Plugins load at server startup. A plugin that raises in setup is logged and skipped — it never takes the server down. Restrict which plugins may load with an allowlist:

HANDS_SECURITY__PLUGIN_ALLOWLIST='["my-plugin"]' hands serve

See docs/plugins.md for the full plugin author's guide, and src/hands/plugins/api.py for the stable import surface (semver-guarded — additive-only within a major version).

Testing

uv run pytest -q                                          # 215 tests, any OS, ~2s
HANDS_CONTRACT_MACOS=1 uv run pytest tests/contract -q    # real macOS driver (needs TCC grants)
HANDS_E2E_MACOS=1 uv run pytest tests/e2e -q              # full stack vs. a real Tk fixture app
uv run pytest tests/perf -m perf --benchmark-only -q      # latency budgets
uv run pytest tests/stress -m stress -q                   # 10k-action soak + concurrency

The contract/e2e suites are opt-in because they genuinely move your mouse and open/close real apps — don't run them unattended. See CONTRIBUTING.md for the full development workflow.

Architecture

types, errors, config, retry
    ↓
driver/base (Protocol) ← fake.py (tests/CI) + macos.py (real hardware)
    ↓
services/  (screenshot, ocr, mouse, keyboard, clipboard, windows, apps,
            waiter, verification)
    ↓
state, permissions (PermissionEngine), audit (hash-chained), metrics, registry
    ↓
dispatcher  (7-phase pipeline: validate → rate-limit → authorize → lock →
             execute-with-retry → observe → audit)
    ↓
tools/      (22 MCP tools)
    ↓
container → server (MCP stdio transport) → cli

Full design rationale: docs/DESIGN.md.

Contributing

Contributions welcome — see CONTRIBUTING.md for the development setup, test-driven workflow, and what a good PR looks like.

License

MIT