PointHand

From a reference to a canvas

PointHand reproduced the Mona Lisa in Microsoft Paint through its own MCP server. The reusable script selects colors and draws mouse strokes into the canvas.

Related MCP server: AiPC

Built for desktop work

The TypeScript MCP server connects your agent to a native Rust host. Windows input, accessibility and the overlay run in that host. The portable package needs Node.js 20+; Rust and Python are only needed for development or optional components.

Release status: v0.2.0-alpha.5 is an alpha for supervised workflows with 28 MCP tools (the next release adds background mode, element search, waits, window management and update notices for 35). Accessibility lines name only the actions an element can run, and start_audio_recording records what Windows is playing. See the release notes and validation limits. Alpha 4 remains available.

Quick start

1. Download the Windows x64 ZIP and its SHA-256 file from Releases.

2. Verify the ZIP with Get-FileHash -Algorithm SHA256, then extract it.

3. Install Node.js 20 or newer and add PointHand to your MCP client:

{
  "mcpServers": {
    "pointhand": {
      "command": "node",
      "args": ["C:/Tools/PointHand/client/dist/index.js"],
      "env": {
        "POINTHAND_ALLOW": "mspaint.exe,Illustrator.exe"
      }
    }
  }
}

Replace the path with the extracted package's client/dist/index.js. Set the allow-list to the apps you want the agent to control; leaving it empty allows every app.

Open a blank Paint canvas and try:

Use PointHand to inspect the Paint window, select a drawing tool and draw a small fish in the center of the canvas.

For the complete portrait workflow, follow the Mona Lisa tutorial, including the reference, ready-made plan and script instructions.

Use with Grok App / Grok CLI

PointHand is a plain stdio MCP server, so any MCP client can load it. Grok App (and its GrokCLI fork) recognise it as a one-click preset:

1. Download and extract this release ZIP, install Node.js 20+.

2. In the app open Settings → Extensions → MCP → Add server → Fill in PointHand. Replace the sample path with your extracted client/dist/index.js, then press Add.

3. Optional: Install PointHand skill writes skills/pointhand/SKILL.md into the user skills folder so the agent knows the look → act → verify workflow. Terminal users can copy that file to ~/.grok/skills/pointhand/SKILL.md and add the same config by hand:

[mcp_servers.pointhand]
command = "node"
args = ["C:/Tools/PointHand/client/dist/index.js"]

[mcp_servers.pointhand.env]
POINTHAND_ALLOW = "mspaint.exe"

Physical Escape still stops PointHand at any time. Full walkthrough: the Grok App docs once the companion PR lands, or the Quick start section above.

Visible control. A clear way to stop.

• Native feedback: a click-through glass frame, activity label, cursor trail and click ripple show PointHand's activity. The overlay targets 60 FPS and is excluded from screenshots by default.

• Stop: once an action has started, mouse movement and clicks do not cancel it. Physical Escape stops further mutations until the server restarts.

• Scoped access: restrict receiving apps with executable names or exact paths in POINTHAND_ALLOW.

• Input cleanup: interrupted strokes release held buttons. Alpha 2 also provides an out-of-band emergency_stop tool that stops active and queued mutations until restart.

Use PointHand in a supervised desktop session. App permissions are a guardrail, not a sandbox against malicious local software. Window screenshots use the window's own pixels when Windows graphics capture is available, so a covered window stays where it is. Pointer actions bring that window forward and do not click a window that remains on top. That refusal names the covering process and returns a fresh screenshot of the target so the action can be retried once. Desktop screenshots are the visible composite. Elevated windows can still reject input. Read the security policy and input behavior guide.

Tested on the desktop

Streamed mixed-DPI overlay runs averaged approximately 57–59 FPS at a 60 FPS target. These are observed renderer timings, not a display-refresh guarantee.

Secure-desktop switching, display disconnection during input, more physical monitor combinations and a clean non-developer PC remain to verify. See the full validation checklist and multi-monitor report.

Help shape PointHand

Contributions are welcome in English or Turkish. You can help with code, documentation, creative examples or compatibility reports.

• New here? Pick a good first issue.

• Have a Windows setup to test? Share a compatibility report.

• Ready to build? Read CONTRIBUTING.md and browse help wanted.

• Have an idea? Propose a workflow or creative example.

We especially need clean-install reports, English Paint checks and more physical-monitor/DPI coverage. Small, reproducible contributions help make the next release stronger.

Develop PointHand

The main implementation uses TypeScript + Rust + Windows APIs. The optional Python server is a reference implementation.

Requires stable Rust and Visual Studio Build Tools with Desktop development with C++, plus Node.js 20+.

./scripts/build-windows.ps1
npm ci --prefix client
npm run build --prefix client

The build writes the native executable and checksum manifest to bin/windows/x64/. Configure your MCP client with the absolute client/dist/index.js path in this checkout. POINTHAND_HOST can select a custom host executable.

To build and verify a portable package:

./scripts/package-windows.ps1
./scripts/verify-package.ps1

npm test --prefix client
node --test tests/js/*.test.mjs
cargo test --release --manifest-path host/Cargo.toml
cargo clippy --release --manifest-path host/Cargo.toml -- -D warnings
node --test scripts/host-regression.mjs

Optional Python checks require python -m pip install -e ".[test]", followed by python -m pytest.

Interactive tests move the real pointer and open a disposable test window. They use PointHand MCP and require Python with Tkinter:

node scripts/native-input-e2e.mjs
node scripts/native-multimonitor-e2e.mjs
node scripts/manual-physical-e2e.mjs

The multi-monitor test requires an extended second display. The manual test requires you to move a physical mouse or touchpad after drawing starts. Instructions and expected results are in the validation checklist.

Documentation