pointman
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@pointman read the controls in my Notepad window"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
pointman
Let an AI agent see and operate apps on your Windows PC, with the limits set by you and not by the agent.
An agent connected to pointman reads a window the way a screen reader does (buttons, text boxes, labels) and can press buttons and fill in text boxes. No screenshots, no mouse takeover. Out of the box it can touch nothing: you name the apps it may read, and you separately switch on the right to act.
Control | What it does |
App allowlist | Only apps you list can be read. Other windows show up as |
Read-only by default | Clicking and typing stay off until you set |
Password boxes | Their contents are never returned, and the agent cannot type into them. |
Kill switch | Create one file and every request is refused until you delete it. |
Tamper-evident log | Every request, allowed or refused, is logged. Editing or deleting a line breaks the chain. |
Windows 10 or 11. It is an MCP server, so it works with Claude Desktop, Claude Code and other MCP clients.
Set it up
Needs uv.
git clone https://github.com/VinceBono/pointman
cd pointman
uv sync
copy pointman.example.toml pointman.toml # then edit itpointman.toml:
allowed_apps = ["notepad.exe"] # executable names; everything else is invisible
allow_actions = false # true lets the agent click and type in those appsAdd it to your MCP client. For Claude Code:
claude mcp add pointman -- uv --directory C:\path\to\pointman run pointman serve --config C:\path\to\pointman\pointman.tomlRelated MCP server: windows-computer-use-mcp
What the agent gets
Tool | Needs | Returns |
| nothing | Open windows with their app name; titles only for allowed apps |
| app allowed | The window's controls, one per line, each with a ref such as |
| app allowed + actions on | Presses the control with that ref |
| app allowed + actions on | Replaces a text box's contents; refuses password boxes |
powershell.exe 'Pointman Test Target'
e1 Edit 'Name' value=''
e2 Edit 'Secret' value=(password, hidden)
e3 Button 'Greet'
e4 Text 'waiting'A refused request comes back as text starting with DENIED: and the reason, and is logged.
Stop it, and check what it did
New-Item $HOME\.pointman\STOP -Force # refuse everything from now on
Remove-Item $HOME\.pointman\STOP # resume
uv run pointman verify-audit --config pointman.tomlThe log is ~/.pointman/audit.jsonl: one line per request with the time, tool, app, target control and
outcome. Text the agent types is not logged, only its length.
Limits
An allowed app is fully allowed. If you allow a browser or a mail client with actions on, the agent can do anything you could do in it. There is no per-button rule and no confirmation step yet. Start read-only.
The limits bind the agent, not someone at your keyboard. Anyone who can edit
pointman.tomlor the log file can change the rules or rebuild the log. The chain detects edits; it does not prevent them.Text on screen can carry instructions. A window the agent reads may contain text written to steer it. pointman limits what a steered agent can reach; it does not detect the steering.
Only apps that expose accessibility information work. Games, some Electron and custom-drawn apps expose little or nothing. Controls that cannot be pressed through the accessibility interface return an error.
Tested on one small test window, on Windows 11 and in CI. Not yet tried against real applications.
Development
uv run python -m unittest discover -s tests -v
python scripts/commitguard.py --allThe desktop tests open a small test window (tests/fixture_app.ps1) and drive it.
License
MIT. See LICENSE.
Available Tools
4 toolsclickB
Press a button or other control from the last read_window.
Inputs: ref (required), handle (required). Returns: result
list_windowsA
Open windows. Titles are shown only for allowed apps, because titles leak document and email names.
Inputs: no named parameters. Returns: result
read_windowB
The window's controls, one per line, each with a ref (e1, e2, ...) for click and set_text.
Inputs: handle (required). Returns: result
set_textA
Replace the contents of a text box from the last read_window. Password boxes are refused.
Inputs: ref (required), text (required), handle (required). Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 4 tools
Each tool maps to a distinct step in desktop automation: list windows, inspect controls, click a control, and set text in a control. The ref-based workflow clearly separates read_window from the action tools, so an agent can tell when to use each.
All names use snake_case and begin with an action verb. list_windows and read_window follow a verb_noun pattern, while click is verb-only and set_text is verb_noun, which is a minor deviation but still predictable.
Four tools are well-scoped for a focused UI automation server. Each tool earns its place by covering a distinct part of the inspect-and-interact workflow without redundancy.
The surface covers core desktop automation: discovering windows, reading controls, clicking, and setting text. Minor gaps like keyboard input, scrolling, or window management are reasonable omissions but could limit some workflows.
Maintenance
Related MCP Connectors
MCP server that lets your AI assistant build and deploy apps, hosting each one at a live URL.
One MCP server for the services your AI needs: memory, files, Microsoft 365, kanban and more.
Zero-log MCP server with 49 tools for AI guardrails, secret scanning, MCP linting & EU compliance.
MCP server for your apps' tools and custom tools, plus hosted AI agents and approval-gated workflows
Related MCP Servers
- AlicenseAqualityDmaintenanceA macOS computer-use MCP server that grants AI agents mouse, keyboard, and screen control with a robust security model including permission profiles, app deny-lists, and audit logging.22MIT
- FlicenseNot gradedqualityDmaintenanceA local, dependency-free MCP server that gives AI agents controlled access to the active Windows desktop, enabling automated interaction with applications through screenshots, clicks, typing, and window management.1-
- AlicenseNot gradedqualityDmaintenanceA local MCP server that gives a cloud AI agent hands on a real machine, enabling file operations, command execution, and screen capture via a secure tunnel with path whitelisting, command guards, and audit logging.MIT
- AlicenseNot gradedqualityDmaintenanceA Windows-only MCP server that gives AI models eyes and hands on a Windows machine, enabling file reading, screenshots, mouse/keyboard injection, and an auto-debug loop with a safety-focused watchdog for protected abort regions.MIT