Skip to main content
Glama

pointman

Let an AI agent see and operate apps on your Windows PC, with the limits set by you and not by the agent.

An agent connected to pointman reads a window the way a screen reader does (buttons, text boxes, labels) and can press buttons and fill in text boxes. No screenshots, no mouse takeover. Out of the box it can touch nothing: you name the apps it may read, and you separately switch on the right to act.

Control

What it does

App allowlist

Only apps you list can be read. Other windows show up as (not allowed) with their titles hidden.

Read-only by default

Clicking and typing stay off until you set allow_actions = true.

Password boxes

Their contents are never returned, and the agent cannot type into them.

Kill switch

Create one file and every request is refused until you delete it.

Tamper-evident log

Every request, allowed or refused, is logged. Editing or deleting a line breaks the chain.

Windows 10 or 11. It is an MCP server, so it works with Claude Desktop, Claude Code and other MCP clients.

Set it up

Needs uv.

git clone https://github.com/VinceBono/pointman
cd pointman
uv sync
copy pointman.example.toml pointman.toml   # then edit it

pointman.toml:

allowed_apps = ["notepad.exe"]   # executable names; everything else is invisible
allow_actions = false            # true lets the agent click and type in those apps

Add it to your MCP client. For Claude Code:

claude mcp add pointman -- uv --directory C:\path\to\pointman run pointman serve --config C:\path\to\pointman\pointman.toml

Related MCP server: windows-computer-use-mcp

What the agent gets

Tool

Needs

Returns

list_windows

nothing

Open windows with their app name; titles only for allowed apps

read_window

app allowed

The window's controls, one per line, each with a ref such as e3

click

app allowed + actions on

Presses the control with that ref

set_text

app allowed + actions on

Replaces a text box's contents; refuses password boxes

powershell.exe 'Pointman Test Target'
e1 Edit 'Name' value=''
e2 Edit 'Secret' value=(password, hidden)
e3 Button 'Greet'
e4 Text 'waiting'

A refused request comes back as text starting with DENIED: and the reason, and is logged.

Stop it, and check what it did

New-Item $HOME\.pointman\STOP -Force     # refuse everything from now on
Remove-Item $HOME\.pointman\STOP         # resume
uv run pointman verify-audit --config pointman.toml

The log is ~/.pointman/audit.jsonl: one line per request with the time, tool, app, target control and outcome. Text the agent types is not logged, only its length.

Limits

  • An allowed app is fully allowed. If you allow a browser or a mail client with actions on, the agent can do anything you could do in it. There is no per-button rule and no confirmation step yet. Start read-only.

  • The limits bind the agent, not someone at your keyboard. Anyone who can edit pointman.toml or the log file can change the rules or rebuild the log. The chain detects edits; it does not prevent them.

  • Text on screen can carry instructions. A window the agent reads may contain text written to steer it. pointman limits what a steered agent can reach; it does not detect the steering.

  • Only apps that expose accessibility information work. Games, some Electron and custom-drawn apps expose little or nothing. Controls that cannot be pressed through the accessibility interface return an error.

  • Tested on one small test window, on Windows 11 and in CI. Not yet tried against real applications.

Development

uv run python -m unittest discover -s tests -v
python scripts/commitguard.py --all

The desktop tests open a small test window (tests/fixture_app.ps1) and drive it.

License

MIT. See LICENSE.

Available Tools

4 tools
clickB

Press a button or other control from the last read_window.

Inputs: ref (required), handle (required). Returns: result

list_windowsA

Open windows. Titles are shown only for allowed apps, because titles leak document and email names.

Inputs: no named parameters. Returns: result

read_windowB

The window's controls, one per line, each with a ref (e1, e2, ...) for click and set_text.

Inputs: handle (required). Returns: result

set_textA

Replace the contents of a text box from the last read_window. Password boxes are refused.

Inputs: ref (required), text (required), handle (required). Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.7/5.0

    Scored across 4 tools

    Disambiguation5/5

    Each tool maps to a distinct step in desktop automation: list windows, inspect controls, click a control, and set text in a control. The ref-based workflow clearly separates read_window from the action tools, so an agent can tell when to use each.

    Naming Consistency4/5

    All names use snake_case and begin with an action verb. list_windows and read_window follow a verb_noun pattern, while click is verb-only and set_text is verb_noun, which is a minor deviation but still predictable.

    Tool Count5/5

    Four tools are well-scoped for a focused UI automation server. Each tool earns its place by covering a distinct part of the inspect-and-interact workflow without redundancy.

    Completeness4/5

    The surface covers core desktop automation: discovering windows, reading controls, clicking, and setting text. Minor gaps like keyboard input, scrolling, or window management are reasonable omissions but could limit some workflows.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      D
      maintenance
      A macOS computer-use MCP server that grants AI agents mouse, keyboard, and screen control with a robust security model including permission profiles, app deny-lists, and audit logging.
      22
      MIT
    • F
      license
      Not graded
      quality
      D
      maintenance
      A local, dependency-free MCP server that gives AI agents controlled access to the active Windows desktop, enabling automated interaction with applications through screenshots, clicks, typing, and window management.
      1
      -
    • A
      license
      Not graded
      quality
      D
      maintenance
      A local MCP server that gives a cloud AI agent hands on a real machine, enabling file operations, command execution, and screen capture via a secure tunnel with path whitelisting, command guards, and audit logging.
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      A Windows-only MCP server that gives AI models eyes and hands on a Windows machine, enabling file reading, screenshots, mouse/keyboard injection, and an auto-debug loop with a safety-focused watchdog for protected abort regions.
      MIT