pointman
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| allowed_apps | No | Executable names of the apps the agent may read (e.g. ["notepad.exe"]); every other window is invisible. Configured in pointman.toml. | |
| allow_actions | No | Set to true to let the agent click and type in the allowed apps. Read-only by default. Configured in pointman.toml. | false |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
Operate Windows apps through their accessibility tree. Call list_windows, then read_window to get element refs, then click or set_text. A reply starting with DENIED is a policy decision by the computer's owner: report it, do not try another route.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| list_windowsA | Open windows. Titles are shown only for allowed apps, because titles leak document and email names. |
| read_windowB | The window's controls, one per line, each with a ref (e1, e2, ...) for click and set_text. |
| clickB | Press a button or other control from the last read_window. |
| set_textA | Replace the contents of a text box from the last read_window. Password boxes are refused. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 4 tools
Each tool maps to a distinct step in desktop automation: list windows, inspect controls, click a control, and set text in a control. The ref-based workflow clearly separates read_window from the action tools, so an agent can tell when to use each.
All names use snake_case and begin with an action verb. list_windows and read_window follow a verb_noun pattern, while click is verb-only and set_text is verb_noun, which is a minor deviation but still predictable.
Four tools are well-scoped for a focused UI automation server. Each tool earns its place by covering a distinct part of the inspect-and-interact workflow without redundancy.
The surface covers core desktop automation: discovering windows, reading controls, clicking, and setting text. Minor gaps like keyboard input, scrolling, or window management are reasonable omissions but could limit some workflows.