DeskPilot
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
Controls the user's real Windows desktop (one chosen screen) with a visible agent cursor. Cheap loop: find/snapshot -> act([...many steps...]) -> read the returned diff. Click by ref (e12, b12 for web pages) or by visible text. Web pages in Chrome/Edge are read precisely (b-refs). Use snapshot(scope='ocr') or screenshot only when nothing else works. Batch steps. For repeated tasks use routine(). Never retry an action the user denied. If PAUSED or STOPPED, stop and tell the user. When the task is finished (or you give up), call done() so the overlay and control panel disappear.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| screen_infoB | Screens, which one you control, mode, top window. op='pick' reopens the user's screen picker; op='switch' + screen ('1','2','all') asks the user to change the controlled screen. |
| snapshotA | Compact list of visible elements with refs, e.g. |
| findA | Find elements by visible text and/or role (button, link, edit, item, tab, checkbox, menuitem...). Falls back to OCR. Cheaper than a full snapshot. |
| actA | Run steps in order with the real mouse/keyboard, then return what changed on screen. Target = ref "e12", visible text "Sign in", or [x,y] pixels of the last screenshot. Steps: {"click":T} {"dblclick":T} {"rclick":T} {"hover":T} {"type":"text","into":T?,"clear":bool?,"enter":bool?} {"key":"ctrl+l"|"enter"|"win+e"...,"repeat":n?} {"key_down":"shift"} {"key_up":"shift"} {"scroll":"down|up|left|right","amount":5,"at":T?} {"drag":T,"to":T} {"wait":"text","timeout":10,"gone":bool?} {"sleep":1} {"focus":"w3|title"} {"launch":"app|url"}. Sensitive steps may ask the user first. |
| launchA | Open an app by name ("Notepad", "Microsoft Store", "chrome"), a URL, a URI (ms-settings:, ms-windows-store:) or a file/folder path. "agent browser " opens the DeskPilot Chrome profile (precise web control). The window is moved to your screen. Returns a snapshot. |
| windowC | Manage windows. target = window ref (w3) or title/app text; default = top window on your screen. move_here brings a window from another screen onto yours. |
| clipboardB | Read or write clipboard text (cheaper than selecting/copying on screen). |
| routineB | Saved, replayable tasks (cheapest way to repeat work). list; run(name, params) replays steps with {{param}} filled in; start = begin recording your act/launch steps, then save(name, description); or save(name, steps=[...]) directly. Use {{param}} in step text for values that change. |
| screenshotA | Last resort image of your screen (costs many tokens). ref=crop around an element (cheap, sharp); region=[x,y,w,h] in previous screenshot pixels; marks=draw refs from the last snapshot. |
| doneA | Call once the task is complete (or you are stopping): hides the agent cursor, screen border and control panel so the user gets their screen back. summary = one short line of what you did. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 10 tools
Each tool targets a fairly distinct capability (inspect, find, act, clipboard, launch, window management, record/replay, screenshot), and descriptions carefully rank cost so agents know when to use find vs snapshot vs screenshot. However, snapshot's filter parameter largely duplicates find, and act's built-in launch/focus/scroll steps overlap with the launch and window tools, creating minor boundary confusion.
All names are short lowercase tokens, but the set mixes nouns (clipboard, window, routine, screenshot) with verbs (find, act, launch, done), and screen_info is the lone snake_case outlier. Readable, but no predictable verb_noun convention to generalize from.
Ten tools is well-scoped for a desktop-control agent: one tool per major capability tier (observe, find, act, transport utilities, automation, cleanup) with no filler tools. Nothing looks padded or missing at the count level.
The surface covers the full perceive-act loop: screen/window enumeration, element discovery, input execution, launching, clipboard, replayable routines, and a proper teardown via done. Minor gaps exist — window management only explicitly advertises move/target (no clear close/minimize/resize), and there's no way to read a titled window's text without OCR or snapshotting.