Skip to main content
Glama
dahx5

sloth-mcp

by dahx5
OverviewSchemaSourceRelated ServersScoreDiscussionsAdmin
Python
Local

sloth-mcp

macbook clicking your stuff.

I'm not writing a proper README. It's called sloth for a reason.

Fine, a bit more.

An MCP server that gives Claude Desktop eyes and hands on a Mac. No Accessibility API, no browser driver, no app plugins — it looks at pixels and clicks like you do, so it works with anything that draws on screen.

The point is not that a model can click. The point is not paying for a screenshot every time it does.

How it goes

Claude sends a batch of semantic steps — click "Save", wait until "Export" appears, read the table — and the server executes the whole batch on its own: finds targets by their visible text, waits for the interface to settle by watching pixels instead of sleeping, and comes back only when the plan and reality disagree. One escalation carries a screenshot, the journey so far and the screen markup, so the next plan starts from the failure instead of from scratch.

It also remembers. Every window it reads becomes a node in a local SQLite graph: what is in it, what clicking each thing led to, when it was last seen. Ask it what it knows about an app and you get a text map with dates — and you can plan a whole route from that map without looking at the screen at all.

Related MCP server: Desktop Pilot MCP

The numbers

  • A screenshot costs ~1400 tokens. The same window from the map costs a few hundred — 6 to 12 times cheaper, and that is the difference between one look per step and one look per plan.

  • A familiar window is recognised by a perceptual hash of its content, so OCR is skipped entirely. On a live run walking System Settings, 16 of 20 reads came from memory — recognition ran 4 times instead of 20.

  • Semantic target matching runs as one batch instead of one call per line: on a 1200-line screen that is 1 model call instead of 828, ~11x faster.

  • Waiting is measured, not guessed. A macOS pane transition is two repaints with a dead-still gap of 115–565 ms between them, so "the screen went quiet" is not the same as "the transition finished" — the server tells them apart by where the pixels moved, not by how long they stayed still.

Running it

uv sync
uv run choto service install
uv run choto service status

The commands are called choto — that was the project's name before it was a sloth, and renaming every entry point is a change for its own sake. Same thing.

Optional: an icon detector. Without one the server reads everything by its visible text, which is most of a Mac; with one it also finds the controls that are only a glyph — the back arrow, the toolbar buttons, the switches with no label. If you have a CoreML icon_detect.mlpackage, point the server at it:

uv run choto model install --icon-detector /path/to/icon_detect.mlpackage

Then point Claude Desktop at the bridge:

"sloth": {
  "command": "/opt/homebrew/bin/uv",
  "args": ["run", "--project", "/path/to/sloth-mcp", "choto-bridge"]
}

The bridge is a thin stdio↔socket shim on purpose: Claude Desktop hands its own TCC identity to child processes, and that identity cannot be granted Accessibility — so the process that actually moves the mouse has to be somebody else. macOS will ask for Screen Recording and Accessibility for the signed Choto.app bundle that service install puts in place.

Kill switch: throw the mouse into the top-left corner, or hit Stop in the overlay frame.

What it deliberately does not do

Accessibility API (universality beats convenience), Windows and Linux, multiple displays, autonomous LLM calls from the server, and — for now — a README longer than this one.

License

MIT — see LICENSE.

One thing that is not: the icon detector. icon_detect.mlpackage is not shipped here and is not downloaded by anything in this repo — you point model install at your own copy. The YOLO-derived weights that name refers to are AGPL-3.0, so whatever you feed it comes with its own terms attached.

Support

There isn't any. Issues are welcome and may sit unread — this is something I built for my own machine and put out because it works, not a product with a roadmap. Forks are the faster path to whatever you need.

Available Tools

12 tools
annotate_iconsA

Ask what the wordless icons of an app are called — the first half of a round trip that has to happen once per drawing, ever. Choto's memory is text: an icon with no label is a place it knows how to click but cannot name, and observe reports those as 'icons: K labeled, M unlabeled'. This tool returns ONE picture (occasionally a few) holding every unnamed glyph of one application as a numbered grid at its real pixel size, plus a text legend giving each cell its context from memory: the app, the window it sits in, roughly where in that window (toolbar strip, sidebar column, bottom-right...), its size, the nearest few texts beside it, and any other windows the same drawing appears in. Read the picture, decide what each numbered control is, and send the answers with submit_icon_labels — that is the whole loop: observe -> annotate_icons -> submit_icon_labels -> click the label like any other text. app_name is optional: without it the app with the most-drawn unnamed glyph is chosen and named in the reply, because a glyph means whatever the program drawing it says it means and one sheet therefore covers one app. Nothing is captured and nothing is clicked — the crops come from memory, so this is safe to call while the user is working. If every glyph already has a name the reply says so plainly; the sheet is capped, and the reply says how many are left for a second call.

ParametersJSON Schema
NameRequiredDescriptionDefault
app_nameNo

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden and does so: crops come from memory, nothing is captured or clicked, it is safe to call while the user is working, the sheet is capped, and the reply reports remaining counts. It even explains why one sheet covers exactly one app.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The opening sentence ('Ask what the wordless icons of an app are called...') is indirect framing rather than a direct statement of what is returned, so the key payload arrives late. The rationale sentence about glyphs meaning whatever the drawing program says adds color but is not strictly needed for correct invocation.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema or annotations exist, yet the description fully covers the return format (picture plus legend fields: app, window, position, size, nearby texts, other windows), edge cases, and the next step. An agent has everything needed to call and interpret it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% for the single app_name parameter, so the description must compensate, and it does: app_name is optional and, when omitted, the app with the most-drawn unnamed glyph is selected and named in the reply. It does not, however, specify accepted name formats or how matching is performed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific action and concrete return artifact: a numbered-grid picture of every unnamed glyph of one application plus a text legend with per-cell context. It also explicitly distinguishes itself from siblings observe and submit_icon_labels by naming them in the round trip.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use (after observe reports 'icons: K labeled, M unlabeled'), the paired follow-up tool (submit_icon_labels), the sequencing loop, and the app_name-omitted behavior. It also states the terminal case (all glyphs named) and the cap with a hint to call again.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

execute_planA

Execute a batch of semantic UI steps and return a compact journey. Each step is an object: {action, target?, path?, modifiers?, text?, keys?, amount?, app_name?, from_text?, to_text?, scroll?, until_readout?, skip_if_absent?, expect?, timeout_ms?}. action is one of click, double_click, right_click, drag, type, hotkey, scroll, wait, focus_app, read, read_clipboard. target={text, scope?, region?, window?, scroll_to_find?} selects an element by its visible text (region is one of top/bottom/left/right/center to disambiguate duplicates); matching is exact, then fuzzy, then semantic. click/double_click/right_click need a target; drag needs path; type needs text; hotkey needs keys (e.g. ['cmd','shift','4']); scroll needs amount (positive scrolls down; add a target to scroll over it); focus_app needs app_name; read takes an optional target (for scope/window/region) and read_clipboard takes nothing. BRINGING TEXT BACK: every other action changes the screen, these two hand you its contents. {action:'read'} returns the text of the window the step works in — laid out as it is drawn, so the columns of a table stay columns — in an 'extracted (step N, window "...")' block after the journey. It is never truncated: read what you need, all of it. On a read, target only says WHERE to read and its text must be EMPTY — {action:'read', target:{text:'', window:'Invoice'}} reads another window, text:'' with scope:'chrome' reads the menu bar or an open menu, and region reads one part of the pane. A read has no element to aim at, so naming one is refused rather than ignored; to bound the reading by text, use from_text/to_text. from_text and to_text narrow the reading to what lies between two pieces of visible text, both lines included — the way you would say 'the table from Subtotal down to Total'. They are matched like any target, and text that is not there fails the step like any missing target. scroll:true reads a document to its END instead of the screenful in front of you: the window is scrolled and re-read until nothing new comes up, the pieces are joined on their overlap with no duplicated lines, and the position is put back where the user had it. Use it for a long page, a chat history, a log; leave it off (the default) for a pane that is fully visible, since each screenful costs a parse. Those scrolls count against the plan's action budget, and the journey says how far it got and why it stopped. {action:'read_clipboard'} returns whatever text is on the system clipboard — the way to harvest text an app will not show in full: select it and copy it with ordinary steps ({action:'hotkey', keys:['cmd','a']}, then {action:'hotkey', keys:['cmd','c']}), then read_clipboard. An empty clipboard or one holding no text is reported in the journey, not treated as a failure. Choto never writes the clipboard, so anything you copy replaces what the user had there — copy only when you need to. STEPS THAT MAY NOT BE THERE: skip_if_absent:true (only on a step with a target) turns 'target not found' from a failure into a skipped step — the journey says 'skipped: target absent', the plan carries on, and the step's expect is not checked because nothing happened. This is for the noise of a real desktop: a cookie banner, an update prompt, a 'what's new' sheet that may or may not appear. Only use it where both branches end on the same screen; a step the plan actually depends on must be allowed to fail. GESTURES (drag): use it for what a click cannot express — rubber-band selection of several items, dragging a file onto a folder or into another pane, moving a slider, resizing a window or a column. path is a list of at least two waypoints: the button goes down on the first, travels through the rest and is released on the last (two is the usual case; more describe a bent route). Each waypoint is EITHER {target:{text,...}, offset?:{dx,dy}} — an element, optionally displaced by dx pixels right and dy pixels down from its center — OR {x, y}, a raw position in the same pixel coordinates the element listing reports. Prefer targets: they are resolved against the screen as it is when the gesture runs, so a list that scrolled still works. offset is how you name a spot with no text on it, which is what a rubber-band selection starts from: to select a block of files, start at the empty margin above-left of the first one, e.g. {target:{text:'report.pdf'}, offset:{dx:-40, dy:-25}}, and end past the last one, {target:{text:'notes.txt'}, offset:{dx:60, dy:25}}. All waypoints must be in the same window, and every one of them is resolved BEFORE anything is pressed: if one cannot be found the step escalates without the gesture starting. Be deliberate — a drag is undone far less often than a click: it moves files, drops them into folders, and reorders things, and there is no generic undo for that. DRAGGING TO A VALUE (until_readout): when what you want is a READING and not a distance — a playhead at 00:12:00, a slider at 50%, a level at -6 dB — do not compute pixels. Add until_readout={watch:{text,...}, value, tolerance?} to the drag: the button goes down on path[0], the pointer is pushed toward path[1] (which is only a DIRECTION and a first guess at how far, not the destination), and after every push the executor reads the instrument and corrects itself until it shows value. The path must be exactly two points. watch names the readout ITSELF, not a label beside it, and it has to be on screen before the gesture starts; value is written the way the app writes it (00:12:00, 50%, -6 dB, 1250) and every reading is then read in that same notation. tolerance (default 0, exact) is in those units — widen it for a control that skips the exact number. The journey says how many pushes it took and what the instrument ended on; if the reading cannot be read, or never arrives, the button is released where it stands and you get the last text that was there. modifiers=['cmd'|'shift'|'alt'|'ctrl'] are held for the whole action and work on click/double_click/right_click/drag: cmd-click to add one item to a selection, shift-click to extend it to a range, alt-drag to copy in Finder instead of moving. SURFACES: a target is looked for on exactly one of two surfaces, and they never mix. target.scope='window' (the default) searches only inside the window the step works in; target.scope='chrome' searches only the menu bar, status items and any open menu or popover. So clicking a menu item is two chrome steps: click {text:'File', scope:'chrome'}, then click {text:'Save As…', scope:'chrome'}. A target missing from the surface you named fails the step and the reply says which surface was searched and which was not — it is never answered from the other one, because the same text is often on both (a keypad '5' and a menu-bar '5'). With scope='chrome' target.window and target.scroll_to_find are rejected: chrome belongs to no window and is not scrolled. Every step works inside ONE window: by default the frontmost window of the app. target.window is a substring of a window's title (case-insensitive) and picks a different one — use it when the app has several windows open and the one you mean is not in front. If no open window matches, the step fails and the reply lists the titles that are open; it never falls back to another window. OFF-SCREEN TARGETS: if the text is not visible, the step does not fail immediately — the cursor is parked over the busiest part of the window and the list is scrolled a few steps down, then back and a few steps up, re-reading each time, so an item just below (or above) the fold is found the way a person would find it. The journey says what happened ('not on screen, scrolled 2x down -> matched ...'), and a search that finds nothing scrolls back to where it started before escalating. Those scrolls count against the plan's action budget. Set target.scroll_to_find=false where scrolling is not a harmless look: an endless feed, a list that loads more as it moves, or anywhere arriving elsewhere is itself a side effect. Drag waypoints are never scrolled for — the whole path is resolved before the button goes down, and scrolling would move the points already resolved. expect={appears?, disappears?, appears_count_increases?, screen_changes?} is a post-condition verified (with retries) within timeout_ms (default 5000); set several fields and all of them must hold. appears: this text must be on screen afterwards — use it when the step opens something new (a dialog, a page, a menu). disappears: this text must be gone afterwards — only for text that really leaves the screen (a closing dialog). Do NOT use it for text you cleared out of an input field: an emptied field shows its placeholder again, so the text is still on screen and the check fails even though the step worked. appears_count_increases: this text must match MORE on-screen lines than it did before the step. This is the right check for adding one more copy of something to a list — sending a chat message, adding a row, appending a tag — where the same text also stays visible elsewhere. Example: to send a message, {action:'type', text:'hi'} then {action:'hotkey', keys:['enter'], expect:{appears_count_increases:'hi'}}. screen_changes: true asserts the screen visibly changed (even slightly, e.g. a typed word), false asserts it did not. On success you get status=success and the journey. If a target cannot be resolved or an expectation is not met in time, execution stops and returns status=escalated with the failed step, a fresh screenshot, the element listing and the closest on-screen texts to the target that missed, so you can correct the plan and resend from the failure. If the user moves the mouse to the top-left corner the run returns status=aborted. EVERY reply — success, escalation or abort — ends with 'windows seen during this run': each distinct window the run passed through, with its app, its title and a sample of its text. That makes a deliberately short 'go and look' plan (focus_app, one click, a wait) the cheap way to explore an unfamiliar interface: you learn it in text instead of screenshots, and everything seen is written to memory, so recall can hand it back later without touching the screen at all. WORKING A MISSION: mission_item_id=N ties this run to item N of the active checklist (mission_start). A successful run marks that item passed and keeps these exact steps as its recipe, with a fingerprint of the world they worked in, so mission_replay(N) can repeat them later; any other outcome marks it failed with this run's own reason. Either way the reply ends with the mission status block — goal, progress, the item now in play — so a long scenario never rests on your memory of it. llm_version_tag is how you label the model driving the run: it is stored with the recipe and compared before a replay. Leave both out for an ordinary run.

ParametersJSON Schema
NameRequiredDescriptionDefault
stepsYes
llm_version_tagNo
mission_item_idNo

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so: escalation vs abort semantics (mouse to top-left), action budgets, scroll_to_find side effects, drag's irreversibility ('no generic undo'), the clipboard being overwritten by the user's copy, expect retries within timeout_ms default 5000, and the standing 'windows seen during this run' tail on every reply. Nothing material about behavior is left implicit.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the step schema and core actions, then organized into clearly headed sections (STEPS THAT MAY NOT BE THERE, GESTURES, SURFACES, OFF-SCREEN TARGETS, WORKING A MISSION). Dense and mostly earning its place, but it is very long and some points (scope/scroll side effects) recur across sections rather than being stated once.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with a nested, undocumented schema and no output schema, the description covers both inputs and returns thoroughly — the journey, 'extracted (step N, window ...)' read blocks, never-truncated reads, escalation payload (failed step, screenshot, element listing, nearest texts) and the mission status block. Nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% and the schema documents nothing, so the description must compensate — and it does exhaustively: the full step object shape, each action's required fields, target/scope/window/region semantics, modifiers, until_readout watch/value/tolerance, expect fields, and the meaning of mission_item_id and llm_version_tag ('leave both out for an ordinary run').

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource up front: 'Execute a batch of semantic UI steps and return a compact journey.' The batch/plan framing (multiple steps, one journey, escalation) clearly separates it from single-action siblings like observe and focus_app.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when/when-not for each sub-capability: scroll:true for 'a long page, a chat history, a log' but 'leave it off (the default) for a pane that is fully visible'; read_clipboard when an app won't show text in full; skip_if_absent only 'where both branches end on the same screen'; drag 'for what a click cannot express.' It also names the cheap alternative path (a short focus_app/click/wait plan) for exploration.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

focus_appA

Bring an application to the front by its name (e.g. 'System Settings', 'Safari'), launching it if it is not already running, and reopening a window if it is running with none. Use before observing or acting on an app you are not sure is frontmost. The name is matched case-insensitively against running apps: exact name first, then a unique prefix or substring, so 'Telegram' finds 'Telegram Lite'. The reply is 'focused ""' only after the app is confirmed frontmost; otherwise it explains what went wrong (ambiguous name — with the candidates listed, no such app, launch failed, or activation not confirmed).

ParametersJSON Schema
NameRequiredDescriptionDefault
app_nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations to lean on, the description carries the full burden and does so well: launch-on-demand, window reopening, case-insensitive matching with exact-then-substring precedence, and the precise success/error reply contract including ambiguous-name candidate listing and unconfirmed activation. This is exactly the behavioral detail an agent needs before invoking a state-changing tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two dense sentences, front-loaded with the action and matching/return semantics, with the error enumeration deferred to the end. No sentence is filler; every clause conveys a distinct behavior or constraint.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has one required parameter, an output schema, and no annotations, and the description covers activation behavior, matching rules, and the reply contract. Nothing an agent needs in order to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does: it explains that app_name is matched case-insensitively, resolved exact-first then by unique prefix or substring, and gives concrete examples ('System Settings', 'Safari', 'Telegram' → 'Telegram Lite'). This is meaning well beyond the bare string type.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource ('Bring an application to the front by its name') along with the side effects (launch if not running, reopen a window if none). An agent can immediately distinguish this activation tool from read-oriented siblings like observe and probe_tooltips.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit trigger condition: 'Use before observing or acting on an app you are not sure is frontmost,' which links it to the observe/act workflow. It does not name a when-not case or an alternative tool for the already-frontmost situation, so it stops short of full alternative routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

mission_finishA

Close the active mission: status=done when there is nothing left to try, abandoned when the checklist was not worked through. Failures do not stop a mission from being done — whether a run with three failed items is finished is your judgement, not a rule the server holds. Closing frees the one active slot, which is what mission_start needs. The reply is the whole checklist as a closing record; nothing is deleted, and the items keep their recipes and their reasons.

ParametersJSON Schema
NameRequiredDescriptionDefault
statusYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does well: it discloses the slot lifecycle effect, that nothing is deleted, that items retain recipes and reasons, and what the response contains. It omits error/permission behavior (e.g. calling with no active mission), which is the main remaining gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action and every sentence adds information (parameter meaning, slot consequence, non-destruction). The sentences are long and clause-heavy, but there is little pure filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be documented, yet the description still characterizes the reply as the full checklist record. Combined with the slot-freeing and non-destructive semantics, this is nearly complete for a one-parameter mutation tool; only the no-active-mission failure case is unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% and the single status parameter has no description or enum, so the description must supply meaning — and it does, defining 'done' and 'abandoned' semantics and clarifying that failure counts do not determine status. It stops short of enumerating whether other values are accepted.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Close the active mission') and immediately ties it to the sibling relationship — closing frees the slot that mission_start needs — so an agent can separate it from mission_start, mission_status, and mission_mark without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear conditions for the required parameter choice (done vs abandoned) and explicitly addresses the ambiguous case of failed items, telling the agent it is a judgement call rather than a server rule. It does not, however, name a sibling alternative or say what to do if no mission is active.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

mission_markA

Mark a checklist item yourself, for the acceptance criteria only you can check — something you read in a screenshot, in text a plan brought back, or judged from the reply. Runs mark their own items (execute_plan with mission_item_id), so use this when there was nothing to execute or when the executor's verdict was not the point. status is passed, failed or skipped. failed REQUIRES reason, and a reason is refused on anything else: 'item 4 failed' with no reason supports no plan except trying the same thing again, and a passed item carrying one asserts two contradictory things. skipped is for an item deliberately not attempted — a precondition never arose, the feature is absent in this build — and is not a defect. The reply is the status block.

ParametersJSON Schema
NameRequiredDescriptionDefault
reasonNo
statusYes
item_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does substantial work: failed REQUIRES reason, reason is refused on other statuses, and the reply format is the status block. It omits idempotency/re-marking behavior and error handling, but the mutation constraints are unusually well disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose is front-loaded and each clause largely earns its place (status semantics, reason rule, skipped definition). It is on the long side and the 'asserts two contradictory things' clause is rhetorically verbose, but nothing is truly redundant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists and the description notes the reply is the status block, so return values needn't be expanded. Coverage of status semantics, reason validity, and sibling routing is strong; only item_id mechanics and re-marking/error behavior are left uncovered.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate. It fully defines the status enum (passed/failed/skipped) and the conditional requirement on reason, including the rationale. item_id is left to inference from 'checklist item,' but the two non-obvious parameters are well covered.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Mark a checklist item yourself') and immediately scopes it to acceptance criteria only the agent can check. Explicitly distinguishes itself from the sibling execute_plan, so an agent can select between them without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit routing rule: runs mark their own items via execute_plan with mission_item_id, so use this 'when there was nothing to execute or when the executor's verdict was not the point.' It also defines when each status applies, including what skipped means and when it is not a defect.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

mission_replayA

Run again, exactly, what already worked for one checklist item. When an item passed through execute_plan(mission_item_id=N), its steps were kept as that item's recipe together with a fingerprint of the world they worked in (the application, the model tag you gave, and the route of windows the run walked through). This replays that recipe — no re-planning, no screenshot, the same execution path any plan takes. BEFORE IT RUNS it compares the recorded world with the world now. If anything comparable differs it does NOT execute a single step: the reply names each field with both values and stops there. A difference is not evidence that the application was updated — it means a window was resized, a dialog is in the way, a different model is driving, or the app really did change, and only you can tell which. Decide and act: run the steps deliberately with execute_plan(mission_item_id=N), which records the world it succeeds in as the item's new truth, or mark the item by hand. Fields nobody can compare are reported too, so a clean preflight is never read as 'nothing changed'. AFTER a successful replay the route it actually walked is held against the recorded one; a different route is reported as a warning on a success, and the item's stored fingerprint is deliberately left as it was. llm_version_tag labels the model driving this replay and is compared with the one stored on the recipe. The reply is the run's journey, the ledger, the preflight notes and the mission status; a failed replay fails the item with the run's own reason.

ParametersJSON Schema
NameRequiredDescriptionDefault
item_idYes
llm_version_tagNo

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden, and it does: preflight world comparison aborts before any step runs, no re-planning or screenshots, route held against the recorded route after success with a warning on divergence, stored fingerprint deliberately left unchanged, and a failed replay fails the item. This is behavior an agent could not infer from the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action and then layered with preflight, post-run and failure semantics in roughly the order the agent needs them. It is long and occasionally editorializes ('only you can tell which'), but nearly every sentence carries real behavioral information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Complete for a complex, side-effecting replay tool with no annotations and no output schema: it documents preconditions, abort behavior, route verification, fingerprint persistence, failure semantics, and what the reply contains. An agent has everything needed to call it and interpret the outcome.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% and neither property has a description, so the description must compensate — it does for llm_version_tag (the model driving this replay, compared against the one stored on the recipe). item_id is only implied through the surrounding references to mission_item_id=N, leaving a small naming ambiguity between the two calls.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource: replaying the exact recorded recipe that previously succeeded for one checklist item. It explicitly distinguishes itself from re-planning and from execute_plan, which the sibling list confirms is a real alternative. An agent can tell mission_replay apart from execute_plan without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives the condition for use (an item that passed through execute_plan(mission_item_id=N)) and names the alternatives explicitly: run deliberately via execute_plan to record a new truth, or mark the item by hand. It also tells the agent what a preflight mismatch means and that the decision is theirs.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

mission_startA

Write down what a long run is for, before you lose it. Halfway through a twenty-window scenario the plan has scrolled out of your context and what is left is a confident recollection of it — a mission is that plan kept outside you, in the server's database. goal is what the whole run is for. items is the checklist in order, each {title, intent, acceptance}: what to do, WHY it is on the list, and what would make it done. All three are required and none may be blank — intent and acceptance are the two you will no longer have when you read this back, and they are the reason the record is worth keeping at all. THE LOOP: mission_start -> work item by item, running each with execute_plan(steps=[...], mission_item_id=N) so the run files its own verdict and its steps onto item N -> the status block comes back on every reply, so you never have to remember where you are -> repeat a recorded pass later with mission_replay(N) -> mission_finish when there is nothing left to try. One mission is active at a time: starting a second while one runs is refused, naming the one to finish or abandon first. The reply is the status block — goal, progress, the item now in play with its intent and acceptance, and any failures with their reasons.

ParametersJSON Schema
NameRequiredDescriptionDefault
goalYes
itemsYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and discloses meaningful behavior: state is persisted in the server's database, only one mission may be active, a conflicting start is refused, and the reply is a status block. It does not cover auth requirements or failure modes beyond the concurrency conflict, but the state-machine disclosure is substantive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The content is dense and mostly earns its place, but the framing is deliberately narrative ('before you lose it', 'a confident recollection of it'), and the context-window motivation sentence spends prose on why rather than what to do. The actionable verb is not front-loaded; the workflow and constraints arrive well into the text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a stateful, multi-step tool with a weak schema and an output schema present, the description supplies the missing nested parameter structure, the concurrency constraint, the reply composition, and the surrounding workflow. Nothing an agent needs to call it correctly is absent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the nested item structure is undefined in the schema, so the description fully compensates: goal is defined as the purpose of the whole run, and items is defined as an ordered checklist of {title, intent, acceptance} with each field's meaning. It further specifies that all three are required and none may be blank — meaning the description adds information the schema does not.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Once parsed, the description clearly states it creates a persistent mission record holding the run's goal and an ordered checklist. It names the resource (mission) and the workflow verb framing ('write down'), and differentiates from siblings by describing the loop through execute_plan, mission_replay, and mission_finish. The core verb is somewhat buried under the 'long run' metaphor, which slightly muddies the first read.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a clear trigger (before a long run / before you lose the plan) and an explicit workflow ordering all related siblings. It also states a hard constraint: one mission active at a time, and a second start is refused naming the mission to finish or abandon. No explicit 'when not to use' beyond that, so it stops just short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

mission_statusA

Where the active mission stands. Without arguments it is the same compact block every execute_plan with a mission_item_id already returns — goal, progress, the current item in full, the failures — so call it when you have lost the thread, not after every run. full=true prints the whole checklist instead: every item with its id and status, nothing elided. That is the view to ask for before deciding a mission is done, and the only one that grows with the list, which is why it is a separate call.

ParametersJSON Schema
NameRequiredDescriptionDefault
fullNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden, and it does substantial work: it enumerates the default return content (goal, progress, current item in full, failures) and discloses that the full mode scales with the checklist. It does not explicitly state that this is a safe read-only call or describe cost/latency beyond the implicit 'not after every run', leaving a small gap given zero annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the purpose, then handles usage and the parameter in a tight sequence. The closing clause ('which is why it is a separate call') is somewhat self-justifying filler, but overall the length is proportionate to the information conveyed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-optional-param read tool with an output schema, the description supplies everything an agent needs: what the two modes return, when to invoke each, and how the tool relates to execute_plan. Nothing required for correct invocation is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% for the single boolean 'full', but the description compensates well, explaining that default yields the compact block while full=true returns every item with id and status and is the only mode that grows with the list. Meaning beyond the bare boolean is clearly added.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opening sentence names the resource and scope ('where the active mission stands') with a concrete read verb implied, and the description explicitly distinguishes it from execute_plan by noting the default output is the same compact block that tool already returns. An agent can tell this apart from mission_start/mission_mark/mission_finish without any schema lookup.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use ('call it when you have lost the thread'), explicit when-not ('not after every run'), and a separate condition for the full=true mode ('before deciding a mission is done'). Alternative-driven reasoning is fully spelled out.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

observeA

Capture the frontmost window and return a screenshot plus a textual listing of its text elements. Use this once before planning a batch of actions. TWO SURFACES: the elements come in two labelled blocks, because a step searches exactly one of them. 'elements in focus window (N)' is the focus window — the default search space of every step, and all a plain target can reach; text in a background window is deliberately not listed and cannot be targeted. 'chrome (menu bar / system, M)' is the macOS menu bar, status items and any open menu or popover; a step reaches those only with target.scope='chrome' — including the step right after opening a menu, since the dropped-down menu is chrome too, not part of the window. The chrome block is printed even when it is empty (M=0), with the reason. The reply also contains: the frontmost app name; which window was read (window); every window that app has open (open_windows) — quote a piece of one of those titles as target.window in execute_plan to work in a window other than the frontmost; whether this window is already known to the interface graph (known_window); the screen size in both pixels (OCR/element coordinates) and logical points (click coordinates); which area was scanned for single-character labels (single_character_scan); and, when there is text on neither surface, how much of it there is (off_surface — counted, not listed, because no step can act on it). Each element reads 'text @ (cx,cy) [conf 0.97]' where (cx,cy) is the element center in pixels. Element text is what you target in execute_plan. A PNG screenshot of the whole display is attached for anything OCR cannot name (icons, images).

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and delivers: it discloses the two labelled surfaces, that background-window text is deliberately excluded and untargetable, that the chrome block prints even when empty (M=0) with a reason, that off-surface text is counted but not listed, and that coordinates are pixels (elements) versus logical points (clicks). This is unusually rich behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is long and dense, but the length is justified given there is no output schema to carry the reply structure. It is front-loaded (purpose, then usage, then surfaces, then reply contents) and uses explicit labels, though a few sentences are packed with multiple clauses that slow scanning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, so the description must describe the return values, and it does so comprehensively: app name, window, open_windows, known_window, pixel vs point screen size, single_character_scan, off_surface, element format, and the attached PNG. Combined with zero parameters and no annotations, nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so the baseline is 4 per the rubric. The description spends its detail budget on the reply contract and targeting semantics rather than on parameters, which is the correct allocation here.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first sentence gives a precise verb and resource: capture the frontmost window and return a screenshot plus a textual listing of text elements. It also clarifies scope (frontmost window, not all windows) and the two output surfaces, so an agent can distinguish it from execute_plan or focus_app without opening anything else.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit when-to-use guidance: 'Use this once before planning a batch of actions.' It also explains the relationship to downstream calls by defining which surface a step can search and how chrome is reached via target.scope='chrome', including the non-obvious case of a just-opened menu.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

probe_tooltipsA

Let the application name its own icons: hover each unnamed icon of the FRONTMOST window and keep the tooltip it shows. This is the free half of icon labelling — annotate_icons asks you what a drawing is, this asks the program that drew it — so try it first on a window whose observe reply says 'icons: K labeled, M unlabeled'. A name it learns is written onto the drawing and onto every place that drawing appears in that app, exactly as submit_icon_labels writes yours, and from then on it is clickable as ordinary text: {action:'click', target:{text:'Share'}}. THIS ONE MOVES THE MOUSE. It is the only icon tool that touches the screen: the pointer visits each icon, waits out the tooltip delay (about a second and a half per icon, so a full call takes tens of seconds) and is put back where the user left it at the end. It works only on the window that is in front right now — bring the app forward with focus_app first — and it stops early, reporting how far it got, if the user pulls the mouse into the top-left corner, presses the stop button, or switches or moves the front window under it. Nothing is clicked and nothing is typed. limit (default 12, max 40) caps how many icons one call visits. Many controls have no tooltip at all; that is normal and the reply counts them ('showed none') — those are the ones to send to annotate_icons. The reply says how many were hovered, how many were named and which names were written, how many showed nothing, and how many drawings on that window still have no name.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so thoroughly: it moves the mouse and is the only icon tool that touches the screen, ~1.5s per icon (tens of seconds per call), pointer restored afterward, frontmost-window-only, early-stop conditions (top-left corner, stop button, window switch/move), and nothing is clicked or typed. Names are written globally across the app, which is important durable side-effect disclosure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose and the routing decision, then caveats; every sentence adds information and the 'THIS ONE MOVES THE MOUSE' emphasis is functional. It is long and slightly dense, but there is little true waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with meaningful side effects and an output schema, the description covers invocation, safety, timing, stopping behavior, and even the shape of the reply reports. Nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, but the description documents the single parameter well: 'limit (default 12, max 40) caps how many icons one call visits', adding the undocumented max of 40 beyond the schema's default. Minor gap only in that no other semantics exist to explain.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a concrete verb+resource ('hover each unnamed icon of the FRONTMOST window and keep the tooltip it shows') and explicitly distinguishes itself from both annotate_icons ('asks you what a drawing is') and submit_icon_labels ('writes yours'). An agent can tell exactly what this tool does and how it differs from its siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit routing: 'try it first' on windows whose observe reply shows unlabeled icons, 'bring the app forward with focus_app first', and directs controls with no tooltip to annotate_icons. When-to-use, prerequisites, and the alternative are all stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

recallA

Ask what Choto already remembers about an interface, as text — call this BEFORE observing or planning. A screenshot costs ~1400 tokens; this map costs a few hundred, so a window that has been visited before does not have to be looked at again to be planned against. The map is BY WINDOW and holds windows only: everything listed is inside one window, which is also where a plain target searches. Chrome — the menu bar, status items, open menus — is never remembered, so its absence here says nothing about the app; reach it with target.scope='chrome' in execute_plan, or see it as its own block in observe. All arguments are optional, and they widen the focus in three steps: with no arguments you get every application known, how many windows each has and when it was last seen; with app_name you get that app's windows plus the contents and outgoing transitions of the one it was last seen in; with app_name and window you get that one window — its text elements and its transitions, each reading 'click X -> window Y'. window is a piece of a window title (case-insensitive) or a node id such as '#5'; app_name matches exactly first, then as a substring ('settings' finds 'System Settings'). depth (default 1, max 3) unfolds that many levels of transitions; a branch leading back to a window already printed is marked '[shown above]' instead of being repeated, so cycles cost nothing. query= asks the other question — WHERE is this? — instead of walking the map yourself. It searches every element label, every named icon and every window title at once, with the same matcher a step's target uses, so a hit means a step aiming at that text would resolve to it. Each hit names the app, the window, what matched, how fresh it is, and the ROUTE to it. Add app_name to narrow the search. Routes read 'in #1 "Main" click "General" -> #2': that is the plan, in order, in the terms execute_plan takes. They start where the app was last seen — the window most recently read, which is where it still is unless something moved it since — unless you name from_window (a title or '#id'). When there is no route the reply names what is missing rather than saying no: which window cannot be reached, which recorded window leads into it (and whether that one was ever entered either), and what the starting window knows — how many windows it reaches, or that nothing has ever been clicked in it. That names the next thing to go and look at instead of leaving you to guess it. COVERAGE — read this before trusting an absence. Every reply carries what the map does NOT know: windows never clicked in, windows seen once, windows with no recorded way in, and 'N of M known control(s) ever clicked'. The denominator is what has been SEEN, never what the app has, so 'not found' means 'never seen here' and never 'does not exist' — the reply says so explicitly when a search comes back empty. A window that has never been clicked in is a photograph, not a map: what its buttons open is unknown, and the way to change that is to go and look. Reading the dates: the reply opens with the server's current time, and every window and transition carries when it was last seen, absolutely (UTC, '07-25 17:29Z', the year omitted while it is the current one) and relatively ('3h ago'). Nothing is hidden for being old, because only you can weigh it: minutes or hours old is normally safe to plan against; days old, or a window marked STALE, means the app may have been updated or rearranged since — either verify with one cheap step (a short plan that just navigates there, whose reply lists the windows it saw) or observe before acting. Element text is what you target in execute_plan; coordinates are deliberately absent, since the executor resolves text to pixels itself. The reply is size-capped and names what it cut ('elements (24 of 40)').

ParametersJSON Schema
NameRequiredDescriptionDefault
depthNo
queryNo
windowNo
app_nameNo
from_windowNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full behavioral burden and does so thoroughly: it discloses the memory model (by-window only, chrome never remembered), freshness semantics (absolute UTC plus relative ages, STALE marking), explicit absence/coverage semantics ('N of M known control(s) ever clicked', 'not found' means 'never seen here'), size-capping behavior, and cycle handling ('[shown above]'). This is rich context an agent could not get from the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The critical guidance is front-loaded in the first two sentences, and nearly every sentence carries decision-relevant information (token economics, chrome exclusion, route syntax, staleness policy). It is, however, an unusually long single block of prose for a five-parameter tool, so structure suffers even though little is pure padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values needn't be explained, yet the description still covers reply shape: timestamps, route notation in execute_plan terms, coverage footers, and truncation notices. For a high-complexity memory-lookup tool with zero annotation coverage, nothing an agent needs to call it correctly appears to be missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate and it does: it explains the three-step widening of focus (no args = all apps, app_name = that app, app_name+window = one window), that window is a title fragment or '#id', that app_name matches exactly-then-substring, depth defaults to 1 with max 3, and what from_window does.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a concrete verb+resource: 'Ask what Choto already remembers about an interface, as text.' It immediately positions the tool against siblings by telling the agent to call it BEFORE observing or planning and contrasting its token cost with a screenshot, so an agent can tell it apart from observe/execute_plan without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly names when to use it ('BEFORE observing or planning'), when not to bother re-looking ('a window that has been visited before does not have to be looked at again'), and which alternatives cover the gap (target.scope='chrome' in execute_plan, or observe). It also routes the agent to a verification step when data is stale.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

submit_icon_labelsA

Send back what the icons on an annotate_icons sheet are called — the second half of the round trip. labels maps each cell's key (the string annotate_icons printed for that cell, NOT the cell number) to the name of the control, in the words a user would use for it ('Share', 'New tab', 'Delete'); app_name is the application annotate_icons reported. Every name is written onto the glyph AND onto every place it is already drawn on the map, and every future sighting of that drawing — in any window of that app — arrives labelled without asking again. After this, target it by the name you gave it: {action:'click', target:{text:'Share'}}. The batch is applied whole or not at all; a key that names no glyph, a blank name or a missing app_name comes back as a sentence saying what to fix rather than a partial write. The reply says how many glyphs and places were named and how many of that app's icons are still unnamed.

ParametersJSON Schema
NameRequiredDescriptionDefault
labelsYes
app_nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so: it discloses the wide-reaching persistence ('written onto the glyph AND onto every place it is already drawn on the map, and every future sighting'), the atomicity guarantee ('applied whole or not at all'), and the failure mode (invalid key, blank name, or missing app_name returns a sentence rather than a partial write). These are exactly the side-effect and error semantics an agent needs for a mutation tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose, then parameter semantics, then persistence/atomicity/error behavior, then the reply — every clause adds information. It is a dense single paragraph with some run-on sentences and no visual structure, which costs a point but nothing is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be spelled out, yet the description still frames them usefully ('how many glyphs and places were named and how many of that app's icons are still unnamed'). For a 2-param mutation tool with zero annotations and zero schema coverage, all decision-relevant context — sequencing, key semantics, persistence, atomicity, errors — is present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate, and it does: it defines the labels key as 'the string annotate_icons printed for that cell, NOT the cell number' and the value as 'the name of the control, in the words a user would use for it' with concrete examples, plus app_name as 'the application annotate_icons reported'. This resolves the most likely ambiguity (key vs. cell index) that the bare object schema leaves open.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('send back what the icons on an annotate_icons sheet are called') and explicitly positions itself as 'the second half of the round trip' relative to the annotate_icons sibling. An agent can distinguish this from annotate_icons and from the other siblings without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The sequencing guidance is explicit: this is the return half of the annotate_icons round trip, and the description even shows the downstream step ('target it by the name you gave it: {action:'click', target:{text:'Share'}}'). It does not state an explicit when-not-to-use or a named alternative for submitting labels differently, so it stops just short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 12 tool updatesv0.1.0
    • First observedannotate_icons
    • First observedexecute_plan
    • First observedfocus_app
    • First observedmission_finish
    • First observedmission_mark
    • First observedmission_replay
    • First observedmission_start
    • First observedmission_status
    • First observedobserve
    • First observedprobe_tooltips
    • First observedrecall
    • First observedsubmit_icon_labels
Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.3/5.0

Scored across 12 tools

Disambiguation4/5

Most tools are clearly distinct: observe (live screen capture) vs recall (stored memory), focus_app, and execute_plan each have unique roles. The only mild overlaps are observe/recall (both surface interface info) and annotate_icons/probe_tooltips (both label icons), but the descriptions explicitly distinguish these as complementary halves of a workflow, so misselection is unlikely.

Naming Consistency3/5

The set mixes conventions: bare verbs (observe, recall), verb_noun (focus_app, execute_plan, annotate_icons, submit_icon_labels, probe_tooltips), and namespace_verb (mission_start, mission_status, mission_finish, mission_replay). The mission_* family is internally consistent and everything remains readable, but no single predictable pattern spans the whole surface.

Tool Count4/5

12 tools is well within a reasonable range for a UI-automation agent, and each earns its place (icon-labelling trio, mission lifecycle family, observation/action core). Slightly top-heavy in that five of twelve tools serve the mission subsystem, but that is a justified feature area rather than filler.

Completeness4/5

The surface covers the agent's lifecycle well: observe/recall for perception, execute_plan for action, focus_app for context, a full icon-labelling loop, and a complete mission lifecycle (start/status/mark/replay/finish). Minor gaps exist — no explicit abort/stop tool (handled via the corner gesture) and no way to prune stale memory — but nothing that would block core workflows.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • Desktop Pilot MCP

    VersoXBT
    A
    license
    Not graded
    quality
    D
    maintenance
    Enables high-speed native macOS automation for Claude by interacting directly with the Accessibility API, AppleScript, and UI trees instead of using screenshots. It allows users to read app states, click elements, and type text semantically across any macOS application.
    14
    10
    MIT

  • Automation MCP

    ashwwwin
    A
    license
    Not graded
    quality
    F
    maintenance
    Enables AI assistants to automate macOS desktop tasks including mouse control, keyboard input, screenshots, window management, and UI interaction.
    7
    415
    MIT