Capture the frontmost window and return a screenshot plus a textual listing of its text elements. Use this once before planning a batch of actions. TWO SURFACES: the elements come in two labelled blocks, because a step searches exactly one of them. 'elements in focus window (N)' is the focus window — the default search space of every step, and all a plain target can reach; text in a background window is deliberately not listed and cannot be targeted. 'chrome (menu bar / system, M)' is the macOS menu bar, status items and any open menu or popover; a step reaches those only with target.scope='chrome' — including the step right after opening a menu, since the dropped-down menu is chrome too, not part of the window. The chrome block is printed even when it is empty (M=0), with the reason. The reply also contains: the frontmost app name; which window was read (window); every window that app has open (open_windows) — quote a piece of one of those titles as target.window in execute_plan to work in a window other than the frontmost; whether this window is already known to the interface graph (known_window); the screen size in both pixels (OCR/element coordinates) and logical points (click coordinates); which area was scanned for single-character labels (single_character_scan); and, when there is text on neither surface, how much of it there is (off_surface — counted, not listed, because no step can act on it). Each element reads 'text @ (cx,cy) [conf 0.97]' where (cx,cy) is the element center in pixels. Element text is what you target in execute_plan. A PNG screenshot of the whole display is attached for anything OCR cannot name (icons, images).

Execute a batch of semantic UI steps and return a compact journey. Each step is an object: {action, target?, path?, modifiers?, text?, keys?, amount?, app_name?, from_text?, to_text?, scroll?, until_readout?, skip_if_absent?, expect?, timeout_ms?}. action is one of click, double_click, right_click, drag, type, hotkey, scroll, wait, focus_app, read, read_clipboard. target={text, scope?, region?, window?, scroll_to_find?} selects an element by its visible text (region is one of top/bottom/left/right/center to disambiguate duplicates); matching is exact, then fuzzy, then semantic. click/double_click/right_click need a target; drag needs path; type needs text; hotkey needs keys (e.g. ['cmd','shift','4']); scroll needs amount (positive scrolls down; add a target to scroll over it); focus_app needs app_name; read takes an optional target (for scope/window/region) and read_clipboard takes nothing. BRINGING TEXT BACK: every other action changes the screen, these two hand you its contents. {action:'read'} returns the text of the window the step works in — laid out as it is drawn, so the columns of a table stay columns — in an 'extracted (step N, window "...")' block after the journey. It is never truncated: read what you need, all of it. On a read, target only says WHERE to read and its text must be EMPTY — {action:'read', target:{text:'', window:'Invoice'}} reads another window, text:'' with scope:'chrome' reads the menu bar or an open menu, and region reads one part of the pane. A read has no element to aim at, so naming one is refused rather than ignored; to bound the reading by text, use from_text/to_text. from_text and to_text narrow the reading to what lies between two pieces of visible text, both lines included — the way you would say 'the table from Subtotal down to Total'. They are matched like any target, and text that is not there fails the step like any missing target. scroll:true reads a document to its END instead of the screenful in front of you: the window is scrolled and re-read until nothing new comes up, the pieces are joined on their overlap with no duplicated lines, and the position is put back where the user had it. Use it for a long page, a chat history, a log; leave it off (the default) for a pane that is fully visible, since each screenful costs a parse. Those scrolls count against the plan's action budget, and the journey says how far it got and why it stopped. {action:'read_clipboard'} returns whatever text is on the system clipboard — the way to harvest text an app will not show in full: select it and copy it with ordinary steps ({action:'hotkey', keys:['cmd','a']}, then {action:'hotkey', keys:['cmd','c']}), then read_clipboard. An empty clipboard or one holding no text is reported in the journey, not treated as a failure. Choto never writes the clipboard, so anything you copy replaces what the user had there — copy only when you need to. STEPS THAT MAY NOT BE THERE: skip_if_absent:true (only on a step with a target) turns 'target not found' from a failure into a skipped step — the journey says 'skipped: target absent', the plan carries on, and the step's expect is not checked because nothing happened. This is for the noise of a real desktop: a cookie banner, an update prompt, a 'what's new' sheet that may or may not appear. Only use it where both branches end on the same screen; a step the plan actually depends on must be allowed to fail. GESTURES (drag): use it for what a click cannot express — rubber-band selection of several items, dragging a file onto a folder or into another pane, moving a slider, resizing a window or a column. path is a list of at least two waypoints: the button goes down on the first, travels through the rest and is released on the last (two is the usual case; more describe a bent route). Each waypoint is EITHER {target:{text,...}, offset?:{dx,dy}} — an element, optionally displaced by dx pixels right and dy pixels down from its center — OR {x, y}, a raw position in the same pixel coordinates the element listing reports. Prefer targets: they are resolved against the screen as it is when the gesture runs, so a list that scrolled still works. offset is how you name a spot with no text on it, which is what a rubber-band selection starts from: to select a block of files, start at the empty margin above-left of the first one, e.g. {target:{text:'report.pdf'}, offset:{dx:-40, dy:-25}}, and end past the last one, {target:{text:'notes.txt'}, offset:{dx:60, dy:25}}. All waypoints must be in the same window, and every one of them is resolved BEFORE anything is pressed: if one cannot be found the step escalates without the gesture starting. Be deliberate — a drag is undone far less often than a click: it moves files, drops them into folders, and reorders things, and there is no generic undo for that. DRAGGING TO A VALUE (until_readout): when what you want is a READING and not a distance — a playhead at 00:12:00, a slider at 50%, a level at -6 dB — do not compute pixels. Add until_readout={watch:{text,...}, value, tolerance?} to the drag: the button goes down on path[0], the pointer is pushed toward path[1] (which is only a DIRECTION and a first guess at how far, not the destination), and after every push the executor reads the instrument and corrects itself until it shows value. The path must be exactly two points. watch names the readout ITSELF, not a label beside it, and it has to be on screen before the gesture starts; value is written the way the app writes it (00:12:00, 50%, -6 dB, 1250) and every reading is then read in that same notation. tolerance (default 0, exact) is in those units — widen it for a control that skips the exact number. The journey says how many pushes it took and what the instrument ended on; if the reading cannot be read, or never arrives, the button is released where it stands and you get the last text that was there. modifiers=['cmd'|'shift'|'alt'|'ctrl'] are held for the whole action and work on click/double_click/right_click/drag: cmd-click to add one item to a selection, shift-click to extend it to a range, alt-drag to copy in Finder instead of moving. SURFACES: a target is looked for on exactly one of two surfaces, and they never mix. target.scope='window' (the default) searches only inside the window the step works in; target.scope='chrome' searches only the menu bar, status items and any open menu or popover. So clicking a menu item is two chrome steps: click {text:'File', scope:'chrome'}, then click {text:'Save As…', scope:'chrome'}. A target missing from the surface you named fails the step and the reply says which surface was searched and which was not — it is never answered from the other one, because the same text is often on both (a keypad '5' and a menu-bar '5'). With scope='chrome' target.window and target.scroll_to_find are rejected: chrome belongs to no window and is not scrolled. Every step works inside ONE window: by default the frontmost window of the app. target.window is a substring of a window's title (case-insensitive) and picks a different one — use it when the app has several windows open and the one you mean is not in front. If no open window matches, the step fails and the reply lists the titles that are open; it never falls back to another window. OFF-SCREEN TARGETS: if the text is not visible, the step does not fail immediately — the cursor is parked over the busiest part of the window and the list is scrolled a few steps down, then back and a few steps up, re-reading each time, so an item just below (or above) the fold is found the way a person would find it. The journey says what happened ('not on screen, scrolled 2x down -> matched ...'), and a search that finds nothing scrolls back to where it started before escalating. Those scrolls count against the plan's action budget. Set target.scroll_to_find=false where scrolling is not a harmless look: an endless feed, a list that loads more as it moves, or anywhere arriving elsewhere is itself a side effect. Drag waypoints are never scrolled for — the whole path is resolved before the button goes down, and scrolling would move the points already resolved. expect={appears?, disappears?, appears_count_increases?, screen_changes?} is a post-condition verified (with retries) within timeout_ms (default 5000); set several fields and all of them must hold. appears: this text must be on screen afterwards — use it when the step opens something new (a dialog, a page, a menu). disappears: this text must be gone afterwards — only for text that really leaves the screen (a closing dialog). Do NOT use it for text you cleared out of an input field: an emptied field shows its placeholder again, so the text is still on screen and the check fails even though the step worked. appears_count_increases: this text must match MORE on-screen lines than it did before the step. This is the right check for adding one more copy of something to a list — sending a chat message, adding a row, appending a tag — where the same text also stays visible elsewhere. Example: to send a message, {action:'type', text:'hi'} then {action:'hotkey', keys:['enter'], expect:{appears_count_increases:'hi'}}. screen_changes: true asserts the screen visibly changed (even slightly, e.g. a typed word), false asserts it did not. On success you get status=success and the journey. If a target cannot be resolved or an expectation is not met in time, execution stops and returns status=escalated with the failed step, a fresh screenshot, the element listing and the closest on-screen texts to the target that missed, so you can correct the plan and resend from the failure. If the user moves the mouse to the top-left corner the run returns status=aborted. EVERY reply — success, escalation or abort — ends with 'windows seen during this run': each distinct window the run passed through, with its app, its title and a sample of its text. That makes a deliberately short 'go and look' plan (focus_app, one click, a wait) the cheap way to explore an unfamiliar interface: you learn it in text instead of screenshots, and everything seen is written to memory, so recall can hand it back later without touching the screen at all. WORKING A MISSION: mission_item_id=N ties this run to item N of the active checklist (mission_start). A successful run marks that item passed and keeps these exact steps as its recipe, with a fingerprint of the world they worked in, so mission_replay(N) can repeat them later; any other outcome marks it failed with this run's own reason. Either way the reply ends with the mission status block — goal, progress, the item now in play — so a long scenario never rests on your memory of it. llm_version_tag is how you label the model driving the run: it is stored with the recipe and compared before a replay. Leave both out for an ordinary run.

Bring an application to the front by its name (e.g. 'System Settings', 'Safari'), launching it if it is not already running, and reopening a window if it is running with none. Use before observing or acting on an app you are not sure is frontmost. The name is matched case-insensitively against running apps: exact name first, then a unique prefix or substring, so 'Telegram' finds 'Telegram Lite'. The reply is 'focused ""' only after the app is confirmed frontmost; otherwise it explains what went wrong (ambiguous name — with the candidates listed, no such app, launch failed, or activation not confirmed).

Ask what Choto already remembers about an interface, as text — call this BEFORE observing or planning. A screenshot costs ~1400 tokens; this map costs a few hundred, so a window that has been visited before does not have to be looked at again to be planned against. The map is BY WINDOW and holds windows only: everything listed is inside one window, which is also where a plain target searches. Chrome — the menu bar, status items, open menus — is never remembered, so its absence here says nothing about the app; reach it with target.scope='chrome' in execute_plan, or see it as its own block in observe. All arguments are optional, and they widen the focus in three steps: with no arguments you get every application known, how many windows each has and when it was last seen; with app_name you get that app's windows plus the contents and outgoing transitions of the one it was last seen in; with app_name and window you get that one window — its text elements and its transitions, each reading 'click X -> window Y'. window is a piece of a window title (case-insensitive) or a node id such as '#5'; app_name matches exactly first, then as a substring ('settings' finds 'System Settings'). depth (default 1, max 3) unfolds that many levels of transitions; a branch leading back to a window already printed is marked '[shown above]' instead of being repeated, so cycles cost nothing. query= asks the other question — WHERE is this? — instead of walking the map yourself. It searches every element label, every named icon and every window title at once, with the same matcher a step's target uses, so a hit means a step aiming at that text would resolve to it. Each hit names the app, the window, what matched, how fresh it is, and the ROUTE to it. Add app_name to narrow the search. Routes read 'in #1 "Main" click "General" -> #2': that is the plan, in order, in the terms execute_plan takes. They start where the app was last seen — the window most recently read, which is where it still is unless something moved it since — unless you name from_window (a title or '#id'). When there is no route the reply names what is missing rather than saying no: which window cannot be reached, which recorded window leads into it (and whether that one was ever entered either), and what the starting window knows — how many windows it reaches, or that nothing has ever been clicked in it. That names the next thing to go and look at instead of leaving you to guess it. COVERAGE — read this before trusting an absence. Every reply carries what the map does NOT know: windows never clicked in, windows seen once, windows with no recorded way in, and 'N of M known control(s) ever clicked'. The denominator is what has been SEEN, never what the app has, so 'not found' means 'never seen here' and never 'does not exist' — the reply says so explicitly when a search comes back empty. A window that has never been clicked in is a photograph, not a map: what its buttons open is unknown, and the way to change that is to go and look. Reading the dates: the reply opens with the server's current time, and every window and transition carries when it was last seen, absolutely (UTC, '07-25 17:29Z', the year omitted while it is the current one) and relatively ('3h ago'). Nothing is hidden for being old, because only you can weigh it: minutes or hours old is normally safe to plan against; days old, or a window marked STALE, means the app may have been updated or rearranged since — either verify with one cheap step (a short plan that just navigates there, whose reply lists the windows it saw) or observe before acting. Element text is what you target in execute_plan; coordinates are deliberately absent, since the executor resolves text to pixels itself. The reply is size-capped and names what it cut ('elements (24 of 40)').

Ask what the wordless icons of an app are called — the first half of a round trip that has to happen once per drawing, ever. Choto's memory is text: an icon with no label is a place it knows how to click but cannot name, and observe reports those as 'icons: K labeled, M unlabeled'. This tool returns ONE picture (occasionally a few) holding every unnamed glyph of one application as a numbered grid at its real pixel size, plus a text legend giving each cell its context from memory: the app, the window it sits in, roughly where in that window (toolbar strip, sidebar column, bottom-right...), its size, the nearest few texts beside it, and any other windows the same drawing appears in. Read the picture, decide what each numbered control is, and send the answers with submit_icon_labels — that is the whole loop: observe -> annotate_icons -> submit_icon_labels -> click the label like any other text. app_name is optional: without it the app with the most-drawn unnamed glyph is chosen and named in the reply, because a glyph means whatever the program drawing it says it means and one sheet therefore covers one app. Nothing is captured and nothing is clicked — the crops come from memory, so this is safe to call while the user is working. If every glyph already has a name the reply says so plainly; the sheet is capped, and the reply says how many are left for a second call.

Send back what the icons on an annotate_icons sheet are called — the second half of the round trip. labels maps each cell's key (the string annotate_icons printed for that cell, NOT the cell number) to the name of the control, in the words a user would use for it ('Share', 'New tab', 'Delete'); app_name is the application annotate_icons reported. Every name is written onto the glyph AND onto every place it is already drawn on the map, and every future sighting of that drawing — in any window of that app — arrives labelled without asking again. After this, target it by the name you gave it: {action:'click', target:{text:'Share'}}. The batch is applied whole or not at all; a key that names no glyph, a blank name or a missing app_name comes back as a sentence saying what to fix rather than a partial write. The reply says how many glyphs and places were named and how many of that app's icons are still unnamed.

Let the application name its own icons: hover each unnamed icon of the FRONTMOST window and keep the tooltip it shows. This is the free half of icon labelling — annotate_icons asks you what a drawing is, this asks the program that drew it — so try it first on a window whose observe reply says 'icons: K labeled, M unlabeled'. A name it learns is written onto the drawing and onto every place that drawing appears in that app, exactly as submit_icon_labels writes yours, and from then on it is clickable as ordinary text: {action:'click', target:{text:'Share'}}. THIS ONE MOVES THE MOUSE. It is the only icon tool that touches the screen: the pointer visits each icon, waits out the tooltip delay (about a second and a half per icon, so a full call takes tens of seconds) and is put back where the user left it at the end. It works only on the window that is in front right now — bring the app forward with focus_app first — and it stops early, reporting how far it got, if the user pulls the mouse into the top-left corner, presses the stop button, or switches or moves the front window under it. Nothing is clicked and nothing is typed. limit (default 12, max 40) caps how many icons one call visits. Many controls have no tooltip at all; that is normal and the reply counts them ('showed none') — those are the ones to send to annotate_icons. The reply says how many were hovered, how many were named and which names were written, how many showed nothing, and how many drawings on that window still have no name.

Write down what a long run is for, before you lose it. Halfway through a twenty-window scenario the plan has scrolled out of your context and what is left is a confident recollection of it — a mission is that plan kept outside you, in the server's database. goal is what the whole run is for. items is the checklist in order, each {title, intent, acceptance}: what to do, WHY it is on the list, and what would make it done. All three are required and none may be blank — intent and acceptance are the two you will no longer have when you read this back, and they are the reason the record is worth keeping at all. THE LOOP: mission_start -> work item by item, running each with execute_plan(steps=[...], mission_item_id=N) so the run files its own verdict and its steps onto item N -> the status block comes back on every reply, so you never have to remember where you are -> repeat a recorded pass later with mission_replay(N) -> mission_finish when there is nothing left to try. One mission is active at a time: starting a second while one runs is refused, naming the one to finish or abandon first. The reply is the status block — goal, progress, the item now in play with its intent and acceptance, and any failures with their reasons.

Where the active mission stands. Without arguments it is the same compact block every execute_plan with a mission_item_id already returns — goal, progress, the current item in full, the failures — so call it when you have lost the thread, not after every run. full=true prints the whole checklist instead: every item with its id and status, nothing elided. That is the view to ask for before deciding a mission is done, and the only one that grows with the list, which is why it is a separate call.

Mark a checklist item yourself, for the acceptance criteria only you can check — something you read in a screenshot, in text a plan brought back, or judged from the reply. Runs mark their own items (execute_plan with mission_item_id), so use this when there was nothing to execute or when the executor's verdict was not the point. status is passed, failed or skipped. failed REQUIRES reason, and a reason is refused on anything else: 'item 4 failed' with no reason supports no plan except trying the same thing again, and a passed item carrying one asserts two contradictory things. skipped is for an item deliberately not attempted — a precondition never arose, the feature is absent in this build — and is not a defect. The reply is the status block.

Close the active mission: status=done when there is nothing left to try, abandoned when the checklist was not worked through. Failures do not stop a mission from being done — whether a run with three failed items is finished is your judgement, not a rule the server holds. Closing frees the one active slot, which is what mission_start needs. The reply is the whole checklist as a closing record; nothing is deleted, and the items keep their recipes and their reasons.

Run again, exactly, what already worked for one checklist item. When an item passed through execute_plan(mission_item_id=N), its steps were kept as that item's recipe together with a fingerprint of the world they worked in (the application, the model tag you gave, and the route of windows the run walked through). This replays that recipe — no re-planning, no screenshot, the same execution path any plan takes. BEFORE IT RUNS it compares the recorded world with the world now. If anything comparable differs it does NOT execute a single step: the reply names each field with both values and stops there. A difference is not evidence that the application was updated — it means a window was resized, a dialog is in the way, a different model is driving, or the app really did change, and only you can tell which. Decide and act: run the steps deliberately with execute_plan(mission_item_id=N), which records the world it succeeds in as the item's new truth, or mark the item by hand. Fields nobody can compare are reported too, so a clean preflight is never read as 'nothing changed'. AFTER a successful replay the route it actually walked is held against the recorded one; a different route is reported as a warning on a success, and the item's stored fingerprint is deliberately left as it was. llm_version_tag labels the model driving this replay and is compared with the one stored on the recipe. The reply is the run's journey, the ledger, the preflight notes and the mission status; a failed replay fails the item with the run's own reason.