execute_plan

Execute a batch of semantic UI steps and return a compact journey. Each step is an object: {action, target?, path?, modifiers?, text?, keys?, amount?, app_name?, from_text?, to_text?, scroll?, until_readout?, skip_if_absent?, expect?, timeout_ms?}. action is one of click, double_click, right_click, drag, type, hotkey, scroll, wait, focus_app, read, read_clipboard. target={text, scope?, region?, window?, scroll_to_find?} selects an element by its visible text (region is one of top/bottom/left/right/center to disambiguate duplicates); matching is exact, then fuzzy, then semantic. click/double_click/right_click need a target; drag needs path; type needs text; hotkey needs keys (e.g. ['cmd','shift','4']); scroll needs amount (positive scrolls down; add a target to scroll over it); focus_app needs app_name; read takes an optional target (for scope/window/region) and read_clipboard takes nothing. BRINGING TEXT BACK: every other action changes the screen, these two hand you its contents. {action:'read'} returns the text of the window the step works in — laid out as it is drawn, so the columns of a table stay columns — in an 'extracted (step N, window "...")' block after the journey. It is never truncated: read what you need, all of it. On a read, target only says WHERE to read and its text must be EMPTY — {action:'read', target:{text:'', window:'Invoice'}} reads another window, text:'' with scope:'chrome' reads the menu bar or an open menu, and region reads one part of the pane. A read has no element to aim at, so naming one is refused rather than ignored; to bound the reading by text, use from_text/to_text. from_text and to_text narrow the reading to what lies between two pieces of visible text, both lines included — the way you would say 'the table from Subtotal down to Total'. They are matched like any target, and text that is not there fails the step like any missing target. scroll:true reads a document to its END instead of the screenful in front of you: the window is scrolled and re-read until nothing new comes up, the pieces are joined on their overlap with no duplicated lines, and the position is put back where the user had it. Use it for a long page, a chat history, a log; leave it off (the default) for a pane that is fully visible, since each screenful costs a parse. Those scrolls count against the plan's action budget, and the journey says how far it got and why it stopped. {action:'read_clipboard'} returns whatever text is on the system clipboard — the way to harvest text an app will not show in full: select it and copy it with ordinary steps ({action:'hotkey', keys:['cmd','a']}, then {action:'hotkey', keys:['cmd','c']}), then read_clipboard. An empty clipboard or one holding no text is reported in the journey, not treated as a failure. Choto never writes the clipboard, so anything you copy replaces what the user had there — copy only when you need to. STEPS THAT MAY NOT BE THERE: skip_if_absent:true (only on a step with a target) turns 'target not found' from a failure into a skipped step — the journey says 'skipped: target absent', the plan carries on, and the step's expect is not checked because nothing happened. This is for the noise of a real desktop: a cookie banner, an update prompt, a 'what's new' sheet that may or may not appear. Only use it where both branches end on the same screen; a step the plan actually depends on must be allowed to fail. GESTURES (drag): use it for what a click cannot express — rubber-band selection of several items, dragging a file onto a folder or into another pane, moving a slider, resizing a window or a column. path is a list of at least two waypoints: the button goes down on the first, travels through the rest and is released on the last (two is the usual case; more describe a bent route). Each waypoint is EITHER {target:{text,...}, offset?:{dx,dy}} — an element, optionally displaced by dx pixels right and dy pixels down from its center — OR {x, y}, a raw position in the same pixel coordinates the element listing reports. Prefer targets: they are resolved against the screen as it is when the gesture runs, so a list that scrolled still works. offset is how you name a spot with no text on it, which is what a rubber-band selection starts from: to select a block of files, start at the empty margin above-left of the first one, e.g. {target:{text:'report.pdf'}, offset:{dx:-40, dy:-25}}, and end past the last one, {target:{text:'notes.txt'}, offset:{dx:60, dy:25}}. All waypoints must be in the same window, and every one of them is resolved BEFORE anything is pressed: if one cannot be found the step escalates without the gesture starting. Be deliberate — a drag is undone far less often than a click: it moves files, drops them into folders, and reorders things, and there is no generic undo for that. DRAGGING TO A VALUE (until_readout): when what you want is a READING and not a distance — a playhead at 00:12:00, a slider at 50%, a level at -6 dB — do not compute pixels. Add until_readout={watch:{text,...}, value, tolerance?} to the drag: the button goes down on path[0], the pointer is pushed toward path[1] (which is only a DIRECTION and a first guess at how far, not the destination), and after every push the executor reads the instrument and corrects itself until it shows value. The path must be exactly two points. watch names the readout ITSELF, not a label beside it, and it has to be on screen before the gesture starts; value is written the way the app writes it (00:12:00, 50%, -6 dB, 1250) and every reading is then read in that same notation. tolerance (default 0, exact) is in those units — widen it for a control that skips the exact number. The journey says how many pushes it took and what the instrument ended on; if the reading cannot be read, or never arrives, the button is released where it stands and you get the last text that was there. modifiers=['cmd'|'shift'|'alt'|'ctrl'] are held for the whole action and work on click/double_click/right_click/drag: cmd-click to add one item to a selection, shift-click to extend it to a range, alt-drag to copy in Finder instead of moving. SURFACES: a target is looked for on exactly one of two surfaces, and they never mix. target.scope='window' (the default) searches only inside the window the step works in; target.scope='chrome' searches only the menu bar, status items and any open menu or popover. So clicking a menu item is two chrome steps: click {text:'File', scope:'chrome'}, then click {text:'Save As…', scope:'chrome'}. A target missing from the surface you named fails the step and the reply says which surface was searched and which was not — it is never answered from the other one, because the same text is often on both (a keypad '5' and a menu-bar '5'). With scope='chrome' target.window and target.scroll_to_find are rejected: chrome belongs to no window and is not scrolled. Every step works inside ONE window: by default the frontmost window of the app. target.window is a substring of a window's title (case-insensitive) and picks a different one — use it when the app has several windows open and the one you mean is not in front. If no open window matches, the step fails and the reply lists the titles that are open; it never falls back to another window. OFF-SCREEN TARGETS: if the text is not visible, the step does not fail immediately — the cursor is parked over the busiest part of the window and the list is scrolled a few steps down, then back and a few steps up, re-reading each time, so an item just below (or above) the fold is found the way a person would find it. The journey says what happened ('not on screen, scrolled 2x down -> matched ...'), and a search that finds nothing scrolls back to where it started before escalating. Those scrolls count against the plan's action budget. Set target.scroll_to_find=false where scrolling is not a harmless look: an endless feed, a list that loads more as it moves, or anywhere arriving elsewhere is itself a side effect. Drag waypoints are never scrolled for — the whole path is resolved before the button goes down, and scrolling would move the points already resolved. expect={appears?, disappears?, appears_count_increases?, screen_changes?} is a post-condition verified (with retries) within timeout_ms (default 5000); set several fields and all of them must hold. appears: this text must be on screen afterwards — use it when the step opens something new (a dialog, a page, a menu). disappears: this text must be gone afterwards — only for text that really leaves the screen (a closing dialog). Do NOT use it for text you cleared out of an input field: an emptied field shows its placeholder again, so the text is still on screen and the check fails even though the step worked. appears_count_increases: this text must match MORE on-screen lines than it did before the step. This is the right check for adding one more copy of something to a list — sending a chat message, adding a row, appending a tag — where the same text also stays visible elsewhere. Example: to send a message, {action:'type', text:'hi'} then {action:'hotkey', keys:['enter'], expect:{appears_count_increases:'hi'}}. screen_changes: true asserts the screen visibly changed (even slightly, e.g. a typed word), false asserts it did not. On success you get status=success and the journey. If a target cannot be resolved or an expectation is not met in time, execution stops and returns status=escalated with the failed step, a fresh screenshot, the element listing and the closest on-screen texts to the target that missed, so you can correct the plan and resend from the failure. If the user moves the mouse to the top-left corner the run returns status=aborted. EVERY reply — success, escalation or abort — ends with 'windows seen during this run': each distinct window the run passed through, with its app, its title and a sample of its text. That makes a deliberately short 'go and look' plan (focus_app, one click, a wait) the cheap way to explore an unfamiliar interface: you learn it in text instead of screenshots, and everything seen is written to memory, so recall can hand it back later without touching the screen at all. WORKING A MISSION: mission_item_id=N ties this run to item N of the active checklist (mission_start). A successful run marks that item passed and keeps these exact steps as its recipe, with a fingerprint of the world they worked in, so mission_replay(N) can repeat them later; any other outcome marks it failed with this run's own reason. Either way the reply ends with the mission status block — goal, progress, the item now in play — so a long scenario never rests on your memory of it. llm_version_tag is how you label the model driving the run: it is stored with the recipe and compared before a replay. Leave both out for an ordinary run.