keyboard
Send text, key combos, or multi-step sequences to a target window. Focuses and guards the window to automate typing, shortcuts, and menu navigation.
Instructions
Purpose: Send keyboard input to a window: 'type' for text, 'press' for key combos, 'sequence' for atomic multi-step chords. Details: action='type' inserts text (auto-clipboard for non-ASCII, bypassing IME conversion). action='press' sends key combos like 'ctrl+c'/'alt+tab'. action='sequence' runs ordered steps in one keyboard lock — use for Alt+letter, letter mnemonic chains where intermediate tool calls would close the menu. windowTitle or hwnd is REQUIRED (blank/whitespace counts as neither) — the server focuses and auto-guards that window (identity, foreground, modal) first, and a call with neither stops with DestinationRequired before any key is sent. Use windowTitle:'@active' to aim at the foreground window on purpose; an hwnd naming a titleless window works only while that window is already foreground. DESKTOP_TOUCH_REQUIRE_DESTINATION=0 downgrades the stop to a warning. Prefer: Set lensId for perception guards. Use desktop_act({action:'setValue'}) for UIA ValuePattern text fields. Caveats: win+r/win+x/win+s/win+l blocked. action='type' does not handle CJK IME composition — use use_clipboard=true or desktop_act({action:'setValue'}); neither lands while an IME composition is pending — commit or cancel it first. hints.clipboard reports the backend and whether the clipboard was restored. Non-ASCII text (CJK / emoji / diacritics / smart-quote-class punctuation) auto-clipboards to prevent silent-drop and Chrome accelerator hijack; pass forceKeystrokes:true to disable. Background (PostMessage/WM_CHAR) auto-engages for terminal-class windows (Windows Terminal / cmd / PowerShell); DTM_BG_AUTO=1 enables globally. Foreground non-terminal type runs a per-chunk leash; user focus-steal mid-stream aborts with FocusLostDuringType + context.typed/remaining; pass abortOnFocusLoss:false to disable. BG type verifies WM_CHAR via UIA TextPattern read-back; mismatch returns BackgroundInputNotDelivered (see SUGGESTS for false-positive notes). BG press read-back is scoped to terminal-class + enter/tab/arrow; other combos return verifyDelivery:'unverifiable', failure returns BackgroundKeyNotDelivered. action='sequence' is FG-only (BG/foreground_flash schema-rejected); emits verifyDelivery:'focus_only'; mid-loop focus theft returns MenuFocusLostMidSequence + context.remaining: Step[]. Win11 FG refusal returns ForegroundRestricted — terminal-class targets auto-engage BG (except a combo with ctrl/shift/alt and type with replaceAll, which go through FG and can return ForegroundRestricted too); non-terminal switch to desktop_act / click_element. Off-desktop target → WindowOnOtherDesktop, not sent. Examples: keyboard({action:'type', text:'hello', windowTitle:'Untitled - Notepad'}) → text injected (guarded) keyboard({action:'type', text:'hello', windowTitle:'@active'}) → typed into the foreground window keyboard({action:'press', keys:'ctrl+c', windowTitle:'Untitled - Notepad'}) → copy keyboard({action:'press', keys:'escape', windowTitle:'Dialog'}) → dismiss dialog keyboard({action:'sequence', steps:[{keys:'alt+i', gapMs:100},{keys:'m'}], windowTitle:'Microsoft Visual Basic'}) → Insert > Module (atomic)
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| hwnd | No | Direct window handle ID (takes precedence over windowTitle). Either this or windowTitle is required. Obtain from desktop_discover response (windows[].hwnd). String type to avoid 64-bit precision issues. A window that has no title can be addressed this way, but only while it is already the foreground window — keyboard focus and guarding cannot target a titleless window yet. | |
| keys | No | Key combo string, e.g. 'ctrl+c', 'alt+tab', 'enter', 'ctrl+shift+s'. Note: win+r, win+x, win+s, win+l are blocked for security. | |
| text | No | The text to type (max 10,000 characters) | |
| fixId | No | 'type': Approve a pending suggestedFix (one-shot, 15s TTL). Pass the fixId returned by a previous failed keyboard(action='type') to re-attempt with guard-validated args. 'sequence': Approve a pending suggestedFix (one-shot, 15s TTL). Only meaningful for GUARD-pre-loop rejections (e.g. unsafe.keyboardTarget). Mid-loop MenuFocusLostMidSequence does NOT issue fixIds — recover by re-calling with context.remaining. | |
| steps | No | Ordered list of key-press steps. Min 1, max 16. Total duration must not exceed 5000ms (excludes settleMs and focus acquisition). N=1 is allowed but inherits the sequence verification contract (hints.verifyDelivery.status='focus_only'); if you want the stricter keyboard:press contract, call keyboard({action:'press', keys}) directly (issue #278, matrix doc §3.1). | |
| action | Yes | Action selector — one of: type, press, sequence. Per-action required fields are enforced at call time (see the tool description); this flat schema lists every action's fields as optional. | |
| lensId | No | 'type': Optional perception lens ID. Guards (safe.keyboardTarget) are evaluated before typing, and a perception envelope is attached to post.perception on success. 'press': Optional perception lens ID. Guards (safe.keyboardTarget) are evaluated before the key press. 'sequence': Optional perception lens ID. Guards (safe.keyboardTarget) are evaluated once before the first step. | |
| method | No | 'type' / 'press': Input routing channel. 'auto' uses background (PostMessage) when the target window is a known terminal class (Windows Terminal / cmd / PowerShell) OR DTM_BG_AUTO=1 is set; else foreground. Terminal auto-detect is HWND-targeted so user-side focus changes mid-stream cannot divert keystrokes (except a combo with ctrl/shift/alt, or type with replaceAll, which 'auto' sends through the foreground). 'background' forces PostMessage-only (no focus change, fails on Chromium/IME). A key combo with ctrl, shift or alt, and type's replaceAll (which selects with Ctrl+A), cannot be posted — the app would see the plain key and type it — so 'background' refuses them with BackgroundModifierComboUnsupported and nothing is sent; 'auto' sends them through the foreground. 'foreground' forces the current behavior (SetForegroundWindow + keystrokes). 'foreground_flash' (ADR-013 Option E) is an explicit opt-in 妥協 BG path for Windows Terminal: temporarily steals foreground (~50-80ms), pastes via clipboard, sends Ctrl+V, restores foreground + clipboard. Single-line + < 5KiB only. Carries `typingLeakRisk: true` in hints because user keystrokes during the flash window can leak to WT. Default 'auto'. 'sequence': Sequence is foreground-only by design — Alt-menu mnemonics need real SendInput. Omit, or pass 'foreground'. method:'background' / 'foreground_flash' are rejected at schema parse time (typed codes BackgroundNotApplicableToSequence / ForegroundFlashNotApplicableToSequence document the rationale for LLMs). | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| narrate | No | Narration level. "rich": include UIA diff in post.rich (appeared/disappeared/valueDeltas, and nameDeltas when an element's name changed, such as a calculator display) — usually removes the need for a verification screenshot. It is withheld, with post.rich.diffDegraded saying why, when the diff cannot be shown to describe the window that was acted on. On click_element and keyboard (and set_element_value where the server registers it), which resolve the target window before acting: another open window's title contains the text this call resolved to, whether you named the hwnd or the server did — "@active" and the dialog rescue both resolve one ("ambiguous_title"); or the target moved between the snapshot and the action ("target_changed"). On those and on mouse_click, retrying with a fixId also withholds it, because the stored fix names a window this cannot see ("fix_target_unknown") — browser tools keep their diff on a fixId retry, because theirs is a tab diff and does not depend on a window title. mouse_click and mouse_drag accept an hwnd and still take their snapshots by windowTitle, so nothing else is withheld for them: verify those with a screenshot. A window too large to read whole is withheld too ("tree_truncated"). The action itself is unaffected in every case; only the diff is. Default: "minimal". | minimal |
| settleMs | No | Milliseconds to wait after the action before checking foreground window (default 300). | |
| forceFocus | No | When true, bypass Windows foreground-stealing protection via AttachThreadInput before focusing the target window. Default: follows env DESKTOP_TOUCH_FORCE_FOCUS (default false). | |
| replaceAll | No | When true, send Ctrl+A to select all existing text before typing. Equivalent to Ctrl+A → keyboard(action='type') in one call (requires field already focused). Default false. | |
| trackFocus | No | When true (default), detect if focus was stolen from the target window after the action. Reports focusLost in the response. Set false to skip. | |
| forceImeOff | No | 'type': Issue #245 系統②: when true, query the target window's IME open-status via Imm32 before typing; if ON, switch OFF for the duration of this call and restore the prior state in `finally`. Prevents silent romaji conversion when the user's Japanese IME is active but the LLM is typing ASCII commands. Requires `windowTitle` or `hwnd` (otherwise no target to query). Default false — existing use_clipboard auto-promotion still handles non-ASCII symbols transparently. No-op when the addon predates the IMM bridge (call proceeds with whatever IME state is in effect). 'sequence': Issue #245 系統②: query the target's IME open-status before the first step; if ON, switch OFF for the whole sequence and restore in finally. Prevents Alt-mnemonic hijack when 日本語 IME is active (the OS routes Alt+letter to IME composition instead of the menu). Requires windowTitle or hwnd. Default false. | |
| windowTitle | No | Partial title of the window that should receive the keystrokes. Required unless you pass hwnd: a call with neither stops with DestinationRequired before any key is sent, and an empty or whitespace-only value counts as neither. The server focuses this window before typing and uses it as the expected target for focusLost detection. Use '@active' to target the current foreground window on purpose. | |
| use_clipboard | No | If true, copy text to clipboard and paste with Ctrl+V instead of simulating keystrokes. Use this when typing URLs, paths, or ASCII text into apps with Japanese IME active — pasted text is not run through IME conversion. Note this does not help while an IME composition is already in progress: the paste keystroke is consumed by the IME and nothing is inserted, though the call still answers ok, so commit or cancel the composition first; when the IME was on at the paste, hints.clipboard.imeOpen says so. Your clipboard is replaced for the duration of the call and put back afterwards; hints.clipboard reports which backend served the paste and whether the restore ran. On builds without the native addon this path is capped at about 12000 characters and fails with code:'ClipboardWriteTooLargeForFallback' above it. Default false. | |
| forceKeystrokes | No | When true, always use keystroke mode even if text contains non-ASCII content (CJK, emoji, diacritics, em-dash, smart quotes, etc.) that would normally trigger auto-clipboard. Default false — auto-clipboard is enabled. | |
| abortOnFocusLoss | No | Focus Leash Phase B: when true, the foreground keystroke send is split into chunks (default 8 chars; override via DTM_LEASH_CHUNK_SIZE env) and the target window's foreground state is verified between chunks. If the user grabs focus mid-stream, the call aborts and returns FocusLostDuringType with context.typed (chars delivered to target) and context.remaining (unsent tail) so the caller can re-focus and retry the unsent portion. Default: true when windowTitle is provided, false otherwise. Has no effect on the clipboard path (atomic Ctrl+V) or the BG (WM_CHAR) path (HWND-targeted, foreground-independent). |