bizhawk_play_input_sequence
Play a series of joypad inputs frame-by-frame in a single round-trip, optionally capturing screenshots and memory reads at fixed intervals, and abort if a specified memory address changes.
Instructions
PURPOSE: Play a pre-built sequence of per-frame joypad inputs back-to-back, advancing one frame per element, ENTIRELY SERVER-SIDE in a single bridge round-trip. Optionally captures screenshots AND labeled memory reads at fixed frame intervals during the play, and optionally aborts play early when a specified memory address changes. All observations come back inline so the agent sees the full trajectory + game state in one tool response. USAGE: Use whenever you have ≥10 frames of inputs to play in order — TAS movie playback, scripted multi-frame sequences, AI-search of input patterns, agent-driven gameplay. ONE bridge round-trip ships N frames instead of the 2N round-trips you'd pay looping bizhawk_press_buttons + bizhawk_frame_advance(1). For sequences over ~200 frames, CHUNK. FOR AGENT-DRIVEN PLAY: combine screenshot_every, observe_memory, and stop_on_memory_change for the killer pattern — 'walk right for up to 200 frames, observing screenshot+x+y+hp every second, but STOP the moment the room ID changes'. The agent sees: where Samus was at each second, AND whether the goal (room transition) was reached, AND screenshots for visual confirmation — all in one tool response. BEHAVIOR: For each frames element, calls joypad.set with that frame's buttons then emu.frameadvance. The bridge's main poll loop is BLOCKED for the duration of the call (no other RPCs, no heartbeat) until the sequence finishes or fails. With screenshot_every, each captured screenshot adds ~1 frame of wall-clock. With observe_memory, each observation also reads the listed memory addresses at the same frame the screenshot was taken — values come back labeled by name in each observation's memory field. With stop_on_memory_change, the bridge records the listed address's value before the first frame, re-reads it after each frame, and aborts the sequence the moment it changes (a final observation is captured at the stop frame regardless of cadence). Returns an error if joypad.set / emu.frameadvance / client.screenshot is missing when needed, if any observe_memory entry references an unknown domain, if any width isn't 'u8' / 'u16' / 'u32', or if any address is out of range. RETURNS: A text summary ('Played N frames. Final framecount: M. Stopped early: yes/no [reason]. Captured K observations with their memory values') followed by K inline image content blocks (one per observation, in frame order). Each observation in the text summary includes its frame_offset and labeled memory values so the agent can correlate the visible screenshot with the game state at that exact frame.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| frames | Yes | Array of per-frame input objects. Each element describes ONE emulated frame: `{"buttons": {"Right": true, ...}, "player": 1}`. Empty `buttons` (or empty object) = no input on that frame. `player` defaults to 1 if omitted. Array order = frame playback order. Chunk longer sequences across multiple calls (≤200 frames each is a reasonable upper bound) to keep the bridge responsive. | |
| screenshot_every | No | Optional. If set, capture a PNG screenshot every N frames during playback (and one extra at the final frame regardless of remainder). Each screenshot costs ~1 wall-clock frame for client.screenshot, so 60 (≈1 sec of game time) is a good default — captures meaningful state changes without doubling batch latency. Omit to skip screenshots. If `observe_memory` is also set, screenshots and memory reads happen at the same observation points. | |
| screenshot_dir | No | Optional. Directory to write screenshot PNGs into when `screenshot_every` is set. Default: C:/temp. Must exist and be writable. Files are named `<prefix>-NNNN.png` where NNNN is the frame offset within the batch (zero-padded to 4 digits). | |
| screenshot_prefix | No | Optional. Filename prefix for screenshots when `screenshot_every` is set. Default: 'obs'. | |
| observe_memory | No | Optional. List of memory reads to perform at each observation point (alongside screenshots if `screenshot_every` is also set). Each result lands in the observation's `memory` field keyed by `name`. Use this to track game-state values per observation — e.g. on Super Metroid, track HP/X/Y/room-ID at each screenshot so you see how state changes across the play batch. | |
| stop_on_memory_change | No | Optional. If set, the bridge reads the specified memory value before the first frame, re-reads it after every frame, and ABORTS the play sequence the moment it changes. The result will have `stopped_early: true` and `stop_reason: 'memory_changed'`. A final observation is captured at the stop frame even if it's not on the normal cadence. Killer use case: watch the room ID — Samus walks through a door, room ID changes, play stops at the exact frame of the transition. |