stream_latest
Retrieve the most recent stream frame from disk and return it as base64 for LLM context input. Use this to obtain pixel data only when necessary.
Instructions
Read the most recent frame of a stream from disk and return it as base64. Use sparingly - this is the path that actually puts pixels into the LLM context.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes |
Implementation Reference
- src/index.ts:213-231 (handler)The handler for the stream_latest tool. It extracts the 'id' argument, calls snapshotStream to get the last frame, reads the file from disk, encodes it as base64, and returns an image content item with metadata.
case "stream_latest": { const id = strArg(args, "id"); const snap = snapshotStream(id, { lastN: 1 }); if (!snap) return text({ error: `Unknown stream id: ${id}` }); const last = snap.frames.at(-1); if (!last) return text({ id, message: "no frames yet" }); const buf = await readFile(last.filePath); const base64 = buf.toString("base64"); return { content: [ { type: "image", data: base64, mimeType: mimeFor(last.format), }, { type: "text", text: JSON.stringify({ id, frame: last }, null, 2) }, ], }; } - src/index.ts:129-139 (schema)Schema and registration for the stream_latest tool. Defines the 'id' (string, required) input parameter and the description that it reads the most recent stream frame from disk and returns it as base64.
{ name: "stream_latest", description: "Read the most recent frame of a stream from disk and return it as base64. Use sparingly - this is the path that actually puts pixels into the LLM context.", inputSchema: { type: "object", required: ["id"], properties: { id: { type: "string" }, }, }, - src/index.ts:37-164 (registration)Tool registration in the ListToolsRequestSchema handler, which declares all tools including stream_latest.
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [ { name: "screenshot", description: "Capture a single screenshot of the desktop. Persists the file to disk and " + "optionally returns a base64 payload. Set cursorRadius>0 to crop a square " + "region around the mouse cursor instead of the full screen.", inputSchema: { type: "object", properties: { cursorRadius: { type: "integer", description: "If >0, crop a square of (2*radius)x(2*radius) px centered on the cursor. 0 = full screen.", default: 0, }, format: { type: "string", enum: ["png", "jpeg", "webp"], default: "jpeg" }, quality: { type: "integer", minimum: 1, maximum: 100, default: 82 }, maxEdge: { type: "integer", description: "Resize the longest edge to this many pixels. 0 disables resizing. " + "Default 2400 for full screen; with cursorRadius>0 the cursor crop is kept at native resolution unless overridden.", }, display: { type: "integer", description: "Optional display index for multi-monitor setups (omit for primary).", }, includeBase64: { type: "boolean", description: "If true, include the image bytes inline in the response. Default true.", default: true, }, }, }, }, { name: "cursor_info", description: "Return the current mouse cursor position, the foreground window title, and the " + "title of the window directly under the cursor (Windows only; other platforms " + "report position only when available).", inputSchema: { type: "object", properties: {} }, }, { name: "stream_start", description: "Start a periodic capture session. Saves frames to disk every intervalSeconds " + "for at most durationSeconds, keeping the last ringCapacity frames in memory. " + "Returns a session id used by stream_status / stream_latest / stream_stop. " + "Streams default to disk-only to keep LLM context lean - call stream_latest " + "with includeBase64=true when you actually want to look at a frame.", inputSchema: { type: "object", required: ["intervalSeconds", "durationSeconds"], properties: { intervalSeconds: { type: "number", minimum: 0.25, description: "Seconds between frames. Minimum 0.25.", }, durationSeconds: { type: "number", minimum: 0.25, description: "Total duration of the stream in seconds.", }, cursorRadius: { type: "integer", default: 0 }, format: { type: "string", enum: ["png", "jpeg", "webp"], default: "jpeg" }, quality: { type: "integer", minimum: 1, maximum: 100, default: 72 }, maxEdge: { type: "integer", description: "Longest edge in px. Default 1920 for full-screen frames; cursor crops keep native resolution unless overridden.", }, ringCapacity: { type: "integer", description: "Maximum number of recent frames kept in memory. Older frames are evicted (still on disk).", default: 60, }, }, }, }, { name: "stream_status", description: "Snapshot of a running or finished stream session - frame count, time remaining, last frames metadata.", inputSchema: { type: "object", required: ["id"], properties: { id: { type: "string" }, lastN: { type: "integer", default: 8 }, }, }, }, { name: "stream_latest", description: "Read the most recent frame of a stream from disk and return it as base64. Use sparingly - this is the path that actually puts pixels into the LLM context.", inputSchema: { type: "object", required: ["id"], properties: { id: { type: "string" }, }, }, }, { name: "stream_stop", description: "Stop a running stream early. Frames already on disk remain.", inputSchema: { type: "object", required: ["id"], properties: { id: { type: "string" } }, }, }, { name: "stream_list", description: "List active and completed stream sessions known to this process.", inputSchema: { type: "object", properties: {} }, }, { name: "stream_drop", description: "Forget a finished stream session (frees its in-memory ring; on-disk files are preserved).", inputSchema: { type: "object", required: ["id"], properties: { id: { type: "string" } }, }, }, ], - src/stream.ts:101-126 (helper)The snapshotStream helper function used by stream_latest. It retrieves a stream session by ID, extracts the last N frames (here N=1), and returns their metadata without base64 payloads (since the handler reads from disk directly).
export function snapshotStream( id: string, options: { withBase64?: boolean; lastN?: number } = {} ): (StreamSnapshot & { latestBase64?: string }) | null { const sess = sessions.get(id); if (!sess) return null; const lastN = options.lastN ?? Math.min(8, sess.frames.length); const slice = sess.frames.slice(-lastN); const summary: StreamSnapshot = { id: sess.id, startedAt: sess.startedAt, done: sess.done, remainingMs: Math.max(0, sess.stopAt - Date.now()), frameCount: sess.frames.length, capacity: sess.capacity, intervalMs: sess.intervalMs, error: sess.error, frames: slice.map(({ base64, ...rest }) => rest), }; if (options.withBase64 && slice.length > 0) { /* Most recent frame only - re-reading from disk avoids holding base64 in * the session object itself. */ return { ...summary, latestBase64: undefined }; } return summary; } - src/index.ts:309-313 (helper)The mimeFor helper function used to determine the MIME type for the image response based on the frame's format.
function mimeFor(format: string): string { if (format === "png") return "image/png"; if (format === "webp") return "image/webp"; return "image/jpeg"; }