get_frame_at

Instructions

Implementation Reference

• src/tools/get-frame-at.ts:11-21 (schema)

  Zod schema for get_frame_at tool: validates url (string URL), timestamp (e.g., '1:23'), and optional returnBase64 boolean.

  const GetFrameAtSchema = z.object({
    url: z.string().url().describe('Video URL (Loom share link or direct mp4/webm URL)'),
    timestamp: z
      .string()
      .describe('Timestamp to extract frame at (e.g., "1:23", "0:05", "01:23:45")'),
    returnBase64: z
      .boolean()
      .default(false)
      .optional()
      .describe('Return frame as base64 inline instead of file path'),
  });

• src/tools/get-frame-at.ts:23-104 (handler)

  Full tool handler registration and execution logic for 'get_frame_at'. Registers the tool via server.addTool with name, description, schema, and execute function. The execute function attempts Strategy 1 (download video + ffmpeg extraction) and falls back to Strategy 2 (browser-based extraction).

  export function registerGetFrameAt(server: FastMCP): void {
    server.addTool({
      name: 'get_frame_at',
      description: `Extract a single video frame at a specific timestamp.
  
  Useful for inspecting what's on screen at a particular moment. The AI reads the transcript,
  identifies a critical moment, and requests the exact frame at that timestamp.
  
  Supports: Loom (loom.com/share/...) and direct video URLs (.mp4, .webm, .mov).
  Requires video download capability — direct URLs work best.
  
  Args:
    - url: Video URL
    - timestamp: Time position (e.g., "1:23", "0:05", "01:23:45")
  
  Returns: A single image of the video frame at the specified timestamp.`,
      parameters: GetFrameAtSchema,
      annotations: {
        title: 'Get Frame at Timestamp',
        readOnlyHint: true,
        destructiveHint: false,
        idempotentHint: true,
        openWorldHint: true,
      },
      execute: async (args, { reportProgress }) => {
        const progress = createProgressReporter(reportProgress);
        const { url, timestamp } = args;
  
        const adapter = getAdapter(url);
  
        await progress(0, 'Starting frame extraction...');
  
        const tempDir = await createTempDir();
  
        // Strategy 1: Download video + ffmpeg extraction
        if (adapter.capabilities.videoDownload) {
          const videoPath = await adapter.downloadVideo(url, tempDir);
  
          if (videoPath) {
            await progress(50, `Extracting frame at ${timestamp}...`);
  
            const frame = await extractFrameAt(videoPath, tempDir, timestamp);
            const optimizedPath = getTempFilePath(tempDir, `opt_frame_at.jpg`);
            await optimizeFrame(frame.filePath, optimizedPath);
  
            await progress(100, 'Frame extracted');
  
            return {
              content: [
                { type: 'text' as const, text: `Frame extracted at ${timestamp}` },
                await imageContent({ path: optimizedPath }),
              ],
            };
          }
        }
  
        // Strategy 2: Browser-based extraction (fallback)
        await progress(30, 'Extracting frame via browser fallback...');
        const seconds = parseTimestamp(timestamp);
        const browserFrames = await extractBrowserFrames(url, tempDir, {
          timestamps: [seconds],
        });
  
        if (browserFrames.length > 0) {
          await progress(100, 'Frame extracted');
          return {
            content: [
              {
                type: 'text' as const,
                text: `Frame extracted at ${timestamp} (via browser)`,
              },
              await imageContent({ path: browserFrames[0].filePath }),
            ],
          };
        }
  
        throw new UserError(
          'Failed to extract frame. Install yt-dlp or Chrome/Chromium for frame extraction.',
        );
      },
    });
  }

• src/tools/get-frame-at.ts:1-9 (helper)

  Imports: getAdapter for URL adapter resolution, extractFrameAt/parseTimestamp from frame-extractor for core ffmpeg logic, extractBrowserFrames as fallback, optimizeFrame for JPEG optimization, createProgressReporter, and createTempDir/getTempFilePath for temp file management.

  import type { FastMCP } from 'fastmcp';
  import { UserError, imageContent } from 'fastmcp';
  import { z } from 'zod';
  import { getAdapter } from '../adapters/adapter.interface.js';
  import { extractBrowserFrames } from '../processors/browser-frame-extractor.js';
  import { extractFrameAt, parseTimestamp } from '../processors/frame-extractor.js';
  import { optimizeFrame } from '../processors/image-optimizer.js';
  import { createProgressReporter } from '../utils/progress.js';
  import { createTempDir, getTempFilePath } from '../utils/temp-files.js';

• src/processors/frame-extractor.ts:144-168 (helper)

  Core helper function extractFrameAt: uses ffmpeg to seek to a specific timestamp (-ss), extract a single frame (-frames:v 1), and save as JPEG. Parses the timestamp string to seconds, then runs ffmpeg with timeout.

  export async function extractFrameAt(
    videoPath: string,
    outputDir: string,
    timestamp: string,
  ): Promise<IFrameResult> {
    const seconds = parseTimestamp(timestamp);
    const outputPath = ffmpegPath_(join(outputDir, `frame_at_${seconds}.jpg`));
  
    try {
      await execFile(
        ffmpegPath,
        ['-ss', String(seconds), '-i', videoPath, '-frames:v', '1', '-q:v', '2', outputPath, '-y'],
        { timeout: 30000 },
      );
    } catch (error: unknown) {
      const msg = error instanceof Error ? error.message : String(error);
      throw new Error(`Frame extraction at ${timestamp} failed: ${msg}`, { cause: error });
    }
  
    return {
      time: timestamp,
      filePath: outputPath,
      mimeType: 'image/jpeg',
    };
  }

• src/processors/frame-extractor.ts:18-31 (helper)

  Helper function parseTimestamp: converts timestamp strings like '1:23' or '01:23:45' into total seconds for ffmpeg seeking.

  export function parseTimestamp(ts: string): number {
    const parts = ts.split(':').map(Number);
    if (parts.some((p) => isNaN(p))) {
      throw new Error(`Invalid timestamp format: "${ts}"`);
    }
  
    if (parts.length === 3) {
      return parts[0] * 3600 + parts[1] * 60 + parts[2];
    }
    if (parts.length === 2) {
      return parts[0] * 60 + parts[1];
    }
    throw new Error(`Invalid timestamp format: "${ts}". Expected "M:SS" or "H:MM:SS".`);
  }