findSimilarFrames

Identify visually similar frames in videos using Apple Vision image feature prints. Provide a reference frame to find matching scenes based on visual content analysis.

Instructions

Find frames that visually look like a reference frame using Apple Vision image feature prints. Accepts a frame assetId or a direct framePath and returns image-backed matches.

Input Schema

TableJSON Schema

Name	Required	Description
`assetId`	No	Reference keyframe asset ID
`framePath`	No	Reference image path on disk
`videoIdOrUrl`	No	Optional video scope for similarity search
`maxResults`	No
`minSimilarity`	No
`dryRun`	No

Implementation Reference

src/lib/visual-search.ts:606-653 (handler)

The findSimilarFrames method within the VisualSearchEngine class, which performs the similarity search using Apple Vision feature vectors and cosine similarity.

async findSimilarFrames(params: FindSimilarFramesParams): Promise<FindSimilarFramesResult> {
  const reference = await this.resolveReferenceFrame(params);
  const referenceVector = reference.featureVector;
  if (!referenceVector || referenceVector.length === 0) {
    throw new Error("Reference frame does not have an Apple Vision feature vector. Re-index the frame or provide a valid image path.");
  }

  const candidates = this.store.listFrames({ videoId: params.videoId }).filter((frame) => {
    if (!existsSync(frame.framePath)) return false;
    if (!frame.featureVector || frame.featureVector.length === 0) return false;
    if (frame.framePath === reference.framePath) return false;
    return true;
  });

  const minSimilarity = params.minSimilarity ?? 0.7;
  const results = candidates
    .map((frame) => ({ frame, similarity: cosineSimilarity(referenceVector, frame.featureVector ?? []) }))
    .filter((item) => item.similarity >= minSimilarity)
    .sort((a, b) => b.similarity - a.similarity)
    .slice(0, clamp(params.maxResults ?? 5, 1, 20))
    .map(({ frame, similarity }) => ({
      similarity: round(similarity, 4),
      videoId: frame.videoId,
      sourceVideoUrl: frame.sourceVideoUrl,
      sourceVideoTitle: frame.sourceVideoTitle,
      frameAssetId: frame.frameAssetId,
      framePath: frame.framePath,
      timestampSec: frame.timestampSec,
      timestampLabel: formatTimestamp(frame.timestampSec),
      explanation: `Apple Vision feature-print similarity ${round(similarity, 3)}${frame.visualDescription ? ` • ${truncate(frame.visualDescription, 140)}` : ""}`,
      ocrText: frame.ocrText,
      visualDescription: frame.visualDescription,
    } satisfies SimilarFrameMatch));

  return {
    reference: {
      assetId: params.assetId,
      framePath: reference.framePath,
      videoId: reference.videoId,
    },
    results,
    searchedFrames: candidates.length,
    limitations: [
      "Similarity is image-to-image only. It finds frames that look alike using Apple Vision feature prints.",
      "Similarity search does not understand transcript text. It only compares visual frame features.",
    ],
  };
}

src/lib/visual-search.ts:137-141 (schema)

Definition of the FindSimilarFramesResult interface describing the output structure.

export interface FindSimilarFramesResult {
  reference: {
    assetId?: string;
    framePath: string;
    videoId?: string;

src/lib/visual-search.ts:115-119 (schema)

Definition of the FindSimilarFramesParams interface describing the input requirements.

export interface FindSimilarFramesParams {
  assetId?: string;
  framePath?: string;
  videoId?: string;
  maxResults?: number;

vidlens-mcp