findSimilarFrames
Identify visually similar frames in videos using Apple Vision image feature prints. Provide a reference frame to find matching scenes based on visual content analysis.
Instructions
Find frames that visually look like a reference frame using Apple Vision image feature prints. Accepts a frame assetId or a direct framePath and returns image-backed matches.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| assetId | No | Reference keyframe asset ID | |
| framePath | No | Reference image path on disk | |
| videoIdOrUrl | No | Optional video scope for similarity search | |
| maxResults | No | ||
| minSimilarity | No | ||
| dryRun | No |
Implementation Reference
- src/lib/visual-search.ts:606-653 (handler)The findSimilarFrames method within the VisualSearchEngine class, which performs the similarity search using Apple Vision feature vectors and cosine similarity.
async findSimilarFrames(params: FindSimilarFramesParams): Promise<FindSimilarFramesResult> { const reference = await this.resolveReferenceFrame(params); const referenceVector = reference.featureVector; if (!referenceVector || referenceVector.length === 0) { throw new Error("Reference frame does not have an Apple Vision feature vector. Re-index the frame or provide a valid image path."); } const candidates = this.store.listFrames({ videoId: params.videoId }).filter((frame) => { if (!existsSync(frame.framePath)) return false; if (!frame.featureVector || frame.featureVector.length === 0) return false; if (frame.framePath === reference.framePath) return false; return true; }); const minSimilarity = params.minSimilarity ?? 0.7; const results = candidates .map((frame) => ({ frame, similarity: cosineSimilarity(referenceVector, frame.featureVector ?? []) })) .filter((item) => item.similarity >= minSimilarity) .sort((a, b) => b.similarity - a.similarity) .slice(0, clamp(params.maxResults ?? 5, 1, 20)) .map(({ frame, similarity }) => ({ similarity: round(similarity, 4), videoId: frame.videoId, sourceVideoUrl: frame.sourceVideoUrl, sourceVideoTitle: frame.sourceVideoTitle, frameAssetId: frame.frameAssetId, framePath: frame.framePath, timestampSec: frame.timestampSec, timestampLabel: formatTimestamp(frame.timestampSec), explanation: `Apple Vision feature-print similarity ${round(similarity, 3)}${frame.visualDescription ? ` • ${truncate(frame.visualDescription, 140)}` : ""}`, ocrText: frame.ocrText, visualDescription: frame.visualDescription, } satisfies SimilarFrameMatch)); return { reference: { assetId: params.assetId, framePath: reference.framePath, videoId: reference.videoId, }, results, searchedFrames: candidates.length, limitations: [ "Similarity is image-to-image only. It finds frames that look alike using Apple Vision feature prints.", "Similarity search does not understand transcript text. It only compares visual frame features.", ], }; } - src/lib/visual-search.ts:137-141 (schema)Definition of the FindSimilarFramesResult interface describing the output structure.
export interface FindSimilarFramesResult { reference: { assetId?: string; framePath: string; videoId?: string; - src/lib/visual-search.ts:115-119 (schema)Definition of the FindSimilarFramesParams interface describing the input requirements.
export interface FindSimilarFramesParams { assetId?: string; framePath?: string; videoId?: string; maxResults?: number;