Text similarity and near-duplicates
ai_text_similarityCompare two or more texts or files to measure wording overlap with Jaccard, MinHash, and cosine scores; rank similar pairs or group near-duplicates using a threshold.
Instructions
Compare two or more texts (or text files) for overlap. For every pair returns: Jaccard similarity of word shingles (k=3 words by default), a MinHash estimate of that Jaccard (128 hashes), and cosine similarity of word frequencies. Words are lower-cased, punctuation and Arabic diacritics ignored. Measures wording overlap, not meaning (no embeddings): paraphrases score low. Mode "pairs" (default) lists pairs sorted by the chosen metric; mode "dedupe" groups near-duplicates whose metric is at least "threshold" (default 0.8; groups are connected, so AB and BC put A, B, C together). Give "texts" (inline strings) and/or "paths" (file paths or base64). Up to 1000 texts.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | "pairs" (similarity of each pair) or "dedupe" (groups of near-duplicates). | pairs |
| paths | No | Or/also: UTF-8 text files (local paths or base64). | |
| texts | No | Texts to compare (inline). | |
| labels | No | Optional names for the texts, in order (texts first, then paths). Default: "#1", "#2"… or the file name. | |
| metric | No | Metric used to sort pairs and to group duplicates. Default jaccard. | jaccard |
| maxPairs | No | Most pairs to return (highest first). Default 200. | |
| numHashes | No | MinHash signature length. Default 128 (error about ±0.09). | |
| threshold | No | Dedupe: pairs at or above this are duplicates. Default 0.8. | |
| shingleSize | No | Words per shingle (k). Default 3. |