dedup_check
Identifies accidentally duplicated or re-ingested content by computing chunk hashes and reporting file pairs with high chunk overlap.
Instructions
Detect near-duplicate documents in the index by computing content hashes for every chunk. Returns file pairs with high chunk overlap, sorted by similarity. Use to identify accidentally duplicated or re-ingested content.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| threshold | No | Similarity threshold (0.5 = 50% chunk overlap, default 0.8). Only pairs above this threshold are reported. |