deduplicate_strings
Eliminate redundant strings and retain semantically diverse subsets using Jina embeddings and submodular optimization. Ideal for deduplication, content sampling, or extracting unique representations from similar text data.
Instructions
Get top-k semantically unique strings from a list using Jina embeddings and submodular optimization. Use this when you have many similar strings and want to select the most diverse subset that covers the semantic space. Perfect for removing duplicates, selecting representative samples, or finding diverse content.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| k | No | Number of unique strings to return. If not provided, automatically finds optimal k by looking at diminishing return | |
| strings | Yes | Array of strings to deduplicate |
Input Schema (JSON Schema)
{
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"k": {
"description": "Number of unique strings to return. If not provided, automatically finds optimal k by looking at diminishing return",
"type": "number"
},
"strings": {
"description": "Array of strings to deduplicate",
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"strings"
],
"type": "object"
}