text_duplicate_word_remover
Removes duplicate words from text, keeping the first occurrence and discarding repeats. Supports case-insensitive comparison, punctuation stripping, sorting, and customizable output formats.
Instructions
Duplicate Word Remover (Deduplicate Words In Text). Removes duplicate words from text, keeping the first occurrence of each word and discarding later repeats. Words are split on whitespace. Use this to dedupe a word list or tags; use text_duplicate_line_remover to dedupe whole lines and text_remove_duplicate_characters to dedupe individual characters. Runs locally on the text you provide: read-only, non-destructive, contacts no external service, and is rate-limited (60 requests/min anonymous). Returns the deduplicated text, before/after statistics (original, unique, removed, reduction percent), and a top-15 analysis of the most-repeated words.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | The text to deduplicate. Split into words on any run of whitespace. | |
| caseSensitive | No | When true, Cat and cat are treated as different words; when false, comparison is case-insensitive. | |
| removePunctuation | No | When true, punctuation is stripped before comparing words so cat. and cat match. | |
| sortResults | No | When true, the surviving unique words are sorted alphabetically; when false, original order is preserved. | |
| outputFormat | No | How to join the unique words in the output: spaces joins with single spaces, lines joins with newlines, commas joins with a comma and space. | spaces |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| success | No | Whether deduplication succeeded. | |
| result | No | The deduplicated output text joined per outputFormat. | |
| stats | No | Before/after word counts. | |
| duplicateAnalysis | No | Up to 15 most-repeated words, sorted by count descending. | |
| options | No | The effective options applied (caseSensitive, removePunctuation, sortResults, outputFormat). |