text_duplicate_line_remover
Remove duplicate lines from text while keeping the first occurrence. Options include case-sensitive matching, whitespace trimming, empty-line retention, and alphabetical sorting.
Instructions
Duplicate Line Remover. Remove duplicate LINES from text, keeping the first occurrence of each, with options for case-sensitive matching, whitespace trimming, empty-line handling, and alphabetical sorting of the result. Operates on whole lines split on newlines - use text_duplicate_word_remover to dedupe individual words, text_remove_duplicate_characters to dedupe characters, or text_sort_lines to only reorder lines without removing duplicates. Runs locally on the text you provide: read-only, non-destructive, contacts no external service, and is rate-limited (60 requests/minute for anonymous callers). Returns the deduplicated text plus before/after line statistics and a top-10 list of the most repeated lines.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| text | No | Multi-line text to deduplicate, split on newline. Blank input returns empty output with zeroed stats. | |
| caseSensitive | No | When true, lines differing only in letter case are kept as distinct; when false (default), comparison is case-insensitive. | |
| trimWhitespace | No | When true (default), leading/trailing whitespace is stripped before comparing and in output; when false, whitespace is significant. | |
| keepEmptyLines | No | When true, blank lines are preserved and deduplicated; when false (default), all empty lines are dropped. | |
| sortResults | No | When true, surviving unique lines are sorted alphabetically (case-aware per caseSensitive); when false (default), original order is preserved. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| success | No | Whether deduplication succeeded. | |
| result | No | The deduplicated text, unique lines joined by newline. | |
| stats | No | Before/after line counts and reduction metric. | |
| duplicateAnalysis | No | Up to 10 most-repeated lines, sorted by descending count. | |
| options | No | The effective options after defaults were applied. |