Remove duplicates
remove_duplicatesRemove duplicate lines from a text list, or duplicate rows from CSV/TSV text such as rows copied from Excel or Google Sheets. Uses the same engine as removeduplicates.org and returns the cleaned text with counts. Stateless: the text is processed in memory for this call only and is never stored.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| keep | No | Which occurrence of a duplicate survives. | first |
| text | Yes | The list or table text: one item per line, or CSV/tab-separated rows. Up to 131072 UTF-8 bytes. | |
| trim | No | Ignore surrounding spaces when comparing and return trimmed values. | |
| order | No | preserve keeps the original order; sort returns a natural (numeric-aware) sort. | preserve |
| format | No | auto detects CSV/TSV tables; lines compares whole lines; table forces CSV/TSV parsing. | auto |
| header | No | Tables only: keep the first row as a header outside duplicate checks and counts. | |
| compare | No | Tables only: "row" compares entire rows; a zero-based column index or an array of indices compares only those columns. | row |
| ignoreCase | No | Treat values such as Apple and apple as duplicates. | |
| removeEmpty | No | Drop blank lines or all-blank rows. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | The cleaned text, rows joined with \n. | |
| stats | Yes | ||
| format | Yes |