Remove Duplicates
remove_duplicatesMerges one or more Apify datasets and removes duplicate rows by chosen fields, applying case, whitespace, and URL normalization to produce deduplicated data.
Instructions
Returns the merged items with duplicates removed (or only the removed duplicates when output is 'duplicatesOnly'). Merges one or more Apify datasets and removes duplicate rows by the fields you choose (for example url, placeId or email), with case, whitespace and URL normalization. Give at least one of: datasetIds. Runs the Apify Actor ucchiii/dataset-dedup-merge on YOUR Apify account (APIFY_TOKEN); you pay that Actor's pay-per-event price, see https://apify.com/ucchiii/dataset-dedup-merge.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "remove_duplicates"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| keep | No | When duplicates are found, keep the 'first' or the 'last' occurrence (datasets are read in the order given). Default: first. | first |
| fields | No | Top-level fields to output. Empty = all fields. | |
| output | No | 'unique' returns the deduplicated items (default). 'duplicatesOnly' returns only the duplicates that were removed (for auditing). | unique |
| datasetIds | No | Dataset IDs (the 'defaultDatasetId' of a finished run of the source Actor, e.g. 'WkzbQMuFYuamGv3YF'). Dataset IDs or names to merge, in order. Items are read in this order (this matters for 'keep first/last').You can also paste the dataset URL from Apify Console or the API (the ID is taken from it). | |
| omitFields | No | Top-level fields to remove from the output. | |
| dedupFields | No | Items with the same values in ALL these fields are duplicates. Use dots for nested fields (e.g. ['url'] or ['address.city']). Empty = compare whole items. | |
| normalizeUrls | No | Treat http/https, www and a trailing slash as the same URL. | |
| addSourceField | No | Adds '_sourceDatasetId' to each output item. | |
| dedupEmptyKeys | No | By default, items whose dedup fields are all empty are kept (never treated as duplicates). | |
| trimWhitespace | No | Treat ' Acme Inc ' and 'Acme Inc' as the same value. | |
| caseInsensitive | No | Treat 'Acme' and 'acme' as the same value. |