Skip to main content
Glama
Internamaro Apify Data MCP

Internamaro Apify Data MCP

Remove Duplicates

remove_duplicates
Read-only

Merges one or more Apify datasets and removes duplicate rows by chosen fields, applying case, whitespace, and URL normalization to produce deduplicated data.

Instructions

Returns the merged items with duplicates removed (or only the removed duplicates when output is 'duplicatesOnly'). Merges one or more Apify datasets and removes duplicate rows by the fields you choose (for example url, placeId or email), with case, whitespace and URL normalization. Give at least one of: datasetIds. Runs the Apify Actor ucchiii/dataset-dedup-merge on YOUR Apify account (APIFY_TOKEN); you pay that Actor's pay-per-event price, see https://apify.com/ucchiii/dataset-dedup-merge.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "remove_duplicates"
  }
}

Input Schema

NameRequiredDescriptionDefault
keepNoWhen duplicates are found, keep the 'first' or the 'last' occurrence (datasets are read in the order given). Default: first.first
fieldsNoTop-level fields to output. Empty = all fields.
outputNo'unique' returns the deduplicated items (default). 'duplicatesOnly' returns only the duplicates that were removed (for auditing).unique
datasetIdsNoDataset IDs (the 'defaultDatasetId' of a finished run of the source Actor, e.g. 'WkzbQMuFYuamGv3YF'). Dataset IDs or names to merge, in order. Items are read in this order (this matters for 'keep first/last').You can also paste the dataset URL from Apify Console or the API (the ID is taken from it).
omitFieldsNoTop-level fields to remove from the output.
dedupFieldsNoItems with the same values in ALL these fields are duplicates. Use dots for nested fields (e.g. ['url'] or ['address.city']). Empty = compare whole items.
normalizeUrlsNoTreat http/https, www and a trailing slash as the same URL.
addSourceFieldNoAdds '_sourceDatasetId' to each output item.
dedupEmptyKeysNoBy default, items whose dedup fields are all empty are kept (never treated as duplicates).
trimWhitespaceNoTreat ' Acme Inc ' and 'Acme Inc' as the same value.
caseInsensitiveNoTreat 'Acme' and 'acme' as the same value.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond readOnlyHint/openWorldHint, it discloses the key operational facts an agent could not infer: the tool runs Actor ucchiii/dataset-dedup-merge on the user's own Apify account, requires APIFY_TOKEN, and incurs that Actor's pay-per-event cost with a link for pricing. It also describes normalization behavior and that dataset order determines 'keep first/last'.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and output semantics are front-loaded, then behavior, then cost/auth. Slightly redundant with the schema on normalization flags and the closing pricing pointer, but no sentence is pure filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 11-parameter, no-output-schema tool it covers the essentials: what is returned, both output modes, the one hard input requirement, and cost/auth implications. It does not mention limits on dataset count or result size, which is a minor gap rather than a blocking one.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, and the description earns a bump by adding example dedup keys (url, placeId, email) and reinforcing the ordering rule that governs keep=first/last. It also states the datasetIds minimum, though that conflicts with the schema's empty required list.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States specific verbs and resources: merges one or more Apify datasets and removes duplicate rows, with an explicit output-mode distinction ('unique' vs 'duplicatesOnly'). An agent can tell exactly what this tool produces without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a hard precondition ('Give at least one of: datasetIds') and explains the output-mode choice, but it never says when to prefer this over the sibling utilities (filter_rows, flatten_json) or how it relates to them. Usage is implied rather than routed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.