deduplicator-merge-advanced

A fair-merge engine for security-audit findings. When several auditors review the same codebase, they file overlapping findings for the same bug — described differently, from different angles, each with its own PoC or fix. This tool reads every finding, groups them by root cause, and produces a plan to merge them fairly: keep the most complete one, credit every contributor, and fold in what the others uniquely add.

It is not a duplicate finder. Finding duplicates is the easy half. The hard, valuable half is merging them without losing content or erasing authorship — that is what this does.

The problem: partial deduplication

Naïve deduplication treats findings as exact matches: find the "duplicates", close them, move on. Real audit findings are partially overlapping — same root cause, different manifestation, different code path, a unique PoC, a different fix angle. Closing them as duplicates:

• loses technical substance — the closed finding's extra affected function, its short PoC, or its alternative fix is gone; and

• erases authorship — the auditor who filed it silently loses credit.

Fair merging solves this. Every merge decision is designed so nothing of value is lost.

Related MCP server: @repomend/mcp

How the fair merge works

Each logical piece exists for a reason:

• Root-cause grouping, not title/label matching. The same bug is written five ways ("TOCTOU", "concurrent claim race", "double-tap"). Grouping reads the actual vulnerability, ignoring template Impact/Likelihood fields and [M-01]-style ID tags, which carry no signal.

• Relation types, because overlap is partial, not binary — exact_dup, partial_dup, subset, same_class, and overlapping_canonical. The last one protects a finding that is canonical elsewhere from being wrongly closed here.

• Canonical selection by completeness. The most complete finding becomes the base (scored on root-cause clarity, code location, attack scenario, PoC, exact-line fix).

• Author transfer — the fairness core. Every member's authors, including overlapping_canonical members, are credited on the canonical. No one's work vanishes.

• Content enhancement (edit TODOs). The canonical is augmented with each duplicate's unique substance — a missing affected function, a materially different fix, a short PoC — as surgical, copy-pasteable edit instructions.

• Severity recalculation. A weighted view across all members, not one submitter's arbitrary label.

• Cross-group protection. A broad finding that is canonical for its own root cause is never destroyed by being closed as a duplicate of another group.

Full write-up: methodology/find-duplicates.md.

Architecture

Findings in, a validated merge plan out; the plan renders to review artifacts or is executed on GitHub.

flowchart LR
  F["Findings<br/>(GitHub issues or files)"] --> IDX[index]
  IDX --> SUM["summarize<br/>normalized per finding"]
  SUM --> GRP["group<br/>by root cause"]
  GRP --> REF["refine<br/>canonical + edit TODOs"]
  REF --> ASM["assemble + validate"]
  ASM --> PLAN[("merge plan")]
  PLAN --> MD["merge_plan.md<br/>final_findings.md"]
  PLAN --> AP["--apply → GitHub<br/>(labels · close · comment · edit)"]

The grouping and refine stages are the model's judgment (the methodology); every other stage is deterministic Python — batching, validation, and the safety net that folds any dropped finding back into standalone.

Two ways to run it

Both share the same brain (the methodology); they differ in the frontend.

1. GitHub mode (MCP)

An MCP server exposing data-access tools so a Claude agent (in Claude Code / Desktop) runs the merge interactively against a GitHub audit repo where findings are Issues (severity label + author <handle> labels). The agent fetches the index, reads bodies on demand, and persists progress to a workspace so a session survives interruptions.

pip install -e ".[mcp]"
python -m dedup_merge.mcp.server        # stdio MCP server

2. Standalone mode (Docker)

A self-contained analyzer: point it at a GitHub repo or a local findings folder, and it writes the merge artifacts — no interactive agent, no write access to anyone's repo.

docker compose run --rm merge --repo owner/name --out /out
# or from local findings (a dir of *.md, or findings.json):
docker compose run --rm merge --input /findings --repo-name my-audit --out /out

Or without Docker:

pip install -e .
dedup-merge --input examples/findings --repo-name example-audit --out ./out

It writes three artifacts to the output directory:

Applying the merge (--apply)

By default the tool only plans. Add --apply (GitHub mode only) to execute the plan on the repo: transfer author <handle> labels onto each canonical, close the merged duplicates (never an overlapping_canonical), recompute the canonical's severity as a weighted vote, post a Deduplicator report comment, and fold the edit TODOs into the canonical's body. It needs a write-scoped GITHUB_TOKEN; without --apply the same invocation prints the exact actions it would take.

dedup-merge --repo owner/name --out ./out            # dry run — prints what it would do
dedup-merge --repo owner/name --out ./out --apply    # executes on GitHub

Authentication

One Anthropic credential is required — either works:

• ANTHROPIC_API_KEY — a standard API key, or

• CLAUDE_CODE_OAUTH_TOKEN — a Claude Code subscription token (sent as an OAuth bearer; the client adds the required beta header automatically).

GitHub mode additionally needs a GITHUB_TOKEN. See .env.example.

Model

Defaults to claude-sonnet-5 (cost-effective for batch analysis). Override with DEDUP_MERGE_MODEL — claude-opus-4-8 for maximum quality, claude-haiku-4-5 for the cheapest passes.

Example

Run the bundled synthetic set:

dedup-merge --input examples/findings --repo-name example-audit --out ./out

It groups the promo-race findings (#101 canonical, #102/#104 merged), keeps the unrelated validation bug (#103) standalone, credits all three authors on #101, and emits an edit TODO folding the webhook angle into the canonical's fix. Sample artifacts are in examples/output/.

Quality

Two layers, because the tool has a deterministic half and a model half:

• Tests (tests/) cover the deterministic plumbing — pipeline wiring, plan validation, source parsing — with a mock LLM, so they run in CI without any credential. Lint (ruff), types (mypy) and tests run on Python 3.10–3.12 in CI.

• Evals (evals/) grade the model's judgment against hand-labeled cases on four metrics: grouping F1, canonical accuracy, author-transfer recall, and relation-type accuracy, with per-case pass/fail thresholds. The bundled cases (a real merge, distinct-bug over-merge stress, and a no-duplicates set) currently pass at 1.0 across all metrics on claude-sonnet-5.

pytest                       # offline, no credential
python evals/run_evals.py    # LLM-graded, needs a credential

License

MIT.