deduplicator-merge-advanced
by danogurtsov
README.md
# deduplicator-merge-advanced
**A fair-merge engine for security-audit findings.** When several auditors review the
same codebase, they file overlapping findings for the same bug — described differently,
from different angles, each with its own PoC or fix. This tool reads every finding,
groups them by **root cause**, and produces a plan to merge them **fairly**: keep the most
complete one, credit every contributor, and fold in what the others uniquely add.
> It is **not** a duplicate finder. Finding duplicates is the easy half. The hard,
> valuable half is merging them without losing content or erasing authorship — that is
> what this does.
## The problem: partial deduplication
Naïve deduplication treats findings as exact matches: find the "duplicates", close them,
move on. Real audit findings are **partially** overlapping — same root cause, different
manifestation, different code path, a unique PoC, a different fix angle. Closing them as
duplicates:
- **loses technical substance** — the closed finding's extra affected function, its short
PoC, or its alternative fix is gone; and
- **erases authorship** — the auditor who filed it silently loses credit.
Fair merging solves this. Every merge decision is designed so **nothing of value is lost**.
## How the fair merge works
Each logical piece exists for a reason:
- **Root-cause grouping, not title/label matching.** The same bug is written five ways
("TOCTOU", "concurrent claim race", "double-tap"). Grouping reads the actual
vulnerability, ignoring template `Impact/Likelihood` fields and `[M-01]`-style ID tags,
which carry no signal.
- **Relation types, because overlap is partial, not binary** — `exact_dup`, `partial_dup`,
`subset`, `same_class`, and `overlapping_canonical`. The last one protects a finding
that is canonical elsewhere from being wrongly closed here.
- **Canonical selection by completeness.** The most complete finding becomes the base
(scored on root-cause clarity, code location, attack scenario, PoC, exact-line fix).
- **Author transfer — the fairness core.** Every member's authors, including
`overlapping_canonical` members, are credited on the canonical. No one's work vanishes.
- **Content enhancement (edit TODOs).** The canonical is augmented with each duplicate's
unique substance — a missing affected function, a materially different fix, a short PoC —
as surgical, copy-pasteable edit instructions.
- **Severity recalculation.** A weighted view across all members, not one submitter's
arbitrary label.
- **Cross-group protection.** A broad finding that is canonical for its own root cause is
never destroyed by being closed as a duplicate of another group.
Full write-up: [`methodology/find-duplicates.md`](methodology/find-duplicates.md).
## Architecture
Findings in, a validated merge plan out; the plan renders to review artifacts or is
executed on GitHub.
```mermaid
flowchart LR
F["Findings<br/>(GitHub issues or files)"] --> IDX[index]
IDX --> SUM["summarize<br/>normalized per finding"]
SUM --> GRP["group<br/>by root cause"]
GRP --> REF["refine<br/>canonical + edit TODOs"]
REF --> ASM["assemble + validate"]
ASM --> PLAN[("merge plan")]
PLAN --> MD["merge_plan.md<br/>final_findings.md"]
PLAN --> AP["--apply → GitHub<br/>(labels · close · comment · edit)"]
```
The **grouping** and **refine** stages are the model's judgment (the methodology); every
other stage is deterministic Python — batching, validation, and the safety net that
folds any dropped finding back into `standalone`.
## Two ways to run it
Both share the same brain (the methodology); they differ in the frontend.
### 1. GitHub mode (MCP)
An MCP server exposing data-access tools so a Claude agent (in Claude Code / Desktop) runs
the merge interactively against a GitHub audit repo where findings are Issues (severity
label + `author <handle>` labels). The agent fetches the index, reads bodies on demand,
and persists progress to a workspace so a session survives interruptions.
```bash
pip install -e ".[mcp]"
python -m dedup_merge.mcp.server # stdio MCP server
```
### 2. Standalone mode (Docker)
A self-contained analyzer: point it at a GitHub repo **or** a local findings folder, and it
writes the merge artifacts — no interactive agent, no write access to anyone's repo.
```bash
docker compose run --rm merge --repo owner/name --out /out
# or from local findings (a dir of *.md, or findings.json):
docker compose run --rm merge --input /findings --repo-name my-audit --out /out
```
Or without Docker:
```bash
pip install -e .
dedup-merge --input examples/findings --repo-name example-audit --out ./out
```
It writes three artifacts to the output directory:
| File | What it is |
|------|-----------|
| `merge_plan.json` | The full machine-readable plan (groups, relations, canonical, scores, author transfers, edit TODOs, standalone, notes) |
| `merge_plan.md` | The reviewable action list — what to close, who's canonical, authors credited, and **how to edit the texts** |
| `final_findings.md` | The merged result — surviving canonicals with combined authors, plus standalone findings |
### Applying the merge (`--apply`)
By default the tool only *plans*. Add `--apply` (GitHub mode only) to execute the plan on
the repo: transfer `author <handle>` labels onto each canonical, close the merged
duplicates (never an `overlapping_canonical`), recompute the canonical's severity as a
weighted vote, post a `Deduplicator` report comment, and fold the edit TODOs into the
canonical's body. It needs a write-scoped `GITHUB_TOKEN`; without `--apply` the same
invocation prints the exact actions it *would* take.
```bash
dedup-merge --repo owner/name --out ./out # dry run — prints what it would do
dedup-merge --repo owner/name --out ./out --apply # executes on GitHub
```
## Authentication
One Anthropic credential is required — either works:
- `ANTHROPIC_API_KEY` — a standard API key, or
- `CLAUDE_CODE_OAUTH_TOKEN` — a Claude Code **subscription** token (sent as an OAuth bearer;
the client adds the required beta header automatically).
GitHub mode additionally needs a `GITHUB_TOKEN`. See [`.env.example`](.env.example).
## Model
Defaults to `claude-sonnet-5` (cost-effective for batch analysis). Override with
`DEDUP_MERGE_MODEL` — `claude-opus-4-8` for maximum quality, `claude-haiku-4-5` for the
cheapest passes.
## Example
Run the bundled synthetic set:
```bash
dedup-merge --input examples/findings --repo-name example-audit --out ./out
```
It groups the promo-race findings (`#101` canonical, `#102`/`#104` merged), keeps the
unrelated validation bug (`#103`) standalone, credits all three authors on `#101`, and
emits an edit TODO folding the webhook angle into the canonical's fix. Sample artifacts
are in [`examples/output/`](examples/output/).
## Quality
Two layers, because the tool has a deterministic half and a model half:
- **Tests** ([`tests/`](tests/)) cover the deterministic plumbing — pipeline wiring,
plan validation, source parsing — with a mock LLM, so they run in CI **without any
credential**. Lint (ruff), types (mypy) and tests run on Python 3.10–3.12 in
[CI](.github/workflows/ci.yml).
- **Evals** ([`evals/`](evals/)) grade the model's judgment against hand-labeled cases on
four metrics: grouping F1, canonical accuracy, author-transfer recall, and
relation-type accuracy, with per-case pass/fail thresholds. The bundled cases (a real
merge, distinct-bug over-merge stress, and a no-duplicates set) currently pass at 1.0
across all metrics on `claude-sonnet-5`.
```bash
pytest # offline, no credential
python evals/run_evals.py # LLM-graded, needs a credential
```
## License
[MIT](LICENSE).
This server cannot be deployed
Maintenance
ActivityStale
ResponsivenessNo issues