Allows retrieving and comparing scientific papers from PubMed, extracting study design details and identifying methodological differences that explain divergent conclusions.
Two well-run papers often reach opposite conclusions. Usually the reason isn't that one is wrong — it's a methodological difference (a cell type, a dose, a follow-up window, an analysis choice) that a reader has to dig out of the methods sections by hand. StudyDiff does that digging. Give it two studies and it extracts each one's design, surfaces the differences that could explain the disagreement, and — critically — grounds every statement in the source text, so it never invents a finding.
It is built for a bench scientist deciding which of two conflicting papers to trust before planning an experiment.
Most "AI literature" tools generate a fluent answer and ask you to trust it. StudyDiff inverts that:
It shows the evidence, then gets out of the way. The design differences up top; every value with the verbatim sentence that supports it underneath.
It refuses to guess. Any field the source doesn't state is shown as not reported, never inferred.
It verifies itself. A deterministic grounding check (no second LLM acting as judge) confirms every extracted value and every explanation is backed by a verbatim quote and traceable numbers. Anything that fails is downgraded before it can be used as a reason.
It knows what it can't do, because it was measured — see below.
Related MCP server: surveyHelper
StudyDiff used to rank the divergent design dimensions and present the top one as the likely driver of a disagreement. We built a benchmark to test that, and it doesn't work.
15 documented contradictions where the literature has since established why the papers disagreed — each label carrying its own citation, the set built blind before any accuracy number existed. Scored against StudyDiff's top-ranked driver:
Top-1 accuracy (strict) 13.3% (95% CI 3.7-37.9%) [2/15]
Baseline "always say assay" 13.3% (95% CI 3.7-37.9%) [2/15]
→ discordant on 0 of 15 cases
Oracle ceiling (reachable) 66.7% (95% CI 41.7-84.8%) [10/15]
Non-assay-labelled cases 0.0% (95% CI 0.0-22.8%) [0/13]The ranking was a fixed prior (DRIVER_RANK in src/compare.mjs) in which assay
outranks everything. Two papers almost always use somewhat different methods, so assay
almost always diverges, so it was picked 13 times out of 15 — and the two hits are exactly
the two assay-labelled cases. It is not merely as good as guessing a constant; it is
behaviourally identical to it on every case in the set.
Fixing grounding first (Phase 2) removed that excuse. Recovering 20 of 26 false-positive rejections doubled the oracle ceiling from 33% to 67% — the established cause is now an available candidate in 10 of 15 cases instead of 5 — and top-1 accuracy did not move at all. The ranker was handed the right answer five more times and took none of them.
What changed as a result: the app no longer nominates a primary driver. It presents the divergent dimensions as an unranked list, because that list is informative (it contains the established cause 10 times in 15) while the ordering is not. Choosing among them needs domain knowledge the tool doesn't have.
What still holds: which dimensions differ, which are identical (ruled out), and the verbatim sentence behind every value. None of that depends on the ranking.
Those 15 cases had by then been read across two phases — failures analysed, spans re-audited — so every post-fix figure from them is development-set accuracy, not a blind measurement. So we built a second set and measured it once.
eval/cases-heldout.json is 15 further documented contradictions, curated to a protocol written
and committed before any case was selected (eval/HELDOUT-PROTOCOL.md),
by a curator kept blind to the dev set's per-case failures, in deliberately different fields:
microbiome, marine ecology, toxicology, psychology, critical care, oncology, infectious disease.
No paper and no contradiction is shared with the dev set — selftest enforces that mechanically.
Top-1 accuracy (strict) 13.3% (95% CI 3.7-37.9%) [2/15]
Baseline "always say assay" 20.0% (95% CI 7.0-45.2%) [3/15]
→ discordant on 1 of 15 cases
Oracle ceiling (reachable) 73.3% (95% CI 48.0-89.1%) [11/15]
Non-assay-labelled cases 0.0% (95% CI 0.0-24.3%) [0/12]On unseen data the prior scores below the constant guess — by exactly one case. The intervals
overlap almost entirely and the two strategies disagree on 1 of 15, so the honest statement is
that it remains indistinguishable from guessing assay every time, not that it is worse.
The line that does not move is the last one. Across both sets, 25 cases where the established
cause was something other than assay, the prior identified none of them. And the ceiling here
is higher than on the dev set — 73.3% against 66.7% — so extraction put the right answer in
front of the ranker more often, and it was taken no more often. That is the Phase 1–2 conclusion
reproduced on data the development loop never saw, which is the only way it could have been
strengthened.
The number is reported as-is, and is never pooled with the dev-set number: summing them into an
"n=30" figure would relaunder read data as blind data. Nothing in src/ was changed on the basis
of it. The set was fetched once on a single pre-registered arm, but the scoring was not one clean
pass — the first run reported n=14 after one paper failed to fetch, and the full set was scored
after retrieving it. Both figures, the sixteen defects an adversarial verification pass found and
corrected before any scoring, and the reasoning behind each label are recorded in the file's own
provenance block rather than summarised away.
npm run eval:heldout # the blind number, offline and free
npm run eval:selftest:heldout # set integrity, incl. zero overlap with the dev setFull method, the pre-registered decisions, and every prediction that turned out wrong:
eval/README.md and eval/PHASE2.md. The benchmark
sets are eval/cases.json (development) and
eval/cases-heldout.json (held-out).
npm run eval # offline, free, no API key — regenerates the numbers above
npm run eval:selftest # validates the harness maths and set integrityeval/cache/ and eval/cache-heldout/ are committed on purpose. They aren't build output,
they're evidence: the published numbers are reproducible from artefacts in the repo rather than
taken on faith.
Under 60 seconds, no API key, no network:
git clone https://github.com/nickjlamb/studydiff && cd studydiff
npm install
npm run demo # explains a real, famous contradiction
npm run demo -- resveratrol-sirt1 # a second worked example
npm run demo -- treg-stability # a third: Treg lineage stabilityRun the web app:
cp .env.example .env # add ANTHROPIC_API_KEY for live comparisons
npm run serve # http://localhost:4173The built-in examples run on cached data and need no key. To compare live, add your key and use the PMID / DOI, Upload PDF, or Paste inputs.
flowchart LR
IN["Two papers<br/>PMID · DOI · PDF · text"] --> R["Retrieve<br/>PubMed / PMC / PDF"]
R --> E["Extract<br/>Claude → structured study cards"]
E --> V["Verify<br/>deterministic grounding"]
V -- "ungrounded → not reported" --> E
V --> C["Compare<br/>divergent vs. shared design"]
C --> X["Present<br/>divergent dimensions + ruled out"]Retrieve — a PubMed/PMC client with a full-text-to-abstract fallback that tags how deep it read (fulltext / abstract / pasted); uploaded PDFs are text-extracted server-side.
Extract — Claude turns each paper into a fixed study card (species, model, assay, dose, timing, endpoint, sample size, statistic, finding, limitations). Every field carries a verbatim supporting quote; absent fields default to not reported.
Verify — grounding runs first: any value whose quote isn't in the source, or whose numbers don't trace, is downgraded to not reported. StudyDiff can't cite a fact it hasn't verified.
Compare — deterministic: which design dimensions are reported by both papers, which of those diverge, and which are identical. Divergence is a token-overlap test on design dimensions and strict inequality on the conclusion itself, because a conclusion's polarity matters where a method list's content does.
Present — the divergent dimensions unranked, each with both papers' values and the verbatim sentence behind them, and the identical ones explicitly ruled out. StudyDiff does not nominate one as the cause: it was measured and it doesn't work.
Two things hold the guarantee up: the API key never leaves the server, and verification is deterministic — grounding and comparison involve no model judgment at all, so the same verified evidence always yields the same divergent dimensions.
StudyDiff's reliability comes from how it uses Claude, not just that it does:
Structured extraction via tool-use. Each paper becomes a study card through a forced Claude (Sonnet) tool schema, so every field returns validated and carries a verbatim supporting quote — no free-text parsing, no "mostly-JSON" failures.
Reasoning that maps claims to evidence. Claude reads the methods/abstract prose and identifies both the design value and the exact sentence that supports it — the hard part of turning unstructured papers into comparable, auditable cards.
Claude proposes, grounding disposes. StudyDiff never uses an LLM as the judge. A deterministic check verifies Claude's output against the source and downgrades anything unsupported before it's shown. Pairing Claude's tool-use extraction with non-LLM verification is what lets the tool trust its own output and sidesteps the usual LLM-as-evaluator pitfalls.
Built with Claude Code. The whole app — pipeline, UI, hardening, deploy — was built iteratively with Claude Code in a single agentic loop.
Question | Papers | What StudyDiff finds |
Do mouse models mimic human inflammation? | Same datasets, opposite conclusions — driven by the gene-selection strategy. | |
Does resveratrol activate SIRT1? | An assay artifact — the Fluor de Lys peptide substrate vs. native substrates. | |
Is the Treg lineage stable in vivo? | A landmark T-cell controversy — stable vs. unstable, driven by the fate-mapping method. |
All three ship as offline demos (npm run demo / npm run demo -- resveratrol-sirt1 / npm run demo -- treg-stability).
Web app (npm run serve) — examples, PMID/DOI lookup, PDF upload, or paste; streams each pipeline step live, and exports a reproducible Markdown report with every value's verbatim supporting sentence.
MCP server (npm run mcp) — lets Claude, or any agent, call the contradiction engine directly. See MCP server.
CLI — node src/cli.mjs --q "Does resveratrol activate SIRT1?" 12939617 19843076
Deploy — see DEPLOY.md (Railway + custom domain).
StudyDiff's engine is also a Model Context Protocol server, so an agent can ask why two papers disagree as a tool call — the same grounded pipeline, no browser.
Published to the official MCP Registry as io.github.nickjlamb/studydiff, and to npm as studydiff-mcp.
Tool | What it does |
| The full pipeline on two papers. Each paper is |
| Runs a cached worked example — no API key, no network. The quickest way to see the grounded output. |
| Lists the built-in worked examples. |
It returns the same auditable report the web app exports: the verdict, the divergent design dimensions (unranked) and the identical ones (ruled out), every value with the verbatim sentence that supports it, the verification counts, and what evidence would resolve the disagreement. Ungrounded fields come back as not reported — never guessed — and the tool never picks a winner, never nominates a primary driver, and never reports a confidence it didn't compute.
Add it to Claude Code:
claude mcp add studydiff -- npx -y studydiff-mcp…or to Claude Desktop (claude_desktop_config.json):
{
"mcpServers": {
"studydiff": {
"command": "npx",
"args": ["-y", "studydiff-mcp"],
"env": { "ANTHROPIC_API_KEY": "sk-ant-..." }
}
}
}ANTHROPIC_API_KEY is only needed for compare_studies. Leave it out and list_examples / compare_example still work — the cached examples run with no key and no network. (To run from a local clone instead, swap the command for node /absolute/path/to/studydiff/src/mcp.mjs.)
Then just ask:
Use studydiff to compare PMID 19633673 and 20929851 — why do they disagree?
No key handy? list_examples, then compare_example("treg-stability") runs entirely offline.
src/ncbi.mjs PubMed / PMC retrieval (+ DOI resolution, source-depth tagging)
src/pdf.mjs PDF text extraction (pure JS)
src/extract.mjs Claude tool-use → structured study cards
src/grounding.mjs deterministic verification (OpenGATE)
src/compare.mjs divergence detection (divergent vs. shared design dimensions)
eval/ driver-ranking benchmark: 15 cited contradictions + scorer
src/gaps.mjs bounded "observed across these papers"
src/pipeline.mjs orchestration: retrieve → extract → verify → compare
src/report.mjs shared Markdown report (answer, drivers, quotes, verification)
src/server.mjs web server + streaming API (+ rate limiting, caching)
src/mcp.mjs MCP server: compare_studies / compare_example / list_examples
public/index.html single-file dashboard UI
fixtures/ cached real papers for the offline demosKeyword search with a results picker, a source viewer that highlights each grounded quote in the original text, batch comparison, and an exportable report. Full list in ROADMAP.md.
Contributions are welcome — see CONTRIBUTING.md for the setup and the invariants that keep the trust guarantee intact.
StudyDiff was built for Anthropic's Built with Claude: Life Sciences hackathon (Builder track). All application code in this repository was written from scratch during the event. Grounding uses OpenGATE and PDF extraction uses unpdf, both as published dependencies. Retrieval uses public NCBI E-utilities; extraction uses the Claude API.
MIT © Nick Lamb
Run one of StudyDiff's built-in worked examples – real, famous contradictions with verbatim abstracts. Runs fully offline against cached papers: no API key and no network needed, so it is the quickest way to see the grounded output. Options: "mouse-inflammation" (Seok 2013 vs Takao 2015 – same datasets, opposite conclusions), "resveratrol-sirt1" (Howitz 2003 vs Beher 2009 – an assay artifact), "treg-stability" (Zhou 2009 vs Rubtsov 2010 – stable vs unstable Treg lineage, driven by the fate-mapping method).
| Name | Required | Description | Default |
|---|---|---|---|
| example | Yes | Which built-in example to run |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral disclosure burden. It clearly discloses that the tool is fully offline, uses cached papers, requires no API key, and needs no network, which are the key behavioral traits an agent needs before invoking it. It does not detail output format or error behavior, but those are minor for a simple example runner.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose, followed by the offline advantage and then a structured list of options. Every sentence earns its place, and the longer option details are necessary because the enum values are otherwise opaque.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter tool with no output schema, the description covers all invocation-critical information: what the tool does, why to use it, and valid parameter values. The only minor gap is that it does not explicitly describe what the returned output looks like, but the phrase 'grounded output' plus the example details make the behavior reasonably clear.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema only lists the enum values with a generic description, while the tool description richly unpacks each option: concrete papers, years, and the nature of each contradiction. This gives an agent far more than the schema alone and makes parameter selection informed and accurate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Run') and a specific resource ('built-in worked examples'), then names the exact examples with the contradictions they demonstrate. It is clearly distinct from a general study-comparison tool because it emphasizes cached, built-in, offline examples.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to prefer this tool: when you want the quickest demonstration of grounded output without an API key or network access. It does not explicitly name alternatives like compare_studies or state when not to use it, so it falls just short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Explain WHY two scientific papers reach different conclusions. Extracts each study design (species, model, intervention, assay, dose, timing, endpoint, sample size, statistic, finding, limitations), verifies every extracted value against the source text with deterministic grounding, then lists the design differences that could drive the disagreement (unranked, plus the dimensions that are identical in both and so ruled out) and suggests what evidence would resolve it. Every claim is traced to a verbatim quote; unsupported fields are returned as "not reported" rather than guessed. Never picks a winner. Each paper is given as {id} (PMID or DOI) or {text}. Requires ANTHROPIC_API_KEY – use compare_example for a no-key demonstration.
| Name | Required | Description | Default |
|---|---|---|---|
| paperA | Yes | First study – {id} (PMID/DOI) or {citation, text} | |
| paperB | Yes | Second study – {id} (PMID/DOI) or {citation, text} | |
| question | No | The question under comparison, e.g. "Is the Treg lineage stable in vivo?" |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of disclosure. It is exceptionally transparent: it lists exactly what is extracted (species, model, intervention, etc.), states that values are verified against source text with deterministic grounding, explains the output structure (unranked differences and ruled-out identical dimensions), commits to verbatim quotes and 'not reported' for unsupported fields, and clearly states it 'Never picks a winner.' This goes far beyond a typical description and leaves no ambiguity about the tool's behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Despite being a long paragraph, every sentence contributes unique information: purpose, extraction details, verification, output structure, limitation handling, and dependency. The critical scoping statement ('Never picks a winner.') is placed near the end but is memorable. There is no redundancy or filler; the density is appropriate for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of comparing two studies and the absence of an output schema, the description is remarkably complete. It covers input requirements, the exact dimensions extracted, the verification process, the output format (unranked differences, ruled-out identicals), and the requirement for an API key. An agent can confidently invoke this tool correctly without needing additional context from elsewhere.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers 100% of parameters, so the baseline is 3. The description adds value by explaining the id/text alternatives for each paper ('Each paper is given as {id} (PMID or DOI) or {text}') and highlighting the need for a citation when supplying raw text. This goes beyond the schema's field descriptions, providing context that helps the agent choose the correct input form.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Explain WHY two scientific papers reach different conclusions.' It clearly defines the tool's core function as comparative explanation, and distinguishes it from siblings by noting the no-key alternative (compare_example). The purpose is unambiguous and not a tautology of the tool name.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use the tool (requires ANTHROPIC_API_KEY) and provides a clear alternative for the no-key case: 'use compare_example for a no-key demonstration.' It also implies the input format (PMID/DOI or text), giving practical guidance on how to supply the two papers. This is sufficient and distinct.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List StudyDiff's cached worked examples (real published contradictions) that can be run with compare_example without an API key.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden. It discloses that examples are cached, are real published contradictions, and require no API key. This adds meaningful context beyond the tool name, though it does not describe output format or pagination.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, information-dense sentence with no filler. The main action and subject are front-loaded, followed by the key qualifiers about cache, authenticity, and API-key-free execution.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, no-output-schema list tool, this description is complete. It tells the agent what is listed, where it comes from, why it matters, how it relates to a sibling tool, and what authentication is required.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and the schema is fully covered, so the description does not need to explain parameter details. It adds useful semantic context by explaining what the returned examples are for, which helps the agent understand the tool's role even without parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description states a specific verb ('List'), a concrete resource ('StudyDiff's cached worked examples'), and adds a distinguishing qualifier ('real published contradictions'). It also links the resource to 'compare_example', making the purpose unmistakable and distinct from sibling comparison tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly implies this tool is the discovery step before running examples with 'compare_example', and notes that no API key is needed. It does not explicitly spell out when not to use it versus 'compare_studies', but the list-versus-compare distinction is evident from the verb and context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.2.0compare_examplecompare_studieslist_examplesScored across 3 tools
Each tool has a clearly distinct purpose: compare_example runs a pre-cached demonstration, list_examples enumerates those demonstrations, and compare_studies handles user-supplied papers. There is no overlap or ambiguity between them.
All tool names follow the consistent verb_noun pattern with snake_case (compare_example, list_examples, compare_studies). The shared 'compare' prefix is semantically appropriate and does not create confusion.
With 3 tools, the surface is tightly scoped to the server's purpose: list examples, run an example, and run a custom comparison. Each tool earns its place and the count is well within the ideal 3-15 range.
The tool set fully covers the intended workflow: users can discover available examples, run them for quick demonstration, and perform a full custom analysis on their own papers. There are no missing operations or dead ends for the stated domain.
Ground answers in scientific literature. Search full text, evaluate trust, access full-text articles
Search 8.5M scientific papers with LLM TLDRs, citations, linked entities, figures, and full text.
Cite the claim, not the paper: 5,033 assertions, each hash-verifiable against its source PDF
Search 340M+ academic papers — citation graphs, semantic similarity, and AI literature reviews.
