OpenPapers

Compare paper to code

compare_paper_to_code

Checks whether a paper's recipe matches the revision-pinned repository configuration, never executing code. Helps confirm reproducibility of documented experiments.

Instructions

Compare an explicit paper recipe with extracted revision-pinned repository configuration; repository content is never executed.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "recipe": "<recipe>",
      "fields": "<fields>",
      "source_url": "<source_url>"
    },
    "name": "compare_paper_to_code"
  }
}

Input Schema

Name	Required	Description	Default
`fields`	Yes
`recipe`	Yes
`commit_sha`	No
`source_url`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

First observedv0.1.0Sep 2, 2026

TDQS

A3.7/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description needs to carry the behavioral burden, and it discloses a key safety trait: 'repository content is never executed.' It also signals that the comparison uses revision-pinned configuration, but it does not describe output, errors, or side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One tight sentence with no filler; the main purpose is front-loaded and the safety qualifier is appended without distracting. It earns every word.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with four parameters (three required), nested objects, no output schema, and no annotations, a single high-level sentence is insufficient. The agent is not told what the comparison returns, what fields represents, or what conditions must be met before calling.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema itself has 0% description coverage and no parameter descriptions, so the description must compensate; it only loosely maps to recipe and repository configuration. It does not explain the required fields array, its lineStart/lineEnd semantics, or the roles of source_url and commit_sha.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific action ('Compare') and targets two distinct resources: a paper recipe and a repository configuration. It clearly distinguishes this tool from siblings like compare_papers or extract_training_recipe by framing the comparison as paper-vs-code.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The context that the user should have an explicit recipe and an extracted, revision-pinned repository configuration is clear, but there is no explicit when-to-use/when-not-to-use guidance or reference to alternative tools. The appropriate usage is implied rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.