ReasonForge

A deterministic, read-only reasoning device for hard questions: eight MCP tools that force a scope lock, a cheapest-disconfirming-step plan, a blind-spot sweep, and a verdict gate that refuses to accept a claim without proof of execution.

English · 한국어 · 日本語 · 简体中文 · 繁體中文 · Español · Português (Brasil) · Deutsch · Français · Русский · Italiano · Bahasa Indonesia · Türkçe

Quickstart · Architecture · Orchestration · Open proposals · Credits

What this is

ReasonForge is a tool, not a product and not an agent: a small vendor-neutral reasoning harness that a coding agent can call, over MCP or a CLI, to impose structure on a hard question. It classifies the prompt, decides whether scaffolding is even worth applying, emits a plan whose first move is the cheapest way to be proven wrong, forces a pre-verdict blind-spot sweep with line-level evidence, and then gates the verdict: a final ACCEPT or REJECT requires real evidence references plus falsification attempts, and a time-sensitive or "verified fact" claim additionally requires proof of execution — actual tool output, not a citation that could be invented. A calibrated DEFER with a concrete fetch plan is a valid answer; a confident answer with nothing behind it is not. It is deliberately read-only: the core and the MCP server do not write files, spawn processes, or touch the network, and that property is enforced by a test rather than by convention.

Related MCP server: verifiable-thinking-mcp

Highlights

• 33 tests across 9 test files, run by Node's built-in test runner, completing in well under a second on this snapshot. Zero runtime dependencies — package.json declares no dependencies at all.

• A purity gate that is a product guarantee, not a lint rule. tests/purity.test.mjs scans all 13 forge/ modules plus the MCP server — 14 files — against 6 forbidden patterns covering node:fs imports, any write/append call, child_process/spawn/exec, fetch, and raw http/https/net/dgram use. If the core ever gains a side effect, the suite goes red.

• Exactly 8 MCP tools, asserted by name in the test suite so the surface cannot drift silently: reason_intake, reason_route, reason_plan, reason_rubric, reason_sweep_checklist, reason_verdict_gate, reason_reflect, reason_eval.

• The CLI and the MCP tool call share one core, and a test asserts they produce the same output — so the two surfaces cannot diverge into two behaviours.

• A router that is allowed to say "skip". Prompts are classified into a task_type and routed to full, light, or skip. Well-specified technical execution is deliberately routed to light touch, because taxing a migration that already has concrete constraints is a cost with no benefit. A scaffold that always fires is a scaffold that cannot be evaluated.

• An ablation harness with 13 probes and 6 variants — control plus full_loop, minus_sweep, minus_verdict_gate, minus_plan, and intake_route_only — so each stage of the loop can be removed and measured rather than assumed useful.

• Blind judging built into the artefact format. A dry run emits shuffled judge packets and a separate answer key with 60 attribution rows and a warning field telling you to keep it away from the judge. The key's own score_source field reads answer-key only; not a judge result.

• A benchmark lane that reports its own failure instead of a number. The GPT lane requires a PTY-capable execution path; when the subprocess transport could not allocate one, harness/gpt-lane-results.json records unavailable: true with the exact reason (tcgetattr/ioctl: Operation not supported on socket) and zero probes, rather than emitting a plausible-looking lift.

• 9 open proposals, none implemented, catalogued in docs/SWEEP_PROPOSALS.md with rationale, effort, risk, and zone — including one that criticises this very README's predecessor for omitting the purity check from its verification section.

Use it

Three surfaces over one shared pure core:

reasonforge intake "Compare MCP hooks and markdown rules for cross-agent reasoning."
reasonforge plan "Find the safest fix for a flaky production retry path."
reasonforge verdict --json '{"verdict":"ACCEPT","evidence_refs":["tests/core.test.mjs"],"falsification_attempts":["missing falsification blocks"]}'
reasonforge verdict --json '{"verdict":"DEFER","fetch_plan":["Browse one current source first."]}'
reasonforge hook --event session_start
reasonforge ab-run --dry-run          # writes the report, blind packets, and answer key
reasonforge rebenchmark-plan          # report-only, owner-gated

Honest labels you should read before quoting any number out of this repository:

• Dry-run and local-live A/B scores are heuristic-directional, and the report says so about itself. A cut, promote, or lift claim requires judging the emitted blind packets with a separate frontier judge and only then deshuffling with the private answer key. The harness will not make that claim for you.

• The checked-in dry-run report covers 12 of the 13 probes. The thirteenth, an anti-gaming probe (gaming-fabricated-verified-fact), was added after that report was generated and does not appear in it. The report is a committed artefact, not a live measurement — regenerate it before trusting it.

• probes/reasoning-golden.json is currently referenced by nothing — no module, no test. It is either a golden set awaiting an evaluator or a leftover; it sits in a frozen zone and is escalated for an owner decision as proposal P4 rather than quietly deleted.

• The GPT lane's recorded result is "unavailable". That is the honest output of a transport that could not run, not a zero-lift finding.

Quickstart

Prerequisites: Node.js 22 or newer (engines.node >= 22), and nothing else — there are no runtime dependencies to install, and npm install has nothing to fetch.

git clone <your-fork-url> reasonforge
cd reasonforge
node --check forge/cli.mjs && node --check mcp/server.mjs   # syntax gate
npm test                                                    # 33 tests
npm run check:pure                                          # read-only guarantee
node bin/reasonforge ab-run --dry-run                        # deterministic fixtures
node mcp/server.mjs                                          # stdio MCP server

The first three commands are exactly what the publication gate runs. ab-run --dry-run writes into harness/, so run it deliberately rather than as part of a check — the publication gate does not run it, precisely so that a gate cannot rewrite a committed artefact behind your back.

Architecture

flowchart TB
  subgraph SURF["Surfaces"]
    CLI["bin/reasonforge — CLI"]
    MCPS["mcp/server.mjs — 8 MCP tools, stdio"]
    HK["hook/reasonforge-hook — fail-open"]
  end
  subgraph CORE["forge/ — 13 pure modules, no fs / spawn / network"]
    IN["reason_intake — classify + scope lock"] --> RT{"reason_route"}
    RT -->|skip| ANS["answer directly"]
    RT -->|light| DIS["cheapest disconfirming step"]
    RT -->|full| PL["reason_plan — six moves, disconfirmation first"]
    PL --> RB["reason_rubric — scoring criteria"]
    RB --> SW["reason_sweep_checklist — blind-spot sweep, line evidence"]
    SW --> VG{"reason_verdict_gate"}
    VG -->|"evidence + falsification + proof of execution"| ACC["ACCEPT / REJECT"]
    VG -->|"missing proof"| DEF["DEFER / BLOCKED + concrete fetch plan"]
    ACC --> RF["reason_reflect — reusable lesson"]
    DEF --> RF
    RF --> EV["reason_eval — read-only ledger row"]
    DIS --> EV
  end
  CLI --> IN
  MCPS --> IN
  HK --> RT
  subgraph HARN["harness/ — the only place side effects live"]
    AB["ab-run: 13 probes x 6 variants"] --> PKT["shuffled blind judge packets"]
    AB --> KEY["answer key, 60 attribution rows, kept from the judge"]
    GL["gpt-lane: records unavailable + reason rather than a number"]
  end
  EV -.->|read-only rows| AB
  PT["tests/purity.test.mjs — 14 files x 6 forbidden patterns"] -.->|guards| CORE

Load-bearing properties:

• Purity is architectural, not stylistic. The side-effecting code lives in harness/, never in forge/ or mcp/. That split is what makes the core safe to call from an agent loop, and the purity test is what stops the split from eroding — a single writeFile in a forge/ module fails the suite.

• The verdict gate's hard part is proof of execution. Requiring "evidence references" is easy to satisfy dishonestly, because a reference can be fabricated. So a time-sensitive assertion, or one asserting verified specifics, additionally requires actual tool output; and citing the scaffold's own verdict as evidence is explicitly invalid. This is the anti-gaming rule, and it exists because a first version was gameable.

• DEFER is a first-class outcome. The gate does not force a decision; it forces an honest one. A calibrated DEFER with a concrete fetch or handoff plan passes. This is the difference between a reasoning aid and a confidence amplifier.

• The ablations are the point of the harness. Removing the sweep, the verdict gate, or the plan and measuring each is what turns "this loop helps" from an assertion into a hypothesis. The harness deliberately withholds the strong claim (heuristic-directional) until a blind judge has been run.

• Version identity is explicit. forge/version.mjs carries a package version, a dated REASON_VERSION for the loop contract, and the MCP protocol revision separately, so a loop change and a package bump are not confused for each other. Changing REASON_VERSION is what triggers the report-only rebenchmark plan.

Project layout

bin/reasonforge          CLI entry point
forge/                   13 pure modules — the shared core
  intake.mjs             prompt classification and scope lock
  route.mjs              full / light / skip decision from task_type and model tier
  plan.mjs               six-move plan, cheapest disconfirming step first
  rubric.mjs             scoring criteria
  sweep.mjs              pre-verdict blind-spot checklist
  disconfirm.mjs         the falsification move
  verdict.mjs            the gate: evidence + falsification + proof of execution
  reflect.mjs eval.mjs   reusable lesson, read-only ledger row
  cli.mjs hook.mjs frame.mjs version.mjs
mcp/server.mjs           stdio MCP server, exactly 8 tools, protocol 2024-11-05
hook/reasonforge-hook    session-start context injection, fail-open
harness/                 the only side-effecting code
  ab.mjs ab-run.mjs      ablation harness: 13 probes x 6 variants
  gpt-lane.mjs           cross-model lane; records unavailability honestly
  rebenchmark.mjs        report-only plan after a REASON_VERSION change
  ab-report.md           committed dry-run artefact (12 probes; see honest labels)
  ab-blind-packets.json  shuffled packets for a blind judge
  ab-answer-key.json     60 attribution rows; keep away from the judge
probes/ab-probes.json    13 A/B probes across moderate, hard, calibration
probes/reasoning-golden.json   3 entries, currently referenced by nothing (P4)
tests/                   9 files, 33 tests, including the purity gate
docs/ORCHESTRATION.md    the loop, the cross-agent shapes, the harness boundaries
docs/SWEEP_PROPOSALS.md  9 proposals, none implemented, each owner-gated

How this was built

The harness exists to try to disprove the product. ReasonForge's thesis is that a structured loop improves hard reasoning. The honest way to hold that thesis is an ablation: remove the sweep, remove the verdict gate, remove the plan, and measure each variant against a control. That is what harness/ab.mjs does with 6 variants over 13 probes — and the results are deliberately labelled heuristic-directional so that the harness's own numbers cannot be quoted as a lift claim. Building the instrument that could embarrass the design was the first real deliverable.

Blindness is enforced by file layout. The dry run emits shuffled judge packets and a separate answer key, and the key carries a warning field saying to keep it from the judge plus a score_source field reading answer-key only; not a judge result. Attribution is only possible after scoring. This is cheap to implement and it removes the most common way a self-evaluation quietly becomes self-congratulation.

An anti-gaming rule that came from being gamed. The verdict gate originally accepted "evidence references", which a model can simply write. The current rule requires proof of execution — real tool output — for time-sensitive or verified-fact ACCEPTs, and explicitly rejects citing the scaffold's own verdict as evidence. The thirteenth probe, gaming-fabricated-verified-fact, exists specifically to attack that path, and its absence from the committed report is called out above rather than glossed over.

A benchmark that refused to fabricate. The GPT lane could not allocate a PTY through its subprocess transport. The recorded outcome is unavailable: true, with the raw error string and zero probes, in both the JSON results and the markdown report. A lane that had emitted lift: 0 would have been worse than useless — it would have looked like a finding. Writing the failure path before the success path is what made that possible.

AI-augmented in bounded lanes, with a purity test as the arbiter. The implementation came from an agent relay: one agent building or auditing a lane, a second cross-checking behaviour and evidence, the author adjudicating. For this repository the decisive arbiter is mechanical — npm test and specifically npm run check:pure. A model can argue that a convenience writeFile in the core is harmless; the purity gate cannot be argued with.

A stability sweep that proposed instead of touching. A later review pass produced 9 proposals and implemented none of them, because each fell outside its allowlist — a behaviour change, a schema change, a frozen zone, or new content. Each is recorded with rationale, effort, risk, and zone, and several are uncomfortable to publish: an MCP input schema that never mentions the tool_evidence field the anti-gaming gate demands, a duplicated flag parser, an unreferenced probe file, and the observation that the previous README omitted the purity check from its verification section. Publishing the list of things you decided not to fix, with the reason, is more useful than a clean-looking repository. (This README documents npm run check:pure, which closes that last one.)

A fail-closed publication gate. This public snapshot is produced by an out-of-repo sanitizing pipeline that never writes to the private source. It archives a ref, applies exclusions, overlays publication-only documentation, then refuses to commit unless every gate is green: secret scanning, credential-shaped-file detection, absolute-home-path and email-shape checks, a fixed-string employer/client wordlist expanded over delimiter variants, exclusion verification, a syntax build, the full test suite, and a claims gate that re-runs the command behind each number here. Two exclusions are worth naming: development handoff documents are removed from the public tree, and a hard-coded ledger path pointing at the author's own filesystem is rewritten to an environment-overridable default inside the repository — a public tool must not assume one machine's directory layout.

Development

npm test                    # 33 tests across 9 files (node --test)
npm run check:pure          # the read-only guarantee: 14 files vs 6 forbidden patterns
node --check forge/cli.mjs  # syntax gate, also run by the publication pipeline
node --check mcp/server.mjs
node bin/reasonforge ab-run --dry-run       # regenerates harness artefacts
node bin/reasonforge rebenchmark-plan       # report-only; owner-gated

ab-run and the GPT lane write into harness/; nothing else in this repository writes anything. If you add a module to forge/, the purity gate picks it up automatically — it enumerates the directory rather than listing files, so a new module cannot be forgotten. If you change REASON_VERSION in forge/version.mjs, run rebenchmark-plan and treat the previous report as stale.

Before proposing a change, read docs/SWEEP_PROPOSALS.md — it may already be there with a reason it was deferred.

Localization

The English README is canonical; 12 translations live under docs/i18n/ and are listed in docs/i18n/README-INDEX.md. Keep commands, paths, tool names, numbers, links, and measured claims synchronized across all of them — an English change that touches a number is not finished until the translations carry the same number. Tool names, verdict tokens (ACCEPT, REJECT, DEFER, BLOCKED), and task_type values are part of the machine contract and are never translated — a translated verdict token would be a bug, not a courtesy.

Credits

See CREDITS.md for the four-tier register covering probe sets, fixtures, and generated reports, the fail-closed rule for unlisted content, and what a fork must remove. Its terms take precedence over the code licence for those files.

License

Source code is licensed under the Apache License 2.0; modification notices are recorded in NOTICE. Probe sets, fixtures, and generated reports are not licensed by Apache-2.0 and remain governed by CREDITS.md.