ReasonForge
by euuuuuuan
README.md
<!-- Rewritten for the public snapshot: the development README was replaced with verified public documentation. Every number below is re-measured by the publication claims gate before a commit is created. -->
<div align="center">
# ReasonForge
**A deterministic, read-only reasoning device for hard questions: eight MCP tools that force a scope lock, a cheapest-disconfirming-step plan, a blind-spot sweep, and a verdict gate that refuses to accept a claim without proof of execution.**
[](LICENSE)
[](package.json)
[](package.json)
[](#development)
[](tests/purity.test.mjs)
[](mcp/server.mjs)
**English** · [한국어](docs/i18n/README.ko_KR.md) · [日本語](docs/i18n/README.ja_JP.md) · [简体中文](docs/i18n/README.zh_CN.md) · [繁體中文](docs/i18n/README.zh_TW.md) · [Español](docs/i18n/README.es.md) · [Português (Brasil)](docs/i18n/README.pt_BR.md) · [Deutsch](docs/i18n/README.de_DE.md) · [Français](docs/i18n/README.fr_FR.md) · [Русский](docs/i18n/README.ru_RU.md) · [Italiano](docs/i18n/README.it_IT.md) · [Bahasa Indonesia](docs/i18n/README.id_ID.md) · [Türkçe](docs/i18n/README.tr_TR.md)
[Quickstart](#quickstart) · [Architecture](#architecture) · [Orchestration](docs/ORCHESTRATION.md) · [Open proposals](docs/SWEEP_PROPOSALS.md) · [Credits](CREDITS.md)
</div>
> [!IMPORTANT]
> Apache-2.0 covers source code only. Probe sets, fixtures, and generated reports are governed by the records in [CREDITS.md](CREDITS.md), which take precedence for those files. Content not listed there is unrecorded, not free to reuse.
## What this is
ReasonForge is a **tool**, not a product and not an agent: a small vendor-neutral reasoning harness that a coding agent can call, over MCP or a CLI, to impose structure on a hard question. It classifies the prompt, decides whether scaffolding is even worth applying, emits a plan whose first move is the cheapest way to be proven wrong, forces a pre-verdict blind-spot sweep with line-level evidence, and then gates the verdict: a final ACCEPT or REJECT requires real evidence references *plus* falsification attempts, and a time-sensitive or "verified fact" claim additionally requires proof of execution — actual tool output, not a citation that could be invented. A calibrated DEFER with a concrete fetch plan is a valid answer; a confident answer with nothing behind it is not. It is deliberately **read-only**: the core and the MCP server do not write files, spawn processes, or touch the network, and that property is enforced by a test rather than by convention.
## Highlights
- **33 tests across 9 test files**, run by Node's built-in test runner, completing in well under a second on this snapshot. **Zero runtime dependencies** — `package.json` declares no `dependencies` at all.
- **A purity gate that is a product guarantee, not a lint rule.** `tests/purity.test.mjs` scans all **13** `forge/` modules plus the MCP server — 14 files — against **6** forbidden patterns covering `node:fs` imports, any write/append call, `child_process`/`spawn`/`exec`, `fetch`, and raw `http`/`https`/`net`/`dgram` use. If the core ever gains a side effect, the suite goes red.
- **Exactly 8 MCP tools**, asserted by name in the test suite so the surface cannot drift silently: `reason_intake`, `reason_route`, `reason_plan`, `reason_rubric`, `reason_sweep_checklist`, `reason_verdict_gate`, `reason_reflect`, `reason_eval`.
- **The CLI and the MCP tool call share one core**, and a test asserts they produce the same output — so the two surfaces cannot diverge into two behaviours.
- **A router that is allowed to say "skip".** Prompts are classified into a `task_type` and routed to `full`, `light`, or `skip`. Well-specified technical execution is deliberately routed to light touch, because taxing a migration that already has concrete constraints is a cost with no benefit. A scaffold that always fires is a scaffold that cannot be evaluated.
- **An ablation harness with 13 probes and 6 variants** — control plus `full_loop`, `minus_sweep`, `minus_verdict_gate`, `minus_plan`, and `intake_route_only` — so each stage of the loop can be removed and measured rather than assumed useful.
- **Blind judging built into the artefact format.** A dry run emits shuffled judge packets and a **separate** answer key with **60** attribution rows and a `warning` field telling you to keep it away from the judge. The key's own `score_source` field reads `answer-key only; not a judge result`.
- **A benchmark lane that reports its own failure instead of a number.** The GPT lane requires a PTY-capable execution path; when the subprocess transport could not allocate one, `harness/gpt-lane-results.json` records `unavailable: true` with the exact reason (`tcgetattr/ioctl: Operation not supported on socket`) and **zero** probes, rather than emitting a plausible-looking lift.
- **9 open proposals, none implemented**, catalogued in [`docs/SWEEP_PROPOSALS.md`](docs/SWEEP_PROPOSALS.md) with rationale, effort, risk, and zone — including one that criticises this very README's predecessor for omitting the purity check from its verification section.
## Use it
Three surfaces over one shared pure core:
| Surface | Entry point | Use when |
|---|---|---|
| CLI | `bin/reasonforge` | Driving the loop by hand or from a script. |
| MCP server | `mcp/server.mjs` (stdio, protocol revision `2024-11-05`) | Registering the eight tools with an MCP-capable agent. |
| Hook helper | `hook/reasonforge-hook` | Injecting routing context at session start in an agent that supports hooks. Fail-open by contract. |
```bash
reasonforge intake "Compare MCP hooks and markdown rules for cross-agent reasoning."
reasonforge plan "Find the safest fix for a flaky production retry path."
reasonforge verdict --json '{"verdict":"ACCEPT","evidence_refs":["tests/core.test.mjs"],"falsification_attempts":["missing falsification blocks"]}'
reasonforge verdict --json '{"verdict":"DEFER","fetch_plan":["Browse one current source first."]}'
reasonforge hook --event session_start
reasonforge ab-run --dry-run # writes the report, blind packets, and answer key
reasonforge rebenchmark-plan # report-only, owner-gated
```
Honest labels you should read before quoting any number out of this repository:
- **Dry-run and local-live A/B scores are `heuristic-directional`,** and the report says so about itself. A cut, promote, or lift claim requires judging the emitted blind packets with a separate frontier judge and only then deshuffling with the private answer key. The harness will not make that claim for you.
- **The checked-in dry-run report covers 12 of the 13 probes.** The thirteenth, an anti-gaming probe (`gaming-fabricated-verified-fact`), was added after that report was generated and does not appear in it. The report is a committed artefact, not a live measurement — regenerate it before trusting it.
- **`probes/reasoning-golden.json` is currently referenced by nothing** — no module, no test. It is either a golden set awaiting an evaluator or a leftover; it sits in a frozen zone and is escalated for an owner decision as proposal P4 rather than quietly deleted.
- **The GPT lane's recorded result is "unavailable".** That is the honest output of a transport that could not run, not a zero-lift finding.
## Quickstart
Prerequisites: **Node.js 22 or newer** (`engines.node >= 22`), and nothing else — there are no runtime dependencies to install, and `npm install` has nothing to fetch.
```bash
git clone <your-fork-url> reasonforge
cd reasonforge
node --check forge/cli.mjs && node --check mcp/server.mjs # syntax gate
npm test # 33 tests
npm run check:pure # read-only guarantee
node bin/reasonforge ab-run --dry-run # deterministic fixtures
node mcp/server.mjs # stdio MCP server
```
The first three commands are exactly what the publication gate runs. `ab-run --dry-run` writes into `harness/`, so run it deliberately rather than as part of a check — the publication gate does **not** run it, precisely so that a gate cannot rewrite a committed artefact behind your back.
## Architecture
```mermaid
flowchart TB
subgraph SURF["Surfaces"]
CLI["bin/reasonforge — CLI"]
MCPS["mcp/server.mjs — 8 MCP tools, stdio"]
HK["hook/reasonforge-hook — fail-open"]
end
subgraph CORE["forge/ — 13 pure modules, no fs / spawn / network"]
IN["reason_intake — classify + scope lock"] --> RT{"reason_route"}
RT -->|skip| ANS["answer directly"]
RT -->|light| DIS["cheapest disconfirming step"]
RT -->|full| PL["reason_plan — six moves, disconfirmation first"]
PL --> RB["reason_rubric — scoring criteria"]
RB --> SW["reason_sweep_checklist — blind-spot sweep, line evidence"]
SW --> VG{"reason_verdict_gate"}
VG -->|"evidence + falsification + proof of execution"| ACC["ACCEPT / REJECT"]
VG -->|"missing proof"| DEF["DEFER / BLOCKED + concrete fetch plan"]
ACC --> RF["reason_reflect — reusable lesson"]
DEF --> RF
RF --> EV["reason_eval — read-only ledger row"]
DIS --> EV
end
CLI --> IN
MCPS --> IN
HK --> RT
subgraph HARN["harness/ — the only place side effects live"]
AB["ab-run: 13 probes x 6 variants"] --> PKT["shuffled blind judge packets"]
AB --> KEY["answer key, 60 attribution rows, kept from the judge"]
GL["gpt-lane: records unavailable + reason rather than a number"]
end
EV -.->|read-only rows| AB
PT["tests/purity.test.mjs — 14 files x 6 forbidden patterns"] -.->|guards| CORE
```
Load-bearing properties:
- **Purity is architectural, not stylistic.** The side-effecting code lives in `harness/`, never in `forge/` or `mcp/`. That split is what makes the core safe to call from an agent loop, and the purity test is what stops the split from eroding — a single `writeFile` in a `forge/` module fails the suite.
- **The verdict gate's hard part is proof of execution.** Requiring "evidence references" is easy to satisfy dishonestly, because a reference can be fabricated. So a time-sensitive assertion, or one asserting verified specifics, additionally requires actual tool output; and citing the scaffold's *own* verdict as evidence is explicitly invalid. This is the anti-gaming rule, and it exists because a first version was gameable.
- **DEFER is a first-class outcome.** The gate does not force a decision; it forces an *honest* one. A calibrated DEFER with a concrete fetch or handoff plan passes. This is the difference between a reasoning aid and a confidence amplifier.
- **The ablations are the point of the harness.** Removing the sweep, the verdict gate, or the plan and measuring each is what turns "this loop helps" from an assertion into a hypothesis. The harness deliberately withholds the strong claim (`heuristic-directional`) until a blind judge has been run.
- **Version identity is explicit.** `forge/version.mjs` carries a package version, a dated `REASON_VERSION` for the loop contract, and the MCP protocol revision separately, so a loop change and a package bump are not confused for each other. Changing `REASON_VERSION` is what triggers the report-only rebenchmark plan.
## Project layout
```text
bin/reasonforge CLI entry point
forge/ 13 pure modules — the shared core
intake.mjs prompt classification and scope lock
route.mjs full / light / skip decision from task_type and model tier
plan.mjs six-move plan, cheapest disconfirming step first
rubric.mjs scoring criteria
sweep.mjs pre-verdict blind-spot checklist
disconfirm.mjs the falsification move
verdict.mjs the gate: evidence + falsification + proof of execution
reflect.mjs eval.mjs reusable lesson, read-only ledger row
cli.mjs hook.mjs frame.mjs version.mjs
mcp/server.mjs stdio MCP server, exactly 8 tools, protocol 2024-11-05
hook/reasonforge-hook session-start context injection, fail-open
harness/ the only side-effecting code
ab.mjs ab-run.mjs ablation harness: 13 probes x 6 variants
gpt-lane.mjs cross-model lane; records unavailability honestly
rebenchmark.mjs report-only plan after a REASON_VERSION change
ab-report.md committed dry-run artefact (12 probes; see honest labels)
ab-blind-packets.json shuffled packets for a blind judge
ab-answer-key.json 60 attribution rows; keep away from the judge
probes/ab-probes.json 13 A/B probes across moderate, hard, calibration
probes/reasoning-golden.json 3 entries, currently referenced by nothing (P4)
tests/ 9 files, 33 tests, including the purity gate
docs/ORCHESTRATION.md the loop, the cross-agent shapes, the harness boundaries
docs/SWEEP_PROPOSALS.md 9 proposals, none implemented, each owner-gated
```
## How this was built
**The harness exists to try to disprove the product.** ReasonForge's thesis is that a structured loop improves hard reasoning. The honest way to hold that thesis is an ablation: remove the sweep, remove the verdict gate, remove the plan, and measure each variant against a control. That is what `harness/ab.mjs` does with 6 variants over 13 probes — and the results are deliberately labelled `heuristic-directional` so that the harness's own numbers cannot be quoted as a lift claim. Building the instrument that could embarrass the design was the first real deliverable.
**Blindness is enforced by file layout.** The dry run emits shuffled judge packets and a separate answer key, and the key carries a `warning` field saying to keep it from the judge plus a `score_source` field reading `answer-key only; not a judge result`. Attribution is only possible after scoring. This is cheap to implement and it removes the most common way a self-evaluation quietly becomes self-congratulation.
**An anti-gaming rule that came from being gamed.** The verdict gate originally accepted "evidence references", which a model can simply write. The current rule requires **proof of execution** — real tool output — for time-sensitive or verified-fact ACCEPTs, and explicitly rejects citing the scaffold's own verdict as evidence. The thirteenth probe, `gaming-fabricated-verified-fact`, exists specifically to attack that path, and its absence from the committed report is called out above rather than glossed over.
**A benchmark that refused to fabricate.** The GPT lane could not allocate a PTY through its subprocess transport. The recorded outcome is `unavailable: true`, with the raw error string and zero probes, in both the JSON results and the markdown report. A lane that had emitted `lift: 0` would have been worse than useless — it would have looked like a finding. Writing the failure path before the success path is what made that possible.
**AI-augmented in bounded lanes, with a purity test as the arbiter.** The implementation came from an agent relay: one agent building or auditing a lane, a second cross-checking behaviour *and* evidence, the author adjudicating. For this repository the decisive arbiter is mechanical — `npm test` and specifically `npm run check:pure`. A model can argue that a convenience `writeFile` in the core is harmless; the purity gate cannot be argued with.
**A stability sweep that proposed instead of touching.** A later review pass produced 9 proposals and implemented **none** of them, because each fell outside its allowlist — a behaviour change, a schema change, a frozen zone, or new content. Each is recorded with rationale, effort, risk, and zone, and several are uncomfortable to publish: an MCP input schema that never mentions the `tool_evidence` field the anti-gaming gate demands, a duplicated flag parser, an unreferenced probe file, and the observation that the previous README omitted the purity check from its verification section. Publishing the list of things you decided not to fix, with the reason, is more useful than a clean-looking repository. (This README documents `npm run check:pure`, which closes that last one.)
**A fail-closed publication gate.** This public snapshot is produced by an out-of-repo sanitizing pipeline that never writes to the private source. It archives a ref, applies exclusions, overlays publication-only documentation, then refuses to commit unless every gate is green: secret scanning, credential-shaped-file detection, absolute-home-path and email-shape checks, a fixed-string employer/client wordlist expanded over delimiter variants, exclusion verification, a syntax build, the full test suite, and a claims gate that re-runs the command behind each number here. Two exclusions are worth naming: development handoff documents are removed from the public tree, and a hard-coded ledger path pointing at the author's own filesystem is rewritten to an environment-overridable default inside the repository — a public tool must not assume one machine's directory layout.
## Development
```bash
npm test # 33 tests across 9 files (node --test)
npm run check:pure # the read-only guarantee: 14 files vs 6 forbidden patterns
node --check forge/cli.mjs # syntax gate, also run by the publication pipeline
node --check mcp/server.mjs
node bin/reasonforge ab-run --dry-run # regenerates harness artefacts
node bin/reasonforge rebenchmark-plan # report-only; owner-gated
```
`ab-run` and the GPT lane write into `harness/`; nothing else in this repository writes anything. If you add a module to `forge/`, the purity gate picks it up automatically — it enumerates the directory rather than listing files, so a new module cannot be forgotten. If you change `REASON_VERSION` in `forge/version.mjs`, run `rebenchmark-plan` and treat the previous report as stale.
Before proposing a change, read [`docs/SWEEP_PROPOSALS.md`](docs/SWEEP_PROPOSALS.md) — it may already be there with a reason it was deferred.
## Localization
The English README is canonical; 12 translations live under [`docs/i18n/`](docs/i18n/README-INDEX.md) and are listed in [`docs/i18n/README-INDEX.md`](docs/i18n/README-INDEX.md). Keep commands, paths, tool names, numbers, links, and measured claims synchronized across all of them — an English change that touches a number is not finished until the translations carry the same number. Tool names, verdict tokens (`ACCEPT`, `REJECT`, `DEFER`, `BLOCKED`), and `task_type` values are part of the machine contract and are **never** translated — a translated verdict token would be a bug, not a courtesy.
## Credits
See [CREDITS.md](CREDITS.md) for the four-tier register covering probe sets, fixtures, and generated reports, the fail-closed rule for unlisted content, and what a fork must remove. Its terms take precedence over the code licence for those files.
## License
Source code is licensed under the [Apache License 2.0](LICENSE); modification notices are recorded in [NOTICE](NOTICE). Probe sets, fixtures, and generated reports are not licensed by Apache-2.0 and remain governed by [CREDITS.md](CREDITS.md).
This server cannot be deployed
Maintenance
ActivitySlowing
ResponsivenessNo issues