Skip to main content
Glama
Peer Review MCP Server

Peer Review MCP Server

by raydollete

Peer Review MCP Server

A provider-agnostic MCP (Model Context Protocol) server that asks a weighted quorum of external LLM peers the same question and returns a consensus answer with a machine-consumable certainty score. It supports any OpenAI-compatible or Anthropic-compatible API, stacks sources in cost tiers, and degrades visibly (never silently) when a source is unavailable.

License: Apache-2.0 Node: >=20

Features

  • Weighted quorum with a certainty score — peers vote with configurable weights; every peer_review response carries a machine-consumable certaintyScore.

  • Provider-agnostic — two wire formats (OpenAI chat/completions, Anthropic v1/messages) cover OpenAI, Gemini, OpenRouter, Ollama, vLLM, Anthropic, and most gateways.

  • Cost tiers with escalation — tier 1 sources are consulted first; higher tiers only fan out on quorum shortfall.

  • Visible degradation — failed or unavailable sources are itemized in the result, never silently dropped; an unmet quorum is a reported outcome, not an error.

  • No vendor SDKs — all providers are reached over plain HTTPS fetch.

Related MCP server: House of Wisdom MCP

Table of Contents

Requirements

  • Node >= 20

Installation

git clone https://github.com/raydollete/peer-review.git && cd peer-review
npm install
npm run build        # emits dist/app.js

Quick Start

Create a config file (default path ./peer-review.config.json):

{
  "thresholds": { "tier1": 2, "tier2": 4 },          // weight targets — see "How tiers, weights, and thresholds interact"
  "arbiter": "gpt-large",                             // source name used for consensus evaluation
  "sources": [
    { "name": "gpt-large", "apiType": "openai",    "baseUrl": "https://api.openai.com/v1",  "model": "gpt-5.2", "apiKeyEnv": "OPENAI_API_KEY",  "weight": 2, "tier": 1, "timeoutMs": 60000 },
    { "name": "claude",    "apiType": "anthropic", "baseUrl": "https://api.anthropic.com",  "model": "claude-fable-5", "apiKeyEnv": "ANTHROPIC_API_KEY", "weight": 2, "tier": 2, "timeoutMs": 120000 }
  ]
}

Then register the server in Claude Code:

claude mcp add peer-review \
  --env PEER_REVIEW_CONFIG=/absolute/path/to/peer-review.config.json \
  --env OPENAI_API_KEY=sk-... \
  -- node /absolute/path/to/peer-review/dist/app.js

Configuration

The server reads a JSON config file from PEER_REVIEW_CONFIG (default ./peer-review.config.json). See peer-review.config.example.json.

Config file fields

Field

Type

Required

Description

thresholds

object

yes

Map of tier → positive integer weight target. Keys may be "tier1" or "1". Every tier used by a source must have a threshold (validated at startup).

arbiter

string

yes

Name of the source used to evaluate agreement between peer responses. Must match a sources[].name.

sources[].name

string

yes

Unique source name (used in query_peer/count_tokens and in quorum reports).

sources[].apiType

"openai" | "anthropic"

yes

Wire format: openai → POST {baseUrl}/chat/completions; anthropic → POST {baseUrl}/v1/messages.

sources[].baseUrl

URL

yes

API base URL. For openai, include the version segment (e.g. https://api.openai.com/v1).

sources[].model

string

yes

Model id sent to the provider. Never client-suppliable.

sources[].apiKeyEnv

string

exactly one of these two

Name of the environment variable holding a static API key. If the variable is unset the source is marked unavailable (the server still starts).

sources[].apiKeyCommand

string

exactly one of these two

Command executed (via sh -c) to mint a short-lived bearer token, e.g. gcloud auth print-access-token for Vertex AI. Output is trimmed and cached (see PEER_CREDENTIAL_TTL_S); a 401 triggers one re-mint + retry.

sources[].weight

positive int

yes

Weight this source contributes to quorum when it agrees with the consensus.

sources[].tier

positive int

yes

Cost tier. Tier 1 is consulted first; higher tiers only on shortfall.

sources[].local

bool

no

Marks a source served by the local inference backend. A caller that itself runs on that backend sets PEER_REVIEW_SKIP_LOCAL so these sources are dropped from its requests (self-contention, not a second opinion). Locality, not model id.

sources[].jsonMode

bool

no

Asks the provider for a JSON-object reply on arbiter evaluations via response_format. OpenAI-compatible sources only — startup rejects it on anthropic. Off = request bodies unchanged. Set it on the arbiter source once you have confirmed its endpoint accepts response_format.

sources[].maxOutputTokens

int

no

Per-source override of PEER_MAX_OUTPUT_TOKENS, sent as max_tokens. Reasoning models bill hidden thinking against it before the answer, so give them a budget above their thinking length (32768 is a safe Anthropic value) and raise timeoutMs with it. A reply stopped at the ceiling fails the source with cause: peer_truncated rather than voting half-written text.

Startup fails fast on: an unreadable/invalid config file, an unknown arbiter name, a source tier without a threshold, duplicate source names, a source named caller (reserved for the callerAnswer rating channel), a source declaring both or neither of apiKeyEnv/apiKeyCommand, a source declaring jsonMode with apiType: "anthropic", or a timeout budget that cannot fit the first tier's wave plus its arbiter evaluation inside PEER_REVIEW_DEADLINE_MS. The first wave is sized to its slowest configured peer, plus a separate arbiter allowance, so a first verdict is always reachable. Higher tiers are not summed: at run time each tier's wave is capped at the time remaining minus one arbiter evaluation, and a tier with no time left is not launched and is itemized skipped. Credential material never appears in config, logs, or responses.

Environment variables

Variable

Default

Description

PEER_REVIEW_CONFIG

./peer-review.config.json

Path to the config file.

PEER_TIMEOUT_MS

120000

Per-source request timeout (includes retries).

PEER_REVIEW_DEADLINE_MS

280000

Hard overall deadline for one peer_review call; on expiry in-flight calls are aborted and the accumulated (possibly degraded) result is returned. MUST stay strictly below the declared PEER_REVIEW_CALLER_TIMEOUT_MS (next row) — above it the caller abandons the call at its own ceiling and reports only its generic timeout, so the partial result arrives after the caller has given up and no verdict anyone computed is read. Each tier's peer wave gets the smaller of its configured timeout and the time remaining minus the arbiter timeout. Also bounds query_peer.

PEER_REVIEW_CALLER_TIMEOUT_MS

300000

Declared MCP client tool-call timeout (ms) — the ceiling this server assumes. The harness registration owner sets the real value (e.g. omp's per-server timeout key); PEER_REVIEW_DEADLINE_MS must stay strictly below it. A deadline at or above it makes the server warn once at startup and report the mismatch (cause: configuration) on list_peers and every failure envelope while it holds.

PEER_REVIEW_SKIP_LOCAL

false

When true (true/1), every source flagged local is excluded from the quorum for this server registration. Set it per-caller in that caller's MCP registration env when the caller itself runs on the local backend, so it does not contend with itself. The class-wide sibling of the per-request callerSource skip.

PEER_MAX_OUTPUT_TOKENS

8192

Max output tokens injected server-side into every peer call; a source may override it with sources[].maxOutputTokens. Hidden reasoning is billed against this budget, so a reasoning model needs a budget above its thinking length or it answers with nothing and the source fails cause: peer_truncated.

PEER_CREDENTIAL_TTL_S

3000

Cache TTL (seconds) for tokens minted via apiKeyCommand.

LOG_LEVEL

info

pino log level (fatal|error|warn|info|debug|trace). Logs go to stderr only.

NODE_ENV

—

production switches the logger from pretty-printing to JSON.

How tiers, weights, and thresholds interact

A peer_review call is a weighted vote. The config decides who votes, in what order, and how many votes win:

  • weight — how much a source's agreement counts. Give larger weights to sources you trust more.

  • tier — cost band. Tier 1 is consulted on every call; tier 2 only when tier 1 falls short; and so on. Put cheap, fast models in tier 1 and expensive frontier models above.

  • thresholds — the agreeing weight required to declare quorum, keyed by target tier.

The one rule that is easy to miss:

A call has a single required weight: the threshold of its target tier. The target tier is the highest configured tier unless the caller caps escalation with the tier input. Lower tiers' thresholds are not applied along the way — thresholds.tier1 only ever binds a call made with "tier": 1.

So with the Quick Start config ({"tier1": 2, "tier2": 4}), a default call must accumulate agreeing weight 4 (tier 2's threshold). Escalation is lazy, not mandatory: if tier 1's agreeing weight alone reaches 4, tier 2 is never contacted.

Execution order for one call:

  1. All sources of the lowest tier are queried in parallel.

  2. The arbiter (at temperature 0) rates each response 0–1 against its own consensus answer. A source agrees when its rating is ≥ 0.7; only agreeing sources' weights count. Evaluation is triggered mid-tier as soon as the responses in hand could arithmetically meet the required weight — and again at the end of every tier, whether or not the required weight was reachable, so the accumulated answers are always rated before escalating.

  3. If agreeing weight ≥ required weight, quorum is achieved: any still-in-flight requests are aborted (cost saved) and the arbiter's consensus answer is returned.

  4. Otherwise the next tier fans out and the arbiter re-rates over all accumulated responses — earlier answers are never discarded.

  5. Tiers that were never reached do not appear in quorum.sources[] at all.

The time budget (the reserve model). sources[].timeoutMs is a per-request ceiling, and the run budgets the tiers that escalation walks in two stages. Startup hard-checks the first tier only: its wave (its slowest source's effective timeout — an unset timeoutMs means the PEER_TIMEOUT_MS default, 120 000 ms) plus one arbiter evaluation must fit strictly inside PEER_REVIEW_DEADLINE_MS, so a first verdict is always reachable. Later tiers reserve at entry: when escalation reaches a tier, the run reserves one arbiter evaluation and clamps that tier's wave to the remaining budget (the smaller of the tier's configured timeout and what the reserve leaves). A tier that cannot start is not billed and not faked as down: it is itemized as skipped with 'Skipped: no time left before the deadline after the arbiter reserve'. The global deadline (the wall) is the backstop that bounds the whole call. The practical rule for the tiers escalation walks: the earlier tiers' slowest timeouts plus one arbiter evaluation each must leave the escalated tier time to launch — budget them against PEER_REVIEW_DEADLINE_MS and recompute whenever you change it (the shipped peer-review.config.example.json budgets 60 s + 60 s before a tier-2 entry under the 280 s default). A higher tier's own timeoutMs may exceed the remaining budget — that value is also its query_peer/preflight ceiling, and the quorum path clamps the wave at run time.

Sizing guidance:

  • Overprovision each tier. A tier whose potential weight exactly equals the threshold escalates the moment any one source fails or dissents. Three weight-1 sources against a threshold of 2 tolerate one outlier; two sources tolerate none.

  • Prefer model diversity within a tier. Two copies of the same model mostly vote together, so their combined weight overstates the independence of the "second opinion".

  • Keep the required weight reachable. The target tier's threshold must be ≤ the summed weight of all sources in tiers ≤ target, or quorum.achieved can never be true.

  • Let lower tiers be able to win. A tier can only short-circuit a call if the weight accumulated through that tier can reach the target tier's threshold. If it can't, that tier is pure overhead on default calls: its peers are billed, the end-of-tier arbiter rating is billed, and escalation happens regardless. With {"tier1": 2, "tier2": 3} over two weight-1 tier-1 sources, a default call can only ever reach weight 2 at tier 1 against a required 3 — so tier 1 never saves anything. Either raise the tier-1 weights, add a tier-1 source, or lower tier2 so the cheap tier has a path to quorum. A tier that can't win is still useful if you want its answers in the mix for the later arbiter pass — just size it knowing that's what you're paying for.

  • Match tier to speed, and isolate slow reasoners. A tier is a cost/latency class, not a redundancy slot: a tier's peers race, and the instant the accumulated agreeing weight meets the target threshold, every still-in-flight peer in that tier is aborted (step 3 above). So a slow reasoning model sharing a tier with a faster one is cancelled before it can answer every time the faster one already suffices — it never contributes, and it surfaces in quorum.sources[] as cancelled. Give a slow reasoner its own higher tier as the sole member, so it is awaited and folded into the re-rating when escalation reaches it. If you must place several reasoners in one tier, set that tier's threshold equal to the summed weight of all its members, so no faster subset can meet the bar and abort the rest.

  • Span providers, not just models. Cross-provider diversity beats cross-model diversity within one provider: models from the same lab share training data and failure modes, so their agreement overstates the independence of the second opinion. Draw a tier's voters from different providers where you can.

  • Never place a local source alone in your top tier. A caller running on the local backend sets PEER_REVIEW_SKIP_LOCAL, which drops every local source from the quorum. If a local model is the sole member of the escalation tier, that tier is empty for that caller and quorum can never escalate into it.

  • Pick a cheap, reliable arbiter. The arbiter is called at least once per peer_review call plus once per escalation, and its tokens count toward tokenUsage. It can double as a regular peer, but note that it then rates its own answer alongside the others — check how much of your threshold that self-rating can supply, and prefer an arbiter that isn't also one of your heaviest voters. If the arbiter's own call fails, the whole evaluation falls back to arbiterFailed: true with certaintyScore: 0 — so give the arbiter role to the source with your most dependable credential.

Example: a three-tier stack

The walkthroughs below all use this config — two cheap flash-class voters in tier 1, a mid-price model in tier 2, a frontier model in tier 3:

{
  "thresholds": { "tier1": 2, "tier2": 3, "tier3": 4 },
  "arbiter": "gpt-mini",
  "sources": [
    { "name": "gemini-flash", "apiType": "openai",    "baseUrl": "https://generativelanguage.googleapis.com/v1beta/openai", "model": "gemini-3-flash-preview", "apiKeyEnv": "GEMINI_API_KEY",    "weight": 1, "tier": 1, "timeoutMs": 45000 },
    { "name": "gpt-mini",     "apiType": "openai",    "baseUrl": "https://api.openai.com/v1",                               "model": "gpt-5-mini",             "apiKeyEnv": "OPENAI_API_KEY",    "weight": 1, "tier": 1, "timeoutMs": 45000 },
    { "name": "gemini-pro",   "apiType": "openai",    "baseUrl": "https://generativelanguage.googleapis.com/v1beta/openai", "model": "gemini-3-pro",           "apiKeyEnv": "GEMINI_API_KEY",    "weight": 2, "tier": 2, "timeoutMs": 60000 },
    { "name": "claude",       "apiType": "anthropic", "baseUrl": "https://api.anthropic.com",                               "model": "claude-fable-5",         "apiKeyEnv": "ANTHROPIC_API_KEY", "weight": 2, "tier": 3, "timeoutMs": 120000 }
  ]
}

A default call targets tier 3, so it needs agreeing weight 4. A "tier": 1 call needs only 2. Note the arbiter is gpt-mini, deliberately not one of the Gemini sources: if GEMINI_API_KEY ever goes missing, evaluation still works (see scenario 4).

What happens when a query comes in

1. Easy factual query — quorum at tier 2, tier 3 never billed

{ "prompt": "What does HTTP status 431 mean? One sentence." }

Tier 1 fans out to gemini-flash and gpt-mini; both agree, but their combined weight (2) can't reach the required 4, so tier 2 is consulted. gemini-pro agrees too: 1 + 1 + 2 = 4 ≥ 4 — quorum. claude is never called and never appears in sources[].

// data
{
  "response": "HTTP 431 (Request Header Fields Too Large) means the server refused the request because its headers are too big.",
  "certaintyScore": 0.98,       // min(1, 4/4) × mean(1.0, 1.0, 0.95), rounded
  "quorum": {
    "achieved": true,
    "tier": 2,                  // tier reached; requiredWeight stays the target tier's threshold
    "requiredWeight": 4,
    "agreeingWeight": 4,
    "sources": [
      { "name": "gemini-flash", "model": "gemini-3-flash-preview", "status": "ok", "weight": 1, "agreement": 1 },
      { "name": "gpt-mini",     "model": "gpt-5-mini",             "status": "ok", "weight": 1, "agreement": 1 },
      { "name": "gemini-pro",   "model": "gemini-3-pro",           "status": "ok", "weight": 2, "agreement": 0.95 }
    ]
  },
  "tokenUsage": { "prompt": 812, "completion": 118, "total": 930 }   // includes both arbiter rounds
}

2. Low-stakes check — cap escalation with "tier": 1

{ "prompt": "Is 'reciept' spelled correctly?", "tier": 1 }

Now the required weight is tier 1's threshold (2). Both flash sources agree, quorum is achieved in one round, and tiers 2–3 stay uncontacted. This is the cheap path for queries where two small models agreeing is good enough. The ceiling holds when every tier-1 source answers — a disagreement-shaped miss is revisable, so escalating it would only cost more. An outage-shaped miss is not: if one tier-1 source errors and the attainable weight (the summed weight of sources that answered) falls below tier 1's required weight, the run escalates to the next configured tier to restore the capacity to reach the bar, while requiredWeight stays tier 1's 2.

// data
{
  "response": "No — the correct spelling is \"receipt\".",
  "certaintyScore": 0.95,       // min(1, 2/2) × mean(1.0, 0.9)
  "quorum": {
    "achieved": true,
    "tier": 1,
    "requiredWeight": 2,
    "agreeingWeight": 2,
    "sources": [
      { "name": "gemini-flash", "model": "gemini-3-flash-preview", "status": "ok", "weight": 1, "agreement": 1 },
      { "name": "gpt-mini",     "model": "gpt-5-mini",             "status": "ok", "weight": 1, "agreement": 0.9 }
    ]
  },
  "tokenUsage": { "prompt": 296, "completion": 41, "total": 337 }
}

3. Contested question — an outlier forces the full stack

{ "prompt": "In Node 20, is structuredClone faster than a JSON round-trip for small flat objects?" }

gemini-flash answers the opposite of everyone else and the arbiter rates it 0.2 — below the 0.7 agreement bar, so its weight is excluded (but still itemized). Agreeing weight climbs 1 → 3 → 5 as tiers 2 and 3 are pulled in, with the arbiter re-rating the accumulated set each round; quorum lands at tier 3.

// data
{
  "response": "For small flat objects a JSON round-trip is usually still faster in Node 20; structuredClone wins once values contain types JSON can't represent.",
  "certaintyScore": 0.9,        // min(1, 5/4) × mean(0.95, 0.9, 0.85) — outlier's 0.2 not averaged in
  "quorum": {
    "achieved": true,
    "tier": 3,
    "requiredWeight": 4,
    "agreeingWeight": 5,        // gpt-mini 1 + gemini-pro 2 + claude 2; gemini-flash excluded
    "sources": [
      { "name": "gemini-flash", "model": "gemini-3-flash-preview", "status": "ok", "weight": 1, "agreement": 0.2 },
      { "name": "gpt-mini",     "model": "gpt-5-mini",             "status": "ok", "weight": 1, "agreement": 0.95 },
      { "name": "gemini-pro",   "model": "gemini-3-pro",           "status": "ok", "weight": 2, "agreement": 0.9 },
      { "name": "claude",       "model": "claude-fable-5",         "status": "ok", "weight": 2, "agreement": 0.85 }
    ]
  },
  "tokenUsage": { "prompt": 2140, "completion": 486, "total": 2626 }
}

4. Degraded run — missing credential, quorum unreachable

Same easy prompt as scenario 1, but GEMINI_API_KEY is unset, taking out both gemini-flash and gemini-pro. The remaining potential weight (gpt-mini 1 + claude 2 = 3) can never reach 4, so the call completes with quorum.achieved: false — still success: true, never an error. The degradation is itemized, not hidden.

// data
{
  "response": "HTTP 431 (Request Header Fields Too Large) means the server refused the request because its headers are too big.",
  "certaintyScore": 0.73,       // min(1, 3/4) × mean(1.0, 0.95), rounded — shortfall caps the score
  "quorum": {
    "achieved": false,
    "tier": 3,
    "requiredWeight": 4,
    "agreeingWeight": 3,
    "sources": [
      { "name": "gemini-flash", "model": "gemini-3-flash-preview", "status": "unavailable", "weight": 1, "agreement": null,
        "error": "Credential not resolvable; source skipped", "cause": "credential", "code": "CONFIGURATION_ERROR" },
      { "name": "gpt-mini",     "model": "gpt-5-mini",             "status": "ok",          "weight": 1, "agreement": 1 },
      { "name": "gemini-pro",   "model": "gemini-3-pro",           "status": "unavailable", "weight": 2, "agreement": null,
        "error": "Credential not resolvable; source skipped", "cause": "credential", "code": "CONFIGURATION_ERROR" },
      { "name": "claude",       "model": "claude-fable-5",         "status": "ok",          "weight": 2, "agreement": 0.95 }
    ]
  },
  "tokenUsage": { "prompt": 640, "completion": 97, "total": 737 }
}

Had the arbiter itself been the casualty (e.g. OPENAI_API_KEY missing in this config), no agreement could be evaluated at all and the response would instead carry the highest-weighted successful answer with "certaintyScore": 0, "quorum": { "arbiterFailed": true, ... }, and a quorum.arbiter row { "name": …, "status": "failed", "error": { "code": …, "cause": "arbiter_unavailable", "detail": … } } naming what its failure was. If instead no source responded at all, the call is the one success: false shape: error.code: "EXTERNAL_SERVICE_ERROR", error.message: "No peer source produced a response", and — since this change — error.cause naming the fault class (credential when every source is credential-unavailable, peer_unreachable when every host refused, configuration when every source was skipped) plus the full per-source roster in error.context.sources[]; see the 401 walkthrough.

5. Checking your own answer — callerAnswer the peers disagree with

{ "prompt": "In JavaScript, does [10, 9, 100].sort() return [9, 10, 100]?", "callerAnswer": "Yes, sort() orders numbers ascending by default." }

You already have an answer and want independent endorsement. The peers never see callerAnswer — they answer the prompt blind. Tier 1's two flash voters agree with each other but can't reach weight 4, so tier 2 is pulled in; gemini-pro agrees too and quorum lands at tier 2 (1 + 1 + 2 = 4). The arbiter separately rates your answer against the peer consensus and returns callerAgreement — low here, because default sort() is lexicographic ([10, 100, 9]), so your answer contradicts what the peers concluded.

// data
{
  "response": "No — Array.prototype.sort() coerces elements to strings by default, so [10, 9, 100].sort() returns [10, 100, 9]. Pass a comparator, e.g. .sort((a, b) => a - b), for numeric order.",
  "certaintyScore": 0.98,       // peers-only: min(1, 4/4) × mean(1.0, 1.0, 0.95) — unaffected by callerAnswer
  "callerAgreement": 0.1,       // arbiter rated your answer against the peer consensus; it contradicts it
  "quorum": {
    "achieved": true,           // the peers reached quorum with each other; your answer carries no weight
    "tier": 2,
    "requiredWeight": 4,
    "agreeingWeight": 4,
    "sources": [
      { "name": "gemini-flash", "model": "gemini-3-flash-preview", "status": "ok", "weight": 1, "agreement": 1 },
      { "name": "gpt-mini",     "model": "gpt-5-mini",             "status": "ok", "weight": 1, "agreement": 1 },
      { "name": "gemini-pro",   "model": "gemini-3-pro",           "status": "ok", "weight": 2, "agreement": 0.95 }
    ]
  },
  "tokenUsage": { "prompt": 934, "completion": 141, "total": 1075 }
}

quorum.achieved: true with callerAgreement: 0.1 is the signal to catch: the peers are confident, but not with you — treat it as a flag to revise your answer. Had you been right, callerAgreement would sit near 1 alongside the same high certaintyScore. Note caller never appears in sources[] and its rating never moved agreeingWeight.

Reading the result: gate any automated decision on quorum.achieved, treat certaintyScore as the confidence dial (it discounts both shortfall and lukewarm agreement), scan sources[] for error/unavailable entries — a degraded quorum is always visible, never silent; every rated row carries reason, the arbiter's one-line explanation of its agreement; agreement: 0 means the arbiter rated that answer as contradicting the consensus, while an ok row with agreement: null was not rated at all, and its note says why (the arbiter returned no rating for it, the arbiter failed, or the run ended before the arbiter rated it) — when a later arbiter pass fails, rows an earlier pass rated keep that rating with a note saying so; a cancelled row is a peer the run no longer needed once quorum was met, not a failure — its reason lives in note and it carries weight: 0, so do not count it as degradation — and, when you passed a callerAnswer, read callerAgreement as the separate "did the peers land where I did?" signal (scenario 5).

A failed quorum withholds its prose. When quorum.achieved is false, response comes back as "" and the envelope carries "responseWithheld": true — text no quorum validated is unvalidated opinion, so the server no longer hands back synthesized consensus (or one peer's representative answer) for a caller to fold into its own reasoning. Everything needed to transcribe the verdict honestly is still returned: quorum with requiredWeight, agreeingWeight, and the full sources[] roster — including each rated source's one-line reason, so you can see why a peer scored low without its prose — certaintyScore, and callerAgreement when a callerAnswer was supplied. An achieved quorum is unchanged — the same response prose as before, with no responseWithheld key at all.

Tools

All tools return the envelope {"success": true, "data": ...} or {"success": false, "error": {"code", "message"}}, JSON-stringified into a single text content block (compatible with the reference Gemini MCP). Error codes: VALIDATION_ERROR, CONFIGURATION_ERROR, TIMEOUT_ERROR, EXTERNAL_SERVICE_ERROR, PEER_API_ERROR, PEER_RATE_LIMIT, TOOL_NOT_FOUND, INTERNAL_ERROR. Model, temperature, and output limits are injected server-side — the input schemas are strict, so client-supplied model/temperature are rejected.

Failure envelopes carry the cause, not just the code. Alongside code and message (unchanged), every failure the server can attribute carries error.cause from the closed vocabulary in the Troubleshooting table, and where applicable error.source (the peer or arbiter at fault), error.detail (one provider-agnostic line — HTTP status or the unwrapped transport errno, never provider body text or credential material), error.fix (the operator action), and, for a peer_review failure with at least one source consulted, error.context — the same weight math and sources[] roster the success path puts in data.quorum, so the no-success response is as diagnosable as the success one. An INTERNAL_ERROR is the redacted catch-all: its message is always the generic line, and its error.context.correlationId points at the full error in the server's stderr log (see the Troubleshooting table). code stays the coarse wire class for readers that switch on it; cause is what a repair guide should read. When a source fails as the arbiter, data.quorum.arbiter (on the arbiter-fallback success path) or error.context.arbiter (on the failure path) reports {name, status: "failed", error: {code, cause, detail}} — the certaintyScore: 0 / arbiterFailed: true flags keep their exact meaning; a row is added, never moved.

Every tool advertises MCP annotations. All four declare readOnlyHint: true — none of them modifies the caller's environment — and openWorldHint: true, except list_peers (openWorldHint: false, it reads configuration only). A harness that gates concurrency on readOnlyHint (Claude Code does) may therefore run several calls issued in one message at once. Inputs, outputs, and quorum behavior are unchanged.

Changes to peer-review.config.json require restarting the server — it is read once at startup, and credential availability is evaluated at boot.

Shipped envelope changes require a rebuild and a harness restart — the harness spawns a long-lived child from dist/app.js (npm run build emits it), so a new error.cause/fix envelope reaches callers only after a rebuild and a harness restart or re-registration.

peer_review

Weighted-quorum consultation. Input: prompt (1–100000 chars), optional history ([{role: "user"|"model", content}]), optional tier (highest tier to escalate to; defaults to the full stack), optional callerAnswer (1–100000 chars — your own answer to the prompt), optional callerSource (the name of a configured source the caller itself is — skipped as a redundant peer for this request, never called, itemized as status: "skipped"; an unknown name is a logged no-op). An unmet quorum is not an error — check quorum.achieved and certaintyScore.

// request
{ "prompt": "What is the capital of France? Answer with just the city name." }
// response data
{
  "response": "Paris",
  "certaintyScore": 1,                    // min(1, agreeingWeight/requiredWeight) × mean agreement of agreeing sources
  "quorum": {
    "achieved": true,
    "tier": 1,
    "requiredWeight": 2,
    "agreeingWeight": 2,
    "sources": [
      { "name": "gemini-a", "model": "google/gemini-3.5-flash", "status": "ok", "weight": 1, "agreement": 1 },
      { "name": "gemini-b", "model": "google/gemini-3.5-flash", "status": "ok", "weight": 1, "agreement": 1 }
    ]
  },
  "tokenUsage": { "prompt": 315, "completion": 45, "total": 360 }   // aggregated across peers + arbiter
}

A third, slower tier-1 peer that was still answering when gemini-a and gemini-b met the bar would appear alongside them as:

{ "name": "slow-reasoner", "model": "glm-flash", "status": "cancelled", "weight": 0, "agreement": null, "note": "Cancelled: quorum was met before this source responded" }

sources[].status is one of:

  • ok — the source answered and the arbiter rated it.

  • error — the source was consulted and failed.

  • unavailable — never called because its credential was unresolvable.

  • skipped — excluded from this request via callerSource/PEER_REVIEW_SKIP_LOCAL; never called.

  • cancelled — still in flight when quorum was met (the early-abort cancelled it) or the caller cancelled the call; a non-failure, not a peer defect — its reason lives in note.

weight is the source's effective contribution to the run: 0 on a cancelled row (a peer the run no longer needed contributed no vote), otherwise its configured weight — so skipped and unavailable keep theirs. note appears only on a cancelled row and carries why it was cancelled (e.g. "Cancelled: quorum was met before this source responded"); it lives outside error, so a reader counting failures off error MUST NOT count these rows. agreement is the arbiter's 0–1 rating (null if the source produced no rated response). If the arbiter itself fails, the response falls back to the highest-weighted successful answer with certaintyScore: 0 and quorum.arbiterFailed: true.

Including your own answer (callerAnswer). When you already have an answer and want to know whether independent peers endorse it, pass it as callerAnswer instead of pasting it into prompt. Peers never see it — they answer the prompt blind, so their independence is preserved and certaintyScore stays an anchoring-free signal. The arbiter is instructed to derive the consensus only from the peer answers and then rate your answer against that consensus, so a confidently-wrong callerAnswer cannot pull the consensus toward itself. The response gains a callerAgreement field (0–1).

How it behaves:

  • Zero quorum weight. callerAgreement is reported alongside the quorum but never counted into agreeingWeight, quorum.achieved, or certaintyScore — your own answer cannot self-certify. Read it as a separate signal: "did independent peers land where I did?"

  • Present only when supplied. The key is omitted entirely from the response when callerAnswer was not passed. When it was passed, the key is always present.

  • null when unrated. callerAgreement is null if the arbiter returned no rating for it — either because the arbiter omitted the caller entry from its ratings, or because there was no trusted evaluation at all (arbiter failed, or quorum ran with no ratable peer response — the same fallback paths that force certaintyScore: 0). A supplied callerAnswer therefore always yields the key, but not always a number.

  • Reserved name. caller is reserved as the internal rating channel and is rejected as a configured sources[].name at startup.

// request
{ "prompt": "What is the capital of France?", "callerAnswer": "Paris" }
// response data — same shape as a normal peer_review, plus callerAgreement:
{
  "response": "Paris",
  "certaintyScore": 1,            // unchanged by callerAnswer — derived from peers only
  "callerAgreement": 1,          // arbiter's 0–1 rating of "Paris" vs the peer consensus
  "quorum": { "achieved": true, "requiredWeight": 2, "agreeingWeight": 2, /* … */ },
  "tokenUsage": { "prompt": 315, "completion": 45, "total": 360 }
}

When your answer diverges from what the peers conclude, callerAgreement drops toward 0 while certaintyScore still reflects the peers' own convergence — e.g. { "callerAgreement": 0.15, "certaintyScore": 0.97 } reads as "the peers strongly agree, but not with you."

query_peer

Single-source escape hatch mirroring the reference query_gemini. Input: prompt, optional history, optional source (defaults to the lowest-tier highest-weight source).

// request
{ "prompt": "In one word, what color is the sky on a clear day?", "source": "gemini" }
// response data
{ "response": "Blue", "model": "google/gemini-3.5-flash", "source": "gemini", "finishReason": "stop",
  "tokenUsage": { "prompt": 14, "completion": 1, "total": 15 } }

list_peers

No input. Returns every configured source with availability (available = credential resolvable):

{ "sources": [ { "name": "gpt-large", "apiType": "openai", "model": "gpt-5.2", "weight": 2, "tier": 1, "available": true } ], "count": 1 }

While PEER_REVIEW_DEADLINE_MS is not strictly below the declared PEER_REVIEW_CALLER_TIMEOUT_MS — the MCP client tool-call timeout this server assumes — the response also carries callerCeiling: { "deadlineMs": …, "callerTimeoutMs": …, "cause": "configuration", "fix": "…" } naming both numbers and the two ways to fix the pair. Above the ceiling the caller abandons the call and reports only its own generic timeout, so no verdict anyone computed is read.

count_tokens

Input: text (1–1000000 chars), optional source. Anthropic-compatible sources use the provider's count endpoint ("method": "api"); OpenAI-compatible sources use a ceil(chars/4) heuristic ("method": "estimate").

{ "totalTokens": 42, "model": "claude-fable-5", "source": "claude", "method": "api" }

Troubleshooting

Failure causes

Every attributable failure names error.cause from this closed set. Read cause, not prose.

cause

meaning

what to do

peer_unreachable

the peer's host refused or failed the connection; error.detail names the unwrapped errno (ECONNREFUSED, ENOTFOUND, …)

check the source's baseUrl and network reachability from the server host, then retry

credential

401/403 from the provider, or the credential is unset/unmappable

set the credential env var or repair apiKeyCommand for the named source in peer-review.config.json, then restart the server

rate_limited

the provider 429'd through its backoff budget

wait before retrying, or lower the weight/tier of this source

peer_bad_response

404/5xx, or a body that is not valid JSON

check the model name and endpoint for the named source in peer-review.config.json, then retry once

peer_truncated

the provider stopped the reply at its own output ceiling (stop_reason: "max_tokens" / finish_reason: "length"), or returned no answer text inside it — the common shape for a reasoning model whose hidden thinking is billed against max_tokens; error.detail names the stop signal and the billed completion tokens

raise maxOutputTokens for the named source (or PEER_MAX_OUTPUT_TOKENS globally) so the budget clears the model's thinking length, and raise timeoutMs with it

arbiter_unavailable

the arbiter source could not be reached

check the arbiter source reachability and credentials in peer-review.config.json

arbiter_malformed

the arbiter answered in prose; JSON extraction failed even after the one re-ask

set jsonMode for the arbiter source or choose a JSON-reliable arbiter

deadline

the per-source timeoutMs or the global wall expired

raise PEER_REVIEW_DEADLINE_MS (staying strictly below PEER_REVIEW_CALLER_TIMEOUT_MS) or lower the per-source timeoutMs

validation

a tool argument failed schema validation; error.fix names the field

correct the named argument and retry

configuration

bad config file or env var, or the ceiling pair is misset

fix the named file or variable, then restart the server

internal

unclassified — the redacted catch-all

inspect the peer-review server stderr log for the correlationId in error.context; rerun with LOG_LEVEL=debug

Walkthrough: 401 from every source

All peers were consulted and every one answered 401. This is not reported as a network outage — cause is credential, the roster is intact, and no credential material ever appears in a caller-visible field:

{
  "success": false,
  "error": {
    "code": "EXTERNAL_SERVICE_ERROR",
    "message": "No peer source produced a response",
    "cause": "credential",
    "detail": "No consulted peer source produced a usable response; see error.context.sources[]",
    "fix": "Set the credential env vars or repair apiKeyCommand for the unavailable sources in peer-review.config.json, then restart the server",
    "context": {
      "tier": 1,
      "requiredWeight": 4,
      "agreeingWeight": 0,
      "sources": [
        { "name": "gpt-large", "model": "gpt-5.2", "status": "unavailable", "weight": 2, "agreement": null,
          "error": "Authentication failed for \"gpt-large\" (HTTP 401)", "cause": "credential", "code": "CONFIGURATION_ERROR" }
      ]
    }
  }
}

Set the env vars (or repair apiKeyCommand) and restart the server — config and credential availability are evaluated once at boot.

Walkthrough: the caller abandoned a slow review

Symptom: the harness reports its own generic "timed out" and never reads a verdict, while the server logs the completed answer after the caller left. Cause: PEER_REVIEW_DEADLINE_MS is not strictly below PEER_REVIEW_CALLER_TIMEOUT_MS (the declared MCP client tool-call timeout — omp's per-server timeout key). Fix either side: lower PEER_REVIEW_DEADLINE_MS, or raise the tool-call timeout in the harness registration and keep the declared value equal to it. The server never dies over this pair: it warns once at startup, list_peers reports the mismatch (callerCeiling, above), and every failure envelope while it holds carries the same fix line.

Per-call server-side logging names the request id, tool, tier, and duration, so "answered after the caller left" is visible in the log rather than guessed.

Migrating from gemini-for-claude-mcp

This server fully supersedes gemini-for-claude-mcp: Gemini becomes just one configured source, reachable via AI Studio or Vertex AI.

It is intended as a drop-in replacement: query_peer accepts the same prompt/history shape (role: "user"|"model" preserved) and returns the same payload fields as query_gemini plus source; list_peers covers list_gemini_models; count_tokens covers count_gemini_tokens (with the method caveat below). Configure Gemini as an ordinary source, either way:

AI Studio (static key):

{ "name": "gemini", "apiType": "openai",
  "baseUrl": "https://generativelanguage.googleapis.com/v1beta/openai",
  "model": "gemini-3-pro", "apiKeyEnv": "GEMINI_API_KEY", "weight": 2, "tier": 1 }

Vertex AI (short-lived OAuth token via gcloud):

{ "name": "gemini", "apiType": "openai",
  "baseUrl": "https://aiplatform.googleapis.com/v1beta1/projects/YOUR_PROJECT/locations/global/endpoints/openapi",
  "model": "google/gemini-3.5-flash",
  "apiKeyCommand": "gcloud auth print-access-token", "weight": 2, "tier": 1 }

Caveat: count_tokens for Gemini uses the ceil(chars/4) estimate ("method": "estimate") rather than Gemini's native count endpoint.

Development

npm run typecheck && npm run lint && npm test   # full suite
npm run dev                                     # tsx watch mode

See ARCHITECTURE.md for layer rules, the quorum algorithm, and runtime guards.

Contributing

Issues and pull requests are welcome. Before submitting a PR, run the full check suite:

npm run typecheck && npm run lint && npm test

License

Apache-2.0 — see LICENSE.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers