Eclipse Benchkit

A free, local SDK a vendor runs against their own MCP server to execute their declared check battery and emit a scorecard in the Assay scorecard format — check ids, verdicts, evidence hashes, environment fingerprint — with a spot-check hook embedded in every check entry as it runs. A separate local harness draws a random sample from the sealed scorecard and re-runs those checks, issuing a counter-signed verification receipt.

Scope label: LOCAL / fixture. v0.1 is exercised only against fixture vendor servers. No live vendor server, no network call, no registry.

Status: v0.1.1 — CERTIFIED (local/fixture). Verity re-bench 2026-10-27: all v0.1.0 FAIL defects (D1/D2 blockers, M1–M4 majors, m1–m5 minors) closed and independently verified; builder suite 27/27. Two new minors (N1 fence-encoding edges, N2 seed-commitment provenance) are recorded for the next fix loop and do not gate this scope. The certification binds exactly this scope: the SDK, emitter, and spot-check harness against fixture vendor servers only. A spot-check receipt is a statement about a sample of one sealed scorecard — not a house bench verdict and not a certification of any vendor server.

v0.1.1 protocol notes

• Commit-then-sample, unpredictable (D1). The verifier nonce is generated verifier-side from a secret seed held in keys/ plus fresh CSPRNG entropy, and revealed only in the sample manifest after the seal is verified. It is never a caller input for a new draw and cannot be precomputed from the scorecard hash or any public state. A supplied nonce is honoured only as a replay key for an already-issued (scorecard_hash, verifier_nonce) pair, which returns the recorded receipt (§3.3); a novel supplied nonce is refused by name.

• Verifier token (§5, M4). Verifier verbs require the verifier token: the fixture token in keys/verifier-token.fixture.key (chmod 600, gitignored), presented per call as verifier_token or via BENCHKIT_VERIFIER_TOKEN in the verifier process.

• Config fence (§7.3, F-B14). Startup hard-exits with CONFIG_FENCE_REFUSED:<class> on a non-local endpoint, a live-key string, a vendor-nonce config, a non-test value_class, or an Assay/Cupel price-or-checkout config; live-URL server descriptors are refused at benchkit_run_begin with the same named code.

• Authenticated artefacts (M1/M2). The seal signature binds the head hash, all counts, and the format version; recompute verifies receipt signatures, the outcome record chain, and the rerun chain head, and rejects extra outcomes.

• Keys (§7.4, M3). All fixture keys are locally generated in keys/ (chmod 600, gitignored, fixture-labelled). No key material exists in source.

Related MCP server: mcp-spec-compliance-mcp

Novel mechanism

Vendor-run scorecards embed spot-check hooks; the house bench re-runs a random sample, so a fabricated self-bench is statistically detectable. The protocol is commit-then-sample: the scorecard is sealed (hash-chained, vendor-signed) before any sample exists, and the sample seed derives from scorecard_hash ‖ verifier_nonce, the nonce revealed only after the seal is presented. The receipt states the arithmetic on its face: at sample size k and fabricated fraction f, detection probability is 1 − (1−f)^k — default k = 30 (or all hook-bearing checks if fewer), powered at f ≥ 0.10 → detection ≥ 95%. A check without a hook is labelled HOOKLESS on the scorecard's face and is outside every sample and every verified claim; a sampled non-deterministic or secret-requiring check resolves to UNVERIFIABLE, never to a silent pass. Receipt verdicts: VERIFIED_SAMPLE · DIVERGENT (mismatched check ids named) · UNVERIFIABLE.

Every vendor scorecard is labelled vendor-run / self-bench on its face. A spot-check receipt says only that a random sample of k of N hook-bearing checks was re-run and matched (or diverged — verdict as issued), powered to detect fabrication at fraction ≥ f with probability p. It is not a house bench verdict and not a certification of the vendor's server, and it never upgrades a self-bench into one. Unverifiable scorecards export only as DOWNGRADED_EXPORT, never upgraded post-hoc.

Format attribution, exact and minimal: scorecard emitted in the Assay scorecard format. Eclipse Assay and Eclipse Cupel are internal tools; Benchkit consumes their scorecard format as a specification only and contains no product surface, price, licence, or buy path for them. The full bench is a paid service engagement; intake via the existing route — Benchkit contains no intake form, no engagement checkout, and no service pricing. Benchkit's own metering point is one spotcheck_verification_completed event per completed verification ($10/verification shape, metering record only, no charge path in this build), keyed to (scorecard_hash, verifier_nonce) so idempotent returns do not double-meter.

Tools

Vendor surface (default, ECLIPSE_BENCHKIT_SURFACE=vendor): benchkit_run_begin, benchkit_check_record, benchkit_scorecard_seal, benchkit_scorecard_get, benchkit_export_get.

Verifier surface (ECLIPSE_BENCHKIT_SURFACE=verifier): spotcheck_request, spotcheck_run, spotcheck_receipt_get, spotcheck_recompute, anomaly_queue_read.

Surfaces are disjoint; cross-surface calls return the identical unknown-tool refusal as a nonsense name. Pre-seal sample attempts, vendor-supplied nonces, and same-nonce re-draws after a verdict are refused by name and receipted in the anomaly queue.

Fixtures

fixtures/battery-*.json — deterministic fixture batteries over fixture vendor servers: good (40 checks), small (8), honest failures (3 genuinely failing checks sealed as failures), hookless (5 of 12), flaky (one non-deterministic check), secret-hook (one un-runnable hook), fabricated (40 claimed checks, 12 fabricated, over the fabricated-source server — the decisive fixture: the seeded spot-check must catch at least one).

Run

python3 server.py            # MCP over stdio JSON-RPC
python3 -m unittest discover -s tests -v

State lives in data/ or BENCHKIT_DATA_DIR. Python 3, standard library only.

Powered by Cyan Eclipse Technologies — https://cetech.ai

MIT License · Copyright © 2026 Cyan Eclipse Technologies LLC · cetech.ai