Skip to main content
Glama

description: Measured self-improvement with paired experiments and a DSH plugin.

evokit

evokit improves the editable parts of an AI system (prompts, skills, code, configuration) and keeps a change only when measurements show it helps.

How it differs. A proposed change is adopted only after:

  • a paired experiment against the current version on the same tasks;

  • a noise-aware gate, with a threshold derived from measured run-to-run variation;

  • a sealed holdout confirmation; when confirmation fails, the operator workflow reverts the adoption.

Each experiment also records cost and latency, and every version records its parent, so lineage is traceable.

Documented result. In one public BFCL pilot, held-out accuracy went from 41 to 47 of 50 after confirmation. This is one small example; it does not establish general gains.

What you supply. Editable file globs, stable tasks, and a runner that executes and scores each task. The task source and scorer stay frozen. A keyless offline example runs without a model or provider key.

Use it as a DeepSeek Harness (DSH) plugin, or through its Python SDK, CLI and MCP server.

Related MCP server: seven-dpt-mcp

Install

Requires Python 3.11+, uv, Node 24, DSH >=0.2.2-rc.1 <0.3.0-0 (0.2.2-rc.1 and later 0.2.x releases, not 0.3) and DSH's pnpm prerequisite. Windows is the verified platform; Linux and macOS runtime integration remains unverified.

dsh plugin --profile evokit add dsh-evokit
uv tool install dsh-evokit
dsh --profile evokit

Run the last command from your scenario directory. EVOKIT_MCP_COMMAND, EVOKIT_WORKSPACE and EVOKIT_SKILL_ROOT are optional: they default to evokit on PATH, the directory where DSH starts, and the bundle's own skills folder. Set them to override these defaults.

Verified alternative: download the wheel, bundle and SHA256SUMS from Releases, compare their SHA-256 hashes, then install the files:

uv tool install ./dsh_evokit-0.1.8-py3-none-any.whl
dsh plugin --profile evokit add ./dsh-evokit-0.1.8.tgz
dsh plugin --profile evokit list

See the DSH guide for an isolated profile and exact PowerShell commands.

Configure the improver model in your DSH profile using your own provider key and your provider's supported model name. Keep the key in DSH's local credential settings or a process environment variable, never in this repository. The optional relay supports file-based keys and accounting; adapt its example endpoint, model identity and illustrative prices before use. No key is included or required for the offline example. Installing or listing the plugin makes no model requests; starting evolution may incur provider charges.

How the loop works

The loop is diagnose → propose → paired experiment → noise-aware gate → holdout confirmation with automatic revert in the operator workflow. The improver sees diagnosis and selection evidence. A separate operator adopts a candidate, checks sealed confirmation aggregates, and calls ledger_revert when confirmation fails; confirm_adoption alone does not move the pointer. An optional final split opens once, with human approval, and never selects changes.

Minimal example

From a source checkout, the Node.js command example needs no model or key:

uv sync --frozen --group dev
uv run python examples/scripted_improver.py command-echo

Its scenario.yaml supplies the three pieces:

editable: [prefix.txt]
tasks: tasks.json
command: [node, runner.mjs]

The runner reads EVOKIT_TASK_FILE, executes against EVOKIT_VERSION_DIR, and writes an outcome and cost to EVOKIT_RESULT_FILE. The scripted example works in a temporary copy. To connect your own system, follow onboarding, then use the bundled onboard, evolve and meta skills.

Safety and limits

Set budgets, anchors, noise policy and a stopping rule before experiments. Repeated mean gates use threshold/√R; majority gates retain the single-repeat threshold. Confirmation has a limited wear budget and is not an untouched final benchmark. The BFCL pilot above is a confirmation result from one small task set.

Scenarios run trusted executable code. Per-task directories are not a security sandbox. Protect sealed tasks and records with OS permissions; MCP filtering cannot hide files an improver can read directly. Review proposed changes for task literals and leakage. The relay accounts reported usage conservatively; it cannot guarantee provider billing or prevent every external cost.

See concepts, tools, storage, caching and recovery and contributing. This project uses the MIT license, Copyright (c) 2026 quei.

Available Tools

16 tools
compare_pairedA
Read-onlyIdempotent

Compare two complete arms task by task: fixes, breaks, net score change, per-task deltas, per-group tables, and cost and latency ratios. Use it to judge a candidate against its parent; use gate_check to record a decision with noise and budget checks. Both arms must be complete, error-free and match on tasks, split, repeats and anchors. Read-only. Returns the comparison object.

Inputs: config (required), child_run (required), parent_run (required), aggregation (required).

failure_cardsA
Idempotent

List failure cards for one arm: every task with outcome below 1, or, with parent_run and change, the tasks this arm fixed or broke against the parent. Use it to diagnose before proposing; use run_results for all rows, and failures_similar (HB workspaces) to search failures across runs. A card has task_id, group, outcome, reason, payload summary, output and trace pointer unless the scenario renders its own. Also saves the cards to the record store for later search, overwriting identical entries; nothing else changes. Returns an array.

Inputs: config (required), run_id (required), group, limit, +5 more.

gate_checkA

Judge whether a proposal's candidate may be adopted and record the verdict. It checks frozen anchors, that the proposal's parent is still the current base, an editable-only diff, a scan for task-specific literals, complete non-smoke selection arms of parent and candidate, cost and latency budgets, scenario quality guards and the recorded noise threshold. Use it after compare_paired looks promising; use ledger_adopt (operator profile) to act on a passing verdict, because this tool never changes the current base. Appends one verdict record per call. Returns {id, verdict, reasons, threshold, comparison, secondary_guards} with verdict promote (net above noise), adopt-no-harm (net >= 0 with a rationale) or reject.

Inputs: config (required), noise_id (required), child_run (required), parent_run (required), +4 more.

ledger_historyA
Read-onlyIdempotent

Read the public lineage: current base, base chain (initial version plus every adoption and revert), queued runner-ups and the append-only public event list. Use it to audit what was adopted and when; use meta_metrics for aggregate rates. Read-only; private confirmation and final events are hidden. Returns {current, base_chain, runner_ups, events}.

Inputs: config (required).

meta_metricsA
Read-onlyIdempotent

Report aggregate improvement metrics: proposal and adoption counts, acceptance rate, mean absolute prediction error, gain per cost unit, total experiment cost and confirmation measurement counts. Use it to judge how well proposals predict and pay off; use ledger_history for individual events. Read-only. Returns one JSON object.

Inputs: config (required).

noise_estimateA

Measure the current base's run-to-run noise from one repeated arm and record the estimate that gate_check requires. Use it once per base, on a complete, non-smoke selection arm of the current version with at least two repeats; use compare_paired for candidate effects. Appends a noise record to the ledger. Returns noise_id, per-task flip rates, net spread, threshold and method; pass noise_id to gate_check.

Inputs: config (required), run_id (required).

proposal_createA

Record a falsifiable proposal for a candidate version before testing it: targets, mechanism, predicted effect, falsification criterion and prior attempt. Use it after version_create and before running and gating the candidate; use proposal_list to read earlier proposals. Checks that targets are editable and cover every changed file, pins the current base as parent and snapshots the improver playbook. Appends to the ledger; runs and adopts nothing. Returns the stored proposal with its new id.

Inputs: config (required), version (required), proposal (required).

proposal_listA
Read-onlyIdempotent

List every recorded proposal in creation order, rejected ones included, with id, parent, version, prediction, playbook version and a not_confirmed flag. Use it to find a proposal_id for gate_check or a prior attempt; use ledger_history for adoptions and the base chain. Read-only. Returns an array.

Inputs: config (required).

run_interleavedA
Idempotent

Run two or more versions as matched arms in rotating task blocks, so each arm sees every task at nearly the same time. Use it for parent-versus-candidate measurements when latency or cost may drift; use run_launch for one arm or a detached run. Blocks until all arms finish. Writes run records, results and traces, and costs whatever the scenario's tasks cost. Repeating the call with identical arms and parameters resumes and skips committed slots; a new plan needs fresh run ids. Returns each arm's run status, keyed by run id.

Inputs: arms (required), config (required), seed, split, +2 more.

run_launchA

Start or resume one arm: a run of one version over the tasks of the diagnosis or selection split, scored by the scenario. Use it to measure a single version; use run_interleaved to run a parent and candidates side by side. Detached by default: it returns at once while a background worker runs, so poll run_status, then read run_results. Writes run records, results and traces, and costs whatever the scenario's tasks cost (for example model calls). Resuming a run_id with the same parameters only fills missing task slots; changed parameters are refused. Returns the run status object.

Inputs: config (required), version (required), wait, smoke, +4 more.

run_resultsA
Read-onlyIdempotent

List one arm's committed task results: task_id, group, repeat, outcome, reason, cost, seconds, output and archived trace path. Use it for raw evidence; use failure_cards for failures only and compare_paired for parent-versus-child deltas. Read-only; partial while the arm is running. Returns an array in stable order.

Inputs: config (required), run_id (required), limit, offset.

run_statusA
Read-onlyIdempotent

Report one arm's progress: state (pending, running, completed or interrupted), completed and expected result counts, and execution errors. Use it to poll a detached run_launch; use run_results for the task rows. Read-only. Returns {run_id, version, split, state, completed, expected, errors}.

Inputs: config (required), run_id (required).

scenario_describeA
Read-onlyIdempotent

Describe the bound scenario: epoch id, editable file globs, frozen scorer version, task counts per split, primer, budgets, replay support and readiness flags. Call it first to learn what may be edited and how large each split is; use version_current for the current version id and ledger_history for lineage. Read-only; never reveals confirmation or final task ids. Returns one JSON object.

Inputs: config (required).

version_createA

Store a candidate version of the editable files, either by snapshotting a complete scenario checkout or by applying a unified patch to a parent version. Use it after editing and before proposal_create and run_launch; use version_diff to review a stored change. Writes content-addressed snapshot files and appends a ledger event; it never adopts or changes the current base. Rejects edits outside the editable globs, unsafe paths and patches that do not apply. Returns {version, parent, changed}; identical content always gets the same version id.

Inputs: config (required), patch, parent, directory.

version_currentA
Read-onlyIdempotent

Return the id of the current base version: the latest adopted or reverted-to snapshot, or the initial snapshot before any adoption. Use it to get the parent for version_create, run_launch and noise_estimate; use ledger_history for the whole base chain. Read-only. Returns {"version": ""}.

Inputs: config (required).

version_diffA
Read-onlyIdempotent

Show the unified text diff of editable files between two stored versions, after verifying both snapshots' content digests. Use it to review a candidate before proposing or gating; use compare_paired to compare measured outcomes instead. Read-only. Returns {parent, child, changed, diff}; binary files are named, not diffed.

Inputs: child (required), config (required), parent (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.4/5.0

    Scored across 16 tools

    Disambiguation5/5

    Each tool targets a distinct resource+action pair, and descriptions explicitly disambiguate near-neighbors (run_launch vs run_interleaved, run_status vs run_results, compare_paired vs noise_estimate vs gate_check, ledger_history vs meta_metrics). The 'use X for… use Y for…' cross-references make selection reliable even in the crowded run/comparison area.

    Naming Consistency4/5

    Names follow a predictable resource-prefix scheme (version_*, run_*, proposal_*, ledger_*, gate_*, compare_*), which is easy to scan and group. A few singular tools (compare_paired, gate_check, meta_metrics, scenario_describe) break the strict prefix family, but the overall convention remains readable and stable.

    Tool Count4/5

    At 16 tools the surface is on the heavy side, but the domain genuinely spans scenario inspection, versioning, execution, analysis, proposal/gating and ledger auditing, and each tool maps to a needed step. Nothing looks redundant or like filler.

    Completeness4/5

    The lifecycle from edit → version_create → run → compare → noise → proposal → gate is well covered, with read-only audit and metrics views. Two gaps: gate_check points to a ledger_adopt tool that is not in this set (adoption is unreachable here), and there is no cancel/abort for a detached run or a general version listing beyond ledger_history.

    Maintenance

    ActivityMaintained
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      C
      maintenance
      Self-improving learning engine for Claude Code. Extracts signals from sessions (corrections, successes,confusion), runs periodic reflections, and evolves behavioral frameworks through evidence. 9 MCP tools, automatichooks, local-only.
      15
      94
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Feynman's twelve-problems method as an MCP server — a persistent set of long-running problems plus an evoke loop that tests each new trick against all of them. Local, TypeScript, no LLM inside, no API key.
      6
      65 npm
      Apache 2.0
    • A
      license
      A
      quality
      B
      maintenance
      Enables coding agents to make offline, zero-cost decisions using schema-safe Choice/Score/Noul primitives, a confidence gatekeeper, planning, adversarial red-teaming, research, and RLVR-based self-improvement via 21 MCP tools.
      24
      7
      MIT