avo
by gatordevin
README.md
# AVO — Agentic Variation Operators
An open reproduction of **[AVO: Agentic Variation Operators for Autonomous
Evolutionary Search](https://arxiv.org/abs/2603.24517)** (Chen, Ye, Xu et al.,
NVIDIA, 2026), runnable on a laptop.
Classical evolutionary search, and the LLM-augmented systems that followed it,
decompose the variation operator into a fixed pipeline:
```
Vary(P_t) = Generate(Sample(P_t))
```
The framework samples parents; the model produces one candidate from them. AVO
replaces that whole decomposition with a single autonomous agent run:
```
Vary(P_t) = Agent(P_t, K, f)
```
The agent sees the full lineage `P_t`, a domain knowledge base `K`, and the
scoring function `f` — and decides for itself what to read, what to change, and
when to measure. It stops being a candidate generator and becomes the variation
operator.
This repo implements that framework, plus the surrounding machinery the paper
describes: a git-backed lineage, a correctness-gated score vector, the
matches-or-improves commit policy, a supervisor that intervenes on stagnation,
and trajectory plots. Two optimisation targets ship with it.
---
## The part that matters: it runs on the session you already have
The default driver does **not** spawn an agent and does **not** call an API. It
hands the variation prompt to the Claude Code session you are already talking
to, and that session does the work. Nothing extra is billed, no `ANTHROPIC_API_KEY`
is needed, and the agent doing the optimising is a real general-purpose coding
agent — which is exactly what the paper used.
Unattended mode (spawn an agent per step and let it run for days, like the
paper's 7-day experiment) is available too, and is opt-in precisely because it
spends quota.
---
## Install
```bash
git clone https://github.com/gatordevin/avo
cd avo
pip install -e ".[all]" # or: pip install -e . for the core only
avo doctor
```
On a system with an externally-managed Python (Homebrew, most Linux distros),
use a virtualenv — the `--system-site-packages` flag reuses a NumPy and
Matplotlib you already have:
```bash
python3 -m venv --system-site-packages .venv
.venv/bin/pip install -e ".[all]"
.venv/bin/avo doctor
```
Requirements: Python 3.10+, `git`, and a C compiler if you want the
`attention_c` target. `numpy` is needed by the bundled targets, `matplotlib` for
plots. The core framework depends only on PyYAML.
---
## Quickstart — drive it from the agent you already have
Full protocol, including Codex and plain-CLI use, in
**[docs/DRIVING.md](docs/DRIVING.md)**.
### Claude Code
Register the MCP server once, at user scope so it is available in every folder:
```bash
claude mcp add avo -s user -- python3 -m avo.mcp_server
# from a virtualenv, point at its interpreter:
claude mcp add avo -s user -- /path/to/avo/.venv/bin/python -m avo.mcp_server
```
`claude mcp list` should show `avo — ✔ Connected`. Optionally install the
bundled skill so `/avo` works anywhere:
```bash
cp -r .claude/skills/avo ~/.claude/skills/avo
```
Then, in a Claude Code session in any directory:
> Use the avo tools to evolve the game2048 target for 10 steps. Call
> `avo_start_run`, then loop: `avo_next_step`, do the work it asks for,
> `avo_evaluate` until you're happy, then `avo_submit`. If it reports a stall,
> call `avo_supervisor_brief`, answer it, and file it with
> `avo_record_supervisor`.
The eleven tools are the whole loop:
| tool | what it does |
|---|---|
| `avo_start_run` | seed `x_0`, score it, measure baselines, open the lineage |
| `avo_next_step` | the variation prompt: `P_t`, the index of `K`, the contract for `f` |
| `avo_evaluate` | run `f` on the work tree — free, call it as often as you like |
| `avo_submit` | end the step: score, then commit or revert per the policy |
| `avo_revert` | abandon an experiment without spending the step |
| `avo_status` / `avo_lineage` | where the run is |
| `avo_supervisor_brief` / `avo_record_supervisor` | the stagnation intervention |
| `avo_plot` | render the trajectory |
| `avo_list_targets` | what can be evolved |
### Codex
Codex CLI speaks MCP and reads `AGENTS.md`, so both halves work:
```bash
codex mcp add avo -- python3 -m avo.mcp_server
```
[`AGENTS.md`](AGENTS.md) at the repo root documents the loop and the rules that
keep a run honest; Codex picks it up automatically when working in this
directory.
### Without MCP
Every tool has a CLI twin, so a plain shell works just as well — this is the
most portable option and works with any agent, or by hand:
```bash
avo start --target game2048 # seeds x0 and prints the first prompt
# ... edit runs/<id>/work/, run runs/<id>/avo-eval as often as you like ...
avo submit -m "expectimax depth 2 with a positional weight matrix"
avo prompt # the next step's prompt
avo status
avo plot -o trajectory.png
```
---
## Worked runs
Two complete runs ship with the repo, both driven in session mode by a Claude
Code session, both including their dead ends.
### `attention_decode` — beating the vendor kernel
**[`examples/attention-decode-run/`](examples/attention-decode-run/)** evolves
the decode step of attention: one query token against a long KV cache, the
computation an LLM runs for **every generated token**. Scored against
`mx.fast.scaled_dot_product_attention` — Apple's own fused Metal kernel.
**0.05 → 1.14× MLX in three steps.** This is the one where the evolved kernel
actually beats the vendor implementation, and the interesting part is *how*:
- **Step 1 was implementation** — split-K flash-decoding took the kernel from
1.6 GB/s to 106 GB/s, about 95% of the machine's streaming limit. That reached
0.95× MLX and exhausted the lever: you cannot read bytes faster than the memory
controller delivers them.
- **Step 2 was mathematics.** The target's gate is an output-error budget rather
than exact equality, so the search could change the computation. Measurement
showed 99.9% of the softmax mass sits in ~11% of keys, so the kernel now scores
every key but reads V only above a threshold derived so the discarded mass is
provably under 0.3%. That crossed 1.0, spending 2% of the error budget.
The lesson generalises: once a bandwidth-bound kernel is at the roofline, the
only remaining lever is to read fewer bytes, and that is an algorithmic change.
[Read the full write-up →](examples/attention-decode-run/RESULTS.md)
### `attention_c` — the paper's own domain
**[`examples/attention-c-run/`](examples/attention-c-run/)** evolves a forward
attention kernel in C, reaching **2.2× a straightforward NumPy/BLAS
implementation** and close to the NEON roofline. Note the honest framing: that
baseline is not a tuned attention library, and this kernel is *slower* than
torch's CPU SDPA and MLX — Apple's AMX matrix units are unreachable from
portable C. The write-up gives the full comparison.
Three findings from it are worth the click:
- **The paper's own algorithm was the wrong answer here.** A FlashAttention-style
tiled kernel with a streaming online softmax measured *worse*, twice. At these
sizes a whole head fits in L2, so blocking for locality buys nothing while the
per-block rescale is pure added work. The cost is arithmetic, not memory.
- **`-ffast-math` silently breaks the standard fast-`exp`**, by algebraically
cancelling the add-magic-constant rounding trick it depends on. The correctness
gate caught it on an N=3 shape; the throughput number never would have.
- **The run forced a target fix.** Scoring raw GFLOP/s on a laptop doing other
work is not a measurement — identical code ranged 44–76 GFLOP/s in twenty
minutes. `eval.py` now times a NumPy/BLAS reference *in the same process,
interleaved with the candidate*, and scores the ratio.
[Read the full write-up →](examples/attention-c-run/RESULTS.md)
### `game2048` — evolving a game-playing policy
**[`examples/game2048-run/`](examples/game2048-run/)** is a complete 8-step run
of the `game2048` target, driven in session mode by a Claude Code session. The
directory holds the unedited output: the evolved policy, the operator's working
notes, the full trajectory, the screening tools it built, and its dead ends.
**876 → 43 826 — 50× the seed, 14× the strongest baseline.** Games reaching 2048:
0% → 77%. Best tile: 512 → 8192. Apple M5, single-threaded, standard library
only.
Improvement arrives in discrete jumps separated by plateaus, matching the
paper's Figure 5. The two flat versions are pure throughput work that bought the
budget the next step spent — the same role the paper's v19→v20 branchless-rescale
change plays.
The largest single gain (+50.5%) was not an optimisation. The benchmark scores
accumulated game points; the heuristic only measured how survivable a board
looked, so nothing in the search knew that merging two 256s banks 512 points.
Four steps of throughput work were worth +27% combined; one step of checking
what was actually being optimised was worth +50%.
[Read the full write-up →](examples/game2048-run/RESULTS.md)
## What ships with it
### `game2048` — evolve a game-playing policy
Evolve `agent.py` into the strongest 2048 player you can, under a hard
thinking-time budget. Scored as the geometric mean of mean game score across
four banks of twelve deterministic seeds. Blowing the 120 s budget scores zero,
not "slightly less" — so search depth, evaluation-function cost, and pruning all
trade against each other, and that trade-off *is* the problem.
Measured on an Apple M5:
| policy | score |
|---|---|
| seed `x_0` (first legal move) | 876 |
| random baseline | 1 076 |
| corner heuristic baseline | 2 565 |
| greedy one-ply baseline | 3 132 |
Strong expectimax players score in the tens of thousands. The worked run above
reached 43 826.
### `attention_c` — evolve a kernel, the paper's own domain
Evolve a single-precision forward attention kernel in C:
`O = softmax(QKᵀ/√D)V`, causal and non-causal, `D = 64`. Gated on agreement with
a float64 reference over eighteen shapes — including prime and off-by-one
sequence lengths, so a kernel that mishandles its tail fails rather than quietly
scoring well.
Scored as **speedup over a NumPy/BLAS reference timed in the same process**,
geometric mean across four sequence lengths × two masking modes. 1.0 is parity
with the library. Scoring a ratio rather than raw GFLOP/s makes the benchmark
immune to whatever else the machine is doing — absolute throughput on a shared
laptop moves by more than most optimisations are worth.
| kernel | score |
|---|---|
| seed `x_0` (naive, materialises the full `N×N` score matrix) | 0.19× |
| NumPy/BLAS baseline — the "cuDNN" of this setup | 1.00× |
| evolved in 3 steps ([write-up](examples/attention-c-run/RESULTS.md)) | **2.11×** |
The knowledge base covers the online-softmax formulation, tiling and block-size
selection, CPU vectorisation, threading, and how to interrogate the host machine
rather than assuming an ISA. Beating BLAS needs most of them.
---
## How it works
### The run directory
```
runs/<run-id>/
work/ the candidate x_t — a standalone git repo whose history IS the lineage
.avo/scores.jsonl every committed version's full score vector
kb/ the knowledge base K, copied in so paths are stable
avo-eval f, as a zero-argument shim the agent can call at will
NOTES.md scratch space that survives across steps
trajectory.jsonl every step, accepted or rejected
rejected/ the diff of each rejected candidate, kept for the record
logs/ evaluator and agent logs
```
Making the lineage a git repo means the agent inspects `P_t` with tools it
already knows — `git log`, `git show v7:attention.c`, `git diff v6 v7` — instead
of a bespoke API. Each accepted version is a commit tagged `vN` whose message
carries the score vector.
### The commit policy
Paper §3.2: a candidate is committed **only if** it passes the correctness gate
*and* matches or improves the best committed score so far. Anything else is
reverted and its diff archived — it stays part of the agent's internal search
trajectory, but never enters the lineage.
Correctness is a gate, not a dimension. A candidate that fails it scores zero
regardless of what it measured (§3.1). In `attention_c` that means a kernel
that is 10× faster and numerically wrong is worth exactly as much as one that
does not compile.
### The score vector
`f(x) = (f_1(x), …, f_n(x))` — one number per benchmark configuration, with the
geometric mean as the scalar being maximised. This is what makes per-config
movement diagnostic: a change that helps `n1024` and hurts `n128` is a blocking
problem, not a win, and the aggregate alone would hide it.
### The supervisor
Paper §3.3: long autonomous runs fail in two ways — the agent *stalls* when it
exhausts its current line of attack, or enters *unproductive cycles* of edits
that keep failing. After N steps without a new best (default 3), AVO stops and
asks for a redirect: a review of the whole trajectory that proposes several
concrete, different optimisation directions. The redirect is injected into the
next variation prompt as a strong prior, and consumed by exactly one step.
In session mode the supervisor is the same session wearing a different hat,
which is cheap enough to actually use. In unattended mode it is a separate agent
run with read-only intent.
### The trajectory
`avo plot` renders the paper's Figure 5/6: running-best geometric mean as a step
function, filled circles at each new best, dotted per-configuration curves, and
the baselines as horizontal lines. Same caveat as the paper — it shows the
*committed* sequence, not the internal search tree explored between commits.
---
## Unattended mode
To reproduce the paper's setup, where the operator is a spawned agent and nobody
is watching:
```bash
avo run --target attention_c --backend claude_cli --max-steps 40 --time 12h
avo run --resume runs/attention_c-20260321-091500 --time 24h
```
Backends: `claude_cli` (Claude Code headless — the closest analogue to the
paper's agent), `api` (a self-contained agent loop on the Messages API, for
people with only an API key), `agent_sdk` (in-process via `claude-agent-sdk`),
and `mock` (a shell command, for testing the machinery without a model).
This spends quota or credits on every step. Session mode does not.
---
## Adding your own target
A target is a directory with a `target.yaml`, a seed program, a knowledge base,
and an evaluator. The evaluator is any executable in any language; the whole
contract is one JSON object on stdout:
```json
{"correct": true,
"metrics": {"config_a": 1520.3, "config_b": 1477.0},
"error": null,
"notes": "shown to the agent"}
```
`correct` is the gate. `metrics` is the score vector. The scalar being optimised
is their geometric mean unless you supply an explicit `primary`.
```yaml
name: my_target
description: One line, shown in `avo targets`.
seed: seed # copied to work/ as x_0
knowledge_base: kb # copied to the run dir as K
entrypoint: kernel.c # informational, used in prompts
evaluate:
command: ["python3", "{target}/eval.py", "--workdir", "{workdir}"]
timeout: 30m
baselines: # optional, measured once before evolution starts
command: ["python3", "{target}/eval.py", "--baselines"]
score:
direction: maximize
agent:
goal: |
What the agent is actually trying to do, and what the trade-offs are.
```
See `docs/TARGETS.md` for the full contract and `tests/fixtures/toy/` for a
minimal working example.
The knowledge base is worth real effort. It is the `K` in `Agent(P_t, K, f)`,
and the difference between an agent that rediscovers tiling from first
principles over ten steps and one that gets there in two.
---
## What is faithful, and what is not
Faithful:
- the operator formulation `Vary(P_t) = Agent(P_t, K, f)` — a real coding agent
with file editing, shell access and persistent memory, given no
task-specific modifications
- single-lineage continuous evolution with git-backed state (§3.3)
- the correctness gate and the n-dimensional score vector (§3.1)
- the matches-or-improves commit policy, with failed attempts excluded from the
lineage (§3.2)
- supervisor intervention on stagnation and unproductive cycles (§3.3)
- geometric-mean aggregation across benchmark configurations, and Figure 5/6
trajectory plots
Not faithful, and deliberately so:
- **The hardware.** The paper evolves attention kernels on B200 GPUs against
cuDNN and FlashAttention-4. `attention_c` is the same problem on a CPU against
NumPy/BLAS. The optimisations transfer in kind (tiling, online softmax,
vectorisation, scheduling), not in magnitude.
- **The scale.** The paper ran 7 days, 40 committed versions, 500+ explored
directions. A session-mode run of 10–20 steps is a demonstration, not a
replication.
- **Population structure.** Like the paper, this implements the single-lineage
case to isolate the operator. Archive- and island-based regimes are compatible
with the formulation but not implemented.
---
## Repo layout
```
src/avo/
types.py Score, LineageEntry, the correctness gate, geomean
config.py target specs and run configuration
lineage.py P_t as git history
scoring.py f as an external process
knowledge.py K
prompts.py the variation and supervisor prompts — the whole framework/agent interface
run.py run state: seed, evaluate, commit policy, trajectory
session.py driver: the session you already have is the operator
loop.py driver: unattended, spawns an agent per step
mcp_server.py the same operations as MCP tools (no dependencies)
cli.py the same operations as subcommands
plot.py Figure 5/6
agents/ backends for unattended mode
targets/
game2048/ policy evolution under a time budget
attention_c/ kernel evolution — the paper's domain, on a CPU
examples/
attention-decode-run/ beats Apple's own fused kernel by changing the maths
attention-c-run/ CPU kernel evolution, with an honest baseline caveat
attention-metal-run/ GPU prefill — every CUDA instinct measured worse
game2048-run/ policy evolution — 50x the seed
docs/
PAPER_MAP.md every section of the paper, and where it lives in the code
TARGETS.md the evaluator contract
DRIVING.md how to drive a run from Claude Code, Codex, or a shell
AGENTS.md cross-agent instructions (read automatically by Codex)
.claude/skills/ the `/avo` skill for Claude Code
```
---
## Citing
This is an independent reproduction. Cite the original work:
```bibtex
@article{chen2026avo,
title = {AVO: Agentic Variation Operators for Autonomous Evolutionary Search},
author = {Chen, Terry and Ye, Zhifan and Xu, Bing and Ye, Zihao and Liu, Timmy
and Hassani, Ali and Chen, Tianqi and Kerr, Andrew and Wu, Haicheng
and Xu, Yang and Chen, Yu-Jung and Chen, Hanfeng and Kane, Aditya
and Krashinsky, Ronny and Liu, Ming-Yu and Grover, Vinod and Ceze, Luis
and Bringmann, Roger and Tran, John and Liu, Wei and Xie, Fung
and Lightstone, Michael and Shi, Humphrey},
journal = {arXiv preprint arXiv:2603.24517},
year = {2026}
}
```
Licensed under Apache-2.0. Not affiliated with or endorsed by NVIDIA.
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessUnresponsive