Blind-Spectrum Memory Training on Fireworks RFT

Training memory use into a small open model (Qwen3-1.7B) with Fireworks Reinforcement Fine-Tuning (GRPO, via eval-protocol), on a task derived from CLBench's blind_spectrum_monitoring.

The question: can RL make a model better at using memory — recalling information from early in a multi-step task to do better later — rather than baking task skill into the weights? We measure it as memory_gain = late-half − early-half performance, and require it to rise under training while the scenario is randomized per rollout (so there is nothing to memorize in the weights).

Status: COMPLETE — see RESULTS_memory.md for the final, controlled result (2026-07-03). Headline: GRPO reproducibly improves the 1.7B's memory recall (carry-rate 0.89 → 0.94 in two independent runs) on never-repeating content (knowledge-baking impossible by construction) with a scrambled-memory control flat (skill drift excluded) — while one paragraph of explicit instruction still installs a stronger policy than 12 epochs of RL reach, and agent-managed notepad memory erodes under prolonged RL. Sections below describe the task and infra; historical results (pre fresh-band rigor) are marked as such.

The task

Each rollout is a series of 12 scans of ONE fixed but unknown band of ~11–14 transmitters. Each scan shows only the ~3 currently-active transmitters' noisy peaks; the rest are dormant (invisible that scan). The agent reports the persistent occupied regions. A memoryless agent (reports only the current peaks) covers ~3/12 of the band; an agent that accumulates across scans covers more — so coverage rising late-vs-early is memory.

• Reward = per-scan occupied-spectrum IoU (spectrum_reward.py), scored in the env vs the hidden ground truth. Memoryless ≈ 0.16, full-memory → ~1.0 — a 4× signal that requires recalling dormant transmitters. (The bench's native available-IoU is a dead end: memoryless already ≈ 0.47, so it barely rewards memory.)

• Proof metric = memory_gain = late-half occ − early-half occ, tracked but never rewarded (rewarding it invites sandbagging — tanking early scans to inflate the delta).

Related MCP server: Arena MCP Server

Final results — RESULTS_memory.md (the definitive account)

The controlled experiment (fresh bands every epoch via epoch-salting, so no band is ever seen twice; scrambled-memory control arm; behavioral carry-rate instrumentation):

Findings: (1) RL trains memory recall — replicated behaviorally, knowledge excluded by construction, skill controlled at zero; (2) the reward correctly ranks memory use within GRPO groups (corr ≈ +0.8), so reward shape was never the bottleneck; (3) instruction ≫ RL at this scale — the explicit prompt installs a stronger policy (0.472) than RL reaches from a weak base in 12 epochs (0.42–0.44); (4) RL preserves env-maintained memory but erodes agent-managed (notepad-tool) memory over long training — the channel-cost dependence result.

Historical note: earlier positive readings (btalo63n, dmzj2mz8, behavioral tables in RUNS_spectrum.md) predate the fresh-band rigor — they reused the same 48 bands across all epochs and are confounded by content repetition; keep them as history, cite RESULTS_memory.md as the result.

The GRPO fix that made anything train (the hard-won one)

GRPO normalizes advantages within a prompt's candidate group — those candidates must face the same task. The env originally drew a fresh random band per rollout, so a row's 12 candidates each got a different band → advantage was band-luck, not policy → zero gradient, frozen policy at any learning rate (six flat runs). Fix: seed the band deterministically from the per-row session_id, so a row's candidates share one band while different rows give different bands. This single change turned flat runs into training. (See _band_seed in spectrum_mcp.py / spectrum_turn_processor.py.)

Why the setup diverges from CLBench (deliberate)

Four choices exist specifically to isolate and reward memory:

1. Reward = occupied-IoU, not the bench's available-IoU (memory-insensitive).

2. ICL-off windowing (spectrum_context_window.py) — the model sees only [system] + [current scan], so its only cross-scan memory is the notepad/scaffold, not re-read history.

3. Forced dormancy (n_active=3 of ~12) — most transmitters dormant each scan, so memory is required.

4. memory_gain (late−early) as the proof metric.

Two rollout interfaces (both cloud-runnable, no hosting)

• McpGym (test_spectrum_rft.py + spectrum_mcp.py) — the proven path. Scans arrive as tool results (agent calls submit_report, gets the next scan back).

• Custom SpectrumTurnRolloutProcessor (test_spectrum_turn.py + spectrum_turn_processor.py) — bench-shaped. Scans arrive as user messages, the agent replies with assistant tool calls, submit_report acks. Confirmed running in-cloud: the cloud runs whatever rollout_processor the uploaded @evaluation_test names, via pytest — no MCP server, no hosted endpoint.

The message-role of the scan (tool vs user) is the only difference between the two; it doesn't change what the model learns. The custom processor exists to match CLBench's user/assistant turn structure.

Repo layout

Current (spectrum) working set:

Earlier pivots (kept for reference, not the current path): poker_*.py / poker_*.jsonl (exploitable- poker memory env — a 1.7B never learned the notepad, pivoted away) and memory_*.py (a synthetic learn-and-recall probe). Plus utilities: plot_eval.py, compare_epochs.py, oracle_diag.py, etc.

How to run

# 1. generate a dataset (N rows; the band is drawn per-row in the env, so rows are interchangeable)
python make_spectrum_dataset.py --n 48 --out spectrum_np48.jsonl

# 2. register the dataset + upload the evaluator
firectl create dataset clbench-spectrum-np spectrum_np48.jsonl
python -m eval_protocol upload --entry test_spectrum_rft.py::test_spectrum_rft --force --yes
#    (or test_spectrum_turn.py::test_spectrum_turn for the bench-shaped user/assistant interface)

# 3. launch the RFT job (Qwen3-1.7B, GRPO)
firectl create reinforcement-fine-tuning-job \
  --base-model accounts/fireworks/models/qwen3-1p7b \
  --dataset clbench-spectrum-np \
  --evaluator accounts/<acct>/evaluators/test-spectrum-rft-test-spectrum-rft \
  --output-model clbench-spectrum-np \
  --epochs 10 --learning-rate 5e-5 --temperature 1.2 \
  --max-output-tokens 1024 --response-candidates-count 12 \
  --max-concurrent-rollouts 96

# 4. watch memory_gain / mean_occ / scans_completed per epoch (dashboard, or poll the job's outputMetrics)

# 5. (optional) evaluate a trained model on the OFFICIAL 90-instance bench schedule
./eval_bench90.sh "<litellm-model-id>" icl_notepad     # see EVAL_bench90.md

Local pytest test_spectrum_rft.py needs a callable model; Qwen3 models are not serverless on the free tier, so end-to-end validation happens in the cloud RFT run (the mechanics are unit-tested locally).

Differences from CLBench blind_spectrum_monitoring

Aligned: user/assistant turn structure (custom processor), scan generation + Jinja rendering (bench's own task + templates), detector noise (p_miss=0.15, p_false_alarm=0.2).

Limitations & honest caveats

• 1.7B ceiling. Agent-controlled notepad memory doesn't train at this size; the working results all rely on the environment helping maintain the memory (scaffold) or on full-context replay. A 4B+ base is the likely unlock (capacity-blocked when tried).

• Not the official bench number. We use random per-row bands + occ-IoU + ICL-off, not the frozen 90-instance corpus + available-IoU. eval_bench90.sh bridges to the official schedule — but note available-IoU under-shows memory (see EVAL_bench90.md).

• Noisy training curves (small magnitude, non-monotonic). Read the trend — early-vs-late halves, slope, cross-run reproducibility, matched control — not peak epochs.

• Capacity. RFT jobs can stall at model-serving deployment creation during Fireworks capacity crunches, independent of the code (a known-good, unrelated job stalls identically). Diagnose with percent / acceleratorSeconds / a deployment check, and verify with an independent control job.

Infra learnings (gotchas worth knowing)

• Fireworks RFT runs eval-protocol (confirmed: eval-protocol-0.3.23 in the cloud streamlogs). It runs whatever rollout_processor the uploaded @evaluation_test names — including a custom in-process one (no MCP server, no hosting). The earlier belief that "the cloud won't drive a custom processor" was a capacity false-negative.

• The in-training model lives in config.completion_params, not row.input_metadata — a custom processor must read it there (this is what McpGym does; reading the wrong place → 404 retry-hang).

• Fireworks rejects tool_choice → set litellm.drop_params = True (or don't pass it).

• totalInputRequests is unreliable (stays 0 even for running jobs) — use percent / acceleratorSeconds / the streamlog instead.

• Qwen3-1.7B thinking exhausts the token budget before the tool call (finish_reason=length) → /no_think.

• RemoteRolloutProcessor also yields user/assistant, but needs a hosted HTTP server; the custom in-process processor is the hosting-free way to the same interface.