clbench-fireworks-rft

Reinforcement Fine-Tuning of Qwen3-8B on the CLBench exploitable_poker task, running on Fireworks infrastructure via eval-protocol MCP-Gym.

This is a port of the sr-networks/clbench-verifiers GRPO setup (Will Brown's verifiers framework + PrimeIntellect hosted training) onto Fireworks RFT. The CLBench poker simulator, action parsing, and reward shaping carry over unchanged; only the RL-framework glue is rewritten.

Why a port and not a copy

verifiers and Fireworks RFT are different abstractions. What transfers vs. what changes:

Key win: because the poker_act tool's parameters are exactly the PokerAction fields (action / thinking / amount), the model emits structured tool calls and malformed-JSON parse failures become impossible — the old guided_json + parse_failure_penalty machinery is no longer needed.

Related MCP server: multivon-mcp

Files

Setup

./setup.sh                                  # deps + bin/python shim
export FIREWORKS_API_KEY="fw_..."           # https://fireworks.ai/account/api-keys
firectl set-api-key "$FIREWORKS_API_KEY"
export PATH="$PWD/bin:$PATH"                 # python3.11 shim first (gym spawns `python server.py`)

firectl (Go binary) install: brew tap fw-ai/firectl && brew trust fw-ai/firectl && brew install fw-ai/firectl/firectl

Launch an RFT job

Two paths. The direct firectl path is what we actually used (it avoids a CLI bug — see Gotchas).

A) Upload the evaluator, then launch with firectl ← used

# 1. upload the evaluator (env + reward) so Fireworks builds the rollout container
eval-protocol create rft \
  --evaluator test_poker_rft.py::test_poker_rft \
  --dataset poker_dataset.jsonl --mcp-server server.py \
  --training-config-base-model accounts/fireworks/models/qwen3-8b \
  --dry-run --skip-validation -y          # uploads evaluator; ignore the poller timeout

# 2. confirm evaluator is ACTIVE, upload dataset, create the job
firectl create dataset clbench-poker-qwen3-8b-data poker_dataset.jsonl
firectl create reinforcement-fine-tuning-job \
  --base-model accounts/fireworks/models/qwen3-8b \
  --dataset clbench-poker-qwen3-8b-data \
  --evaluator accounts/<ACCOUNT>/evaluators/test-poker-rftpytest-poker-rft \
  --output-model clbench-poker-qwen3-8b \
  --epochs 2 --learning-rate 1e-6 --temperature 1.0 \
  --max-output-tokens 1024 --response-candidates-count 8

B) Pure eval-protocol (once the poller bug is fixed upstream)

eval-protocol create rft --evaluator test_poker_rft.py::test_poker_rft \
  --dataset poker_dataset.jsonl --mcp-server server.py \
  --training-config-base-model accounts/fireworks/models/qwen3-8b \
  --training-config-output-model clbench-poker-qwen3-8b \
  --training-config-epochs 2 --training-config-learning-rate 1e-6 \
  --inference-parameters-temperature 1.0 --inference-parameters-max-output-tokens 1024 \
  --inference-parameters-response-candidates-count 8

Config mapping from the Prime TOML

rollouts_per_example=8 → --response-candidates-count 8 (GRPO group size) · temperature=1.0 · max_tokens=1024 → --max-output-tokens 1024 · enable_thinking=false baked into the gym prompt · guided_json → tool-call schema (free).

Training runs

See RUNS.md for the full log. Summary:

Monitor: firectl get reinforcement-fine-tuning-job <job-id> · dashboard: https://app.fireworks.ai/dashboard

Gotchas (hard-won)

• from __future__ import annotations breaks eval-protocol. It stringifies annotations, so FastMCP tool registration (issubclass("str", Context)) and the @evaluation_test signature validator both fail. Do not use it in poker_mcp.py or test_poker_rft.py.

• FastMCP (this version) crashes on Optional[int] tool params while locating the Context arg. poker_act uses a plain int = -1 sentinel instead.

• firectl needs firectl set-api-key; it does not read FIREWORKS_API_KEY automatically. (firectl whoami additionally needs OIDC signin — ignore it.)

• eval-protocol create rft has a poller bug: it polls …/evaluators/<file>.py::<func> — the .py:: makes the URL malformed → HTTP 400 → false 10-minute "evaluator not ready" timeout. The evaluator is actually ACTIVE; launch via firectl.

• macOS python is often 2.7. The gym spawns python server.py, so bin/python must shim to the 3.11 interpreter that has the deps and be first on PATH.

• Rollouts run on Fireworks, in a container built from requirements.txt — so a local serverless deployment of the base model is not required for training (only for local pytest rollouts).