FIREFIGHTER

The problem

Run three or four agents at once and each one decides on its own to train a model, run an eval sweep, build a Docker image or render something. They all grab the same GPU. VRAM runs out (CUDA out-of-memory, or silent spill into system RAM on Windows), two trainings slow each other to a crawl, your game stutters, audio crackles, and nobody knows who's using what or for how long.

Related MCP server: agent-bus-mcp

What firefighter does

• 🎤 You always come first. It watches your keyboard and mouse, fullscreen apps, games (3D-engine use), and every process it didn't start: Docker containers, Kubernetes pods, local model servers, your own apps. It keeps a dynamic reserve of CPUs, RAM and VRAM for you that grows within half a second when you sit down or start a game and gives capacity back slowly when you leave.

• 🎟️ Agents take turns instead of fighting. Heavy work waits in a queue. A GPU job starts only when its VRAM need fits, and that need is learned from the command's past runs. Projects that have used the least go first (least attained service), with a 45-minute cap on any wait.

• 💹 A market maker, not a bidding war. Agents don't bid against each other. firefighter posts prices for three tiers: spot (leftovers, ⅓ price), normal, and urgent (2×, only when you're waiting on the result). Every reply an agent already gets ends with one line: when each tier would start, what it costs against the agent's budget, the odds the GPU is busy over the next 2 hours, a quiet hour to aim for, and a do: with the one recommended action. The advice never tells an agent to outbid anyone, so the whole market costs about 60 tokens and there's no bidding war to win.

• 🛡️ Fallbacks for careless or rogue agents. Heavy processes an agent starts outside firefighter are found and adopted anyway: idle priority, billed 1.5×, preempted first. That makes skipping the queue strictly worse than using it. Other guards cover queue flooding (per project and in total), new project names that try to get a fresh budget (all new projects share one small daily pool), chronic under-estimates (×1.25), VRAM hoarding (×3), lease overruns (×2, preempted first), leases over 24 h, extensions after a short backfill, and one agent cancelling or re-bidding another agent session's job (you, from a terminal or editor, can manage any job). Text an agent sends (plans, names) is stripped of control characters before other agents or your terminal see it.

• 🫡 Graceful for the agent. When you need the machine back, the job's own client says why, and the job gets the whole grace period (60 s) to save and exit (checkpoint protocol). On macOS/Linux a job still running 10 s before the deadline gets SIGINT (Ctrl+C) before SIGTERM; on Windows it keeps the full grace period, then is stopped. It's requeued under the same id and the agent is told why and that there's nothing to resubmit.

• ⏸️ Polite preemption. The ladder goes: shrink the job's CPU set → cooperative pause at a step boundary → suspend (for jobs that ignore the pause) → checkpoint-and-exit with automatic requeue when you need the memory back → kill, but only when RAM is critical.

• 🌐 Network too. Downloads and uploads (docker pull, pip/npm/cargo installs, git clone, model and dataset downloads; detected automatically, or --net) take turns, wait while you're using the link (a call, a stream: network.human_busy_pct), and pause while you game.

• 🧮 Any machine. Any number of GPUs from any vendor (NVIDIA, AMD, Intel; mixed is fine), any CPU count (Windows processor groups past 64 included), any RAM size. Each GPU job is placed on the card it was admitted on, in that vendor's own numbering. On mixed machines a job goes to the NVIDIA cards unless it says otherwise (--gpu-vendor amd|intel|any). Every PC runs its own firefighter; nothing goes over the network.

• 🔌 Set and forget. setup starts the guard at logon and, for every agent it finds installed, registers the MCP server and adds a short rule block to its global instructions, so every new session (including T3 Code threads) already knows the protocol: Claude Code, Codex, Gemini CLI, Antigravity, Cursor, Grok Build, OpenCode, Kimi CLI and Kimi Code, Pi, Muse Code, fx, Kiro, Windsurf/Devin, Copilot CLI, Amp, Qwen Code, Crush, Cline, Goose and Droid. Each in its own format; uninstall removes all of it.

Install (one command)

Windows (PowerShell):

irm https://raw.githubusercontent.com/JoshWright22/FIREFIGHTER/main/scripts/install.ps1 | iex

macOS / Linux:

curl -fsSL https://raw.githubusercontent.com/JoshWright22/FIREFIGHTER/main/scripts/install.sh | sh

The installer checks the release checksum, puts one ~8 MB binary on your PATH, and runs firefighter setup. With Go 1.23+: go install github.com/JoshWright22/firefighter/cmd/firefighter@latest && firefighter setup.

Requirements: Windows 10/11, macOS 12+ or Linux, on x64 or arm64. No admin rights needed. For GPU numbers you need nvidia-smi (NVIDIA driver) or rocm-smi (AMD). On Linux, idle detection uses GNOME, KDE or another freedesktop ScreenSaver desktop, or xprintidle; headless machines count you as "present".

What install changes (each config file gets a one-time *.firefighter.bak backup first):

Verify: open a new terminal and run firefighter status. Remove: firefighter uninstall undoes autostart, the rule blocks and the MCP entries. Then delete the binary and, if you want to forget budgets and history, the state folder.

Use

firefighter run --gpu --minutes 40 --vram 10 --name "nanoGPT train" -- python train.py
firefighter run --gpu --urgent --minutes 10 -- python eval.py          # you're waiting on it: 2x
firefighter run --gpu --spot --minutes 120 -- python sweep.py          # can wait: 1/3 price
firefighter run --minutes 5 -- npm run build                          # long CPU work
firefighter run --gpu --minutes 20 -- docker run --gpus all my/eval    # containers are governed too
id=$(firefighter run --detach --gpu --minutes 90 -- python sweep.py) && firefighter wait "$id"
firefighter quote --gpu --minutes 30 --vram 8                          # the market, no side effects
firefighter bid <id> 10                                                # make a waiting job urgent (5 = normal)
firefighter status                                                     # who's on stage until when, the queue, budgets
firefighter plan "eval sweep" --when tonight --minutes 120 --gpu 90 --vram 12
firefighter extend <id> 15                                             # running long? say so

run waits for a turn, then runs your command in your own environment (venv, conda, uv, nvm and docker all just work), passes its output and exit code through, and ends with a receipt:

◆ 3f2a91c0 nanoGPT train waiting #2, starts ~14:05 (in 22m): GPU turn queue
   gpu 40m: start ~22m, cost ~42 of 212 (urgent ~6m ~84) (spot ~70m ~14) | busy 15/60/120m 92/35/12% | quiet 02:00 (usually 8% busy) | do: queue it at normal (starts ~22m); keep working meanwhile
◆ nanoGPT train: spotlight's yours (lease 40 min, budget left 212.0)
◆ nanoGPT train: curtain call, ok (exit 0) in 37.2 min (declared 40), waited 6.1, charged 31.4 machine-min, peak 3.1 GB RAM / 9.4 GB VRAM

MCP tools (one-line text replies plus structuredContent): firefighter_submit, firefighter_wait, firefighter_status (which also gives quotes), firefighter_job (extend, urgent/normal, cancel) and firefighter_plan. firefighter_config is listed only when you allow agents to change the policy.

How it decides (and how to change it)

Zero config gives you the researched defaults below. You can swap each part by name, live, with firefighter config set <key> <value>. Agents can read the policy through the firefighter_config MCP tool, and can change it only if you set guard.agents_can_configure true.

Presets: balanced (default), gamer, throughput, fair → firefighter config set preset gamer.

The preemption ladder

Steps run in order, and each one fires only if the trigger is still there:

1. CPU set. Jobs always live outside your CPU reserve (Windows, Linux).

2. Cooperative pause (GPU jobs, while your state holds the GPU): firefighter creates the file at FIREFIGHTER_PAUSE_FILE. A job that honors it creates FIREFIGHTER_PAUSED_FILE and sleeps until the pause file is gone.

3. Suspend: if the job hasn't acknowledged within 10 s (or, where per-process GPU counters exist, acknowledged but kept computing), it's suspended. It stays suspended while you need the GPU. Past 60 s, if the memory it holds is what you're short of, it's stopped and requeued (it ignored the pause, so it can't checkpoint). --evict never jobs stay suspended until you're done.

4. Checkpoint: when RAM or VRAM (per card) stays under your reserve for 20 s, or when a spot job has run its 10-minute minimum and an in-budget job is waiting on what it holds, firefighter lifts any pause and creates FIREFIGHTER_CHECKPOINT_FILE. The job gets 60 s to save and exit. On Unix it then gets a SIGTERM, and 10 s later it's killed. A job that exits with 75, or exits at all after the request, is put back in line under the same id. --evict force jobs are killed at once (and put back in line the same way).

5. Kill: if available RAM drops under 3%, one job is killed at once, even mid-grace, and the next only after 5 s (memory takes a moment to come back).

Victims are picked so one eviction fixes the problem: first jobs whose memory covers the shortfall, then the highest band (spot, past their lease, adopted), then the heaviest. One eviction at a time. --evict never jobs are only ever touched when RAM is critical. CUDA can't be checkpointed from outside on Windows or macOS, so the job has to cooperate. firefighter protocol prints a short Python helper you call once per step.

Per-OS support

CLI reference

Files in the state folder: firefighter.toml (your settings), state.json (budgets, learned history, receipts, the hour-of-week occupancy), STATUS.md (a status snapshot rewritten every 15 s for agents that read files), daemon.log, logs/<id>.log (detached jobs, kept 7 days), signals/ (pause and checkpoint files), daemon.json and daemon.lock.

Editors and agent apps

• Every agent setup finds (see the list above, T3 Code's providers included): MCP and the rules are wired up automatically.

• Claude Code plugin: a live status-line entry, /firefighter, and a guard that sends heavy Bash commands back to the agent to rerun through the queue: /plugin install firefighter --marketplace JoshWright22/FIREFIGHTER

• VS Code, Cursor, Windsurf extension: a live status bar item (the .vsix is on every release).

• Anything else: firefighter statusline gives one line for any status bar (Claude Code's statusLine, tmux, starship), and firefighter events streams NDJSON for editor plugins. There's also a local HTTP API. See docs/integrations.md.

How it compares

Prior art was surveyed in Nov 2026 (about 65 projects read in full); nothing found does more than about half of this, and none on all three OSes. Details: docs/research.md.

FAQ

How do I stop several Claude Code or Codex agents from fighting over my GPU? Install firefighter. New sessions read the rule block and register the MCP server, so they queue GPU work through firefighter run or firefighter_submit and wait their turn.

Why do I get CUDA out of memory when two AI agents train at once? Each agent sees the whole card free and starts its own job, so together they ask for more VRAM than the card has. firefighter starts a GPU job only when its VRAM need fits on one card next to what's already running and your reserve; the need is learned from that command's past runs.

Can several AI agents share one GPU? Yes. They take turns, or run side by side when the card has room (gpu.max_jobs per card). With more than one GPU, each card takes jobs on its own and a job goes to the least busy card that fits.

Is there an MCP server for queuing GPU jobs? firefighter mcp, registered for every agent setup finds. Tools: firefighter_submit, firefighter_wait, firefighter_status, firefighter_job, firefighter_plan.

Will it slow down my games? It's designed to do the opposite. Fullscreen apps and 3D-engine use switch you to the "game" state (on macOS and Linux, list your games in reserve.exclusive_apps). Running GPU jobs get a cooperative pause; jobs that ignore it are suspended. If your game then needs the memory a job holds, that job checkpoints (or, if it ignored the pause, is stopped) and requeues. No new GPU jobs start until you're done.

Does it work with Docker and Kubernetes? Containers you run yourself count as your load and are protected by the reserve. firefighter run -- docker run ... labels the container so firefighter can pin, pause and stop it like any other job.

What if an agent ignores the rules? If an agent CLI starts something that stays heavy for 60 seconds outside firefighter, it gets adopted. It's put on idle priority, charged to its project, and evicted first under pressure.

Can agents cheat the market? Prices are posted, not auctioned, so there's no bidding game. A bid of 10 costs double and buys at most 30 minutes of lead. Budgets belong to projects, so running more jobs doesn't buy more turns. Runtimes are checked against each command's history. Agents can read the policy but not change it unless you set guard.agents_can_configure. An agent with shell access could still edit firefighter's files directly. firefighter is a referee for cooperative agents, not a security boundary.

Does it need an NVIDIA GPU? No. It reads NVIDIA (nvidia-smi plus Windows GPU counters) and AMD (rocm-smi). On Apple silicon it schedules unified memory as RAM. Without a GPU it still schedules CPU and RAM.

Research

docs/how-it-works.md maps every paper's idea to the code that uses it, with the numbers, and lists what isn't implemented and why. The policy comes from a survey of about 40 repos and 30 papers. Among them: DRF (NSDI '11), Tiresias (NSDI '19), Themis (NSDI '20), Gavel (OSDI '20), Karma (OSDI '23), PerfIso (ATC '18), PilotFish (ATC '22), lottery and stride scheduling (OSDI '94), BOINC, KAI-Scheduler, AgentCgroup and AIOS. The full report with sources is in docs/research.md.

Testing

go test ./... runs the unit tests and an end-to-end suite. The suite builds the binary, starts a real daemon, and scripts the human, memory pressure and a GPU through FIREFIGHTER_FAKE_FILE. It covers GPU turns that never overlap, VRAM admission, game pause and resume, RAM-pressure checkpoint and requeue, cancel, quotes and re-bids, adoption of rogue processes, daemon restart, the MCP server, and setup/uninstall. CI runs it on Windows (x64 and ARM), macOS and Linux (x64 and ARM) VMs, then tests both one-command installers.

Security notes

• The guard, setup and both installers refuse to run as administrator or root: any agent of yours can read the token and write its files, so anything elevated would lend those rights to every agent. It runs as you, binds only 127.0.0.1, and keeps its folder private (0700/0600 on Unix).

• It only acts on what a job's own client started (the caller is identified by its exact TCP connection, not by what it claims; only that client can start the job), on containers labeled for that job, and on processes adopted by you or, for its own session's work only, by an agent. Ancestry and ownership checks ignore reused pids. "You" means a caller whose unbroken process chain reaches a terminal window (Windows Terminal, iTerm, GNOME Terminal, ...; or sshd) with no agent CLI, script runtime (node) or IDE in between (guard.human_hosts, guard.agent_hosts). The terminal must be installed by an administrator: the program and every folder above it owned by the system (root on macOS/Linux; TrustedInstaller, SYSTEM or Administrators on Windows), or a real sshd session started by root (a Linux text-console login, whose parent is init, doesn't count: an agent's detached process ends up there too). A terminal installed just for your user (a kitty or WezTerm download, Scoop, an app you dragged into /Applications, a Nix profile) doesn't count. The desktop shell, a classic console started from it, tmux and screen don't count either (an agent can open those silently); from them, add --force for other projects' jobs and for adopt. Anything else, including detached helpers and agents hosted in node or an IDE, is an agent session limited to its own work and its project; from an IDE terminal, add --force to manage another project's job. An agent that opens a window in a terminal you already have running (wt.exe, open -a Terminal, a silent ssh localhost) passes for you there too. These are guardrails against accidents, not a boundary against a hostile agent with your shell.

• firefighter_submit runs a command as you, outside an agent's own sandbox; its MCP annotations say so, so hosts can ask for approval as they do for a shell.

• Releases are built by GitHub Actions with SHA-pinned actions, the Go toolchain from go.mod, a locked npm toolchain for the extension, and least-privilege tokens, and carry signed build provenance. The installers check it automatically when the GitHub CLI is installed and signed in; by hand: gh attestation verify firefighter_windows_amd64.zip --repo JoshWright22/FIREFIGHTER.

• Any agent can stop the guard (it's your process; it could kill it anyway). It comes back on the next firefighter call, and the new guard thaws anything the old one had suspended.

• Not a sandbox: an agent with your shell can do anything you can. firefighter makes the polite path the cheap one and catches the careless one; it doesn't contain a hostile one.

• Windows: a job's whole process tree is in a job object, so nothing it starts outlives it by accident. Shared build servers on the guard.keep_alive list (MSBuild nodes, VBCSCompiler, ...) are spared when the job ends normally; if its client is killed, they go with it.