GPUPlane

Agent-native training control plane for personal & small-scale GPU environments. A control plane for personal GPU training: job queueing and scheduling, real-time metrics/logs, checkpoint registration, event diagnostics — built for Agents (MCP) and browsers, no longer reliant on SSH + tmux + eyeballing loss.

Product documentation site: https://ericyuan2007.github.io/GPUPlane/ (Quick Start / Guides / MCP tool reference); design docs in docs/ (product research / product design / system design / MVP roadmap), also readable in the design doc archive on the documentation site.

Features (v0.1)

• Zero intrusion into training code: python train.py always remains independently runnable; GPUPlane only does process hosting and observation, the SDK is fully optional and never throws into the training process.

• GPU slot scheduling: exclusive allocation per card (no idle guessing from utilization), priority queue, automatic retries on failure (except OOM), automatic CUDA_VISIBLE_DEVICES injection.

• Three-level metrics ingestion: real-time tail of TensorBoard directory (L1, zero changes), SDK direct reporting (L2, from gpuctl import run), NVML system metrics (L3).

• Logs: full agent persistence to disk + server tail of last 2000 lines + SSE real-time stream, gpuctl logs -f tails directly.

• Checkpoint discovery: debounced watch-directory scan, associating best metrics by step.

• Event semantics layer: LOSS_NAN / OOM / DISK_LOW / lifecycle events, deterministic rule-engine evaluation.

• SQLite single file: no Kafka/Redis/Postgres, WAL mode, one-command online backup.

• Offline resilience: local jsonl spool on the agent, replay on disconnect; idempotent cursor dedup on the server, no loss and no duplication across agent restarts.

Related MCP server: Train in Silence

Features (v0.2, accepted)

• Experiment management: Project/Experiment CRUD and run grouping; Web UI Experiments page + run comparison view (multi-run metric overlay, sorted by best).

• Evaluation closed loop: POST /checkpoints/{id}/evaluations queues an EVALUATE job (inherits the training job's working_dir/resources) → results reattached → GET /checkpoints:recommend recommends a checkpoint by primary metric.

• Complete event rules: LOSS_SPIKE / OVERFITTING_SUSPECTED / GPU_UNDERUTILIZED / DISK_LOW, in-process rolling window + debounce.

• Failure-classified retry: OOM / EXIT_CODE / DISPATCH_FAILED are not auto-retried, the rest are requeued per max_attempts.

• DockerRunner: same semantics as ProcessRunner (logs/exit code/slots), --gpus injection, docker kill for cancellation.

• Event forwarding: server.yaml configures ntfy/Bark webhook (severity/type filtering, best-effort); gpuctl event-hook wakes local scripts from the event stream.

• Framework callbacks: gpuctl.callbacks provides thin wrappers for Lightning / HF Trainer, optional import with zero dependencies.

• MetricDefinition UI: visible editing of direction and global/project/experiment primary; resolution order experiment → project → global, driving checkpoint recommendation.

Features (v0.3, accepted)

• Agent-native (MCP): gpuctl-mcp runs as a standalone process/package (fastmcp 3.4.7 pinned) exposing 21 tools — 11 observe + 5 control + 5 semantic; core is streamable HTTP stateless. A failure does not affect the server (design §15).

• Semantic layer (no LLM): diagnose_run / compare_runs / compare_checkpoints / get_best_checkpoint / explain_failure — five pure Python functions (rules + statistics), with REST and MCP both as thin exposure layers; each returns next_actions to guide the next Agent step, no polling needed.

• 3 skills: run-experiment / monitor-experiment / analyze-results (.claude/skills/gpu-training/), 6-field frontmatter, allowed-tools pre-authorizes MCP tools, encoding the “submit → monitor → compare → recommend” experiment loop.

• Plugin packaging: .claude-plugin/plugin.json (stdio MCP + skills + hooks distributable); .mcp.json (http, gitignored).

• AGENTS.md: repository-level agent rules (canonical, mirrored by CLAUDE.md) — bare process + SQLite red lines, experiment loop, metric/job/ckpt conventions, OOM runbook.

• Read/write scope tiers: GET /auth/whoami exposes {name, scope, can_write}; write operations (submit/cancel/retry/evaluate) require write-scope tokens, read-only tokens return WriteScopeError.

Acceptance: an Agent completes the full “submit training → monitor → compare → recommend” loop using only natural language + MCP, with no manual platform interaction. See docs/08-v0.3-acceptance.md.

Quick Start (30-minute standard deployment)

Environment: Python ≥3.12, uv.

git clone <repo> && cd GPUPlane
uv sync                          # 安装全部组件（server/agent/cli/sdk）

# 1. 启动 server（GPU 机器上；首启自动生成 admin token 写入 ~/.gpuctl/server.yaml）
uv run gpuctl-server

# 2. 启动 agent（同机；token 从 server.yaml 复制到 ~/.gpuctl/agent.yaml）
uv run gpuctl-agent

# 3. 提交训练（cpu_only 示例先跑通，再上 GPU）
uv run gpuctl job submit -n mnist -g 1 \
  -d "$PWD/examples/mnist" --watch checkpoints \
  -- python train.py --epochs 3

# 注意：-d/--working-dir 按【agent 所在机器】解释，CLI 不做本地改写；
# 从笔记本向远端提交时传远端绝对路径。

# 4. 观测
uv run gpuctl status             # 节点/队列总览
uv run gpuctl job list           # 任务状态
uv run gpuctl logs -f <job-id>   # 实时日志
uv run gpuctl run show <run-id>  # 指标摘要（latest/best/trend）

# 5. Web UI（server 自动托管 web/dist；也可用 GPUCTL_WEB_DIST 指定）
open http://<gpu-host>:8600      # 输入 token 登录

Import historical experiments (TB events + checkpoints → IMPORTED run):

uv run gpuctl import-run ~/experiments/old-run --project legacy

Restricted/offline network installation

Dependencies are only about 50 small wheels in total (torch is not a platform dependency; the existing environment on the training machine is fine). When the network is restricted, pre-download on a machine with internet access, then copy to the target machine and install offline:

# 在能上网的机器（如 Mac）上，为 Linux x86_64 + py3.12 下载
uv export --format requirements-txt --locked --no-hashes --no-dev -o /tmp/reqs.txt
grep -v '^-e ' /tmp/reqs.txt > /tmp/reqs-clean.txt
uv run --python 3.12 --with pip python -m pip download -r /tmp/reqs-clean.txt hatchling editables \
  --python-version 312 --only-binary=:all: \
  --platform manylinux_2_28_x86_64 --platform manylinux_2_17_x86_64 \
  --platform manylinux2014_x86_64 -d ./wheels
rsync -az ./ ./wheels/ gpu-host:~/GPUPlane-wheels/   # 含仓库本体

# 目标机（离线）
cd ~/GPUPlane && uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python --no-index --find-links ~/GPUPlane-wheels \
  -r ~/GPUPlane-wheels/reqs-clean.txt hatchling editables
uv pip install --python .venv/bin/python --no-index --no-build-isolation \
  -e ./packages/common -e ./packages/tbreader -e ./packages/sdk \
  -e ./packages/server -e ./packages/agent -e ./packages/cli

Two deployment topologies

Same machine (recommended for start): server + agent both run on the GPU machine, a laptop browser/CLI accesses over LAN (server.yaml sets host: 0.0.0.0, token auth).

Separated: server runs on an always-on lightweight machine (even a Mac mini/NAS), agent runs on each GPU machine, agent.yaml’s server_url points to the server’s ws://<ip>:8600. The agent makes a single outbound long-lived connection; GPU machines need no inbound port, and auto-reconnect + spool replay cover disconnects.

Event push to phone / local hooks (v0.2)

Add a webhook to ~/.gpuctl/server.yaml (ntfy example; Bark uses kind: bark + device URL):

webhooks:
  - url: "https://ntfy.sh/my-gpu-topic"   # 手机装 ntfy 订阅同一 topic
    kind: ntfy
    min_severity: warning                  # info|warning|critical，低于此不推
    # types: ["OOM", "LOSS_NAN"]           # 可选：只推这些事件类型

Local automation (run a command on the current machine when an event arrives, e.g. wake a local agent):

gpuctl event-hook --severity critical -- /path/to/on-event.sh
# 事件经 GPUCTL_EVENT_TYPE/SEVERITY/MESSAGE/RUN_ID/... 环境变量 + stdin JSON 传入

Agent-native: drive the experiment loop with natural language (v0.3)

gpuctl-mcp is a standalone adapter process (does not depend on server/agent) that exposes the control plane as 21 MCP tools. Once configured, Claude Code (or any MCP client) completes “submit training → monitor anomalies → compare checkpoints → give recommendations” in natural language, without touching Web/CLI:

# 1. 起 adapter（独立进程；指向 server，带 write-scope token）
GPUCTL_SERVER_URL=http://127.0.0.1:8600 GPUCTL_MCP_TOKEN=<write-token> \
  gpuctl-mcp serve --port 18602            # streamable HTTP, stateless
# stdio 形态（插件用）：gpuctl-mcp stdio

# 2. 让 Claude Code 发现它（仓库根 .mcp.json，已 gitignore）
cat > .mcp.json <<'JSON'
{ "mcpServers": { "gpuctl": { "type": "http",
  "url": "http://127.0.0.1:18602/mcp",
  "headers": { "Authorization": "Bearer <write-token>" } } } }
JSON

# 3. 自然语言驱动（skill 自动加载，无需手点工具）
claude -p "提交一个 mnist 训练，跑完告诉我结果，再对比最近两次 run 给我最好的 checkpoint"

Read/write scope tiering: under a read-only token, submit_job/cancel_job/retry_job/evaluate_checkpoint/set_primary_metric return WriteScopeError. The three skills (.claude/skills/gpu-training/) encode the experiment loop and pre-authorize MCP tools; see docs/08-v0.3-acceptance.md.

Training-side SDK (optional)

from gpuctl import run

run.init(project="qwen-sft", experiment="lr-2e5", config={...})   # 平台 job 内自动 attach，可省略
run.log({"train/loss": loss.item()}, step=step)                   # 有界队列，绝不阻塞/抛错
run.log_checkpoint(path, step=step)                               # 只登记，不搬运文件
run.finish()

Zero configuration in platform-dispatched jobs (env auto-injected); a bare run outside the platform automatically registers a run with source=sdk; without a server it silently falls back to a local jsonl spool (~/.gpuctl/spool/), so training-script behavior is completely unchanged.

Architecture

┌────────────┐   WS (出站)   ┌──────────────┐    REST/SSE    ┌──────────┐
│  Agent(s)  │ ───────────► │    Server    │ ◄────────────  │ CLI/Web  │
│ monitor/   │  heartbeat   │  scheduler   │                │ (同源)   │
│ runner/tb  │  metrics/logs│  SQLite(WAL) │ ◄──── HTTP ─── │ SDK      │
└────────────┘              └──────────────┘                └──────────┘

monorepo: packages/{common,server,agent,sdk,cli,mcp,tbreader} + web/ + examples/mnist + .claude/skills/ (agent skills) + .claude-plugin/. packages/mcp is a standalone adapter process, independent of server/agent.

Common Operations

uv run gpuctl backup             # 在线备份 SQLite 到 <data_dir>/backups/
uv run gpuctl backup-agent       # 在每台 Agent 主机归档完整 job 日志/runtime/spool
# 常驻运行见 deploy/systemd/（user unit + enable-linger）
uv run gpuctl job retry <id>     # 失败任务重新排队
uv run gpuctl job cancel <id>    # SIGTERM → 5s → SIGKILL（整进程组）
uv run pytest tests/ -q          # Python 测试（当前 145 个用例）
cd web && pnpm test:e2e          # 浏览器 smoke（2 个用例）
uv run ruff check . && uv run mypy

Docker runner is an optional capability; Process runner remains the default. Before using GPU containers, the Agent host must first install the Container Toolkit per the NVIDIA official install guide and register the runtime in the Docker configuration:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker       # 先确认没有运行中的容器
docker run --rm --gpus all <cuda-image> nvidia-smi

On WSL2, also confirm that host passthrough works first (nvidia-smi will show the GPU); the Toolkit only exposes the already-passthrough GPU to containers, and cannot replace the Windows/WSL driver.

Design Red Lines (read before contributing)

1. Training-code independence: python train.py must always remain usable outside the platform.

2. best-effort telemetry: any reporting failure only goes to a local buffer, never raising exceptions in the training process.

3. Process runner is a first-class citizen; Docker/Git are not forced.

4. Scheduling is by exclusive GPU slots, never guessing idleness from utilization.

5. Job ≠ Run: a job is the scheduling unit, a run is the semantic unit of training semantics (one Run per attempt).

6. SQLite + no message queue; a single worker on the server (in-memory pub/sub is deliberate).

License

GPUPlane is licensed under the Apache License 2.0. Attribution information is in NOTICE.