# Agent GPU Pool

**Give your AI agent a persistent compute queue—not another notebook to babysit.**

[![Python](https://img.shields.io/badge/python-3.11%2B-blue)](pyproject.toml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

[中文说明](README.zh-CN.md) · [Quick start](#try-it-without-credentials) · [Kaggle setup](docs/kaggle.md) · [Architecture](docs/architecture.md)

Agent GPU Pool is a local, persistent compute broker for **Codex, Claude Code, and MCP clients**. Your agent describes a job's requirements; the broker selects an eligible worker, tracks execution, and collects results. A fresh agent session can discover those results without repeating the experiment.

```text
Codex / Claude Code / MCP client
                │ submit → job ID
                ▼
    Persistent broker + hardware-aware scheduler
                │
       ┌────────┼─────────┐
       ▼        ▼         ▼
     Kaggle   Local    Fake demo
       └────────┼─────────┘
                ▼
       Artifacts + metrics + SHA-256 manifest
                │
                ▼
       Resume from another agent session
```

## Why this exists

A chat session should not be the only place that remembers an experiment. GPU access also varies: a worker with two small cards cannot serve a job requiring one large-memory card.

- **Recover across sessions.** SQLite stores jobs, remote IDs, reservations, and results. A detached broker continues after the MCP client disconnects.
- **Match real hardware requirements.** Filter by model, device count, and memory **per device**. Hardware options share their worker's quota and concurrency limit.
- **Preserve scarce capacity.** Prefer a sufficient worker while leaving more flexible hardware available for demanding jobs, when other ranking criteria are equal.
- **Collect evidence.** Artifacts have manifests and verified hashes. Execution success and scientific conclusions are separate records.
- **Avoid accidental duplicate runs.** Persist identity before dispatch; reconcile ambiguous launches instead of blindly submitting again.
- **Use one interface.** A CLI and 13 MCP tools share the same broker state. Credentials live in the OS keyring.

## Try it without credentials

Requires Python **3.11+** (macOS, Linux, or **Windows 10/11**) and [uv](https://docs.astral.sh/uv/getting-started/installation/).

```sh
git clone https://github.com/AlfredQuan/agent-gpu-pool.git
cd agent-gpu-pool
uv sync --extra dev
source .venv/bin/activate

gpupool --home "$PWD/.demo-pool" demo
gpupool --home "$PWD/.demo-pool" jobs
```

This runs a **synthetic demo**, with no Kaggle account, GPU allocation, or paid cloud resource. It exercises the real scheduler, persistence, and artifact flow. A worker with 4 hours cannot take the simulated 8-hour job; the scheduler selects an eligible worker instead. Metrics marked `synthetic: true` are not training results. Use a fresh demo directory when repeating it.

## How other software calls this

Nothing is bound to a chat process. Any program can reach the same broker through the shared home directory (`~/.local/share/agent-gpu-pool`) using one of three interfaces. Clients that share a home directory share all jobs, results, and authorizations.

### 1. MCP server (13 tools over stdio)

Run the broker as a stdio subprocess that any MCP client can connect to:

```sh
gpupool mcp
# or, identical server:
agent-gpu-pool
```

In your client's MCP configuration, point the server command at the absolute path of `gpupool`/`agent-gpu-pool` inside your virtual environment. The server exposes these tools:

`pool_overview`, `list_workers`, `sync_quotas`, `submit_job`, `get_job`, `list_jobs`, `cancel_job`, `get_job_logs`, `list_ready_results`, `list_artifacts`, `fetch_artifact`, `record_experiment_result`, `get_project_runs`.

Admin actions (credential profiles, worker/project registration, project policies) are deliberately **not** MCP tools — run those with the CLI so an administrator stays in control.

### 2. CLI (for shells, scripts, and schedulers)

```sh
gpupool pool
gpupool workers
gpupool submit examples/job.yaml --idempotency-key my-exp-v1
gpupool jobs
gpupool job JOB_ID
gpupool logs JOB_ID --tail 50
gpupool ready --project my-project
gpupool fetch JOB_ID metrics.json --content
```

Every command prints JSON, so it composes directly with `jq`, `PowerShell`, or any scripting layer.

### 3. Python API (the same `BrokerService`)

```python
from pathlib import Path
from agent_gpu_pool.broker import BrokerService

broker = BrokerService(Path("~/.local/share/agent-gpu-pool").expanduser(), autostart=True)
print(broker.pool_overview())
```

## Connect your agent

With the virtual environment activated:

```sh
gpupool agent install codex
# Or:
gpupool agent install claude
```

Each command **prints** an installation command with your absolute executable and state paths. Run the printed command, then start a new agent session. Clients must use the same broker home to share jobs and results. Default: `~/.local/share/agent-gpu-pool`.

Try asking your agent:

> Inspect the GPU pool and check for finished results for my project before proposing another run. Use the broker to choose hardware that meets the job requirements.

MCP makes the tools available; the client must load the server, and the agent must choose to use them. Copy the relevant workflow from [AGENTS.md](AGENTS.md) or [CLAUDE.md](CLAUDE.md) into your project instructions.

## Use authorized Kaggle resources

Install the optional adapter dependency and authenticate locally:

```sh
uv sync --extra dev --extra kaggle
gpupool credential add kaggle --name kaggle-main --oauth
gpupool credential test kaggle-main
```

Then [register a worker and authorize a project](docs/kaggle.md). New projects permit only fake workers; real uploads and execution require an administrator-applied policy. Register only resources and sources you are permitted to use. Never paste tokens into chats or commit credentials.

[Hardware configuration](docs/account-hardware.md) explains multiple options per worker. Example templates are unverified by default: they do not grant or establish provider access.

## Inspect and recover work

```sh
gpupool pool
gpupool jobs
gpupool job JOB_ID
gpupool logs JOB_ID --tail 50
gpupool artifacts JOB_ID
gpupool fetch JOB_ID metrics.json --content
```

`SUCCEEDED` means the program exited successfully. `artifacts_ready` separately indicates that required results were collected. Missing metrics trigger collection retries, not another training run.

## Validation and limits

**Early release, not a claim of production readiness.** The development suite contains 50 tests covering scheduling, quota reservations, restarts, real stdio MCP connections, client disconnects, local execution, artifacts, credential isolation, and mocked Kaggle CLI behavior.

| Backend | Status |
| --- | --- |
| Fake | End-to-end synthetic scheduling and artifact flow tested |
| Local | Trusted Python execution; timeout and cancellation tested |
| Kaggle | CLI adapter implemented; authentication/quota reads checked during development; broker-driven GPU launch and collection remain unverified on real hardware |
| SSH / RunPod / Vast | Disabled extension placeholders |

- Windows 10/11 + Python 3.11+ is supported. `flock` is replaced by `msvcrt` byte-range locks (see `src/agent_gpu_pool/_platform.py`); `start_new_session`/`os.killpg` map to `CREATE_NEW_PROCESS_GROUP` + `taskkill /T`; local cancellation uses a `cancel.requested` marker file. Known limit: when a client connects through the MCP `stdio_client`, a disconnect can terminate the broker daemon via the Windows Job Object (`KILL_ON_JOB_CLOSE`) — the CLI (`gpupool`) is unaffected, and the related tests skip on Windows.
- This is a trusted single-user service, **not a code sandbox** or multi-tenant platform.
- Kaggle cancellation cannot safely stop a kernel through the checked CLI interface; the broker records the request and directs the user to the provider UI.
- Hardware access is administrator-attested. Provider availability, competition permissions, and runtime compatibility may change.
- Broker-owned reservations do not constitute a complete inventory of externally launched jobs.
- Source snapshots use an explicit allowlist capped at 20 MiB. Large datasets should use provider sources.
- Hash checking and read-only files provide application-level integrity, not tamper-proof storage.
- No leaderboard submission, paid provisioning, or automatic training-script parallelization.

```sh
uv run pytest -q
uv run ruff check src tests
```

## Contribute

Useful next steps include real Kaggle integration evidence, clearer first-run onboarding, and provider adapters with reliable reconciliation. See [CONTRIBUTING.md](CONTRIBUTING.md) and the [security model](docs/security.md).

If this solves a workflow you recognize, a star helps others find it. Bug reports and reproducible examples are especially welcome.

MIT licensed. Independent project; not affiliated with OpenAI, Anthropic, Kaggle, or NVIDIA.