# Agent GPU Pool

**Give your AI agent a persistent compute queue—not another notebook to babysit.**

[![CI](https://github.com/HarrisonYangKuang/agent-gpu-pool/actions/workflows/ci.yml/badge.svg)](https://github.com/HarrisonYangKuang/agent-gpu-pool/actions/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.11%2B-blue)](pyproject.toml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

[中文说明](README.zh-CN.md) · [Quick start](#try-it-without-credentials) · [Kaggle setup](docs/kaggle.md) · [Reliability](docs/reliability.md) · [Architecture](docs/architecture.md)

Agent GPU Pool is a local, persistent compute broker for **Codex, Claude Code, and MCP clients**. Your agent describes a job's requirements; the broker selects an eligible worker, tracks execution, and collects results. A fresh agent session can discover those results without repeating the experiment.

```text
Codex / Claude Code / MCP client
                │ submit → job ID
                ▼
    Persistent broker + hardware-aware scheduler
                │
       ┌────────┼─────────┐
       ▼        ▼         ▼
     Kaggle   Local    Fake demo
       └────────┼─────────┘
                ▼
       Artifacts + metrics + SHA-256 manifest
                │
                ▼
       Resume from another agent session
```

## Why this exists

A chat session should not be the only place that remembers an experiment. GPU access also varies: a worker with two small cards cannot serve a job requiring one large-memory card.

- **Recover across sessions.** SQLite stores jobs, remote IDs, reservations, collection retries, and results. A detached broker continues after the MCP client disconnects.
- **Match real hardware requirements.** Filter by model, device count, and memory **per device**. Hardware options share their worker's quota and concurrency limit.
- **Preserve scarce capacity.** Prefer a sufficient worker while leaving more flexible hardware available for demanding jobs, when other ranking criteria are equal.
- **Collect evidence.** Artifacts have manifests and verified hashes. Execution success and scientific conclusions are separate records.
- **Avoid accidental duplicate runs.** Persist identity before dispatch; reconcile ambiguous launches instead of blindly submitting again.
- **Expose stalled and collecting work.** An opt-in progress timeout flags silent runs, while artifact retrieval retries with persistent exponential backoff.
- **Use one interface.** A CLI and 14 MCP tools share the same broker state. Credentials live in the OS keyring.

## Try it without credentials

Requires **macOS or Linux**, Python **3.11+**, and [uv](https://docs.astral.sh/uv/getting-started/installation/).

```sh
git clone https://github.com/HarrisonYangKuang/agent-gpu-pool.git
cd agent-gpu-pool
uv sync --extra dev
source .venv/bin/activate

gpupool --home "$PWD/.demo-pool" demo
gpupool --home "$PWD/.demo-pool" jobs
```

This runs a **synthetic demo**, with no Kaggle account, GPU allocation, or paid cloud resource. It exercises the real scheduler, persistence, and artifact flow. A worker with 4 hours cannot take the simulated 8-hour job; the scheduler selects an eligible worker instead. Metrics marked `synthetic: true` are not training results. Use a fresh demo directory when repeating it.

## Connect your agent

With the virtual environment activated:

```sh
gpupool agent install codex
# Or:
gpupool agent install claude
```

Each command **prints** an installation command with your absolute executable and state paths. Run the printed command, then start a new agent session. Clients must use the same broker home to share jobs and results. Default: `~/.local/share/agent-gpu-pool`.

Try asking your agent:

> Inspect the GPU pool and check for finished results for my project before proposing another run. Use the broker to choose hardware that meets the job requirements.

MCP makes the tools available; the client must load the server, and the agent must choose to use them. Copy the relevant workflow from [AGENTS.md](AGENTS.md) or [CLAUDE.md](CLAUDE.md) into your project instructions.

## Use authorized Kaggle resources

Install the optional adapter dependency and authenticate locally:

```sh
uv sync --extra dev --extra kaggle
gpupool credential add kaggle --name kaggle-main --oauth
gpupool credential test kaggle-main
```

Then [register a worker and authorize a project](docs/kaggle.md). New projects permit only fake workers; real uploads and execution require an administrator-applied policy. Register only resources and sources you are permitted to use. Never paste tokens into chats or commit credentials.

[Hardware configuration](docs/account-hardware.md) explains multiple options per worker. Example templates are unverified by default: they do not grant or establish provider access.

## Inspect and recover work

```sh
gpupool pool
gpupool jobs
gpupool job JOB_ID
gpupool logs JOB_ID --tail 50
gpupool artifacts JOB_ID
gpupool fetch JOB_ID metrics.json --content
```

`SUCCEEDED` means the program exited successfully. The user-facing `phase` separately distinguishes `RUNNING`, `STALLED`, `COLLECTING`, `COMPLETED`, and failure states. Missing metrics or temporary provider API lag trigger persistent collection retries, not another training run. See [reliability and recovery](docs/reliability.md).

## Validation and limits

**Early release, not a claim of production readiness.** The development suite contains 52 tests covering scheduling, quota reservations, restarts, real stdio MCP connections, client disconnects, local execution, artifact recovery, credential isolation, and mocked Kaggle CLI behavior.

| Backend | Status |
| --- | --- |
| Fake | End-to-end synthetic scheduling and artifact flow tested |
| Local | Trusted Python execution; timeout and cancellation tested |
| Kaggle | CLI adapter implemented; authentication/quota reads checked during development; broker-driven GPU launch and collection remain unverified on real hardware |
| SSH / RunPod / Vast | Disabled extension placeholders |

- POSIX process locks are required; Windows execution is not supported.
- This is a trusted single-user service, **not a code sandbox** or multi-tenant platform.
- Kaggle cancellation cannot safely stop a kernel through the checked CLI interface; the broker records the request and directs the user to the provider UI.
- Hardware access is administrator-attested. Provider availability, competition permissions, and runtime compatibility may change.
- Broker-owned reservations do not constitute a complete inventory of externally launched jobs.
- Source snapshots use an explicit allowlist capped at 20 MiB. Large datasets should use provider sources.
- Hash checking and read-only files provide application-level integrity, not tamper-proof storage.
- No leaderboard submission, paid provisioning, or automatic training-script parallelization.

```sh
uv run pytest -q
uv run ruff check src tests
```

## Contribute

Useful next steps include real Kaggle integration evidence, clearer first-run onboarding, and provider adapters with reliable reconciliation. See [CONTRIBUTING.md](CONTRIBUTING.md) and the [security model](docs/security.md).

If this solves a workflow you recognize, a star helps others find it. Bug reports and reproducible examples are especially welcome.

MIT licensed. Independent project; not affiliated with OpenAI, Anthropic, Kaggle, or NVIDIA.