autodl-research-pilot
README.md
<p align="center">
<img src="assets/logo.svg" width="680" alt="AutoDL Research Pilot">
</p>
<p align="center">
<strong>Task-aware AutoDL orchestration for Codex research workflows.</strong><br>
Size the run, choose a region and GPU, inspect the worker, then keep a human at the billing boundary.
</p>
<p align="center">
<a href="https://github.com/chengxi271-commits/autodl-research-pilot/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/chengxi271-commits/autodl-research-pilot/actions/workflows/ci.yml/badge.svg"></a>
<img alt="Python 3.11+" src="https://img.shields.io/badge/Python-3.11%2B-3776AB">
<a href="LICENSE"><img alt="Apache-2.0" src="https://img.shields.io/badge/License-Apache--2.0-green.svg"></a>
</p>
<p align="center">
<strong>English</strong> · <a href="README.zh-CN.md">简体中文</a>
</p>
# AutoDL Research Pilot
AutoDL Research Pilot gives Codex enough context to make a useful rental decision: the experiment contract, peak disk demand, prepared data regions, transfer conditions, current stock, expected throughput, budget, and deadline. It produces a reviewable plan, creates the resource only after confirmation, and can inspect the allocated worker over SSH before the expensive run begins.
Version 0.3 adds the pieces that turn allocation into an operating workflow:
- resilient Developer API reads with structured failure diagnosis;
- OS-keyring credentials and an explicit console compatibility mode for Pro;
- one typed view of Pro and Elastic resources;
- workload admission for disk, persistence, region, runtime, and network;
- live filesystem, CPU, memory, GPU, and bounded network probes;
- preview → confirm → receipt for paid operations.
The project uses the [official AutoDL Developer API](https://www.autodl.com/docs/common_api/) as its default control plane. Browser-session compatibility is separate, visible, and opt-in.
## Quick start
### Install
```bash
git clone https://github.com/chengxi271-commits/autodl-research-pilot.git
cd autodl-research-pilot
python -m venv .venv
```
Windows PowerShell:
```powershell
.\.venv\Scripts\Activate.ps1
python -m pip install -e ".[full]"
```
Linux / macOS:
```bash
source .venv/bin/activate
python -m pip install -e '.[full]'
```
`full` installs the MCP, OS-keyring, and SSH extras. Install `.[auth]`, `.[remote]`, or `.[mcp]` when only one integration is needed.
### Save the Developer Token
The recommended login reads the token without echoing it and stores it under the fixed `autodl-research-pilot` service in the operating-system keyring:
```bash
autodl-pilot auth login
autodl-pilot auth status
autodl-pilot doctor --live
```
Remove the saved Developer Token with `autodl-pilot auth logout`.
For an ephemeral shell, use `AUTODL_TOKEN` instead. The environment takes precedence over the keyring and is resolved again for every request.
```powershell
$env:AUTODL_TOKEN = "YOUR_TOKEN"
```
```bash
export AUTODL_TOKEN='YOUR_TOKEN'
```
The token stays out of project TOML, plans, receipts, logs, and request previews.
### Describe the run before renting
```bash
cp examples/elastic.project.toml my-run.project.toml
cp examples/elastic.catalog.toml my-run.catalog.toml
autodl-pilot workload-inspect --project my-run.project.toml
```
The project file records four small contracts:
- `runtime`: `mode`, an `audition` or `full` phase, and a launch policy. Use
`product_default` when comparing products: it resolves to
`manual_after_preflight` on Pro and `embedded_preflight` on Elastic;
- `capacity`: system runtime, hot workspace, cache, persistent output and
headroom as separate byte demands, plus workspace/persistent inode counts,
CPU, total and working-set RAM, GPU count, and VRAM;
- `storage`: the concrete workspace, cache and persistent-output paths, which
determine which filesystem pays each capacity demand, plus prepared regions;
- `network`: total download volume, every relevant source, the primary source
used for transfer ETA, cache state, and measured throughput per region.
`workload-inspect` reports the compatible product set. It recommends a backend
only when exactly one product fits. When both Pro and Elastic fit,
`recommended_backend` remains null and `requires_cost_time_comparison=true`:
build both candidate plans and run `portfolio-plan` instead of defaulting to
Elastic. Missing measurements remain visible; they do not turn into optimistic
defaults.
### Build and compare candidate plans
```bash
autodl-pilot discover --catalog my-run.catalog.toml --output live-context.json
autodl-pilot live-plan \
--project my-run.project.toml \
--catalog my-run.catalog.toml \
--profile balanced \
--output elastic.plan.json
# Local request preview: no mutation
autodl-pilot apply-live-plan --plan elastic.plan.json
```
The live planner treats each GPU-profile/region pair as a separate candidate. The selected request is bound to one observed region and the catalog's `price_to` ceiling. Confirmation refreshes wallet, stock, deployment-name, image, workload, region-storage, budget, and deadline conditions before calling the create endpoint.
With `launch_policy = "product_default"`, the Elastic path requires
`preflight_cmd` in the catalog and sends one foreground command equivalent to
`preflight_cmd && cmd`. The embedded preflight must check mounts, actual free
space/inodes, runtime and fixed inputs before a large download or model load. A
failed preflight prevents the scientific command from starting and lets the
finite container stop.
If workload inspection reports both products compatible, build a Pro/local
plan from current quotes under the same project contract and compare the two:
```bash
autodl-pilot plan \
--project my-run.project.toml \
--offers current-offers.toml \
--profile balanced \
--output pro.plan.json
autodl-pilot portfolio-plan \
--plan pro.plan.json \
--plan elastic.plan.json \
--profile balanced \
--output portfolio.json
```
Follow `portfolio.json.next_action`. A candidate that was not the selected,
execution-ready result in its source plan must be replanned in that source
planner before preview. Once the chosen source plan is current, confirm only
that plan; for an Elastic winner:
```bash
autodl-pilot apply-live-plan --plan elastic.plan.json
autodl-pilot apply-live-plan \
--plan elastic.plan.json \
--receipt runs/my-run.receipt.json \
--confirm
```
Candidates that miss VRAM, budget, deadline, inventory, or research-admission requirements are removed before scoring. For the rest, total time includes queue, setup, transfer, and workload-adjusted compute time; total cost uses the conservative billed time and approved hourly ceiling. For AutoDL Pro offers, the configured transfer time is floored at the slowest measured primary-source ETA among the allowed data centers, so a cheap offer cannot hide a region's cold-start download. `economy` chooses the lowest conservative cost, `time` chooses the fastest feasible run, `balanced` uses 45% cost and 55% time regret on the Pareto frontier, and `custom` accepts project weights. Performance factors should come from a short run of the same model, precision, batch semantics, and data path.
### Inspect the allocated runtime
```bash
autodl-pilot resources
autodl-pilot resource-inspect elastic DEPLOYMENT_UUID \
--container-uuid CONTAINER_UUID \
--runtime
autodl-pilot runtime-preflight elastic DEPLOYMENT_UUID \
--container-uuid CONTAINER_UUID \
--project my-run.project.toml \
--plan elastic.plan.json \
--network
autodl-pilot resource-watch elastic DEPLOYMENT_UUID \
--container-uuid CONTAINER_UUID \
--samples 6 \
--interval 60 \
--runtime
```
`runtime-preflight` checks the actual filesystems containing the configured
workspace, cache and persistent output, groups paths that share a device, and
compares their combined byte/inode demand with real free capacity. Passing the
approved `--plan` also binds the observation to its GPU count/model and region:
Elastic must match its exact selected region, while Pro must be inside the
approved `data_center_list`. The SSH probe collects CPU/load, memory,
filesystem state, and GPU utilization through NVML, `nvidia-smi`, or AutoPanel
when available. Connection credentials live only long enough to open the
session; public output is sanitized.
For a full Pro run, `product_default` resolves to
`manual_after_preflight`. Leave `start_command` unset, create or power on the
instance, pass its approved plan to `runtime-preflight`, and start the workload
over SSH only after that gate passes. AutoDL documents that failure of a Pro
`start_command` does not fail instance startup, so provider state `running`
cannot stand in for this gate.
## When the API “suddenly stops working”
A failed call does not identify its cause by itself. v0.3 keeps the useful distinctions in every `AutoDLError`:
| Category | Typical signal | Client behavior |
|---|---|---|
| `configuration` | No usable credential source | Stop before network access and point to `auth login` or `AUTODL_TOKEN` |
| `auth` | `AuthorizeFailed`, HTTP 401, invalid token | No retry; request a human-supplied Developer Token |
| `permission` | HTTP 403 or product/subaccount access error | Keep other capabilities usable and report the affected product permission |
| `transient` | Transport failure, HTTP 408/429/5xx | Retry read-only calls up to three attempts with backoff |
| `request-shape` / `contract` | Rejected request form, changed response, endpoint drift | Preserve `code`, `msg`, HTTP status, and `request_id` for diagnosis |
Mutations never retry automatically. A timed-out create, power, stop, or delete call may already have reached AutoDL; the next step is state reconciliation, not a duplicate request.
The [Pro API documentation](https://www.autodl.com/docs/instance_pro_api/) specifies `snapshot` and `status` as GET requests carrying a JSON body. Some HTTP paths reject that shape. The default `auto` mode first sends the documented form, then tries a query-string compatibility form only after an explicit 400, 405, 422, or request-shape error. Authentication and permission failures never trigger that switch. Set `AUTODL_GET_PAYLOAD_MODE=json` to require the official form or `query` to select compatibility mode directly.
AutoDL's public API documentation describes where to obtain a Developer Token; it does not publish a refresh endpoint or token-lifetime contract. v0.3 reloads the environment/keyring value on every request and leaves replacement to `autodl-pilot auth login`. The client does not scrape the console for a new token.
`autodl-pilot auth status --live` and `doctor --live` probe capabilities separately. A working wallet call no longer implies that Pro instances, Elastic deployments, images, and stock all share the same permission. AutoDL documents separate eligibility for [Pro](https://www.autodl.com/docs/instance_pro_api/), [Elastic](https://www.autodl.com/docs/esd_api_doc/), and [subaccounts](https://www.autodl.com/docs/c_user/).
`auth status` also reports the credential source that is actually active. When
`AUTODL_TOKEN` and keyring both exist, the environment wins and
`keyring_shadowed_by_environment=true`; after updating keyring, unset the stale
environment variable before expecting the new token to take effect.
`auth status` may also display the JWT-shaped `aud`, `tenant`, and `exp` fields
as unverified routing hints. Account identifiers and the credential itself are
omitted, and these hints never replace a live capability probe.
## Explicit console compatibility for Pro
If the documented Pro endpoint is unavailable while the logged-in website still manages the same instance, v0.3 can use a human-supplied browser-session credential:
```bash
autodl-pilot auth console-login --app-version APP_VERSION
# Developer API first; console fallback for eligible Pro read failures
autodl-pilot --pro-control-plane auto resources
autodl-pilot --pro-control-plane auto resource-inspect pro PRO_UUID --runtime
# Select the compatibility client directly
autodl-pilot --pro-control-plane console api instances
# Recovery for an existing stopped Pro root filesystem; preview the price first
autodl-pilot --pro-control-plane auto pro-disk-expand PRO_UUID \
--required-free-gib REQUIRED_FREE_GIB
autodl-pilot --pro-control-plane auto pro-disk-expand PRO_UUID \
--required-free-gib REQUIRED_FREE_GIB \
--current-free-gib CURRENT_FREE_GIB --confirm
autodl-pilot auth console-logout
```
`console-login` prompts for the current browser-session `Authorization` value and stores it with `AppVersion` in the OS keyring. Set `AUTODL_CONSOLE_SUB_USER=1` for the observed subaccount route.
On a stopped Pro instance, `--required-free-gib` is the desired free space on the root filesystem, not the
create-time expansion increment. The command reads current root free space from
the snapshot and expands by `ceil(required - current)`; pass
`--current-free-gib` only when `runtime-preflight` measured it but the snapshot
did not expose usable root usage. It does not resize `/root/autodl-tmp` or an
external mount.
These frontend routes are **not part of AutoDL's published Developer API**. They may change with the website, have no documented stability promise, and remain disabled unless `--pro-control-plane console` or `auto` is supplied. For Pro reads, `auto` falls back after auth, permission, contract, or request-shape failures; a transient Developer API outage remains visible. Confirmed power operations are never resent through another control plane. Inspect state, then explicitly select `console` and preview again if needed. Elastic operations and Pro creation continue through the documented Developer API.
## Three resource surfaces
AutoDL exposes three operational surfaces with different identifiers and lifecycle rules:
| Surface | How this project sees it | Suitable use |
|---|---|---|
| **Container Instance Pro** | `/api/v1/dev/instance/pro/*`; API list/status/snapshot and lifecycle | Restartable or interactive work, Jupyter, system-disk sizing at creation |
| **Enterprise Elastic** | `/api/v1/dev/deployment/*`; deployments, containers, events, and filtered stock | Batch jobs and elastic services with explicit command lifecycle |
| **Ordinary marketplace instance** | Manual console surface; no public Developer API enumeration is documented | Manual host selection and the marketplace's expandable data-disk workflow |
The official [Elastic overview](https://www.autodl.com/docs/elastic_deploy/) documents the differences from marketplace instances: Elastic containers have a fixed 50 GB local data disk, no Jupyter entry, no restart after stop, and no durable local state after the lifecycle ends. Pro has its own endpoint family and `pro-*` identifiers. An instance missing from the ordinary “Container Instances” page can still exist and accrue usage under another control plane.
`autodl-pilot resources` merges Pro instances and Elastic deployments/containers into typed JSON while preserving `kind`, `resource_id`, parent deployment, state, region, GPU, and control-plane provenance. It also reports `manual_market_instances.discovered=false`, keeping the undocumented inventory gap explicit.
## Storage and region admission
Disk demand is decided before the rental request:
- **Path mapping:** `system_required_gb` always belongs to the root filesystem.
`local_hot_required_gb`, `cache_required_gb`, and
`persistent_required_gb` are charged to the filesystems containing
`storage.workspace`, `storage.cache`, and `storage.persistent_output`.
`/root/autodl-tmp` is the local data role; `/root/autodl-fs` and
`/root/autodl-nas` are external durable roles; other writable paths are on the
system role. The runtime gate groups paths that resolve to the same device,
sums their requirements, and adds headroom once to that device.
- **Pro:** create-time system expansion is
`ceil(system-role demand + headroom - 30 GB)`, clamped at zero. Moving the
cache or workspace between the system disk and `/root/autodl-tmp` therefore
changes the result. The [Pro create API](https://www.autodl.com/docs/instance_pro_api/)
accepts 0–500 GB of system-disk expansion. A data-role demand above the
default 50 GB is not rescued by system expansion because the Developer API
has no data-disk size field.
- **Elastic:** all workspace and cache demand mapped to `/root/autodl-tmp`, plus
headroom, must fit its fixed 50 GB local disk. Persistent output must map to a
prepared durable mount in the selected region. Jobs that fail either rule are
excluded before allocation.
- **File storage:** `/root/autodl-fs` is region-local shared storage. It must be initialized in each candidate region; it is durable and convenient, with lower I/O performance than local disk and a documented 200,000-inode limit. See [file storage](https://www.autodl.com/docs/fs/) and the [instance filesystem layout](https://www.autodl.com/docs/env/).
The Elastic catalog may list several `dc_list` regions. Stock is queried per region, each `(profile, region)` result stays separate, and a plan with persistent output cannot execute until its chosen region appears in `prepared_regions`.
For a Pro plan, start from a fresh account quote because the public Pro API has no pre-allocation stock or market-price endpoint:
```bash
autodl-pilot plan \
--project examples/project.toml \
--offers current-offers.toml \
--profile balanced \
--output pro.plan.json
autodl-pilot apply-plan --plan pro.plan.json
autodl-pilot apply-plan \
--plan pro.plan.json \
--receipt runs/pro-run.receipt.json \
--confirm
```
The workload admission is embedded in the plan, including the derived system-disk expansion. A successful confirmed create writes the provider UUID and approved selection to a non-secret receipt, giving a later Codex session a stable handoff record.
## Network-aware staging
AutoDL uses shared regional bandwidth, documented at roughly 3–10 Gbps for a region, so the path seen by one worker varies with load ([network](https://www.autodl.com/docs/network/)). `workload-inspect` therefore asks for download volume, all source classes, `primary_download_source`, cache state, and any available `observed_mbps_by_region`. The `observed_mbps` shorthand is valid only when one region is prepared. Runtime preflight requires every declared source to have a built-in or caller-supplied probe and tests them separately; only the primary source's regional observation is converted into the download ETA. A full live plan with uncached downloads and no primary-source observation for its selected region asks for a short network audition.
After a comparable slice finishes, convert its measured compute and transfer
time into regional scheduler inputs, inspect them, then rerun the plan:
```bash
autodl-pilot audition-calibrate \
--project my-run.project.toml \
--profile-id fast-24gb \
--region westDC2 \
--work-fraction 0.1 \
--compute-seconds 900 \
--download-bytes 1073741824 \
--download-seconds 120
```
The download sample in this command must come from the project's
`primary_download_source`; the resulting regional Mbps/ETA is labelled with
that source rather than generalized to every origin.
Add `--network` to `runtime-preflight` to probe the declared entries in
`sources`. For a standalone `resource-inspect`, pass one credential-free
`--network-url`. Each route uses `Range: bytes=0-1048575`, accepts at most 1 MiB,
and uses a 10-second CLI timeout; the underlying bounded probe accepts at most
30 seconds, with at most eight sources in one preflight. Hugging Face and GitHub may test both direct
and command-scoped `/etc/network_turbo`, so their maximum is 2 MiB per source;
other origins use one route. Every declared source is required. A source without
a built-in mapping blocks the network preflight until its real,
credential-free probe object is supplied; repeat the option for multiple custom
sources:
```bash
autodl-pilot runtime-preflight elastic DEPLOYMENT_UUID \
--container-uuid CONTAINER_UUID \
--project my-run.project.toml --plan elastic.plan.json --network \
--network-source-url private_oss=https://HOST/PATH/probe.bin
```
Omit `--network` only for a general capacity/telemetry inspection, not as proof
that a full download is ready. Proxy variables do not leak into later package
installs or training.
For a large HTTP object, the bundled downloader verifies byte-range support, keeps completed chunks across interruptions, and publishes the assembled file with an atomic rename:
```bash
autodl-resumable-http \
"https://HOST/PATH/model.bin" \
"/root/autodl-tmp/models/model.bin" \
--workers 4
```
If the probe selected the academic route for GitHub or Hugging Face, scope it to this command:
```bash
bash -lc 'source /etc/network_turbo && autodl-resumable-http "https://HOST/PATH/model.bin" "/root/autodl-tmp/models/model.bin" --workers 4'
```
The [academic acceleration guide](https://www.autodl.com/docs/network_turbo/) limits that service to supported academic domains and gives no stability guarantee. Package installs should use an appropriate domestic [pip/conda source](https://www.autodl.com/docs/source/) without inherited proxy variables. When `/root/autodl-tmp` exists and has sufficient capacity, set `HF_HOME=/root/autodl-tmp/cache` to keep Hugging Face caches off the system disk ([Hugging Face guide](https://www.autodl.com/docs/huggingface/)).
For a non-empty object, resume is admitted only when the server supplies a
strong ETag or Last-Modified validator, or the caller supplies an authoritative
`--sha256`. The `.parts/object-identity.json` binding records the sanitized URL,
length, validator and optional expected digest; range requests use `If-Range`,
and changed identity refuses old chunks. The query string and URL credentials
are never written to that record or JSON output. For bulk datasets, pre-stage
to same-region file storage or object storage and copy the hot subset to local
disk. No checksum sidecar is created when an expected digest was not supplied.
## Observation and lifecycle
Pro snapshots expose CPU, memory, root/data-disk usage, connection details, and image progress. Elastic list/event APIs expose allocation and lifecycle state. Dynamic GPU metrics require the data plane: SSH, NVML/`nvidia-smi`, or AutoDL's in-container endpoint at `127.0.0.1:2022` for eligible enterprise accounts ([performance monitoring](https://www.autodl.com/docs/metric_monitor/)).
`resource-watch` is bounded to 1–60 samples. `--hourly-price-cny` means the
price of **one GPU-hour**; pass the approved `--gpu-count`, and the estimate is
`elapsed_hours × hourly_price_cny × gpu_count`. Optional billing-start, budget,
and deadline arguments produce review alerts; the command reports
`automatic_stop=false`.
```bash
autodl-pilot resource-watch pro PRO_UUID \
--samples 3 --interval 60 --runtime \
--hourly-price-cny PRICE_PER_GPU --gpu-count GPU_COUNT \
--billing-started-at ISO_TIME --budget-cny BUDGET
```
Stopping still uses the preview/confirm flow:
```bash
autodl-pilot container-stop CONTAINER_UUID
autodl-pilot container-stop CONTAINER_UUID --no-cache --confirm
autodl-pilot deployment-stop DEPLOYMENT_UUID
autodl-pilot deployment-stop DEPLOYMENT_UUID --confirm
autodl-pilot power-off PRO_UUID
autodl-pilot power-off PRO_UUID --confirm
```
Elastic `cmd` is the container lifecycle: the foreground command exits, then the container stops. Persist checkpoints during the run; the documented shutdown hook has a five-second limit. A reuse cache may contain previous files, so reusable deployments should initialize a unique run directory before training. See the [Elastic operating guide](https://www.autodl.com/docs/elastic_deploy_practice/).
## CLI map
| Command | Purpose |
|---|---|
| `auth`, `doctor` | Manage secret sources and diagnose per-product API capability |
| `workload-inspect` | Validate runtime, disk, persistence, region, and network demand |
| `audition-calibrate` | Convert a comparable short run into regional performance inputs |
| `portfolio-plan` | Compare Pro/local and Elastic candidates when both products fit |
| `discover`, `live-plan`, `apply-live-plan` | Read Elastic stock, select a plan, preview or create |
| `plan`, `apply-plan` | Compare offline/local/Pro candidates, preview or create Pro |
| `resources` | List Pro and Elastic under one typed inventory |
| `resource-inspect`, `runtime-preflight` | Read control-plane state and optionally inspect the worker over SSH |
| `resource-watch` | Collect a finite series of observations and budget/deadline alerts |
| `api ...` | Direct read access to documented wallet, Pro, and Elastic endpoints |
| `power-on`, `power-off`, `image-save`, `release` | Preview or confirm Pro lifecycle actions |
| `pro-disk-expand` | Explicit console-compatible price preview and Pro system-disk expansion |
| `container-stop`, `deployment-stop`, `deployment-delete` | Preview or confirm Elastic lifecycle actions |
All ordinary results and controlled failures are JSON. Run `autodl-pilot --help` for the complete argument reference.
## Codex and MCP
Load the repository as a local Codex plugin after installing the `mcp` extra. [`.mcp.json`](.mcp.json) targets Windows; [`docs/mcp.unix.json`](docs/mcp.unix.json) is the Unix manifest. The Codex workflow lives in [`skills/autodl-research-pilot/SKILL.md`](skills/autodl-research-pilot/SKILL.md).
MCP `preflight_run` defaults to `network=false`, so an ordinary capacity probe
does not contact download origins. For a full network gate, pass
`network=true`, the approved plan as `approved_plan`, and any custom mappings as
`network_source_urls={"SOURCE":"https://HOST/PATH/probe.bin"}`. Every source
declared by the workload must then have a probe result.
A useful first prompt is:
> Inspect this project's workload contract, compare the prepared AutoDL regions, and show me the selected request and runtime preflight. Ask before any paid or destructive action.
## Current boundaries
- The live scheduler currently creates one-GPU Elastic `Container` deployments. Job, ReplicaSet, and multi-GPU policy need separate workload evidence.
- The public Pro API does not expose a live pre-allocation catalog; Pro planning uses supplied, current offers.
- Ordinary marketplace instances are not enumerated because no public Developer API for them is documented.
- Console compatibility covers implemented Pro reads and lifecycle operations only. It is an observed frontend contract, requires current human-supplied credentials, and may change without notice.
- Runtime inspection executes a fixed probe set. It is not a general remote shell or training runner.
- `resource-watch` raises review alerts and never stops a resource automatically.
- Provider price fields and purchased duration packages are not treated as a complete billing ledger. Confirm the platform bill for long runs.
Endpoint and unit notes are collected in [AutoDL API Boundaries](docs/AUTODL_API_BOUNDARIES.md). This project is community-maintained and is not affiliated with AutoDL.
## Development
```bash
python -m unittest discover -s tests -v
```
See [CONTRIBUTING.md](CONTRIBUTING.md), [SECURITY.md](SECURITY.md), and the [issue tracker](https://github.com/chengxi271-commits/autodl-research-pilot/issues).
## License
[Apache-2.0](LICENSE)
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo issues