[READ] Attribute a $/1M-token unit cost from live vLLM throughput.

Multiplies the current generation-throughput gauge by the supplied GPU hourly cost to derive the cost of serving 1M tokens; degrades to an insufficient-data forecast when no throughput metric is present.

Args: gpu_hourly_cost: Hourly cost of a single GPU (e.g. cloud on-demand rate). num_gpus: Number of GPUs backing the deployment; defaults to 1. target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Deploy a Serve application from an import path.

The app is merged into the cluster's declarative Serve config (existing apps preserved). Replica counts come from the app's own config; set them afterward with serve scale (deployment names are known only once the app materialises).

Args: application: Serve application name to create/replace. import_path: Python import path of the Serve app (e.g. 'module:app'). target: Inference target name from config; omit for the default.

[WRITE][risk=high] Tear down a whole Serve application (removes all deployments).

Irreversible without the original import path — pass dry_run=True to preview.

Args: application: Serve application name (from serve_deployment_list). dry_run: If True, preview without undeploying. target: Inference target name from config; omit for the default.

[WRITE][risk=high] Force a deployment to re-apply new config.

Applies the new config immediately and can drop unfinished requests — pass dry_run=True to preview.

Args: application: Serve application name. deployment: Deployment name within the application. dry_run: If True, preview without redeploying. target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Switch a deployment's request-routing policy.

NOTE: Ray Serve's request-routing policy is a code-level option on the @serve.deployment decorator, NOT a field settable over the Dashboard REST API — there is no REST endpoint for it. This tool therefore refuses with a teaching error rather than pretend; change the policy in the deployment's source and redeploy.

Args: application: Serve application name. deployment: Deployment name. policy: Desired routing policy (prefix_aware / round_robin / session_affinity). target: Inference target name from config; omit for the default.

[READ] Liveness of the serving engine (vLLM / SGLang / TGI) via its health probe.

Args: target: Inference target name from config; omit for the default.

[READ] Running-model identity + engine server info (engine-agnostic).

vLLM / SGLang report served ids from /v1/models; TGI's single model id comes from /info.

Args: target: Inference target name from config; omit for the default.

[READ] TTFT / TPOT / e2e latency + generation-token totals (where the engine exposes them).

Args: target: Inference target name from config; omit for the default.

[READ] Running vs waiting requests — the leading backpressure signal (any engine).

Args: target: Inference target name from config; omit for the default.

[READ][RCA] Rank the probable cause of a latency spike for any serving engine.

Correlates whichever signals the engine exposes (queue backpressure, KV/token-cache pressure, cache locality) into a ranked cause + the knob to turn — the engine-agnostic counterpart to diagnose_latency_spike (vLLM).

Args: target: Inference target name from config; omit for the default.

[READ] vLLM TTFT / TPOT / e2e latency + generation-token totals.

Args: target: Inference target name from config; omit for the default.

[READ] Running vs waiting requests — the leading backpressure signal.

Args: target: Inference target name from config; omit for the default.

[READ] KV-cache utilisation, prefix-cache hit rate, and preemption count.

Args: target: Inference target name from config; omit for the default.

[READ][RCA] Rank the probable cause of a TTFT/latency spike + the knob to turn.

Correlates queue depth, KV-cache pressure/preemption, and prefix-cache locality into a ranked cause list — call this first on "why is inference slow".

Args: target: Inference target name from config; omit for the default.

[READ][RCA] Explain an under-used GPU (batching / idle / overprovision).

Args: target: Inference target name from config; omit for the default.

[READ] All served vLLM models, LoRA adapters flagged.

Args: target: Inference target name from config; omit for the default.

[READ] One model's config: max length, root/parent, permission (best-effort).

Args: model_id: Model id (from model_list). target: Inference target name from config; omit for the default.

[READ] Whether the vLLM engine is suspended in Sleep Mode.

Returns isSleeping: true (suspended, serving nothing), false (serving), or null when the engine did not report it — null means UNKNOWN, not awake.

Sleep Mode exists only on servers started with VLLM_SERVER_DEV_MODE=1; on any other server this reports that the route is absent rather than failing vaguely.

Args: target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Hot-load a LoRA adapter onto the running engine.

Args: lora_name: Adapter name to register. lora_path: Local path or HF repo id of the adapter weights. target: Inference target name from config; omit for the default.

[WRITE][risk=high] Hot-unload a LoRA adapter (traffic on it starts failing).

Pass dry_run=True to preview.

Args: lora_name: Adapter name to unload (from model_list). dry_run: If True, preview without unloading. target: Inference target name from config; omit for the default.

[WRITE][risk=high] Suspend the engine via Sleep Mode (it stops serving requests).

Frees GPU memory between bursts. level=1 offloads the weights to CPU RAM and wakes fast; level=2 discards them, so waking reloads from disk. The engine serves nothing until model_wake. Pass dry_run=True to preview.

Sleep Mode exists only on servers started with VLLM_SERVER_DEV_MODE=1; on any other server this reports that the route is absent rather than failing vaguely.

Args: level: 1 to offload weights to CPU RAM, 2 to discard them (default 1). dry_run: If True, preview without suspending. target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Resume serving after Sleep Mode (weights return to GPU).

The inverse of model_sleep. Pass dry_run=True to preview. Records no undo: vLLM never reports which sleep level the engine was at, so re-sleeping would have to guess between level 1 and level 2 — call model_sleep with the level you want instead.

Sleep Mode exists only on servers started with VLLM_SERVER_DEV_MODE=1; on any other server this reports that the route is absent rather than failing vaguely.

Args: dry_run: If True, preview without waking. target: Inference target name from config; omit for the default.

[READ] Cluster-wide CPU/GPU capacity + headroom + pending placement groups.

Args: target: Inference target name from config; omit for the default.

[READ] Serve controller health + app/deployment counts.

Args: target: Inference target name from config; omit for the default.

[READ] Submitted Ray jobs: id, status, entrypoint, start time.

Returns {"jobs": [...], "returned": N, "limit": L, "truncated": bool}. When truncated is true there are more jobs than were returned — re-run with a higher limit rather than treating the result as the whole history.

Args: target: Inference target name from config; omit for the default. limit: Maximum job rows to return. Default 100.

[READ] Per-node GPU count, utilisation %, and memory used/total.

Args: target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Stop a submitted/running Ray job.

Args: job_id: Job id (from ray_job_list). target: Inference target name from config; omit for the default.

[WRITE][risk=high] Restart one Serve replica — NOT available over Ray's REST API.

Ray Serve exposes no per-replica restart endpoint over REST; the controller respawns unhealthy replicas on its own. This tool refuses with a teaching error. To force a cycle, scale the deployment down then up, or redeploy. The dry_run preview reports the same unavailability rather than a false green.

Args: application: Serve application name. deployment: Deployment name. replica_id: Replica id (from replica_list). dry_run: If True, report availability without attempting a restart. target: Inference target name from config; omit for the default.

[READ] All Ray Serve deployments: status, replica count, target.

Args: target: Inference target name from config; omit for the default.

[READ] One deployment's status + current/target replica count.

Args: application: Serve application name (from serve_deployment_list). deployment: Deployment name within the application. target: Inference target name from config; omit for the default.

[READ] Replica states for one deployment (running/starting/draining).

Args: application: Serve application name. deployment: Deployment name. target: Inference target name from config; omit for the default.

[READ] Autoscale bounds (min/max replicas, target ongoing requests).

Args: application: Serve application name. deployment: Deployment name. target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Raise a deployment's replica count (reversible → prior).

Pass dry_run=True to preview: it reads the deployment's current count, so the preview reports the real from→to rather than only the requested target — and it fails here, not mid-write, if the deployment does not exist.

Args: application: Serve application name. deployment: Deployment name. num_replicas: New (higher) replica count. dry_run: If True, preview without scaling. target: Inference target name from config; omit for the default.

[WRITE][risk=high] Lower a deployment's replica count on prod (reversible → prior).

Fewer replicas can strand in-flight requests — pass dry_run=True to preview.

Args: application: Serve application name. deployment: Deployment name. num_replicas: New (lower) replica count. dry_run: If True, preview without scaling. target: Inference target name from config; omit for the default.

[WRITE][risk=high] Park a deployment at 0 replicas (reversible → prior count).

Stops the cost bleed but adds cold-start latency and can strand the ingress — pass dry_run=True to preview.

Args: application: Serve application name. deployment: Deployment name. dry_run: If True, preview without scaling to zero. target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Live-tune autoscale bounds without a redeploy (reversible).

Args: application: Serve application name. deployment: Deployment name. min_replicas: New floor (omit to leave unchanged). max_replicas: New ceiling (omit to leave unchanged). target_ongoing_requests: New per-replica concurrency target (omit to leave). target: Inference target name from config; omit for the default.

[WRITE][risk=high] Drain one replica — NOT available over Ray's REST API.

Ray Serve exposes no per-replica drain endpoint (only the whole-cluster declarative config); individual-replica draining is a Python-API capability only. This tool refuses with a teaching error. To retire surplus replicas, scale the deployment down — the controller drains them gracefully. The dry_run preview reports the same unavailability rather than a false green.

Args: application: Serve application name. deployment: Deployment name. replica_id: Replica id (from replica_list). dry_run: If True, report availability without attempting a drain. target: Inference target name from config; omit for the default.

[READ] List recorded, not-yet-applied undo tokens (most recent first).

Each entry names the original tool, the inverse tool that undo_apply would run, and a human note. Use the undoId with undo_apply.

Returns {"undos": [...], "returned": N, "limit": L, "truncated": bool}. truncated is measured (one extra row is fetched), not guessed from a length coincidence: when it is true there are MORE tokens than shown, so re-run with a higher limit rather than reporting the list as complete.

Each entry carries effectVerified. False means the original write lost its response, so the change it reverses is PROBABLE, not confirmed — check the live state before applying, and do not report the result as a restore of a state that may never have been reached.

Args: limit: Max rows to return (default 50, capped at 500). target: Unused (undo state is host-local); accepted for CLI uniformity.

[WRITE][risk=medium] Apply a recorded undo by dispatching its inverse tool.

The inverse runs through its own governed tool, so it is audited under its own name and risk level. Pass dry_run=True to preview the inverse call without executing it. A token can only be applied once.

Args: undo_id: The undoId from undo_list (or an _undo_id in a write result). dry_run: If True, preview the inverse tool + params without running it. target: Passed through to the inverse tool when it accepts a target.