Tunnel, certificate expiry and an SSH test. Call first in every session. (Starts the tunnel if it is down; with wireproxy, a stall before login restarts it, at most once a minute, also from other tools that run commands on the login node, not yet from the copying in push_files/fetch_files: recovery.) The first call of a server run also carries banner (Amadeus' ASCII art): show it to the human once.

Tokens (credits) used by the whole project since a date, per user, from sacct (CPU nodes included; B200 as node-hours, no published price) and what is left of the configured budget. The projects portal is authoritative. Without a configured account: only the user's own jobs.

Cancel a job. Jobs submitted through this server can be cancelled directly; any other job needs the human's approval in approval (AGENTS.md §9).

Check a job before submitting: runbook rules (account, partition/QoS, time, dev limits, CPU limits, script pitfalls), cost estimate at the time limit, whether a human must approve it (counting your running and pending jobs), and sbatch --test-only. script is a path relative to cwd on the cluster (default: the configured workdir), never the script's text (refused without touching the cluster); or pass the script itself as script_text (e.g. a draft_job result) to check it without pushing it first (cwd is still where it would run). Pass each sbatch option and its value as separate list items. The test-only start time is unreliable. Also tested on the cluster: that the models, containers, envs and inputs the script reads exist (missing_paths; submit_job refuses them unless the job has a --dependency), and for vLLM on the dev QoS whether a compile cache is warm. asc: what ASC's submit filter changed in the request (comfort mode). expected_minutes: your runtime estimate; a --time far above it is flagged.

Nodes per partition: allocated, idle (free now), planned (held free by the scheduler for jobs next in line: not available to new jobs, except a short one backfilled before their planned start; plain sinfo %A/%F counts them as idle), other (down, drained, reserved, maintenance, not responding) and total, plus states (node count per raw Slurm state); pending and running jobs per partition; my fair-share and the priority factors of my pending jobs (low fair-share means long waits; idle GPU nodes can take CPU work at 4×).

Draft a Slurm job script for MUSICA from the runbook's tested patterns; nothing is sent to the cluster. kind: cpu (part of a CPU node: cores, mem_gb), cpu-node (whole CPU nodes), gpu (1-3 GPUs on a shared node: gpus; shared nodes can take hours to start), gpu-node (whole H100 nodes; with vllm_model: vLLM servers on every node, vllm_servers_per_node 1/2/4 = TP4/TP2/TP1, and command as their client on the first node). command: the work (shell lines); with nodes > 1 it runs once on every node ($NODE_RANK, $NNODES, $HEAD). time: minutes or HH:MM:SS; dev=True uses the dev QoS (at most 10 min, 2 nodes): test there first. array: e.g. 0-9%2 ($TASK in the script). python_env: a venv path (e.g. $DATA/envs/train); container: a .sif (prefix commands with run); modules: Lmod modules to load. runs (gpu-node, 1 node): up to 8 commands in parallel on one whole node, run r on GPU r % 4 ($RUN, own caches, log logs/--r.log; the job fails if any run failed; command runs first as setup); run_minutes: one estimate per run (refused if the slowest exceeds time). array_args: one argument string per array task (sets --array=0-(n-1)%throttle; the task gets $ARGS and $OUT=results/--a); array_throttle: tasks at once (0 = within the approval limits); not together with array. vllm_preset: tested serve settings for some models, 'qwen38-fp8' or 'glm53-fp8' (flags for one TP4 server per node; model and image default to $DATA/models/ and $DATA/containers/, which you must download first, or set vllm_model/vllm_image; vllm_args are appended, later flags win); load_cap_s: drop a vLLM server whose weights are not loaded this many seconds after its node's launch (60-3600). Drafts check at job start that the env, container and model exist (exit 2 otherwise), set HF_HUB_OFFLINE=1 on GPUs and OMP_NUM_THREADS=1 for vLLM. Returns the script, the static check (estimate, approvals needed) and notes; then check_job(script_text=...) for sbatch --test-only and the path checks on the cluster.

Copy a file or directory from the cluster into local_dir on host (tar over SSH, retried; overwrites files of the same name). Refuses more than max_mb (fetch subdirectories of big results). host as in push_files.

State, exit code, times, time limit, nodes and cost of a job (every array task and requeue attempt), its sbatch command line, the reason (and dependency) a pending job waits, and the cleaned tails of its output files (at most 20 files). An ended job that failed, or completed with errors in its log, gets a diagnosis: category (infrastructure = cluster side, resubmit; script; program; benign), summary, known signatures with their meaning and action (transient = a resubmit usually works), and the first real exceptions; harmless noise is not counted.

Tail of a job's stdout (stream='out'), stderr ('err') or of file: a path or glob relative to the job's working directory (%j %x %A %a allowed, e.g. 'logs/%x-%j-r*.log' or 'results/run-%j/train.log'; at most 20 files). For array jobs pass the task id (123_4) to see one task. The text is cleaned: ANSI colours and binary bytes removed, \r progress updates collapsed to their last state (raw=True: as written). grep: an extended regex, matched per line (also on \r updates and in binary logs). progress=True: instead of text, the latest state of each progress bar per file (label, n, total, percent, rate, elapsed, eta; kind loading / warmup / work: 'Loading weights 723/723' is not training). hide_benign=True: drop known-harmless lines and tracebacks (vLLM usage-stats TLS error, shutdown noise, srun's 'forcing job termination', ...); counted in benign_hidden. A log missing while the job is pending or in its first minutes is normal (node prolog).

Tested job scripts to copy from: Amadeus's jobs/ and the project's template directories (config). Without name: the list, each with a summary from its comments and its resources. With name (as listed, or the file name): the script and the helper scripts it uses. Prefer draft_job for a new job; use these for patterns it does not cover (vLLM multi-node benchmarks, DDP training, CPU-node arrays, container pulls).

The last submissions, updates, cancellations and approved commands made through this server (local ledger, ~/.local/state/amadeus/jobs.jsonl by default; AMADEUS_LEDGER overrides).

Issues filed through report_issue or automatically. status: open | in-progress | needs-human | fixed | wontfix | duplicate | all.

My running and pending jobs (arrays as one row), with the purpose recorded at submission (if submitted through this server), and what they hold together: nodes at once and the tokens they can still spend at their limits. For pending jobs start is Slurm's estimate (unreliable) and reason why it waits.

Run a short diagnostic command on the node of one of your RUNNING jobs (ssh to compute nodes is refused on MUSICA; this uses srun --jobid=<id> --overlap from the login node and shares the job's resources). Typical: nvidia-smi, top -b -n1 | head -30, free -g, ps -u $USER -o pid,pcpu,rss,etime,args --forest, $DATA/envs/tools/bin/py-spy dump --pid <pid>, tail of a node-local log, df -h /local. node: one of the job's nodes (default: its first). timeout_s: 5 to 120. No background processes, no srun. The same rules as run_command apply; deleting files or signalling processes (kill, pkill, fuser -k) needs the human's approval in approval, in their words. The check goes by command word (also behind timeout, nice, env, xargs, bash -c); killing processes by any other means (e.g. os.kill in a python -c) needs that approval too.

Copy a directory to the cluster (tar over SSH, retried; staged, so a failed push changes nothing; overwrites files of the same name, never deletes others). mode='tracked': git-tracked and untracked-but-not-ignored files plus a REVISION file, for code repositories; mode='all': everything except .git, venvs, caches and *.sif. host: the machine local_dir is on ('' = the machine the server runs on; default: the server's client machine, see the result's 'host'). local_dir must be inside the local root there (the home directory, outside hidden directories); remote_dir under $DATA, $SCRATCH or ~/ (e.g. '$DATA/'). Code only: big data is fetched in jobs. Warns (does not refuse) when your pending or running jobs work in or around remote_dir: Slurm keeps only a copy of the batch script, so pending jobs run the pushed code and running ones may read changed files mid-run.

Report a problem with this server or the cluster interface, for the maintainer session that fixes them. kind: bug | wrong-result | missing-feature | docs | cluster | other. severity: high (blocks work) | medium | low. Give the tool name, the arguments you used, what came back, what you expected, and your project (repository). No secrets. Returns the issue id; list_issues shows its status later.

Run a shell command on the login node (bash, non-interactive). $DATA, $SCRATCH, $USER expand there. Light work only: no compute, builds or big downloads (use a job). lmod=True initialises Lmod first (for module). Refused: Slurm-changing commands (use submit_job, cancel_job, update_job; sacctmgr only in its read-only show/list forms, e.g. sacctmgr -nP show qos format=Name,MaxWall), ssh to compute nodes and srun (use node_exec) and changes to SSH/shell/VPN config. Deleting or moving files (rm, mv, rsync --delete, ...) needs the human's approval in approval, in their words. The command is not repeated after an SSH failure; when recovery says the tunnel was restarted after a stall (nothing ran remotely), run it again.

Start a certificate login (or an early renewal while the current one is still valid) and return the device link. Show the link to the human immediately; it expires after about 60 s. Only one login runs at a time. Call it only when answering a message from the human, not from a cron, /loop or stop-hook turn (AGENTS.md §3).

Submit a batch job after the runbook checks (see check_job) and record it in the local ledger. script: the path of the job script on the cluster, relative to cwd (default: the configured workdir); script text is refused (check a draft with check_job(script_text=...), then save it in the project, push_files, submit). The job is submitted on hold, checked against Slurm's own reading of it, and released (or cancelled if Slurm reads it differently and it then breaks the rules). purpose: one line on what the job is for. approval: the human's explicit approval in their words (who, what, limit); required above the approval limits ([limits] in the config; by default ≥ 4 nodes at once, > 20 tokens including your other jobs), for B200 and long/priority QoS; a job of 16 or more nodes (headsup_nodes) needs an approval that names its node count, even inside an approved budget. Refused when a model, container, env or input the script reads does not exist on the cluster (unless the job has a --dependency, or allow_missing=True for a job that fetches or creates them itself before use: then a warning). Pass each sbatch option and its value as separate items. expected_minutes: your runtime estimate (a --time far above it is flagged).

Change a job: a new time limit (e.g. '02:00:00') or node count (pending jobs only), or release / hold it. A time limit can be raised only while the job is pending: MUSICA denies raising a running job's limit (lowering works). The new limits are re-checked like a submission (≥ 4 nodes at once, > 20 tokens need approval, the human's words). Jobs not submitted through this server need approval for any change.

Wait for one job (job_id) or several (job_ids) and return at the first event that matters, with the new log lines since the last call. Returns at the latest after max_wait_s (10 to 3600).

until: 'end' / 'all_end' = all jobs ended (default), 'any_end' = the first job ends, 'any_fail' = the first job ends other than COMPLETED (or all ended), 'start' = a job starts (or ends). Alarms always return at once:

• milestones: [{pattern, within_s, file}]: a regex (Perl/Python style) grepped in the job's 'out' (default), 'err' or a path relative to its working directory (%j %x %a allowed). A match returns MILESTONE; no match within_s seconds after the job's start returns LATE (e.g. vLLM 'ready after \d+s' within 600).

• quiet_alarm_s: QUIET when a RUNNING job's logs have not grown for that long (0 = off). vLLM compile and graph capture can be silent for about 3 min: use 300 for big runs, 120-180 while testing.

• expect: output paths relative to the working directory (%j %x %a, globs): OUTPUTS_MISSING when a job ended without files written since its start (a COMPLETED state can hide a failed run). Events: STARTED, ENDED, FAILED, MILESTONE, LATE, QUIET, OUTPUTS_MISSING; failed jobs come with a diagnosis. finished: True = all ended, False = still running (an event or max_wait_s), None = cluster or job not found. Pass next_cursor back as log_cursor: you then get only new log lines and events you have not seen yet. State lives in Slurm: after a restart, a dropped connection or a compaction just call again with the same ids; alarms this server already returned for these jobs then come back as info and do not end the call at once. Never build sleep/sacct watcher loops or background monitors. Each poll is one SSH command for all jobs.