Tunnel, certificate expiry and an SSH test. Call first in every session. (Starts the tunnel if it is down; with wireproxy, a stall before login restarts it, at most once a minute, also from other tools that run commands on the login node, not yet from the copying in push_files/fetch_files: recovery.) The first call of a server run also carries banner (Amadeus' ASCII art): show it to the human once.

Start a certificate login (or an early renewal while the current one is still valid) and return the device link. Show the link to the human immediately; it expires after about 60 s. Only one login runs at a time. Call it only when answering a message from the human, not from a cron, /loop or stop-hook turn (AGENTS.md §3).

Run a shell command on the login node (bash, non-interactive). $DATA, $SCRATCH, $USER expand there. Light work only: no compute, builds or big downloads (use a job). lmod=True initialises Lmod first (for module). Refused: Slurm-changing commands (use submit_job, cancel_job, update_job; sacctmgr only in its read-only show/list forms, e.g. sacctmgr -nP show qos format=Name,MaxWall), ssh to compute nodes and srun (use node_exec) and changes to SSH/shell/VPN config. Deleting or moving files (rm, mv, rsync --delete, ...) needs the human's approval in approval, in their words. The command is not repeated after an SSH failure; when recovery says the tunnel was restarted after a stall (nothing ran remotely), run it again.

My running and pending jobs (arrays as one row), with the purpose recorded at submission (if submitted through this server), and what they hold together: nodes at once and the tokens they can still spend at their limits. For pending jobs start is Slurm's estimate (unreliable) and reason why it waits.

State, exit code, times, time limit, nodes and cost of a job (every array task and requeue attempt), its sbatch command line, the reason (and dependency) a pending job waits, and the cleaned tails of its output files (at most 20 files). An ended job that failed, or completed with errors in its log, gets a diagnosis: category (infrastructure = cluster side, resubmit; script; program; benign), summary, known signatures with their meaning and action (transient = a resubmit usually works), and the first real exceptions; harmless noise is not counted.

Tail of a job's stdout (stream='out'), stderr ('err') or of file: a path or glob relative to the job's working directory (%j %x %A %a allowed, e.g. 'logs/%x-%j-r*.log' or 'results/run-%j/train.log'; at most 20 files). For array jobs pass the task id (123_4) to see one task. The text is cleaned: ANSI colours and binary bytes removed, \r progress updates collapsed to their last state (raw=True: as written). grep: an extended regex, matched per line (also on \r updates and in binary logs). progress=True: instead of text, the latest state of each progress bar per file (label, n, total, percent, rate, elapsed, eta; kind loading / warmup / work: 'Loading weights 723/723' is not training). hide_benign=True: drop known-harmless lines and tracebacks (vLLM usage-stats TLS error, shutdown noise, srun's 'forcing job termination', ...); counted in benign_hidden. A log missing while the job is pending or in its first minutes is normal (node prolog).

Wait for one job (job_id) or several (job_ids) and return at the first event that matters, with the new log lines since the last call. Returns at the latest after max_wait_s (10 to 3600).

until: 'end' / 'all_end' = all jobs ended (default), 'any_end' = the first job ends, 'any_fail' = the first job ends other than COMPLETED (or all ended), 'start' = a job starts (or ends). Alarms always return at once:

• milestones: [{pattern, within_s, file}]: a regex (Perl/Python style) grepped in the job's 'out' (default), 'err' or a path relative to its working directory (%j %x %a allowed). A match returns MILESTONE; no match within_s seconds after the job's start returns LATE (e.g. vLLM 'ready after \d+s' within 600).

• quiet_alarm_s: QUIET when a RUNNING job's logs have not grown for that long (0 = off). vLLM compile and graph capture can be silent for about 3 min: use 300 for big runs, 120-180 while testing.

• expect: output paths relative to the working directory (%j %x %a, globs): OUTPUTS_MISSING when a job ended without files written since its start (a COMPLETED state can hide a failed run). Events: STARTED, ENDED, FAILED, MILESTONE, LATE, QUIET, OUTPUTS_MISSING; failed jobs come with a diagnosis. finished: True = all ended, False = still running (an event or max_wait_s), None = cluster or job not found. Pass next_cursor back as log_cursor: you then get only new log lines and events you have not seen yet. State lives in Slurm: after a restart, a dropped connection or a compaction just call again with the same ids; alarms this server already returned for these jobs then come back as info and do not end the call at once. Never build sleep/sacct watcher loops or background monitors. Each poll is one SSH command for all jobs.

Run a short diagnostic command on the node of one of your RUNNING jobs (ssh to compute nodes is refused on MUSICA; this uses srun --jobid=<id> --overlap from the login node and shares the job's resources). Typical: nvidia-smi, top -b -n1 | head -30, free -g, ps -u $USER -o pid,pcpu,rss,etime,args --forest, $DATA/envs/tools/bin/py-spy dump --pid <pid>, tail of a node-local log, df -h /local. node: one of the job's nodes (default: its first). timeout_s: 5 to 120. No background processes, no srun. The same rules as run_command apply; deleting files or signalling processes (kill, pkill, fuser -k) needs the human's approval in approval, in their words. The check goes by command word (also behind timeout, nice, env, xargs, bash -c); killing processes by any other means (e.g. os.kill in a python -c) needs that approval too.

Nodes per partition: allocated, idle (free now), planned (held free by the scheduler for jobs next in line: not available to new jobs, except a short one backfilled before their planned start; plain sinfo %A/%F counts them as idle), other (down, drained, reserved, maintenance, not responding) and total, plus states (node count per raw Slurm state); pending and running jobs per partition; my fair-share and the priority factors of my pending jobs (low fair-share means long waits; idle GPU nodes can take CPU work at 4×).

Tokens (credits) used by the whole project since a date, per user, from sacct (CPU nodes included; B200 as node-hours, no published price) and what is left of the configured budget. The projects portal is authoritative. Without a configured account: only the user's own jobs.

Check a job before submitting: runbook rules (account, partition/QoS, time, dev limits, CPU limits, script pitfalls), cost estimate at the time limit, whether a human must approve it (counting your running and pending jobs), and sbatch --test-only. script is a path relative to cwd on the cluster (default: the configured workdir), never the script's text (refused without touching the cluster); or pass the script itself as script_text (e.g. a draft_job result) to check it without pushing it first (cwd is still where it would run). Pass each sbatch option and its value as separate list items. The test-only start time is unreliable. Also tested on the cluster: that the models, containers, envs and inputs the script reads exist (missing_paths; submit_job refuses them unless the job has a --dependency), and for vLLM on the dev QoS whether a compile cache is warm. asc: what ASC's submit filter changed in the request (comfort mode). expected_minutes: your runtime estimate; a --time far above it is flagged.

Draft a Slurm job script for MUSICA from the runbook's tested patterns; nothing is sent to the cluster. kind: cpu (part of a CPU node: cores, mem_gb), cpu-node (whole CPU nodes), gpu (1-3 GPUs on a shared node: gpus; shared nodes can take hours to start), gpu-node (whole H100 nodes; with vllm_model: vLLM servers on every node, vllm_servers_per_node 1/2/4 = TP4/TP2/TP1, and command as their client on the first node). command: the work (shell lines); with nodes > 1 it runs once on every node ($NODE_RANK, $NNODES, $HEAD). time: minutes or HH:MM:SS; dev=True uses the dev QoS (at most 10 min, 2 nodes): test there first. array: e.g. 0-9%2 ($TASK in the script). python_env: a venv path (e.g. $DATA/envs/train); container: a .sif (prefix commands with run); modules: Lmod modules to load. runs (gpu-node, 1 node): up to 8 commands in parallel on one whole node, run r on GPU r % 4 ($RUN, own caches, log logs/--r.log; the job fails if any run failed; command runs first as setup); run_minutes: one estimate per run (refused if the slowest exceeds time). array_args: one argument string per array task (sets --array=0-(n-1)%throttle; the task gets $ARGS and $OUT=results/--a); array_throttle: tasks at once (0 = within the approval limits); not together with array. vllm_preset: tested serve settings for some models, 'qwen38-fp8' or 'glm53-fp8' (flags for one TP4 server per node; model and image default to $DATA/models/ and $DATA/containers/, which you must download first, or set vllm_model/vllm_image; vllm_args are appended, later flags win); load_cap_s: drop a vLLM server whose weights are not loaded this many seconds after its node's launch (60-3600). Drafts check at job start that the env, container and model exist (exit 2 otherwise), set HF_HUB_OFFLINE=1 on GPUs and OMP_NUM_THREADS=1 for vLLM. Returns the script, the static check (estimate, approvals needed) and notes; then check_job(script_text=...) for sbatch --test-only and the path checks on the cluster.

Tested job scripts to copy from: Amadeus's jobs/ and the project's template directories (config). Without name: the list, each with a summary from its comments and its resources. With name (as listed, or the file name): the script and the helper scripts it uses. Prefer draft_job for a new job; use these for patterns it does not cover (vLLM multi-node benchmarks, DDP training, CPU-node arrays, container pulls).

Submit a batch job after the runbook checks (see check_job) and record it in the local ledger. script: the path of the job script on the cluster, relative to cwd (default: the configured workdir); script text is refused (check a draft with check_job(script_text=...), then save it in the project, push_files, submit). The job is submitted on hold, checked against Slurm's own reading of it, and released (or cancelled if Slurm reads it differently and it then breaks the rules). purpose: one line on what the job is for. approval: the human's explicit approval in their words (who, what, limit); required above the approval limits ([limits] in the config; by default ≥ 4 nodes at once, > 20 tokens including your other jobs), for B200 and long/priority QoS; a job of 16 or more nodes (headsup_nodes) needs an approval that names its node count, even inside an approved budget. Refused when a model, container, env or input the script reads does not exist on the cluster (unless the job has a --dependency, or allow_missing=True for a job that fetches or creates them itself before use: then a warning). Pass each sbatch option and its value as separate items. expected_minutes: your runtime estimate (a --time far above it is flagged).

Change a job: a new time limit (e.g. '02:00:00') or node count (pending jobs only), or release / hold it. A time limit can be raised only while the job is pending: MUSICA denies raising a running job's limit (lowering works). The new limits are re-checked like a submission (≥ 4 nodes at once, > 20 tokens need approval, the human's words). Jobs not submitted through this server need approval for any change.

Cancel a job. Jobs submitted through this server can be cancelled directly; any other job needs the human's approval in approval (AGENTS.md §9).

The last submissions, updates, cancellations and approved commands made through this server (local ledger, ~/.local/state/amadeus/jobs.jsonl by default; AMADEUS_LEDGER overrides).

Copy a directory to the cluster (tar over SSH, retried; staged, so a failed push changes nothing; overwrites files of the same name, never deletes others). mode='tracked': git-tracked and untracked-but-not-ignored files plus a REVISION file, for code repositories; mode='all': everything except .git, venvs, caches and *.sif. host: the machine local_dir is on ('' = the machine the server runs on; default: the server's client machine, see the result's 'host'). local_dir must be inside the local root there (the home directory, outside hidden directories); remote_dir under $DATA, $SCRATCH or ~/ (e.g. '$DATA/'). Code only: big data is fetched in jobs. Warns (does not refuse) when your pending or running jobs work in or around remote_dir: Slurm keeps only a copy of the batch script, so pending jobs run the pushed code and running ones may read changed files mid-run.

Copy a file or directory from the cluster into local_dir on host (tar over SSH, retried; overwrites files of the same name). Refuses more than max_mb (fetch subdirectories of big results). host as in push_files.

Report a problem with this server or the cluster interface, for the maintainer session that fixes them. kind: bug | wrong-result | missing-feature | docs | cluster | other. severity: high (blocks work) | medium | low. Give the tool name, the arguments you used, what came back, what you expected, and your project (repository). No secrets. Returns the issue id; list_issues shows its status later.

Issues filed through report_issue or automatically. status: open | in-progress | needs-human | fixed | wontfix | duplicate | all.