wait_for_job
Waits for one or more Slurm jobs and returns at the first relevant event with new log lines since the last call, enabling progress monitoring without sleep or sacct watcher loops.
Instructions
Wait for one job (job_id) or several (job_ids) and return at the first event that matters, with the new log lines since the last call. Returns at the latest after max_wait_s (10 to 3600).
until: 'end' / 'all_end' = all jobs ended (default), 'any_end' = the first job ends, 'any_fail' = the first job ends other than COMPLETED (or all ended), 'start' = a job starts (or ends). Alarms always return at once:
milestones: [{pattern, within_s, file}]: a regex (Perl/Python style) grepped in the job's 'out' (default), 'err' or a path relative to its working directory (%j %x %a allowed). A match returns MILESTONE; no match within_s seconds after the job's start returns LATE (e.g. vLLM 'ready after \d+s' within 600).
quiet_alarm_s: QUIET when a RUNNING job's logs have not grown for that long (0 = off). vLLM compile and graph capture can be silent for about 3 min: use 300 for big runs, 120-180 while testing.
expect: output paths relative to the working directory (%j %x %a, globs): OUTPUTS_MISSING when a job ended without files written since its start (a COMPLETED state can hide a failed run). Events: STARTED, ENDED, FAILED, MILESTONE, LATE, QUIET, OUTPUTS_MISSING; failed jobs come with a diagnosis. finished: True = all ended, False = still running (an event or max_wait_s), None = cluster or job not found. Pass next_cursor back as log_cursor: you then get only new log lines and events you have not seen yet. State lives in Slurm: after a restart, a dropped connection or a compaction just call again with the same ids; alarms this server already returned for these jobs then come back as info and do not end the call at once. Never build sleep/sacct watcher loops or background monitors. Each poll is one SSH command for all jobs.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| until | No | end | |
| expect | No | ||
| job_id | No | ||
| poll_s | No | ||
| job_ids | No | ||
| log_cursor | No | ||
| max_wait_s | No | ||
| milestones | No | ||
| quiet_alarm_s | No |