node_exec
Run diagnostic commands on a compute node of a running MUSICA job without SSH by using srun to share job resources for checks like nvidia-smi, top, or logs.
Instructions
Run a short diagnostic command on the node of one of your RUNNING jobs (ssh to compute nodes is refused on
MUSICA; this uses srun --jobid=<id> --overlap from the login node and shares the job's resources).
Typical: nvidia-smi, top -b -n1 | head -30, free -g, ps -u $USER -o pid,pcpu,rss,etime,args --forest,
$DATA/envs/tools/bin/py-spy dump --pid <pid>, tail of a node-local log, df -h /local.
node: one of the job's nodes (default: its first). timeout_s: 5 to 120. No background processes, no srun.
The same rules as run_command apply; deleting files or signalling processes (kill, pkill, fuser -k) needs the
human's approval in approval, in their words. The check goes by command word (also behind timeout, nice, env,
xargs, bash -c); killing processes by any other means (e.g. os.kill in a python -c) needs that approval too.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| node | No | ||
| job_id | Yes | ||
| command | Yes | ||
| approval | No | ||
| timeout_s | No |