Args:
    command: Shell command to execute (e.g. "python train.py --epochs 100")
    name: Job name for identification in SLURM queue
    nodes: Number of compute nodes to allocate
    gpus_per_node: Number of GPUs per node (0 for CPU-only)
    ntasks_per_node: Number of tasks per node
    cpus_per_task: Number of CPUs per task
    memory_per_node: Memory per node (e.g. "32GB", "64G")
    time_limit: Wall time limit (e.g. "4:00:00", "1-00:00:00")
    partition: SLURM partition name (e.g. "gpu", "cpu")
    nodelist: Specific nodes to use (e.g. "node001,node002")
    conda: Conda environment name to activate before running
    venv: Path to Python virtual environment to activate
    env_vars: Additional environment variables as key-value pairs
    log_dir: Directory for stdout/stderr log files
    work_dir: Working directory for the job (defaults to cwd)
    transport: Cluster selector — omit / "local" for local SLURM, or an
        SSH profile name to submit to that remote cluster.

Args:
    transport: Cluster selector — omit / "local" for local SLURM, or an
        SSH profile name to query that remote cluster.

Args:
    job_id: SLURM job ID to check
    transport: Cluster selector — omit / "local" for local SLURM, or an
        SSH profile name to query that remote cluster.

Args:
    job_id: SLURM job ID to cancel
    transport: Cluster selector — omit / "local" for local SLURM, or an
        SSH profile name to cancel on that remote cluster.

Args:
    job_id: SLURM job ID
    job_name: Optional job name to help locate log files
    transport: Cluster selector — omit / "local" for local SLURM, or an
        SSH profile name to fetch logs from that remote cluster.

Args:
    partition: Specific partition to check (None for all partitions)
    transport: Cluster selector — omit / "local" for local SLURM, or an
        SSH profile name to query that remote cluster.

Read-only: this never transfers, deletes, or creates anything on the
cluster. Call it freely, including before a sync you are unsure about.

Its main job is answering a question ``sync_files`` cannot: **what is on the
cluster that no longer exists locally?** A sync is additive, so those files
stay — including code deleted in a local refactor, which a job on the
cluster can still import and run. They are listed here as
``mirror_delete_candidate_paths``.

Those candidates mix two kinds of thing:

* **produced by jobs** — checkpoints, logs, outputs. Must NOT be deleted.
* **left over locally** — stale modules, renamed files. Usually should be.

``stale_upload_paths`` is the second group on its own: paths srunx recorded
uploading that are no longer present locally. Job output was never uploaded,
so it does not appear there — which holds even when the mount's exclude list
misses an output directory, the case that otherwise buries a few stale
scripts among dozens of artifacts.

One exception: output pulled into the local tree with ``srunx ssh sync
--pull`` becomes a file the next push manages, so it is recorded like any
other and can be reported as stale once its local copy is removed. Excluding
the output directories on the mount avoids that, and is worth doing anyway.

**Check ``stale_uploads_known`` first.** When it is false the record could
not answer (nothing uploaded with tracking yet, an unreadable record, or a
changed exclude filter), and ``stale_uploads: 0`` means "could not tell",
not "nothing is stale" — ``stale_uploads_unknown_reason`` says which. Fall
back to reading the full candidate list yourself in that case.

Args:
    transport: SSH profile name to inspect. Required — there is no local
        inspection, and (unlike the CLI) no implicit current-profile
        fallback. Call ``list_ssh_profiles`` for the available profiles and
        the mounts each defines.
    mount: Mount name from that SSH profile.
    max_paths: Cap on how many paths to list. Counts stay exact regardless;
        past the cap the list is omitted rather than shortened, and
        ``mirror_delete_candidate_paths_omitted`` says so.

Returns:
    ``files_would_transfer``, ``mirror_delete_candidates`` (a count),
    ``mirror_delete_candidate_paths``, whether that list was omitted,
    ``effective_exclude_patterns``, and the stale-upload fields described
    above (``stale_uploads_known`` / ``stale_uploads`` /
    ``stale_upload_paths`` / ``stale_uploads_unknown_reason``).

    The exclude list matters for reading the result: excluded paths are
    invisible to this inspection *and* protected from a mirror's deletions,
    so something absent from the candidates may simply be excluded rather
    than in sync.

Copies new and changed files only. Files that exist on the cluster but not
locally are left untouched unless ``delete=True``.

That means a file deleted locally stays on the cluster, where a job can
still pick it up. This tool does not report those — call ``inspect_mount``
to see them. It is read-only, so it is safe to call before or after a sync;
reach for it rather than setting ``delete=True`` to find out what is stale.

Args:
    transport: SSH profile name to sync against. Required and must name
        an SSH profile — there is no local-to-local sync, and (unlike
        the CLI) no implicit current-profile fallback. ``"local"`` is
        rejected. Call ``list_ssh_profiles`` to see profiles and the
        mounts each one defines.
    mount: Mount name from that SSH profile. Only pre-registered mounts
        can be synced; arbitrary paths are not accepted.
    dry_run: Preview only. Reports exactly what would be transferred and
        deleted without touching the cluster. Prefer this first whenever
        you are unsure, and always before a ``delete=True`` run.
    delete: Mirror the mount — also DELETE cluster files that no longer
        exist locally. **This destroys remote-only data** such as
        training checkpoints, job logs, and outputs written by jobs on
        the cluster, which by definition do not exist locally. Leave it
        off unless the user explicitly asked for a mirror, and preview
        with ``dry_run=True`` before running it.
    max_delete: Refuse the mirror, without changing anything, if it would
        delete more than this many **entries**. Entries are files *and*
        directories, matching rsync's own ``--max-delete`` unit: removing
        a directory holding two files counts as three entries (both files
        plus the directory), so set this above the file count you have in
        mind. Guards against mirroring from a wrong or half-populated
        local directory. Must be >= 1; to sync without deleting, leave
        ``delete`` off. Only applies to a real ``delete=True`` run — a
        ``dry_run`` preview is never capped, so it can show the whole list.

Returns:
    On success: ``files_transferred``, ``entries_deleted``, and the
    ``deleted_paths`` list. Past a very large number of deletions the list
    is omitted and ``deleted_paths_omitted`` is set — the count stays
    exact, and no list is ever silently shortened.

    The two counts use different units on purpose, because that is what
    rsync reports: ``entries_deleted`` includes removed directories, while
    ``files_transferred`` counts only regular files whose data actually
    crossed the wire — matching rsync's own "regular files transferred"
    statistic. Directory creations, symlinks, devices, hard links and
    attribute-only touch-ups move no data and are excluded, so a sync can
    legitimately change the remote while reporting zero transfers.

    Counts are reliable; path *strings* have one documented limit. rsync
    separates its flag block from the filename with whitespace whose width
    varies by version, so a filename that itself begins with spaces cannot
    be told apart from that separator, and those leading spaces are lost
    from the reported string. Such a deletion is still counted, so the
    cap and the refusal logic are unaffected.

Generates a YAML workflow definition that can be executed with run_workflow.
Each job in the workflow can depend on other jobs, forming a DAG.

Args:
    name: Workflow name for identification
    jobs: List of job definitions. Each job dict should contain:
        - name (required): Job identifier
        - command (required for regular jobs): Command as string or list of strings
        - script_path (required for shell jobs): Path to shell script
        - depends_on: List of job names this job depends on (e.g. ["preprocess"])
          Supports dependency types: "afterok:job_a", "after:job_a", "afterany:job_a"
        - retry: Number of retry attempts on failure (default 0)
        - retry_delay: Seconds between retries (default 60)
        - resources: Dict with nodes, gpus_per_node, ntasks_per_node,
          cpus_per_task, memory_per_node, time_limit, partition, nodelist
        - environment: Dict with conda, venv, env_vars, container
        - log_dir: Log directory path
        - work_dir: Working directory path
    output_path: File path to write the YAML workflow (e.g. "workflow.yaml")
    args: Optional template variables for Jinja2 templating in job definitions
    default_project: Default SSH project/mount name for file syncing

Checks for valid YAML syntax, correct job structure, dependency resolution,
and circular dependency detection.

Args:
    yaml_path: Path to the YAML workflow file to validate

Jobs are executed in dependency order - independent jobs run in parallel,
dependent jobs wait for their prerequisites to complete.

Args:
    yaml_path: Path to the YAML workflow file
    from_job: Start execution from this job (skip earlier jobs)
    to_job: Stop execution at this job (skip later jobs)
    single_job: Execute only this specific job, ignoring dependencies
    dry_run: If true, show what would be executed without actually running
    args: Optional mapping merged over the YAML ``args`` section before
        Jinja rendering. ``python:`` prefix values are rejected.
    sweep: Optional sweep spec: ``{"matrix": {...}, "fail_fast": bool,
        "max_parallel": int}``. When present, the request goes through
        :class:`SweepOrchestrator` and the response contains
        ``sweep_run_id``.
    transport: Cluster selector — omit / "local" for local SLURM, or an
        SSH profile name to run against that remote cluster. Orthogonal to
        ``mount``: ``transport`` picks *which* cluster, ``mount`` picks the
        path-translation root within it.
    mount: Optional mount name within the SSH profile, enabling mount-aware
        path translation for ``work_dir`` / ``log_dir``. Requires an SSH
        ``transport``; passing ``mount`` with a local transport is an error.

Scans the directory for YAML files that contain a valid srunx workflow
structure (must have 'name' and 'jobs' keys).

Args:
    directory: Directory to search for workflow files (default: current directory)

Args:
    yaml_path: Path to the YAML workflow file