sync_files
Sync local files to a remote SLURM cluster, transferring only new or changed files. Use dry-run to preview; optionally delete remote-only files with max-delete safeguard.
Instructions
Sync a configured mount from this machine to a remote SLURM cluster.
Copies new and changed files only. Files that exist on the cluster but not
locally are left untouched unless ``delete=True``.
That means a file deleted locally stays on the cluster, where a job can
still pick it up. This tool does not report those — call ``inspect_mount``
to see them. It is read-only, so it is safe to call before or after a sync;
reach for it rather than setting ``delete=True`` to find out what is stale.
Args:
transport: SSH profile name to sync against. Required and must name
an SSH profile — there is no local-to-local sync, and (unlike
the CLI) no implicit current-profile fallback. ``"local"`` is
rejected. Call ``list_ssh_profiles`` to see profiles and the
mounts each one defines.
mount: Mount name from that SSH profile. Only pre-registered mounts
can be synced; arbitrary paths are not accepted.
dry_run: Preview only. Reports exactly what would be transferred and
deleted without touching the cluster. Prefer this first whenever
you are unsure, and always before a ``delete=True`` run.
delete: Mirror the mount — also DELETE cluster files that no longer
exist locally. **This destroys remote-only data** such as
training checkpoints, job logs, and outputs written by jobs on
the cluster, which by definition do not exist locally. Leave it
off unless the user explicitly asked for a mirror, and preview
with ``dry_run=True`` before running it.
max_delete: Refuse the mirror, without changing anything, if it would
delete more than this many **entries**. Entries are files *and*
directories, matching rsync's own ``--max-delete`` unit: removing
a directory holding two files counts as three entries (both files
plus the directory), so set this above the file count you have in
mind. Guards against mirroring from a wrong or half-populated
local directory. Must be >= 1; to sync without deleting, leave
``delete`` off. Only applies to a real ``delete=True`` run — a
``dry_run`` preview is never capped, so it can show the whole list.
Returns:
On success: ``files_transferred``, ``entries_deleted``, and the
``deleted_paths`` list. Past a very large number of deletions the list
is omitted and ``deleted_paths_omitted`` is set — the count stays
exact, and no list is ever silently shortened.
The two counts use different units on purpose, because that is what
rsync reports: ``entries_deleted`` includes removed directories, while
``files_transferred`` counts only regular files whose data actually
crossed the wire — matching rsync's own "regular files transferred"
statistic. Directory creations, symlinks, devices, hard links and
attribute-only touch-ups move no data and are excluded, so a sync can
legitimately change the remote while reporting zero transfers.
Counts are reliable; path *strings* have one documented limit. rsync
separates its flag block from the filename with whitespace whose width
varies by version, so a filename that itself begins with spaces cannot
be told apart from that separator, and those leading spaces are lost
from the reported string. Such a deletion is still counted, so the
cap and the refusal logic are unaffected.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mount | Yes | ||
| delete | No | ||
| dry_run | No | ||
| transport | Yes | ||
| max_delete | No |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||