SFTP Helper

🇫🇷 · 🇬🇧

SFTP Helper belongs to a collection of libraries called AI Helpers, built for developing Artificial Intelligence applications.

SFTP — the SSH File Transfer Protocol — moves files to and from a remote server over the same encrypted channel used for a remote terminal login (SSH).

No password is needed if you would rather not use one: a key pair proves who you are, the same way a physical key proves you are allowed into a building. The server only needs to know your key's public half.

An AI pipeline that produces files — a rendered video, a trained model, a batch of transcripts — usually needs to hand them off somewhere else without a human clicking through a file manager. sftp-helper is that hand-off, done from Python, with the file-integrity checks a script running unattended needs and a human with a mouse does not: resumable transfers, content hashing, atomic writes.

This toolbox requires:

• a settings.yaml for the sftp parameters (or JSON or environment variables or .env)

• that you previously added you SSH key of your local machine in the SFTP server

🌍 AI Helpers

SFTP Helper is a Python library that provides utility functions for working with SFTP servers via the system OpenSSH sftp client. Host key verification is on by default: ~/.ssh/known_hosts is consulted and unknown hosts are rejected.

The Promise

Remote by design. sftp-helper exists to move data to and from a remote server, so it is deliberately not local-first and ships no GUI. For cloud object storage (S3 / GCS / Azure / MinIO) use bucket-helper; for downloading media from a URL use youtube-helper.

Battle-tested. Eighteen tagged releases, a green CI on every push (unit tests plus a linter that blocks the merge exactly like a failing test), and a dedicated security pass (v3.2.1) that closed a path-injection gap in the batch-command builder before it ever shipped to a user. Nothing here is promised without a check backing it: the numbers above come straight from this repository's own tags and CHANGELOG.md, not from a marketing claim.

Related MCP server: MCP Filesystem Server

Documentation

💻 Documentation

🗺️ Landscape

📋 Examples

Features

• Upload a local file or directory to the server.

  • Destination. Pass an explicit sftp://host/path, or omit it (single file only) to get a deterministic content-hashed name under sftp_destination_path — identical bytes de-duplicate to the same path.

  • Directories travel as one stream. upload_many packs and ships a single tar.gz (one put plus one remote tar extraction) whenever the server accepts exec, instead of one round trip per file.

  • Knobs. overwrite skips destinations already present with a matching size (incremental sync), resume discards a stale partial transfer, progress suppresses the progress bar.

• Download a remote file or directory to disk (defaults to the remote basename for a single file), archive-accelerated the same way via download_many, with the same overwrite/resume/progress knobs.

• Delete a remote file, idempotent: removing an absent file succeeds.

• Existence checks for a remote file (remote_file_exists) and a remote directory (remote_dir_exist).

• List a remote directory (list_dir, optionally recursive) or list with per-entry size/mtime in one pass (list_dir_stat); stat a single remote file (remote_stat).

• Create remote directories with mkdir -p semantics (make_remote_directory): every missing intermediate level is created.

• Path helpers: normalize_path (single leading /, no trailing /) and strip_sftp_path (drop the sftp:// scheme + host).

• remote_tempfile context manager: reserve a unique random remote path (optionally under a subdir, optionally with an extension) that is auto-deleted on block exit, even if an exception propagates; hands back both the sftp:// address and its public HTTPS URL.

• Credentials loader (credentials) resolving JSON / YAML / directory / SFTP_* env vars / .env, with a masked show-credentials view.

• Strict host-key verification, always on: OpenSSH StrictHostKeyChecking=yes, no opt-out; trust an extra key via the optional sftp_known_hosts credential.

• Three surfaces, one behavior: Python library, a click CLI (sftp-helper), and a FastAPI HTTP surface. See the multi-surface section.

• Trigger catalogue in TRIGGERS.md.

How it works

Installation

Prerequisites: Python 3.10–3.13 and git, cross-platform:

• 🍎 macOS (Homebrew): brew install python git

• 🐧 Ubuntu/Debian: sudo apt update && sudo apt install -y python3 python3-pip git

• 🪟 Windows (PowerShell): winget install Python.Python.3.12 Git.Git

We recommend using Python environments. Check this link if you're unfamiliar with setting one up: 🥸 Tech tips.

From PyPI (recommended)

# Core SFTP utilities (library + click CLI)
pip install sftp-helper

# Optional surfaces
pip install "sftp-helper[api]"       # FastAPI HTTP surface

From source (no PyPI)

git clone https://github.com/warith-harchaoui/sftp-helper.git
cd sftp-helper
pip install -e .

# Optional surfaces
pip install -e ".[api]"

Write your own configuration file

A ready-to-fill template is committed at settings.yaml.example, with inline comments explaining every key and how to obtain its value. Copy it and edit in place; the real settings.yaml is gitignored so you cannot accidentally commit secrets:

cp settings.yaml.example settings.yaml
# then edit settings.yaml with your credentials

You may also provide a JSON file, environment variables, or an .env file; sftp-helper falls back in that order via os_helper.get_config:

Only three fields are required: sftp_host, sftp_login, sftp_https. Authenticate with an SSH key (recommended: no password) by pointing sftp_key at your public key (~/.ssh/id_ed25519.pub); OpenSSH lets your SSH agent or hardware token do the signing, so no private-key material is ever named in this file. Or load your key into the SSH agent and leave sftp_key empty. sftp_destination_path is optional and defaults to the server root /.

YAML (settings.yaml)

sftp_host: "<sftp_host>"
sftp_login: "<sftp_login>"
sftp_https: "<sftp_https>"
sftp_key: "~/.ssh/id_ed25519.pub" # optional public key; empty -> SSH agent + default keys
# sftp_passwd: "<sftp_passwd>"    # optional fallback (needs `sshpass`)
# sftp_destination_path: "/base"  # optional; empty -> server root "/"
# sftp_port: "2022"               # optional; default 22

or

JSON

{
    "sftp_host": "<sftp_host>",
    "sftp_login": "<sftp_login>",
    "sftp_https": "<sftp_https>",
    "sftp_key": "~/.ssh/id_ed25519.pub"
}

or

ENVIRONMENT VARIABLES

SFTP_HOST="<sftp_host>" \
SFTP_LOGIN="<sftp_login>" \
SFTP_HTTPS="<sftp_https>" \
SFTP_KEY="~/.ssh/id_ed25519.pub" \
python <your_python_script>

or

.env

SFTP_HOST                = <sftp_host>
SFTP_LOGIN               = <sftp_login>
SFTP_HTTPS               = <sftp_https>
SFTP_KEY                 = ~/.ssh/id_ed25519.pub

Where to find these (in your favorite FTP tool; mine is FileZilla):

• <sftp_host> is the server host, e.g. sftp.example.com

• <sftp_login> is your username

• <sftp_https> corresponds to the web URL of sftp_destination_path

• sftp_key points at the public half of the key you already use to ssh/sftp into the server (that same public key must be installed in the server's authorized_keys, and the private key loaded in your SSH agent); or leave it empty and rely on your SSH agent. Only if you run no agent, point it at the private key (~/.ssh/id_ed25519) instead.

• <your_python_script> is your python script :)

No SSH key yet?

The ssh-keygen command is identical on every OS: it writes the private key to ~/.ssh/id_ed25519 and the public key to ~/.ssh/id_ed25519.pub:

ssh-keygen -t ed25519 -C "you@example.com"

Load the private key into your SSH agent so the public key can sign, then install the public key on the server:

# Load the private key into the agent
ssh-add --apple-use-keychain ~/.ssh/id_ed25519          # macOS
eval "$(ssh-agent -s)" && ssh-add ~/.ssh/id_ed25519     # Ubuntu / Linux
Start-Service ssh-agent; ssh-add $HOME\.ssh\id_ed25519  # Windows (PowerShell)

# Install the public key on the server (~/.ssh/authorized_keys)
ssh-copy-id -i ~/.ssh/id_ed25519.pub your-login@sftp.example.com   # macOS / Ubuntu
type $HOME\.ssh\id_ed25519.pub | ssh your-login@sftp.example.com "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"  # Windows

Quick start

For the full catalog of recipes (uploads, downloads, existence checks, recursive directory creation, temporary remote files with auto-cleanup, strict host-key verification), see 📋 EXAMPLES.md.

Here's an example of how to use SFTP helper (won't work without a valid path/to/settings.yaml):

import sftp_helper as sftph
import os_helper as osh

# Write a small text file
local_file = "example.txt"
with open(local_file, "wt") as f:
    f.write("A small example of text")

# Load creds from JSON / YAML file, or fall back to .env / environment vars.
cred = sftph.credentials("path/to/settings.yaml")

remote_file = cred["sftp_destination_path"] + "/" + local_file
url = cred["sftp_https"] + "/" + local_file

# upload() raises on failure and returns the destination URL on success.
sftph.upload(local_file, cred, remote_file)
print(f"Uploaded {local_file} to {remote_file}")
# Uploaded example.txt to /remote/base/path/example.txt

assert osh.is_working_url(url), f"URL not reachable: {url}"
print(f"URL is live: {url}")
# URL is live: https://files.example.com/example.txt

Temporary remote files

If you need a unique remote path that gets cleaned up automatically, use the remote_tempfile context manager:

import sftp_helper as sftph
import os_helper as osh

credentials = sftph.credentials("path/to/settings.yaml")

with sftph.remote_tempfile(credentials, ext="txt") as (sftp_address, url):
    sftph.upload("local.txt", credentials, sftp_address)
    assert osh.is_working_url(url)
# On exit, the remote file is deleted.

Host key verification

A server's host key is how your machine recognises it. The first time you connect, OpenSSH records that key in ~/.ssh/known_hosts; every later connection checks the key still matches.

Without that check, someone who can intercept your network traffic could impersonate the server and read everything you upload or download. The check is what makes that attack fail loudly instead of succeeding silently.

sftp_helper never disables host key verification. Every sftp invocation passes StrictHostKeyChecking=yes, and ~/.ssh/known_hosts is consulted automatically, so a host whose key you have not already accepted is rejected. To trust a server whose key lives elsewhere, point at an extra known_hosts file through the optional sftp_known_hosts credential.

When a connection stalls

Two different failures need two different bounds, and OpenSSH does not conflate them:

• A host that never answers is bounded by ConnectTimeout=30, which covers the TCP connect and the handshake.

• A session that comes up and then goes quiet is bounded by ServerAliveInterval=15 with ServerAliveCountMax=4. OpenSSH documents ConnectTimeout as applying only "when connecting to the SSH server", so without the keepalives a transfer that stalls after a successful handshake is waited on indefinitely — no read timeout exists. 15 × 4 means a dead session fails in about a minute.

Both are passed on every ssh and sftp invocation. Tune or disable them:

export SFTP_HELPER_SERVER_ALIVE_INTERVAL=30   # seconds; 0 disables keepalives
export SFTP_HELPER_SERVER_ALIVE_COUNT_MAX=4   # unanswered probes before giving up

This matters more here than it would in a one-shot client, because connection multiplexing is on (ControlMaster=auto): a wedged master connection does not only hang its own transfer — every later call in the run attaches to the same control socket and hangs with it.

Multi-surface exposure

sftp-helper is not just a library: the same functions are exposed as a CLI, a FastAPI HTTP surface, and MCP tools:

# Python library (default)
import sftp_helper as sftph

# the CLI (installed automatically)
sftp-helper upload      --config settings.yaml --input local.txt --remote /uploads/local.txt
sftp-helper upload      --config settings.yaml --input ./local_dir --remote /uploads/dir
sftp-helper download    --config settings.yaml --remote /uploads/local.txt --output out.txt
sftp-helper exists      --config settings.yaml --remote /uploads/local.txt
sftp-helper list        --config settings.yaml --remote /uploads --recursive
sftp-helper remote-stat --config settings.yaml --remote /uploads/local.txt
sftp-helper mkdir       --config settings.yaml --remote /uploads/a/b/c

# FastAPI HTTP surface (needs the [api] extra)
pip install "sftp-helper[api]"
SFTP_HELPER_CONFIG=./settings.yaml uvicorn sftp_helper.api:app --port 8000
# → OpenAPI docs at http://localhost:8000/docs

# MCP tools for any MCP-aware agent host (needs the [mcp] extra), the same app
# with an added /mcp endpoint
pip install "sftp-helper[mcp]"
SFTP_HELPER_CONFIG=./settings.yaml sftp-helper-mcp

Docker image (HTTP on port 8000):

docker build -t sftp-helper .
docker run --rm -p 8000:8000 \
  -v $PWD/settings.yaml:/app/settings.yaml:ro \
  -e SFTP_HELPER_CONFIG=/app/settings.yaml \
  sftp-helper

See TRIGGERS.md for the exhaustive catalogue of phrasings, commands, and functions that invoke it (and when to reach for bucket-helper / youtube-helper instead).

There is no GUI. A forward-looking dashboard design plan (pipeline dashboard, storage-health panel, live transfer feed) lives in GUI.md, but no such code ships today.

Use it as an agent skill

skills/sftp-helper/ packages sftp-helper as a Claude Skill and an OpenCode skill, so an agent can reach for it on your behalf. See skills/README.md to install it (symlink into ~/.claude/skills/ and ~/.opencode/skills/), and TRIGGERS.md for the exhaustive trigger catalogue — scripts/check_triggers.py keeps the skill's description in sync with it, and the test suite fails if a capability stops being named.

Author

• Warith HARCHAOUI

Acknowledgements

Special thanks to Mohamed Chelali and Bachir Zerroug for fruitful discussions.

License

This project is licensed under the BSD-3-Clause License; see the LICENSE file for details.