Bucket Helper

🇫🇷 · 🇬🇧

Bucket Helper belongs to a collection of libraries called AI Helpers developed for building Artificial Intelligence, each published on PyPI behind its own green CI gate (pytest plus ruff, both blocking) and semantic-versioned releases.

Utility functions for AWS S3 and any S3-compatible object storage: MinIO, Backblaze B2 S3 API, DigitalOcean Spaces, Cloudflare R2, Wasabi, and friends. Built on boto3. Same shape as sftp-helper: a credentials() loader, the usual CRUD (upload / download / delete / exists / list_prefix), and a remote_tempfile context manager for stage-and-share flows.

Object storage keeps files as flat, addressable blobs — a bucket plus a key, such as my-bucket/folder/file.txt — instead of a nested folder tree on a hard drive. Nothing to create in advance, no limit on how many files pile up in one place, and every object reachable straight from a URL.

Amazon Web Services built the first popular version of this, S3 (Simple Storage Service), and its wire protocol became the de facto standard. MinIO, Backblaze B2, DigitalOcean Spaces, Cloudflare R2 and Wasabi all speak the same S3 API, so bucket-helper runs unchanged against any of them. Only the endpoint URL changes.

🌍 AI Helpers

The Promise

Remote by design. bucket-helper exists to move data to and from object storage you choose: AWS, or any S3-compatible endpoint you point it at (including a MinIO instance on your own network). It is deliberately not local-first and ships no GUI. For a remote reached over SFTP instead of S3, use sftp-helper; for downloading media from a URL, use youtube-helper.

That remote reach is also where "battle-tested" has to mean something checkable rather than a slogan.

Every push runs a blocking CI gate: the test suite exercises the S3 client against a moto-mocked backend, then ruff checks style. Nothing merges to main on a red run.

The package has shipped through fourteen semantic-versioned releases on PyPI, from v0.2.2 to the current v1.3.3 — the tag history is visible with git tag. It depends on os-helper, the small foundation package the whole AI Helpers suite shares for logging and file handling; nothing here reinvents that layer.

Related MCP server: MinIO MCP Server

Documentation

💻 Documentation

🗺️ Landscape

📋 Examples

🎯 Triggers

Features

• CRUD against AWS S3 or any S3-compatible endpoint: upload, download, delete, exists, list_prefix.

• Works against any S3-compatible provider, MinIO, Backblaze B2 S3 API, DigitalOcean Spaces, Cloudflare R2, Wasabi, by pointing the s3_endpoint_url credential at it; no code changes per provider.

• Credentials loader (credentials) resolving a JSON / YAML file, then .env, then environment variables, in that fallback order.

• remote_tempfile context manager for stage-and-share flows: upload, hand back the object, auto-delete on block exit, no manual cleanup.

• Four surfaces, one behavior: Python library, a click CLI, a FastAPI HTTP surface ([api] extra), and an MCP server ([mcp] extra). See the multi-surface section.

• Docker image ships the HTTP server ready to run.

How it works

Installation

Prerequisites: Python 3.10–3.13 and git, cross-platform:

• 🍎 macOS (Homebrew): brew install python git

• 🐧 Ubuntu/Debian: sudo apt update && sudo apt install -y python3 python3-pip git

• 🪟 Windows (PowerShell): winget install Python.Python.3.12 Git.Git

We recommend using Python environments. Check this link if you're unfamiliar with setting one up: 🥸 Tech tips.

From PyPI (recommended)

# Core library (credentials loader + CRUD + remote_tempfile)
pip install bucket-helper

# Optional surfaces
pip install "bucket-helper[api]"       # FastAPI HTTP surface
pip install "bucket-helper[mcp]"       # MCP server (pulls [api] in)

From source (no PyPI)

git clone https://github.com/warith-harchaoui/bucket-helper.git
cd bucket-helper
pip install -e .

# Optional surfaces
pip install -e ".[api]"
pip install -e ".[mcp]"

The CLI ships with the core install.

Configuration

A ready-to-fill template is committed at settings.yaml.example. Copy it to settings.yaml and edit in place: settings.yaml is gitignored, so you cannot accidentally commit secrets.

cp settings.yaml.example settings.yaml
# then edit settings.yaml with your AWS / MinIO / R2 / B2 credentials

You may also write JSON instead of YAML, use a .env, or set environment variables; bucket-helper falls back in that order via os_helper.get_config. Required keys:

{
  "s3_access_key": "AKIA...",
  "s3_secret_key": "...",
  "s3_bucket":     "my-bucket",
  "s3_https":      "https://my-bucket.s3.eu-west-3.amazonaws.com"
}

Optional keys:

From the environment, every key is read under its upper-case name (S3_BUCKET, S3_ENDPOINT_URL) or exactly as written (s3_bucket).

Endpoint URLs for common S3-compatible storage

Set s3_endpoint_url to:

Quick start

For the full catalog of recipes (uploads / downloads / listings, S3-compatible endpoints such as MinIO / R2 / B2 / Spaces / Wasabi, temporary remote keys with auto-cleanup, mirroring with sftp-helper), see 📋 EXAMPLES.md.

import bucket_helper as bh

# Load creds: JSON / YAML file, then .env, then environment (auto-fallback in that order)
cred = bh.credentials("path/to/settings.yaml")

# Upload a local file
uri = bh.upload("local.txt", cred, "folder/uploaded.txt")
# uri == "s3://my-bucket/folder/uploaded.txt"

assert bh.exists(uri, cred)

# Download
bh.download(uri, "downloaded.txt", cred)

# List (returns at most max_keys, 1000 by default)
for key in bh.list_prefix("folder/", cred):
    print(key)

# List and find out whether anything was left behind
keys, truncated = bh.list_prefix_page("folder/", cred, max_keys=100)
if truncated:
    print("there are more than 100 objects under folder/")

# Delete
bh.delete(uri, cred)

Listing is capped, and says so

list_prefix returns at most max_keys keys (1000 by default). That is a deliberate bound, not a bug — S3 pages its responses and an unbounded listing of a large prefix is rarely what a caller wants.

What it cannot do is tell you whether the cap was reached. A response of 1000 keys looks identical whether the prefix holds exactly 1000 objects or fifty thousand, which matters as soon as a program acts on the list rather than a human reading it. Use list_prefix_page when that distinction counts:

keys, truncated = bh.list_prefix_page("logs/", cred, max_keys=500)
if truncated:
    ...  # there is more behind these 500

It asks S3 for one key beyond the cap and reports whether it arrived, so the extra cost is a single key. bucket-helper list prints a warning to stderr when a listing is truncated (keys stay on stdout, so pipelines are unaffected), and POST /list returns truncated alongside keys and count.

MinIO example

cred = {
    "s3_access_key": "minioadmin",
    "s3_secret_key": "minioadmin",
    "s3_bucket": "uploads",
    "s3_https": "http://minio.example.com:9000/uploads",
    "s3_endpoint_url": "http://minio.example.com:9000",
    "s3_use_path_style": "true",
    "s3_region": "us-east-1",  # MinIO accepts any region string
}

bh.make_bucket("uploads", cred)
bh.upload("file.bin", cred, "file.bin")

Stage-and-share with remote_tempfile

Drop a generated file at a unique random key, hand the public URL to a downstream worker / webhook, and the object is deleted on block exit (even if the body raises):

import bucket_helper as bh
import requests

cred = bh.credentials("path/to/settings.yaml")

with bh.remote_tempfile(cred, ext="json", prefix="runs") as (s3_addr, public_url):
    bh.upload("payload.json", cred, s3_addr, content_type="application/json")
    # Hand the URL to something that fetches it once.
    requests.post(
        "https://hook.example.com/process", json={"input_url": public_url}
    ).raise_for_status()
# Object is gone here, no manual cleanup.

Multi-surface exposure

The eight storage verbs — upload, download, exists, list, delete, make-bucket, tempfile, strip-path — are each reachable three ways beyond the Python functions themselves:

• CLI: bucket-helper <subcommand>, built with click (installed by default).

• FastAPI HTTP: uvicorn bucket_helper.api:app --host 0.0.0.0 --port 8000 (install [api] extra).

• MCP: bucket-helper-mcp exposes the same HTTP surface as MCP tools for any MCP-aware agent host (install [mcp] extra). It serves the API and the MCP endpoint /mcp on 127.0.0.1:8000; BUCKET_HELPER_HOST and BUCKET_HELPER_PORT change that address.

There is one command line, built with click: the bucket-helper-click twin and the argparse implementation it shadowed were both removed in v1.3.0, and bucket-helper-click <verb> is now spelled bucket-helper <verb>. Its eight verb names are the ones the FastAPI surface introspects, so the two doors cannot answer differently.

credentials and get_client_s3 stay library-only, deliberately: one resolves secrets, the other hands back a live boto3 client. Neither belongs behind an HTTP endpoint.

The exhaustive catalogue of what triggers the toolkit (natural-language phrasings, commands, functions, address cues, and explicit SKIP rules) lives in TRIGGERS.md.

Use it as an agent skill

skills/bucket-helper/ packages bucket-helper as a Claude Skill and an OpenCode skill, so an agent can reach for it on your behalf. See skills/README.md to install it (symlink into ~/.claude/skills/ and ~/.opencode/skills/), and TRIGGERS.md for the exhaustive trigger catalogue — scripts/check_triggers.py keeps the skill's description in sync with it, and the test suite fails if a capability stops being named.

CLI examples

bucket-helper upload      --config settings.yaml --input local.txt --key folder/uploaded.txt
bucket-helper exists      --config settings.yaml --key folder/uploaded.txt
bucket-helper download    --config settings.yaml --key folder/uploaded.txt --output back.txt
bucket-helper list        --config settings.yaml --prefix folder/
bucket-helper delete      --config settings.yaml --key folder/uploaded.txt
bucket-helper make-bucket --config settings.yaml --bucket new-bucket
bucket-helper tempfile    --config settings.yaml --ext json --prefix runs
bucket-helper strip-path  --config settings.yaml --address s3://my-bucket/path/to/obj

HTTP server

# Serve HTTP (default credentials picked up from BUCKET_HELPER_CONFIG)
BUCKET_HELPER_CONFIG=$PWD/settings.yaml uvicorn bucket_helper.api:app --host 0.0.0.0 --port 8000
# → Swagger UI at http://localhost:8000/docs

Per-request credentials can also be sent as multipart form fields (s3_access_key / s3_secret_key / s3_bucket / s3_https / …).

Routes: GET /health, then one POST per verb (/upload, /download, /delete, /exists, /list, /make-bucket, /tempfile, /strip-path), all taking form fields. Each answers JSON except /download, which streams the file back. A malformed s3:// address or a missing credential returns 400; a failure on the S3 side returns 502.

Docker

docker build -t bucket-helper .
docker run --rm -p 8000:8000 \
  -e BUCKET_HELPER_CONFIG=/config/settings.yaml \
  -v $PWD/settings.yaml:/config/settings.yaml:ro \
  bucket-helper

Or through Compose, which carries the port, the credential mount and a health probe for you — and, behind --profile dev, a local MinIO to talk to:

docker compose up --build
docker compose --profile dev up --build

See also: TRIGGERS.md (what invokes the toolkit) and GUI.md (visual product design plan; no GUI ships, bucket-helper is remote object-storage plumbing).

Author

• Warith HARCHAOUI

Acknowledgements

Special thanks to Mohamed Chelali and Bachir Zerroug for fruitful discussions.

License

This project is licensed under the BSD-3-Clause License; see the LICENSE file for details.