# Bucket Helper

[🇫🇷](https://github.com/warith-harchaoui/bucket-helper/blob/main/LISEZMOI.md) · [🇬🇧](https://github.com/warith-harchaoui/bucket-helper/blob/main/README.md)

[![CI](https://github.com/warith-harchaoui/bucket-helper/actions/workflows/ci.yml/badge.svg)](https://github.com/warith-harchaoui/bucket-helper/actions/workflows/ci.yml) [![License: BSD-3-Clause](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://github.com/warith-harchaoui/bucket-helper/blob/main/LICENSE) [![Python](https://img.shields.io/badge/python-3.10%E2%80%933.13-blue.svg)](#)

`Bucket Helper` belongs to a collection of libraries called `AI Helpers` developed for building Artificial Intelligence, each published on PyPI behind its own green CI gate (pytest plus ruff, both blocking) and semantic-versioned releases.

Utility functions for **AWS S3** and any **S3-compatible object storage**: MinIO, Backblaze B2 S3 API, DigitalOcean Spaces, Cloudflare R2, Wasabi, and friends. Built on [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html). Same shape as [sftp-helper](https://github.com/warith-harchaoui/sftp-helper): a `credentials()` loader, the usual CRUD (`upload` / `download` / `delete` / `exists` / `list_prefix`), and a `remote_tempfile` context manager for stage-and-share flows.

Object storage keeps files as flat, addressable blobs — a bucket plus a key, such as `my-bucket/folder/file.txt` — instead of a nested folder tree on a hard drive. Nothing to create in advance, no limit on how many files pile up in one place, and every object reachable straight from a URL.

Amazon Web Services built the first popular version of this, S3 (Simple Storage Service), and its wire protocol became the de facto standard. MinIO, Backblaze B2, DigitalOcean Spaces, Cloudflare R2 and Wasabi all speak the same S3 API, so `bucket-helper` runs unchanged against any of them. Only the endpoint URL changes.

[🌍 AI Helpers](https://deraison.ai/ai-helpers)

[![logo](https://raw.githubusercontent.com/warith-harchaoui/bucket-helper/main/assets/logo.png)](https://deraison.ai/ai-helpers)

## The Promise

**Remote by design.** `bucket-helper` exists to move data to and from *object
storage you choose*: AWS, or any S3-compatible endpoint you point it at
(including a MinIO instance on your own network). It is deliberately **not**
local-first and ships **no GUI**. For a remote reached over SFTP instead of
S3, use `sftp-helper`; for downloading media from a URL, use `youtube-helper`.

That remote reach is also where "battle-tested" has to mean something checkable rather than a slogan.

Every push runs a blocking CI gate: the test suite exercises the S3 client against a moto-mocked backend, then ruff checks style. Nothing merges to `main` on a red run.

The package has shipped through fourteen semantic-versioned releases on PyPI, from `v0.2.2` to the current `v1.3.3` — the tag history is visible with `git tag`. It depends on `os-helper`, the small foundation package the whole AI Helpers suite shares for logging and file handling; nothing here reinvents that layer.

## Documentation

[💻 Documentation](https://deraison.ai/ai-helpers/docs/bucket-helper-doc/)

[🗺️ Landscape](https://github.com/warith-harchaoui/bucket-helper/blob/main/LANDSCAPE.md)

[📋 Examples](https://github.com/warith-harchaoui/bucket-helper/blob/main/EXAMPLES.md)

[🎯 Triggers](https://github.com/warith-harchaoui/bucket-helper/blob/main/TRIGGERS.md)

## Features

- **CRUD** against AWS S3 or any S3-compatible endpoint: `upload`, `download`,
  `delete`, `exists`, `list_prefix`.
- **Works against any S3-compatible provider**, MinIO, Backblaze B2 S3 API,
  DigitalOcean Spaces, Cloudflare R2, Wasabi, by pointing the `s3_endpoint_url`
  credential at it; no code changes per provider.
- **Credentials loader** (`credentials`) resolving a JSON / YAML file, then
  `.env`, then environment variables, in that fallback order.
- **`remote_tempfile`** context manager for stage-and-share flows: upload,
  hand back the object, auto-delete on block exit, no manual cleanup.
- **Four surfaces, one behavior**: Python library, a click CLI, a FastAPI
  HTTP surface (`[api]` extra), and an MCP server (`[mcp]` extra). See the
  [multi-surface section](#multi-surface-exposure).
- **Docker image** ships the HTTP server ready to run.

## How it works

![How bucket-helper works: credentials in, objects in the storage you chose out](https://raw.githubusercontent.com/warith-harchaoui/bucket-helper/main/assets/diagrams/how-it-works.png)

## Installation

**Prerequisites**: **Python 3.10–3.13** and **git**, cross-platform:

- 🍎 **macOS** ([Homebrew](https://brew.sh)): `brew install python git`
- 🐧 **Ubuntu/Debian**: `sudo apt update && sudo apt install -y python3 python3-pip git`
- 🪟 **Windows** (PowerShell): `winget install Python.Python.3.12 Git.Git`

We recommend using Python environments. Check this link if you're unfamiliar with setting one up: [🥸 Tech tips](https://deraison.ai/4ml/#install).

### From PyPI (recommended)

```bash
# Core library (credentials loader + CRUD + remote_tempfile)
pip install bucket-helper

# Optional surfaces
pip install "bucket-helper[api]"       # FastAPI HTTP surface
pip install "bucket-helper[mcp]"       # MCP server (pulls [api] in)
```

### From source (no PyPI)

```bash
git clone https://github.com/warith-harchaoui/bucket-helper.git
cd bucket-helper
pip install -e .

# Optional surfaces
pip install -e ".[api]"
pip install -e ".[mcp]"
```

The CLI ships with the core install.

## Configuration

A ready-to-fill template is committed at [`settings.yaml.example`](https://github.com/warith-harchaoui/bucket-helper/blob/main/settings.yaml.example). Copy it to `settings.yaml` and edit in place: `settings.yaml` is gitignored, so you cannot accidentally commit secrets.

```bash
cp settings.yaml.example settings.yaml
# then edit settings.yaml with your AWS / MinIO / R2 / B2 credentials
```

You may also write JSON instead of YAML, use a `.env`, or set environment variables; `bucket-helper` falls back in that order via `os_helper.get_config`. Required keys:

```json
{
  "s3_access_key": "AKIA...",
  "s3_secret_key": "...",
  "s3_bucket":     "my-bucket",
  "s3_https":      "https://my-bucket.s3.eu-west-3.amazonaws.com"
}
```

Optional keys:

| Key | Default | Notes |
|---|---|---|
| `s3_region` | `"us-east-1"` | AWS region; mostly cosmetic for MinIO / R2 |
| `s3_endpoint_url` | empty (= AWS S3) | Set this for S3-compatible backends: see table below |
| `s3_prefix` | empty | Default key prefix added by `upload(...)` when no destination is given, and by `remote_tempfile` |
| `s3_use_path_style` | `"false"` | Force path-style addressing (`endpoint/bucket/key` instead of `bucket.endpoint/key`). Typical for MinIO with custom domains. |
| `s3_verify_ssl` | `"true"` | Disable only for dev MinIO with self-signed certs |

From the environment, every key is read under its upper-case name (`S3_BUCKET`,
`S3_ENDPOINT_URL`) or exactly as written (`s3_bucket`).

## Endpoint URLs for common S3-compatible storage

Set `s3_endpoint_url` to:

| Provider | Endpoint |
|---|---|
| **AWS S3** | leave empty / unset |
| **MinIO** | `http://minio.example.com:9000` (or `https://...` with TLS) |
| **DigitalOcean Spaces** | `https://nyc3.digitaloceanspaces.com` (region in subdomain) |
| **Cloudflare R2** | `https://<account_id>.r2.cloudflarestorage.com` |
| **Backblaze B2 (S3 API)** | `https://s3.<region>.backblazeb2.com` |
| **Wasabi** | `https://s3.<region>.wasabisys.com` |

## Quick start

For the full catalog of recipes (uploads / downloads / listings, S3-compatible endpoints such as MinIO / R2 / B2 / Spaces / Wasabi, temporary remote keys with auto-cleanup, mirroring with sftp-helper), see [📋 EXAMPLES.md](https://github.com/warith-harchaoui/bucket-helper/blob/main/EXAMPLES.md).

```python
import bucket_helper as bh

# Load creds: JSON / YAML file, then .env, then environment (auto-fallback in that order)
cred = bh.credentials("path/to/settings.yaml")

# Upload a local file
uri = bh.upload("local.txt", cred, "folder/uploaded.txt")
# uri == "s3://my-bucket/folder/uploaded.txt"

assert bh.exists(uri, cred)

# Download
bh.download(uri, "downloaded.txt", cred)

# List (returns at most max_keys, 1000 by default)
for key in bh.list_prefix("folder/", cred):
    print(key)

# List and find out whether anything was left behind
keys, truncated = bh.list_prefix_page("folder/", cred, max_keys=100)
if truncated:
    print("there are more than 100 objects under folder/")

# Delete
bh.delete(uri, cred)
```

## Listing is capped, and says so

`list_prefix` returns **at most `max_keys` keys** (1000 by default). That is a
deliberate bound, not a bug — S3 pages its responses and an unbounded listing of
a large prefix is rarely what a caller wants.

What it cannot do is tell you whether the cap was reached. A response of 1000
keys looks identical whether the prefix holds exactly 1000 objects or fifty
thousand, which matters as soon as a program acts on the list rather than a
human reading it. Use `list_prefix_page` when that distinction counts:

```python
keys, truncated = bh.list_prefix_page("logs/", cred, max_keys=500)
if truncated:
    ...  # there is more behind these 500
```

It asks S3 for one key beyond the cap and reports whether it arrived, so the
extra cost is a single key. `bucket-helper list` prints a warning to **stderr**
when a listing is truncated (keys stay on stdout, so pipelines are unaffected),
and `POST /list` returns `truncated` alongside `keys` and `count`.

## MinIO example

```python
cred = {
    "s3_access_key": "minioadmin",
    "s3_secret_key": "minioadmin",
    "s3_bucket": "uploads",
    "s3_https": "http://minio.example.com:9000/uploads",
    "s3_endpoint_url": "http://minio.example.com:9000",
    "s3_use_path_style": "true",
    "s3_region": "us-east-1",  # MinIO accepts any region string
}

bh.make_bucket("uploads", cred)
bh.upload("file.bin", cred, "file.bin")
```

## Stage-and-share with `remote_tempfile`

Drop a generated file at a unique random key, hand the public URL to a
downstream worker / webhook, and the object is deleted on block exit
(even if the body raises):

```python
import bucket_helper as bh
import requests

cred = bh.credentials("path/to/settings.yaml")

with bh.remote_tempfile(cred, ext="json", prefix="runs") as (s3_addr, public_url):
    bh.upload("payload.json", cred, s3_addr, content_type="application/json")
    # Hand the URL to something that fetches it once.
    requests.post(
        "https://hook.example.com/process", json={"input_url": public_url}
    ).raise_for_status()
# Object is gone here, no manual cleanup.
```

## Multi-surface exposure

The eight storage verbs — `upload`, `download`, `exists`, `list`, `delete`,
`make-bucket`, `tempfile`, `strip-path` — are each reachable three ways
beyond the Python functions themselves:

- **CLI**: `bucket-helper <subcommand>`, built with click (installed by default).
- **FastAPI HTTP**: `uvicorn bucket_helper.api:app --host 0.0.0.0 --port 8000` (install `[api]` extra).
- **MCP**: `bucket-helper-mcp` exposes the same HTTP surface as MCP tools for
  any MCP-aware agent host (install `[mcp]` extra). It serves the API and the
  MCP endpoint `/mcp` on `127.0.0.1:8000`; `BUCKET_HELPER_HOST` and
  `BUCKET_HELPER_PORT` change that address.

There is one command line, built with click: the `bucket-helper-click` twin and
the argparse implementation it shadowed were both removed in `v1.3.0`, and
`bucket-helper-click <verb>` is now spelled `bucket-helper <verb>`. Its eight
verb names are the ones the FastAPI surface introspects, so the two doors cannot
answer differently.

`credentials` and `get_client_s3` stay library-only, deliberately: one resolves
secrets, the other hands back a live boto3 client. Neither belongs behind an
HTTP endpoint.

The exhaustive catalogue of what triggers the toolkit (natural-language
phrasings, commands, functions, address cues, and explicit SKIP rules) lives in
[TRIGGERS.md](https://github.com/warith-harchaoui/bucket-helper/blob/main/TRIGGERS.md).

### Use it as an agent skill

`skills/bucket-helper/` packages bucket-helper as a **Claude Skill** *and* an **OpenCode
skill**, so an agent can reach for it on your behalf. See
[`skills/README.md`](https://github.com/warith-harchaoui/bucket-helper/blob/main/skills/README.md) to install it (symlink into
`~/.claude/skills/` and `~/.opencode/skills/`), and
[`TRIGGERS.md`](https://github.com/warith-harchaoui/bucket-helper/blob/main/TRIGGERS.md) for the exhaustive trigger catalogue —
`scripts/check_triggers.py` keeps the skill's description in sync with it, and
the test suite fails if a capability stops being named.

## CLI examples

```bash
bucket-helper upload      --config settings.yaml --input local.txt --key folder/uploaded.txt
bucket-helper exists      --config settings.yaml --key folder/uploaded.txt
bucket-helper download    --config settings.yaml --key folder/uploaded.txt --output back.txt
bucket-helper list        --config settings.yaml --prefix folder/
bucket-helper delete      --config settings.yaml --key folder/uploaded.txt
bucket-helper make-bucket --config settings.yaml --bucket new-bucket
bucket-helper tempfile    --config settings.yaml --ext json --prefix runs
bucket-helper strip-path  --config settings.yaml --address s3://my-bucket/path/to/obj
```

## HTTP server

```bash
# Serve HTTP (default credentials picked up from BUCKET_HELPER_CONFIG)
BUCKET_HELPER_CONFIG=$PWD/settings.yaml uvicorn bucket_helper.api:app --host 0.0.0.0 --port 8000
# → Swagger UI at http://localhost:8000/docs
```

Per-request credentials can also be sent as multipart form fields
(`s3_access_key` / `s3_secret_key` / `s3_bucket` / `s3_https` / …).

Routes: `GET /health`, then one `POST` per verb (`/upload`, `/download`,
`/delete`, `/exists`, `/list`, `/make-bucket`, `/tempfile`, `/strip-path`),
all taking form fields. Each answers JSON except `/download`, which streams the
file back. A malformed `s3://` address or a missing credential returns 400; a
failure on the S3 side returns 502.

## Docker

```bash
docker build -t bucket-helper .
docker run --rm -p 8000:8000 \
  -e BUCKET_HELPER_CONFIG=/config/settings.yaml \
  -v $PWD/settings.yaml:/config/settings.yaml:ro \
  bucket-helper
```

Or through Compose, which carries the port, the credential mount and a health
probe for you — and, behind `--profile dev`, a local MinIO to talk to:

```bash
docker compose up --build
docker compose --profile dev up --build
```

See also: [TRIGGERS.md](https://github.com/warith-harchaoui/bucket-helper/blob/main/TRIGGERS.md) (what invokes the toolkit) and
[GUI.md](https://github.com/warith-harchaoui/bucket-helper/blob/main/GUI.md) (visual product design plan; no GUI ships, bucket-helper is remote object-storage plumbing).

## Author

 - [Warith HARCHAOUI](https://linkedin.com/in/warith-harchaoui)

## Acknowledgements

Special thanks to [Mohamed Chelali](https://mchelali.github.io) and [Bachir Zerroug](https://www.linkedin.com/in/bachirzerroug) for fruitful discussions.

## License

This project is licensed under the BSD-3-Clause License; see the [LICENSE](https://github.com/warith-harchaoui/bucket-helper/blob/main/LICENSE) file for details.