agent-bucket

Make your object store searchable by AI agents.

A lightweight retrieval sidecar for existing S3-compatible object stores. Sync a bucket prefix, then retrieve source-backed chunks through REST, MCP or a small web console. Keep your source objects where they are; parsed text and vectors live in a local derived index.

Status: V0.1 starter / developer preview. Single-node first, scale-out by design. This repository supplies a runnable foundation and an explicit hardening roadmap. It is not a stable production release. No throughput or scale claims have been published.

The problem

Your knowledge already lives in an object store. Giving an agent useful context often means writing custom ingestion code and running a separate search stack. agent-bucket packages those boundaries into a sidecar you can inspect, run and extend.

Related MCP server: minio-mcp-server

Quickstart: offline demo

Prerequisites: Docker Engine + Compose, or Python 3.12, uv, Node 22.13+ and npm.

git clone https://github.com/Sanjeev-Karthick/agent-bucket.git
cd agent-bucket
cp .env.example .env
docker compose up --build -d

Open http://localhost:8000. Use the token from .env (local-demo-token-change-me for the supplied demo), click Sync source, then search How do I rotate access keys?. Results include source URI, chunk and score. The initial image build installs optional ML/parser packages and may take several minutes.

The default demo uses hash vectors: lexical fixtures, not semantic embeddings. It needs no model downloads. BM25 works normally; vector/hybrid modes exercise the pipeline with fixture vectors. Enable FastEmbed before evaluating semantic retrieval.

Without Docker:

make setup
make build
make demo

Open http://localhost:8000. For UI development, run make ui in another terminal and open the Vite URL; its /v1 requests proxy to the backend. Sync is explicit: the service does not ingest or schedule jobs automatically.

Demo from the API

export AB_API_TOKEN=local-demo-token-change-me
curl -sS -X POST http://localhost:8000/v1/sync \
  -H "Authorization: Bearer $AB_API_TOKEN"
curl -sS http://localhost:8000/v1/jobs \
  -H "Authorization: Bearer $AB_API_TOKEN"

Wait for the job to report succeeded, then query:

curl -sS http://localhost:8000/v1/search \
  -H "Authorization: Bearer $AB_API_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"query":"rotate access keys","mode":"keyword","limit":3}'

Interactive REST reference: http://localhost:8000/docs. GET /v1/status reports the source, embedder and counts. POST /v1/sync returns a durable job; concurrent active sync requests share the same job. GET /v1/jobs/{id} reports progress at completion.

Connect an agent through MCP

Use a client supporting Streamable HTTP at http://localhost:8000/mcp/ and set Authorization: Bearer <AB_API_TOKEN> in its request headers. Client configuration syntax varies; do not put production secrets in committed client configuration.

Tools: bucket_status and search_objects. Example search_objects arguments:

{"request":{"query":"rotate access keys","mode":"hybrid","limit":5}}

Both tools are retrieval-only. Starting a sync uses the operator REST API/UI. Treat retrieved content as untrusted input, preserve citations and keep authorization decisions outside document instructions.

Use your S3-compatible store

Install real inference and parsing for a local development deployment:

uv sync --frozen --extra dev --extra inference --extra parsing

Set the following in .env, provide read-only credentials via boto3's standard credential chain, then run make dev. The Docker image already includes both extras.

AB_SOURCE=s3
AB_S3_BUCKET=your-knowledge-bucket
AB_S3_PREFIX=docs/
# Omit AB_S3_ENDPOINT for AWS S3; set it for a compatible provider.
AB_EMBEDDING_PROVIDER=fastembed
AB_DATA_DIR=.data-real
AB_API_TOKEN=replace-with-a-long-random-secret

Use a fresh data directory when changing source, model, dimensions or chunking. For Compose, choose a new named data volume in compose.yaml; host AB_DATA_DIR does not override its /data mount. Model downloads occur on first use. Check operations for configuration, backup and troubleshooting and SECURITY.md before using private data.

Architecture

flowchart TB
  store[S3-compatible store / demo files] --> adapter[StorageAdapter]
  subgraph sidecar[One container · one process · one local volume]
    adapter --> sync[SyncPlanner + one worker]
    state[SQLite WAL: jobs and manifest] <--> sync
    sync --> parser[Text / Docling parser]
    parser --> chunks[Character chunks + FastEmbed]
    chunks --> index[LanceDB: BM25 + vectors]
    index --> retrieve[RetrievalService: keyword / vector / hybrid]
    retrieve --> rest[FastAPI]
    retrieve --> mcp[Official Python MCP SDK]
    rest --> ui[React / Vite / TypeScript UI]
  end
  mcp --> agents[AI agents]

See architecture and ADRs for job recovery, publication semantics and adapter boundaries.

Current scope

The byte/object caps are input limits, not a tested capacity or a performance promise. Index data is a sensitive derived copy; deletion does not purge historical snapshots/backups.

Scale-out story

V0.1 has a vertical ceiling: one machine's CPU, RAM and disk. Clear StorageAdapter, JobRepository, SyncPlanner and RetrievalService boundaries leave room to later separate indexing and query workers. That work still requires an external control plane, leases/fencing, immutable index generations and consistent query publication. Running several replicas against the local volume is unsupported. Read the roadmap for the proposed path and unresolved design work.

Develop and contribute

make setup
make check
docker compose -f compose.minio.yaml up -d
make integration

The default suite uses synthetic data and no model downloads. See test coverage and MinIO plan, benchmark methodology, contribution guide, starter issues, release checklist and launch workspace.

backend/agent_bucket/   adapters, control state, worker, retrieval, REST/MCP
frontend/              React console, component tests, build tooling
tests/                 unit/API tests and MinIO integration harness
examples/demo-data/    synthetic, safe-to-share demo corpus
scripts/               local MinIO data seeding
benchmarks/            repeatable harness and methodology; no claimed results
docs/                  architecture, ADRs, operations, release and launch notes
.github/               CI, issue/PR templates, dependency updates

Licensed under Apache-2.0. Community help: SUPPORT.md. Please follow our code of conduct. Report vulnerabilities privately using SECURITY.md.