wiki4llm

Git-to-RAG knowledge for coding agents: a Python ingestion CLI, an authenticated MCP gateway, and a reusable GitHub Actions workflow in one repository.

• Deploy the stack — OpenAI, Cloudflare, credentials and first index.

• Connect a repository — workflow inputs and optional GitHub token.

• Operate the gate — Cloudflare or native Node/QMD runtime.

• Develop and verify — local checks and public release process.

The Python command remains wiki-rag. It ingests Git-tracked text and explicitly selected PDFs into OpenAI Retrieval and publishes an active-file registry for agents. Cloudflare stores the remote journal and serves the MCP endpoint; no persistent indexing machine is required.

Requires Python 3.14, managed with uv, and a POSIX host (macOS or Linux). The lockfile pins application and development dependencies. The CLI is an ephemeral process; the separately deployed gate lives in gate/. The repository does not include a hosted account or service subscription.

For CI, the shared GitHub Actions workflow accepts a committed TOML and gate/OpenAI secrets from each caller. Its separate GitHub checkout token is optional. Consumer installation and credential setup are explicit operator steps.

Quick start

uv sync --locked
cp wiki-rag.example.toml wiki-rag.toml
uv run wiki-rag --config wiki-rag.toml plan

Edit the configuration to reference a local Git repository, your OpenAI Project and an existing empty vector store in that project. Set the configured WIKI_RAG_INGEST_KEY environment variable through your credential-management workflow. Never put a key in TOML or command arguments.

uv run wiki-rag index --config wiki-rag.toml --apply
uv run wiki-rag update --config wiki-rag.toml
uv run wiki-rag update --config wiki-rag.toml --apply
uv run wiki-rag status --config wiki-rag.toml

--config works before or after the subcommand. In local-state mode, plan, status, and bare index/update make zero remote requests and require no credentials. Remote-state previews make read-only authority requests. index --apply initializes new ownership state; thereafter index --apply and update --apply converge using that state. update --apply refuses missing state. JSON output reports selection counts/bytes, additions, replacements, deletions, skipped files and reasons. Exit code 2 means the operation failed; interruption returns 130. In local mode, cleanup failures after a successful publication are reported as cleanup_pending in otherwise successful JSON output. Remote-mode cleanup failures exit with code 2 and report the committed publication ID.

For installation from a built wheel:

uv build
uv tool install --python 3.14 dist/wiki_rag-0.1.0-py3-none-any.whl
wiki-rag --help

Related MCP server: CodeBrain

Configuration and selection

See wiki-rag.example.toml. TOML schema version 1 is strict: unknown keys and literal API-key fields are rejected. Roots and the state directory resolve relative to the configuration file. Parent-relative paths such as ../handbook and ../state are normalized lexically; every existing component is checked first, so a symlink in a segment later cancelled by .. is rejected. Each named collection has an independent stable repository identity, local root, optional branch, and existing vector_store_id; these are distinct concepts. Multiple collections can contain README.md without collision.

Only Git-tracked paths are considered. In local mode, working-tree bytes are captured, hashed, and uploaded; staged versions do not override working-tree contents. Each file's source_commit is set only when those exact bytes match its HEAD blob; otherwise it is null. Repository HEAD is separately recorded for ancestry and concurrent-change guards. A configured branch must be the checked-out branch; detached HEAD is rejected when branch is configured. Remote mode requires full history, an index equal to HEAD and selected literal HEAD blob bytes; see the remote source policy.

Both the include glob and extension allowlist must match. Excludes always win. Globs are case-sensitive, root-relative POSIX paths: *, ?, and bracket classes match within one segment; a standalone ** segment matches zero or more segments. **/*.md includes root-level Markdown. There is no brace expansion, negation, shell expansion or configuration interpolation.

The scanner skips symlinks, Git submodules, nested repositories, .git, local state, .env*, common credential/key names, recognized private keys and common API-token patterns. It never follows symlinks or executes repository hooks or fsmonitor code. These checks are a conservative baseline, not a general-purpose secret detector; review the selected source before uploading it.

Text handling accepts UTF-8. Known text suffixes include Markdown, plain text, reStructuredText, common source code, JSON, YAML, TOML, HTML, XML and CSV; see TEXT_SUFFIXES in selection.py for the exact set. Eligible text is sent unchanged as a uniquely named .txt file. The original path and kind remain in attributes and the manifest. LFS pointers, NUL-containing or non-UTF-8 text, and unsupported formats are reported as skipped.

PDFs are supported only when .pdf is explicitly present in the collection's extension allowlist and its include globs match. They are sent with their original raw bytes, a .pdf filename and application/pdf MIME type. A bounded check requires a PDF 1.0–1.7 or 2.0 header and a trailing %%EOF marker within the last 1024 bytes. This is a sanity check, not structural PDF validation; provider completed-ingestion status is still required. Invalid PDFs are reported as skipped. PDF parsing/extraction is provider-owned (pdf-native-v1); no local parser runs. Office documents, images, archives and other unsupported binary formats are never relabelled as text.

Selected unreadable or oversized files abort before uploads. Defaults are 2 MB per file, 50 MB total, and 10,000 selected files per run; configurable ceilings are 20 MB, 200 MB, and 100,000 files. Limits are decimal bytes. Chunking is explicitly static: default 800 tokens with 400 overlap, configurable within 100–4096 tokens and overlap at most half the chunk size. Content hash and per-file preprocessing and chunking configuration jointly determine a file version.

Durability and recovery

In local mode, the private state directory is mode 0700; JSON and lock files are created with mode 0600. One POSIX process lock serializes writes. Keep the entire directory, including hidden .initialized, together on durable storage. state.json is a checksummed ownership journal. manifest.json is the independently atomically replaced published snapshot. State is not an evictable cache.

Initial indexing requires empty vector stores to avoid silently replacing a lost journal or duplicating an existing corpus. Missing, corrupt or differently bound ownership state fails closed. Collection names, repository identities, branch bindings, project and vector stores cannot be changed under an existing state directory. Restore the original configuration/state or create fresh empty stores with new state. Moving a local repository root is allowed. Adding/removing collections under existing state is deliberately outside this MVP.

New and changed files are uploaded once, attached, and polled until completed. All configured collections publish together only after every selected file succeeds and a second source scan matches the captured selection. The previous manifest survives prepublication failure. Unchanged updates upload nothing and retain the publication identity. Source revisions must descend from the last published repository commit; stale, divergent and unavailable ancestry fails closed. Local uncommitted updates are serialized and rescanned, but independent workers must share one durable authority rather than clone stale state.

Creates are never automatically retried after ambiguous transport/HTTP failures. Intent is persisted before each create and resource IDs immediately afterward. Reruns reconcile unknown uploads using their persisted unique filename and exact content hash, and unknown attachments using store/file IDs and operation metadata. If no unique matching result is visible, the run stops instead of guessing. Retry reconciliation later; unresolved outcomes require operator investigation. Explicit rejected requests such as HTTP 403/429 may be attempted again on a subsequent run. Reads and idempotent deletes retry at most three times with bounded backoff. Polling makes at most 60 status checks, separated by two seconds (and bounded read retries); resume a timed-out poll with the same state.

Confirmed terminal failed/cancelled ingestion is journaled separately from unknown poll outcomes. The failing run preserves the prior publication. On a subsequent run, its owned failed resources are retired before a new upload/attach attempt begins. There are at most three ingestion attempts for an unsuccessful logical version, counted durably across restarts. Exhaustion stops for operator investigation; it never loops indefinitely. Timeouts, transport failures and missing/ambiguous status do not authorize a fresh create.

After publication, cleanup deletes only journal-owned obsolete attachments and files; IDs retained by any collection are protected. Both attachment and File deletion intents are persisted before their respective remote requests. A version whose retirement started cannot be reused until those idempotent deletions finish; if deletion is unavailable, the prior publication remains active. Retired ownership receipts are preserved before reusing a logical version key. Cleanup is restartable and may remain pending. Unresolved obsolete create intents are retained and reported. Do not manually share these owned file IDs with unrelated stores or modify managed remote resources out of band. Deleted IDs remain in the journal as recovery history; this MVP does not compact it.

For ephemeral CI runners, use the remote authority backend. Local mode requires a durable shared state volume or a central ingestion worker. Restoring a stale backup while newer writers exist can defeat ordering; do not run multiple independent state copies against the same stores. Native Windows and distributed/network-filesystem locking have not been qualified.

Permission-gate integration

The read-only contract is docs/manifest-contract.md. In local mode, import one complete manifest.json snapshot into the gate's own authorization registry. Remote mode publishes atomically through its ingestion authority and does not use a separate import step. Never import the private journal or infer identity from filenames. Apply project/collection authorization and active-file filtering before any retrieved content reaches a model or caller.

Remote replacement is not atomic. Staged files are remotely searchable before publication, and provider deletion is eventually consistent. The registry lets a gate reject both staged and stale IDs. Use separate ingestion and gate credentials, constrained to the intended OpenAI Project. This tool sends an explicit OpenAI-Project header to the fixed https://api.openai.com/v1/ origin; base-URL overrides, ambient proxy settings and redirects are not used. It needs normal project Files/Vector Stores permissions, never organization admin keys.

Development and release checks

uv sync --locked
uv run python --version
uv run pytest -q
uv run ruff check .
uv run ruff format --check .
uv build
uv run python scripts/check_public.py

Stage the intended payload before running the public check. It snapshots the index with git write-tree and scans the resulting immutable Git blobs and modes, not working-tree bytes. The output identifies the exact checked tree SHA; compare it with the eventual commit tree (git rev-parse 'HEAD^{tree}') and rerun after restaging. Unstaged sanitizing edits cannot hide staged sensitive content. Pass repeatable --forbid arguments for organization-specific names or private paths. The check reports filenames and reasons, never suspected credential contents. Review the full staged diff, Git author metadata, dependency lock, wheel and source archive before publication. Automated pattern checks do not establish absence of all sensitive information. No remote publication is performed by these commands.

Tests use synthetic Git repositories, a recording fake provider and HTTPX's mock transport. They cover selection, preview isolation, lock contention, incremental changes, ownership, failure preservation, recovery, source provenance, fixed origin/headers, pagination and bounded retries. A manual synthetic live procedure is in docs/live-smoke.md; it is opt-in and may incur API storage charges. Local tests do not claim live cloud acceptance.

API grounding: Files upload, Vector Store attachments, and Retrieval lifecycle.

Licensed under MIT.

Ephemeral runners and remote state

Use [remote_state] instead of state to keep the ingestion journal, lease and atomic gate publication in a provisioned Cloudflare authority. A fresh runner needs only full committed source checkouts, TOML and environment-backed credentials. index --apply safely initializes or resumes/converges under the authoritative lease; update --apply remains strict on uninitialized state. Remote previews make read-only state requests. See remote setup, recovery and source policy and the shared-workflow caller template.