ContextBank

ContextBank is a local-first personal context layer for AI-assisted work. It stores saved links, files, and X bookmarks in SQLite, mirrors them into readable Markdown cards, and exposes bounded read-only retrieval tools through MCP.

This repository starts with the CLI-first MVP spine from the product brief:

• SQLite as the canonical store.

• Markdown cards as a transparent, regenerable mirror.

• Manual URL, pasted text, bulk URL list, and file import that work without X credentials.

• Cost-aware X connector scaffolding for the official bookmark API.

• Offline full-text search and context-pack generation.

• Read-only MCP server tools for agents.

• Open-source, BYOK AI configuration with no bundled provider API keys.

Quick Start

python -m pip install -e ".[dev,mcp]"
contextbank init
contextbank add https://docs.x.com/x-api/users/get-bookmarks
contextbank add https://x.com/example_user/status/1234567890
printf "A useful pasted note" | contextbank add - --title "Pasted note"
contextbank import-list urls.md
contextbank import docs/example.md
contextbank search "local-first retrieval"
contextbank review list
contextbank mark <card-or-source-id> --priority high --pin
contextbank project link <card-or-source-id> --project . --reason "Useful for this repo"
contextbank project use <card-or-source-id> --project . --outcome-note "Applied to the agent flow"
contextbank mcp config
contextbank mcp instructions --client all --project . --write
contextbank project autoload . --task "current project goal"
contextbank readiness --project . --json
contextbank backup create
contextbank backup verify ~/.contextbank/exports/backup --json
contextbank mcp serve

By default, data lives in ~/.contextbank. Set CONTEXTBANK_HOME to use another directory.

Related MCP server: knowledgine

See What ContextBank Has

Use sync status for a quick inventory of the local vault:

contextbank sync status --json
contextbank source list --counts-only --json
contextbank source list --type x --json

sync status includes source_items, documents, knowledge_cards, and per-connector sync state. For X bookmarks, a successful limited sync should show an x.bookmarks entry with status such as paused or complete, last_error: null, and either a saved cursor for resumable sync or full_sync_completed: true. source list is the safer human inventory view: it shows counts by source type and availability, and can filter to --type x to confirm how many X bookmark sources are currently saved without exposing raw local paths.

Search the saved library directly:

contextbank search "agent memory" --limit 10
contextbank search "agent memory" --limit 10 --json
contextbank show <card-or-source-id> --json

search returns source-linked cards. The CLI can show local card text because it runs on the user's machine; MCP responses are stricter and expose source-authored text through delimited source_packet fields so agents treat it as evidence, not instructions.

Preview exactly what an agent should receive for a project task without using MCP:

contextbank project autoload /path/to/project --task "what I am working on" --json
contextbank context "what I am working on" --project /path/to/project --json

Those commands do not sync X, fetch web pages, call cloud AI, or write to the project. They only search already saved local ContextBank items and return a bounded MCP-safe context pack. Source authored card fields are redacted from ordinary JSON fields and available through source_packet. Pass --raw only for explicit local human/debug inspection.

contextbank import-list accepts newline text, Markdown links, CSV files with a URL-like column, or JSON arrays/objects containing url, source_url, href, or link. Web capture only fetches http(s) URLs and blocks obvious local/private network targets by default. contextbank source adapters --json lists the built-in source adapter contract and capabilities for web, file, text, and x.bookmarks, including which adapters require credentials. X post URLs are captured as partial X records keyed by post id, so later OAuth sync can hydrate the same item instead of creating a duplicate. When a new import matches an existing saved item by source hash, document hash, or canonical URL, ContextBank preserves the new record for provenance but marks it as a possible duplicate and hides it from normal agent retrieval. Review the relationship with contextbank review approve-duplicate <new-id> --duplicate-of <existing-id> or contextbank review reject-duplicate <new-id> --not-duplicate-of <existing-id>.

contextbank mcp config prints a local MCP client snippet. By default it emits the Codex TOML [mcp_servers.contextbank] shape; pass --client mcp-json, --client claude, --client claude-code, --client claude-desktop, or --client cursor for agents that use the JSON mcpServers shape. Use --home /path/to/vault when an agent should read a specific ContextBank vault instead of the default ~/.contextbank. MCP snippets include only CONTEXTBANK_HOME; provider and X credentials remain BYOK environment or token-store concerns. contextbank mcp manifest --json prints the command, args, tools, autoload contract, and read-only safety metadata for custom MCP setup flows. contextbank readiness --project . --json runs a no-network preflight that reports whether the local SQLite store, MCP manifest, project autoload, BYOK config, and X OAuth setup are ready. It separates ready_for_local_agent from ready_for_x_sync, so local agent use can be ready even before the live X OAuth flow is finished. The JSON payload also includes next_actions, a machine-readable list of commands that unblock any remaining local-agent or X-sync gates.

For project startup, contextbank project autoload . --task "..." builds the same source-linked context pack that MCP agents can request with autoload_project_context. The MCP server publishes startup instructions telling compatible agents to call this tool at the start of a project session, then use the returned cards before planning or implementing. If a client does not surface server instructions, run contextbank mcp instructions --client codex --project . or contextbank mcp instructions --client claude-code --project . and paste the managed block into the agent's project instructions. Add --write to write the block into the conventional local file such as AGENTS.md or CLAUDE.md. Use contextbank mcp instructions --client all --project . --write to install managed autoload blocks for the common Codex, Claude, Cursor, and generic project-instruction files in one step.

Generated instruction files are local setup files. They may contain an absolute path to the user's repository, so do not publish them to a public repo unless you intentionally sanitize or replace that path.

Using ContextBank From Codex Or Another Agent

First connect the MCP server to the agent. For Codex-style TOML:

contextbank mcp config --client codex --command "$(command -v contextbank)"

For JSON-based MCP clients such as Claude Code, Claude Desktop, Cursor, or generic MCP UIs:

contextbank mcp config --client claude-code --command "$(command -v contextbank)"
contextbank mcp config --client cursor --command "$(command -v contextbank)"

Then initialize the target project and install local startup instructions if the client does not automatically surface MCP server instructions:

cd /path/to/project
contextbank project init . --name "My Project" --autoload
contextbank project autoload-config . --enabled --token-budget 1500 --output-type implementation
contextbank mcp instructions --client codex --project . --write

At the start of a session, the agent should call the MCP tool:

autoload_project_context({
  "project_path": "/path/to/project",
  "task": "the user's current request",
  "token_budget": 1500,
  "desired_output_type": "implementation"
})

ContextBank inspects the project profile, lightweight project signals, explicit project links, and the saved local vault, then returns source-linked cards, stale warnings, source references, and retrieval metadata. The agent uses that pack before planning or editing. It does not receive write tools by default and autoload does not perform network or cloud calls.

Autoload can be controlled per repository through .contextbank/project.yml:

contextbank project init . --name "My App" --autoload
contextbank project autoload-config . --disabled
contextbank project autoload-config . --enabled --token-budget 1200 --output-type implementation

Autoload reads already saved/imported/synced ContextBank items, explicit project links, the project profile, and safe local project signals such as common manifest metadata and README headings. It does not sync X, fetch bookmarks, call cloud AI providers, or refresh web pages during agent startup. Project links marked rejected or obsolete, cards marked rejected or hidden, and profile exclude_topics are excluded from default agent retrieval.

Use contextbank project link <card-or-source-id> --project . --reason "..." to persist why a saved item matters to a repository. Use contextbank project use <card-or-source-id> --project . after an item actually helps; it marks the project link used, records an optional outcome note, and teaches future autoload that this item was useful for that repository. Project links are explicit local writes, mirrored into the Markdown card, and searchable with contextbank search "query" --project <name>. contextbank reprocess <card-or-source-id> regenerates source-derived fields while preserving explicit user decisions such as project links, pin/usefulness signals, hidden/rejected review state, and approved or rejected skill review.

Skill candidates stay inactive until reviewed. Use contextbank review approve-skill <id> or contextbank review reject-skill <id> to persist the decision. User signals such as --pin, --usefulness, --used-in-project, --obsolete, --incorrect, and --hide are recorded through contextbank mark; hidden items remain showable by direct id but are excluded from normal agent retrieval. Use contextbank review correct <card-or-source-id> to override generated card fields after human review. Corrections are stored in provenance with previous and corrected values, update the Markdown card, and survive later reprocessing. contextbank review list --queue all shows local library-health queues including skill review, duplicates, low confidence, stale, partial, failed, missing-source, hidden, and unprocessed items.

Source lifecycle commands keep local retention explicit:

contextbank source unavailable <card-or-source-id> --reason "gone upstream"
contextbank source revalidate <card-or-source-id> --reason "check freshness"
contextbank source revalidate <card-or-source-id> --fetch --reason "refresh saved source"
contextbank source delete-raw <card-or-source-id> --yes
contextbank source delete <card-or-source-id> --yes --detach-notes

source revalidate without --fetch is local lifecycle bookkeeping only. --fetch rereads the exact saved web URL or local file path, and can refresh one X post by ID through the official Post lookup endpoint when an OAuth user token is configured. It preserves user/project review signals; manual text reports not-refetchable because there is no external source to reread. Raw-data and full-item deletion require --yes. User notes, corrections, and project-use signals are preserved after source deletion only when --detach-notes is supplied.

Backups copy the SQLite database with SQLite's online backup API, include the Markdown card mirror, and write a sanitized config.toml without secret-like keys or secret-looking scalar values. Run contextbank backup verify <backup-dir> --json after creating or moving a backup to check database integrity, schema version, config parseability, secret-key/value absence, and card-file presence.

X Credentials

ContextBank does not store X access tokens in config files. X bookmarks require an OAuth 2.0 user access token, not an app-only Bearer token. The token needs tweet.read, users.read, and bookmark.read; add offline.access only if you want refresh-token sync.

In the X Developer Console, configure OAuth 2.0 as a Native App and allowlist:

http://127.0.0.1:8765/oauth/x/callback

Then run:

export CONTEXTBANK_X_CLIENT_ID="..."
# or persist the public OAuth Client ID locally:
contextbank config set x.client_id "..."
contextbank auth x-setup
contextbank auth x
contextbank sync x --dry-run --limit 25
contextbank sync x --limit 25
contextbank sync status

For automation, CONTEXTBANK_X_BEARER_TOKEN is still supported as an environment-token override. contextbank auth x-setup --json includes the secret-free Developer Console checklist, next_actions, and a data_use_answer draft for X's data-protection form. contextbank sync x validates that token by calling X. Local readiness commands intentionally report env-only bearer tokens as scope-unverified because they do not contact X; contextbank auth x stores local scope metadata that readiness can verify.

contextbank sync x calls /2/users/me first and uses that authenticated user id unless you set x.user_id; if you do set it, ContextBank verifies it matches the token before fetching bookmarks. Pagination cursors are persisted after each page so interrupted or deliberately limited syncs can resume on the next run. Use --no-resume to start from the beginning without clearing saved state, or --reset-cursor to discard the saved cursor before syncing. Use --reconcile-removed only after a full sync from the beginning; it soft-marks local X bookmark records that are absent from the completed API result instead of deleting source data. Use --fetch-linked-pages to also import expanded URLs found inside synced bookmarks. Linked pages use the same guarded web ingestion path as contextbank add, so blocked/private URLs are reported without stopping the rest of the sync. Rate-limit responses are retried with bounded exponential backoff before surfacing a typed error that includes reset timing when X returns it. When available from X, sync preserves conversation IDs, referenced posts, expanded links, media metadata, image alt text, and public metrics as metadata. Media transcription/OCR is not performed; that absence is recorded instead of pretending media was fully processed.

The X API uses authenticated user access for bookmarks and current pay-per-use pricing. Always run dry runs before large backfills. Use --cost-basis standard when estimating sync for users who do not own the developer app.

BYOK AI Providers

ContextBank’s core is open source and works with ai.mode = "none" by default. Optional local or cloud AI providers are bring-your-own-key: configure provider names, models, local endpoints, and credential environment-variable names in config.toml, but keep actual API key values in your shell, OS keychain, or secret manager.

contextbank config set ai.mode cloud
contextbank config set ai.generation_provider openai
contextbank config set ai.generation_model gpt-4.1-mini
contextbank config set ai.generation_credential_env OPENAI_API_KEY
contextbank ai status

contextbank ai status reports whether the named environment variables are present, but never prints their values. contextbank config set refuses secret-like config keys such as tokens, passwords, and API-key fields, and rejects values that look like accidentally pasted API keys or bearer tokens. Configure environment variable names, not the secret values themselves. When generation is configured, enrich one saved item explicitly with:

contextbank ai generate-card <card-or-source-id>

This calls the configured BYOK provider, requires ai.allow_cloud = true for non-loopback endpoints, wraps source text as untrusted evidence, and validates the returned JSON against ContextBank's card schema before saving it.

For offline semantic-style retrieval, build the local hash embedding index:

contextbank ai index-embeddings
contextbank search "related implementation idea" --semantic

The built-in local-hash embedding provider is deterministic and local. It makes no cloud calls and uses no bundled API keys.

External OpenAI-compatible embedding providers are explicit BYOK adapters. Configure a provider, model, endpoint if needed, and the name of an environment variable that already contains your key:

contextbank config set ai.embedding_provider openai
contextbank config set ai.embedding_model text-embedding-3-small
contextbank config set ai.embedding_credential_env OPENAI_API_KEY
contextbank config set ai.allow_cloud true
contextbank ai index-embeddings --provider configured

ai.allow_cloud is required for non-loopback endpoints. Loopback OpenAI-compatible endpoints can use openai-compatible with ai.embedding_base_url and no bundled key.