ContextMine

What is ContextMine?

ContextMine indexes your documentation and code repositories, making them searchable via the Model Context Protocol (MCP). Connect it to Claude Desktop, Cursor, or any MCP-compatible AI assistant to provide rich context for code understanding, documentation lookup, and codebase exploration.

Key features:

• Hybrid search - Full-text + vector similarity with RRF ranking for accurate retrieval

• Deep research agent - Multi-step AI agent with LSP and Tree-sitter for complex codebase questions

• Code intelligence - Symbol extraction, code outlines, and structural navigation via Tree-sitter

• Architecture Cockpit - Read-only extracted Twin views per collection/scenario (Overview, Topology, Deep Dive, C4 Diff, Exports)

• Strict real metrics - File-level LOC/complexity/coupling/coverage for GitHub sources with explicit availability status

• Web crawling - Index documentation sites automatically

• Git indexing - Index GitHub repositories with incremental updates

• Self-hosted - Your data stays on your infrastructure

Deep Research Agent

The deep research agent goes beyond simple search to answer complex questions about your codebase. It uses an iterative approach with multiple tools:

The agent collects evidence from multiple sources, verifies findings, and synthesizes a comprehensive answer with citations.

Related MCP server: docs-mcp

Quick Start

Choose your deployment method:

• Docker Compose (recommended for local development)

• Kubernetes (Helm) (recommended for production)

Docker Compose

# Clone the repository
git clone https://github.com/mayflower/contextmine.git
cd contextmine

# Copy environment template and configure
cp .env.example .env
# Edit .env with your API keys (see Configuration section)

# Start all services
docker compose up -d

# Run database migrations
docker compose exec api sh -c "cd /app/packages/core && alembic upgrade head"

Kubernetes (Helm)

For production deployments, use the Helm chart from GHCR:

# Create a values file with your configuration
cat > my-values.yaml << EOF
api:
  image:
    repository: ghcr.io/mayflower/contextmine-api
    tag: latest
worker:
  image:
    repository: ghcr.io/mayflower/contextmine-worker
    tag: latest
config:
  publicBaseUrl: "https://contextmine.example.com"
secrets:
  github:
    clientId: "your-github-client-id"
    clientSecret: "your-github-client-secret"
  sessionSecret: "$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
  tokenEncryptionKey: "$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
  openaiApiKey: "sk-..."
EOF

# Install from OCI registry
helm install contextmine oci://ghcr.io/mayflower/contextmine -f my-values.yaml

# Access the application
kubectl port-forward svc/contextmine-api 8000:8000

See deploy/helm/contextmine/README.md for full configuration options.

2. Create Your First Collection

1. Open the admin UI at http://localhost:8000

2. Log in with GitHub OAuth

3. Create a new Collection (e.g., "My Docs")

4. Add a Source:

   • Web: Enter a documentation URL (e.g., https://docs.python.org/3/)

   • GitHub: Enter owner/repo (e.g., fastapi/fastapi)

5. Click Sync to start indexing

3. Connect Your AI Assistant

Configure your MCP client to connect to ContextMine. Authentication is handled via GitHub OAuth automatically.

Claude Desktop (~/.config/claude/claude_desktop_config.json on Linux, ~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "contextmine": {
      "url": "http://localhost:8000/mcp"
    }
  }
}

When you first connect, your MCP client will redirect to GitHub for authentication.

Cursor: Settings → MCP → Add server with URL http://localhost:8000/mcp

4. Start Using It

In your AI assistant, you can now:

Search the FastAPI docs for information about dependency injection

What authentication methods does this codebase support?

Show me the outline of src/auth/handlers.py

Architecture Cockpit (Extracted Views)

The web app includes an Architecture Cockpit for project/collection-level Twin inspection in the browser.

Views

1. Overview - City KPIs and hotspot analysis.

2. Topology - Layered architecture graph view.

3. Deep Dive - Large graph slices for dependency/controlflow inspection.

4. Evolution - Investment/Utilization, Knowledge Islands, Temporal Coupling, Fitness Functions.

5. C4 Diff - AS-IS / TO-BE Mermaid compare with selectable C4 view level.

6. Exports - Generate cc_json, cx2, jgf, lpg_jsonl, mermaid_c4.

C4 View Controls

GET /api/twin/collections/{collection_id}/views/mermaid supports:

1. c4_view=context|container|component|code|deployment

2. c4_scope (optional focus selector for component/code/deployment)

3. max_nodes (diagram cap for large code/deployment views)

The response includes warnings (and in compare mode as_is_warnings / to_be_warnings) when views are generated in best-effort mode due sparse source signals.

Real Metrics Semantics

Overview uses GET /api/twin/collections/{collection_id}/views/city and reads:

{
  "summary": {
    "metric_nodes": 120,
    "coverage_avg": 71.4,
    "complexity_avg": 9.8,
    "coupling_avg": 3.2,
    "change_frequency_avg": 4.6,
    "churn_avg": 21.3
  },
  "metrics_status": {
    "status": "ready|unavailable",
    "reason": "ok|no_real_metrics|awaiting_ci_coverage|coverage_ingest_failed",
    "strict_mode": true
  },
  "hotspots": [
    {
      "node_natural_key": "file:src/main.py",
      "loc": 210,
      "symbol_count": 12,
      "coverage": 73.2,
      "complexity": 16.1,
      "coupling": 5,
      "change_frequency": 8,
      "churn": 46
    }
  ]
}

Rules:

1. ready: real metric snapshots are available.

2. unavailable: no valid real metrics for the selected scenario.

3. reason=awaiting_ci_coverage: structural metrics are ready, coverage has not been ingested yet.

4. reason=coverage_ingest_failed: the latest coverage ingest job failed or was rejected.

5. UI shows N/A for unavailable KPI values (not placeholder 0.00).

GitHub Actions Coverage Ingest (CI Push)

Coverage is no longer discovered from repository files. CI pushes raw coverage reports to ContextMine.

One-time setup

1. Identify your GitHub source ID (/api/collections/{collection_id}/sources).

2. Rotate the ingest token once as source owner:

   • POST /api/sources/{source_id}/metrics/coverage-ingest-token/rotate

3. Store returned token in GitHub Secrets as CONTEXTMINE_INGEST_TOKEN.

4. Store source ID in GitHub Secrets as CONTEXTMINE_SOURCE_ID.

GitHub Actions example

- name: Push coverage to ContextMine
  if: always()
  env:
    CONTEXTMINE_URL: https://contextmine.example.com
    CONTEXTMINE_SOURCE_ID: ${{ secrets.CONTEXTMINE_SOURCE_ID }}
    CONTEXTMINE_INGEST_TOKEN: ${{ secrets.CONTEXTMINE_INGEST_TOKEN }}
  run: |
    curl --fail-with-body \
      -X POST "$CONTEXTMINE_URL/api/sources/$CONTEXTMINE_SOURCE_ID/metrics/coverage-ingest" \
      -H "X-ContextMine-Ingest-Token: $CONTEXTMINE_INGEST_TOKEN" \
      -F "commit_sha=${{ github.sha }}" \
      -F "branch=${{ github.ref_name }}" \
      -F "workflow_run_id=${{ github.run_id }}" \
      -F "provider=github_actions" \
      -F "reports=@coverage/lcov.info" \
      -F "reports=@coverage/coverage.xml"

Notes:

1. commit_sha must exactly match the current source cursor SHA.

2. Multiple report files are supported and merged by file-level average.

3. Supported protocols (Core 6): lcov, Cobertura XML, JaCoCo XML, Clover/PHPUnit XML, OpenCover XML, generic-file-coverage-v1 JSON.

4. Check job status via GET /api/sources/{source_id}/metrics/coverage-ingest/{job_id}.

Available MCP Tools

Context Retrieval

Code Intelligence

Advanced Research

Configuration

Copy .env.example to .env and configure these variables:

Required

Optional

Setting Up GitHub OAuth

1. Go to https://github.com/settings/developers

2. Click New OAuth App

3. Fill in:

   • Application name: ContextMine (or your preferred name)

   • Homepage URL: http://localhost:8000

   • Authorization callback URL: http://localhost:8000/api/auth/callback

4. Copy the Client ID and Client Secret to your .env

Note: Both the admin UI and MCP clients use the same callback URL. The server automatically routes OAuth flows to the appropriate handler.

Generating Secure Keys

# Generate session secret
python -c "import secrets; print(secrets.token_urlsafe(32))"

# Generate encryption key
python -c "import secrets; print(secrets.token_urlsafe(32))"

Adding Sources

Web Documentation

Best for: API docs, guides, reference documentation

1. Create a collection in the admin UI

2. Add a source with type Web

3. Enter the base URL (e.g., https://docs.example.com/)

4. The crawler follows links within the same domain

GitHub Repositories

Best for: Source code, README files, inline documentation

1. Add a source with type GitHub

2. Enter the repository as owner/repo

3. Optionally specify:

   • Branch: defaults to the default branch

   • Path filter: limit to specific directories (e.g., src/, docs/)

4. Code files are parsed for symbols (functions, classes, methods)

Supported languages for symbol extraction: Python, TypeScript, JavaScript, Go, Rust, Java, C, C++, Ruby, PHP

Supported languages for strict real metrics (Twin/City): Python, TypeScript, JavaScript, Java, PHP

Strict metrics gate behavior for GitHub sources:

1. Sync computes structural metrics (loc, complexity, coupling) without blocking on coverage.

2. Coverage is ingested asynchronously from CI and bound to exact commit SHA.

3. Coverage ingest is strict: invalid token/payload/SHA mismatch/path mismatch fails the ingest job.

4. City metrics become fully ready only after successful coverage ingest.

Architecture

┌───────────────────────────────┐     ┌─────────────┐
│     FastAPI + React SPA       │────▶│  PostgreSQL │
│  /api/* /mcp/* /* (frontend)  │     │    pg4ai    │
└───────────────────────────────┘     └─────────────┘
               │
        ┌──────┴──────┐
        ▼             ▼
  ┌─────────┐   ┌─────────┐
  │ Prefect │   │ spider  │
  │ Worker  │   │   _md   │
  └─────────┘   └─────────┘

• API (apps/api): FastAPI serving REST API at /api/*, MCP at /mcp/*, and React frontend at /*

• Web (apps/web): React admin console (built and served by API)

• Worker (apps/worker): Background sync jobs using Prefect

• Core (packages/core): Shared models, database, and utilities

Development

Prerequisites

• Python 3.12+

• Node.js 20+

• uv for Python dependency management

• Docker (for pg4ai: PostgreSQL + pgvector + Apache AGE)

Local Development Setup

# Start database
docker compose up -d postgres

# Optional: verify vector + graph capabilities in postgres
./scripts/docker/smoke-pg4ai.sh

# Install Python dependencies
uv sync --all-packages

# Run migrations
cd packages/core
DATABASE_URL=postgresql+asyncpg://contextmine:contextmine@localhost:5432/contextmine \
  uv run alembic upgrade head
cd ../..

# Build frontend (one-time, or after frontend changes)
cd apps/web && npm install && npm run build && cd ../..

# Start API server (serves both API and frontend)
STATIC_DIR=apps/web/dist uv run uvicorn apps.api.app.main:app --reload --port 8000

For frontend development with hot reload, run the Vite dev server separately:

# Terminal 1: API server
uv run uvicorn apps.api.app.main:app --reload --port 8000

# Terminal 2: Frontend dev server (proxies API requests to :8000)
cd apps/web && npm run dev

Running Tests

# All tests
uv run pytest -v

# Specific test file
uv run pytest packages/core/tests/test_treesitter.py -v

# With coverage
uv run pytest --cov=contextmine_core --cov-report=term-missing

Code Quality

# Linting
uv run ruff check .

# Type checking
uvx ty check

# Auto-format
uv run ruff format .

# Pre-commit hooks
uv run pre-commit install
uv run pre-commit run --all-files

Container Images

Pre-built images are available from GitHub Container Registry:

docker pull ghcr.io/mayflower/contextmine-api:latest
docker pull ghcr.io/mayflower/contextmine-worker:latest
docker pull ghcr.io/mayflower/contextmine-web:latest

Troubleshooting

"No collections found" in MCP client

1. Ensure you've created at least one collection in the admin UI

2. Check that the collection visibility is set to Global (or you're authenticated)

3. Verify you've completed the GitHub OAuth flow when prompted by your MCP client

Sync not finding documents

1. Check the Prefect UI at http://localhost:4200 for job status

2. For GitHub sources, ensure the repository is accessible

3. For web sources, verify the URL is reachable and returns HTML

Symbols not being extracted

Symbol extraction works for supported languages only. Check that:

1. The file has a recognized extension (.py, .ts, .js, .go, etc.)

2. The sync has completed (symbols are extracted during sync)

Cockpit Overview shows N/A metrics

1. Inspect GET /api/twin/collections/{collection_id}/views/city.

2. Check metrics_status.reason:

   • awaiting_ci_coverage: CI has not pushed coverage yet.

   • coverage_ingest_failed: review ingest job diagnostics.

   • no_real_metrics: no structural metric snapshots were produced.

3. Verify latest ingest job:

   • GET /api/sources/{source_id}/metrics/coverage-ingest/{job_id}

4. Re-run CI upload with matching commit_sha=${{ github.sha }} and valid reports.

License

MIT License - see LICENSE for details.