# AI Square Documentation MCP Server

The official, documentation-first Model Context Protocol (MCP) service for AI Square. It is designed to let MCP-compatible clients retrieve authoritative AI Square documentation, API reference material, SDK examples, guides, and troubleshooting information with traceable citations.

> **Current status:** Phase 8 complete. The repository can crawl and incrementally index the configured AI Square documentation scope, then serve citation-safe hybrid retrieval through ten read-only MCP documentation tools and eight static `docs://` resources, as well as its internal API. Bounded TTL/LRU caches cover search outcomes, query embeddings, exact pages, and categories; Redis is an optional shared L2 backend for container and production deployments. Structured latency/error metrics and deterministic unit, integration, retrieval-quality, CLI, and Streamable HTTP MCP tests run in CI with an 80% coverage floor. Operator runbooks, deployment guidance, maintenance workflows, and troubleshooting are included below.

## Design principles

- Documentation is the source of truth; the server does not synthesize facts from model memory.
- Every indexed chunk retains a stable source URL, document identity, structure, and filterable metadata.
- Interfaces separate transports, application services, and infrastructure so providers can be replaced without changing MCP tools.
- Configuration and secrets come from environment variables; no credentials are committed.
- Async I/O, structured logs, health checks, and container-first operation are baseline concerns rather than later additions.

## Architecture

See [the architecture guide](docs/ARCHITECTURE.md) for the design and the executable Phase 1–8 boundaries.

```text
MCP clients / internal callers
          │
          ├── MCP transport (stdio or Streamable HTTP)
          └── FastAPI internal API
                         │
                    application ports
                         │
 crawler → parser → structural chunker → embedding adapter → Qdrant
                                      └──────────────────→ PostgreSQL
                                                              │
 query ──► BM25 + dense search ──► reciprocal-rank fusion ──► reranker
                                                              │
                                                   cited, compact context
```

The FastAPI API is for operational and internal integrations. MCP transports are the public AI-client interface. They share application services but neither transport is allowed to depend directly on provider-specific indexing or future retrieval code.

## Repository layout

```text
app/
  api/             FastAPI application, routes, and HTTP contracts
  config/          Typed settings and logging configuration
  mcp/             Official MCP SDK host, public retrieval adapter, and transports
  models/          Shared domain models (expanded with later phases)
  services/        Application services for ingestion, indexing, and retrieval
  crawler/         Documentation acquisition (Phase 2)
  parser/          Structured document extraction (Phase 2)
  chunking/        Heading-aware document chunking (Phase 3)
  embeddings/      Provider-neutral embedding contracts and adapters (Phase 3)
  retrieval/       Hybrid retrieval and context building (Phase 4)
  database/        PostgreSQL metadata and Qdrant vector adapters (Phase 3)
  cache/           Bounded TTL/LRU cache, safe codecs, and optional Redis L2 (Phase 7)
tests/             Unit, integration, and transport tests
docs/              Architecture, engineering, and operator documentation
docker/            Container runtime support files
scripts/           Explicit operational commands (added as phases need them)
```

## Quick start

Prerequisites: Python 3.12+, [uv](https://docs.astral.sh/uv/), Docker with Compose, and an embedding-provider API key for indexing.

```bash
# Run from the repository root.
cp .env.example .env
uv sync --all-groups
uv run aisquare-docs-api
```

The API is available at `http://127.0.0.1:8000`. Useful endpoints are:

- `GET /health/live` — process liveness, with no downstream checks.
- `GET /health/ready` — process/configuration readiness; it is degraded when local hybrid-search configuration is incomplete, while storage connectivity is verified by an index or retrieval run.
- `GET /api/v1/status` — service and phase capability status.
- `POST /api/v1/index` — incrementally index crawl artifacts through the internal API.
- `POST /api/v1/reindex` — force a complete indexing pass through the internal API.
- `GET /api/v1/search` — hybrid documentation search with filters and cited context.
- `GET /api/v1/page` — exact indexed-page lookup by canonical URL.
- `GET /api/v1/examples` — code-focused hybrid search.
- `GET /api/v1/categories` — available indexed source categories.
- `GET /docs` — OpenAPI documentation in development.

Run a respectful documentation crawl to create parsed artifacts:

```bash
uv run aisquare-docs-crawl
```

It writes deterministic structured JSON files and a run manifest under `data/documents` by default. Fetched HTML is used only in memory and is never written to an artifact. See [the ingestion guide](docs/INGESTION.md) before changing crawl limits or the user agent.

To index the artifacts, put an embedding key in `.env` (for the default adapter, set `AISQUARE_OPENAI_API_KEY`), then start PostgreSQL and Qdrant and run the incremental indexer:

```bash
docker compose up -d postgres qdrant
uv run aisquare-docs-index
# or: make index
```

Use `uv run aisquare-docs-index --force` (or `make reindex`) to run a complete forced pass. Once PostgreSQL and Qdrant contain that index, the internal API can retrieve it with hybrid search. See [the indexing guide](docs/INDEXING.md) for storage and re-indexing behavior, and [the retrieval guide](docs/RETRIEVAL.md) for search configuration and endpoint use.

For example, after indexing and starting the API:

```bash
curl -G http://127.0.0.1:8000/api/v1/search \
  --data-urlencode 'query=How do I authenticate?'
```

Set `AISQUARE_REQUIRE_API_KEY=true` and send `X-API-Key` outside a trusted local environment. The default Cohere reranking stage needs `AISQUARE_COHERE_API_KEY`; Voyage and Jina are selectable alternatives using their provider keys. Set `AISQUARE_RERANKER_ENABLED=false` to return deterministic RRF results without a reranker.

Run the MCP server over standard input/output for desktop MCP clients:

```bash
uv run aisquare-docs-mcp
```

Phase 5 exposes ten read-only documentation tools—search, exact page retrieval,
API/reference, code/example, related-page, category, SDK, guide, and
troubleshooting access—plus static `docs://` resources for the main
documentation areas. See [the MCP usage guide](docs/MCP.md) for the complete
tool/resource contract, aliases, and stdio or Streamable HTTP setup.

For containerized local dependencies and the API:

```bash
docker compose up --build
```

See [development and configuration instructions](docs/DEVELOPMENT.md) for local setup and verified settings, [the operations runbook](docs/OPERATIONS.md) for Docker and production deployment, and [the MCP usage guide](docs/MCP.md) for client setup.

## Documentation

- [Architecture](docs/ARCHITECTURE.md) explains the system boundaries, data flow, storage model, and retrieval guarantees.
- [Running locally and configuration](docs/DEVELOPMENT.md) covers setup, every supported environment variable, indexing, retrieval, and verification.
- [Adding or updating documentation and embeddings](docs/INDEXING.md) explains the safe crawl-to-index workflow, compatibility boundaries, and collection migration steps.
- [Crawler and artifact behavior](docs/INGESTION.md) documents source scope, robots handling, parsed artifacts, and failure behavior.
- [Retrieval behavior](docs/RETRIEVAL.md) covers hybrid search, citations, reranking, and caching.
- [MCP client use](docs/MCP.md) documents stdio and Streamable HTTP transports, tools, resources, and diagnostics.
- [Docker and production operations](docs/OPERATIONS.md) provides deployment, startup, reindex, recovery, scaling, and rollback runbooks.
- [Troubleshooting](docs/TROUBLESHOOTING.md) maps safe symptoms, checks, and remediation steps to each subsystem.
- [Contributing](CONTRIBUTING.md) describes the development workflow, quality gates, and repository invariants.

## Phase plan

1. **Foundation (complete):** package, configuration, API/MCP hosts, Docker topology, logging, health checks, CI, and tests.
2. **Ingestion (complete):** respectful sitemap/robots-aware crawler, structured parser, and durable raw-HTML-free JSON document artifacts.
3. **Indexing (complete):** heading-aware chunks, provider-neutral embedding contracts, OpenAI embeddings, Qdrant vectors, PostgreSQL metadata, and incremental re-indexing.
4. **Retrieval (complete):** BM25 + dense retrieval, reciprocal-rank fusion, metadata filters, optional reranking, and citation-preserving context through the internal API.
5. **MCP surface (complete):** ten read-only documentation tools and static resources backed by the citation-safe retrieval service.
6. **Quality (complete):** integration, retrieval-quality, CLI, and end-to-end Streamable HTTP MCP suites, with an enforced 80% coverage floor.
7. **Optimization (complete):** bounded search/page/category/query-embedding caches, optional Redis L2 coherence and invalidation, cache metrics, and OpenTelemetry retrieval/embedding/error instrumentation.
8. **Operations documentation (complete):** runbooks, deployment guidance, contribution workflows, and troubleshooting.

## Security notes

Use a separate API key per environment. Bind internal HTTP endpoints to a private network in production and put Streamable HTTP MCP behind TLS and your identity provider. The project deliberately keeps OAuth and tenant context at the boundary so they can be introduced without rewriting retrieval services.

Never use `.env` files as an artifact or image layer, and rotate any key that has been exposed.

## Contributing

```bash
uv run ruff check .
uv run black --check .
uv run mypy app
uv run pytest
uv run pytest -m retrieval_quality
uv run pytest -m mcp_e2e
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for the full contribution workflow and [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) for the coding, testing, and configuration conventions.