# Aegis — Agent-Ready Knowledge Compiler

**Aegis** is a durable, evidence-grounded knowledge compiler that transforms instructional video content (starting with YouTube playlists in v1) into versioned, structured, and validated knowledge packages exposed to AI agents via the Model Context Protocol (MCP 2026-07-28).

---

## Key Capabilities

* **Deterministic 10-Stage Pipeline**: Discover $\rightarrow$ Acquire $\rightarrow$ Evidence Extraction $\rightarrow$ Segmentation $\rightarrow$ Structured Extraction $\rightarrow$ Evidence Binding $\rightarrow$ Enhancement $\rightarrow$ Validation $\rightarrow$ Vector Indexing $\rightarrow$ Atomic Publication.
* **PostgreSQL 18 + pgvector Single Source of Truth**: All stage executions, worker leases, idempotent operation identities, evidence citations, and knowledge items reside in PostgreSQL.
* **Content-Addressed Immutable Artifacts**: All intermediate outputs and raw transcripts are hashed (SHA256) and stored in S3/MinIO.
* **Hybrid Semantic & Lexical Retrieval**: Combines dense `text-embedding-3-small` vector similarity with PostgreSQL full-text search and evidence-support boosting.
* **Read-Only MCP 2026-07-28 Server**: 5 read-only tools exposed over `stdio` and Streamable HTTP for seamless integration with Claude Desktop, Cursor, and custom agent SDKs.
* **Operator Console**: Next.js 16 + React 19 web dashboard for ingestion, run tracking, evidence timeline exploration, and human-in-the-loop review queue resolution.

---

## Monorepo Layout

```text
aegis/
├── apps/
│   ├── api/          # FastAPI 0.139 Control REST API (:8000)
│   ├── worker/       # Background processing daemon & reconciler
│   ├── mcp/          # Read-only MCP 2026-07-28 Server (:8001)
│   └── web/          # Next.js 16 + React 19 Operator Console (:3000)
├── packages/
│   ├── domain/       # Pure domain models (zero framework dependencies)
│   ├── schemas/      # Pydantic v2 validation models & extraction schemas
│   ├── database/     # SQLAlchemy 2.0 async models (16 tables) & repositories
│   ├── pipeline/     # Idempotency engine, lease manager, 10-stage graph
│   ├── providers/    # External AI (OpenAI GPT-5.6) & YouTube (yt-dlp) adapters
│   ├── retrieval/    # Hybrid vector + lexical search engine
│   └── storage/      # Content-addressed S3/MinIO storage backend
├── migrations/       # Alembic database migrations
├── infra/
│   ├── compose/      # Docker Compose definition & overrides
│   ├── docker/       # Production multi-stage Dockerfiles
│   └── scripts/      # Backup, restore, migration, and healthcheck utilities
├── tests/            # Unit, integration, contract, evaluation, and crash recovery tests
└── docs/             # Architecture, REST API, operations runbook, and MCP guide
```

---

## Technology Stack

| Layer | Technology |
| :--- | :--- |
| **Backend Runtime** | Python 3.13 |
| **Dependency Manager** | `uv` (Python) + `pnpm` (Node.js) |
| **Control API** | FastAPI 0.139, Pydantic v2 |
| **Orchestration & State** | LangGraph 1.2, SQLAlchemy 2.0 Async, Alembic |
| **Database & Search** | PostgreSQL 18 + pgvector 0.8.6 |
| **Object Storage** | AWS S3 / MinIO (Content-Addressed) |
| **LLM & Embeddings** | OpenAI Responses API (GPT-5.6) / `text-embedding-3-small` |
| **Media Extraction** | yt-dlp 2026.7.4 |
| **Protocol Integration** | Model Context Protocol (MCP 2026-07-28 Python SDK v2) |
| **Operator Console** | Next.js 16, React 19, TypeScript, Tailwind CSS |
| **Quality & Linters** | pytest, pytest-asyncio, Ruff |

---

## Quickstart

### Option A: One-Liner Production Installation (curl)

Bootstrap dependencies, virtual environment, and configuration with a single command:

```bash
# Automated installer (fetches uv, clones/syncs repo, initializes .env with secure keys, checks Docker)
curl -fsSL https://raw.githubusercontent.com/Demi8-patch/aegis/main/install.sh | bash

```

Or execute locally from the repository root:
```bash
./install.sh
```

---

### Option B: Quickstart with `uv`

```bash
# 1. Clone and Bootstrap Environment
cp .env.example .env
uv sync
pnpm install

# 2. Verify System Health & Diagnostics
uv run aegis doctor

# 3. Start PostgreSQL 18 (with pgvector) and MinIO
docker compose -f infra/compose/docker-compose.yml up -d

# 4. Apply Database Migrations
uv run aegis migrate
```

---

## Unified Aegis CLI Reference

Aegis provides a comprehensive command-line tool `aegis` accessible via `uv run aegis <command>` (or directly as `aegis` when installed):

| Command | Description | Example |
| :--- | :--- | :--- |
| `aegis doctor` | Probes environment, database, pgvector, S3 storage, and OpenAI API keys. | `uv run aegis doctor` |
| `aegis init` | Safely creates `.env` with a cryptographically secure 256-bit `SECRET_KEY`. | `uv run aegis init` |
| `aegis ingest <url>` | Compiles a YouTube playlist or video end-to-end through the 10-stage pipeline. | `uv run aegis ingest "https://youtube.com/playlist?list=..."` |
| `aegis mcp` | Starts the read-only Model Context Protocol (MCP 2026-07-28) server. | `uv run aegis mcp --port 8001` or `uv run aegis mcp --stdio` |
| `aegis api` | Launches the FastAPI Control REST API server with Swagger docs. | `uv run aegis api --port 8000` |
| `aegis worker` | Runs the durable background worker daemon and periodic lease reconciler. | `uv run aegis worker` |
| `aegis eval` | Runs continuous retrieval evaluation benchmarks or records regression cases. | `uv run aegis eval --list-cases` |
| `aegis migrate` | Applies latest Alembic database migrations. | `uv run aegis migrate` |
| `aegis backup` | Creates a compressed PostgreSQL database dump and syncs S3 artifacts. | `uv run aegis backup --backup-dir ./backups` |
| `aegis restore` | Restores database schema and object storage artifacts from a backup archive. | `uv run aegis restore backups/aegis_db_*.sql.gz` |

---

### Launching Development Services

```bash
# Terminal 1: FastAPI Control API (:8000)
uv run aegis api --port 8000 --reload

# Terminal 2: Aegis Background Worker & Reconciler
uv run aegis worker

# Terminal 3: Read-Only MCP Server (:8001 / stdio)
uv run aegis mcp

# Terminal 4: Operator Web Console (:3000)
pnpm --filter aegis-web dev
```

The Operator Console will be live at `http://localhost:3000`, the Control API at `http://localhost:8000`, and the MCP endpoint at `http://localhost:8001/mcp`.

---

## MCP Tools Reference

The Aegis MCP server exposes 5 read-only tools conforming to the MCP 2026-07-28 specification:

| MCP Tool | Description |
| :--- | :--- |
| `search_knowledge` | Hybrid vector + lexical search across published knowledge items with evidence citations. |
| `get_knowledge` | Fetches a structured knowledge item by UUID with bound citations and semantic relationships. |
| `get_evidence` | Retrieves the raw transcript segment, start/end video seconds, and confidence score. |
| `search_sources` | Searches imported playlists and videos by title, URL, channel, or video ID. |
| `get_source` | Fetches source metadata, video durations, and published package versions. |

For client integration setup (Claude Desktop, Cursor, Windsurf, custom agents), see the [MCP Client Integration Guide](docs/operations/mcp-client-guide.md).

---

## Testing & Quality Assurance

Aegis includes a comprehensive test suite with 82+ tests covering unit logic, integration flows, crash recovery, prompt injection defense, and retrieval quality.

```bash
# Run full pytest suite (82 passed)
uv run pytest tests/

# Run Ruff linter and style checks
uv run ruff check .
uv run ruff format --check .

# Build Wheel & Source Distribution
uv build

# Build Next.js Operator Web Console
pnpm --filter aegis-web build
```


---

## Production Operations & Disaster Recovery

* **Database & Artifact Backup**:
  ```bash
  ./infra/scripts/backup.sh
  ```
* **Database & Artifact Restore**:
  ```bash
  ./infra/scripts/restore.sh backups/aegis_db_YYYYMMDD_HHMMSS.sql.gz backups/aegis_artifacts_YYYYMMDD_HHMMSS.tar.gz
  ```
* **Comprehensive Documentation**:
  * [Architecture Specification](docs/architecture/architecture.md)
  * [REST API Reference](docs/api/api-reference.md)
  * [Operations Runbook](docs/operations/runbook.md)
  * [MCP Client Guide](docs/operations/mcp-client-guide.md)

---

## License
Apache-2.0