git clone https://github.com/bholagabbar/deeplore.git
cd deeplore
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env
# Fill in .env:
#   OPENROUTER_API_KEY=sk-or-...      required — LLM for notes generation + answers
#   APIFY_API_TOKEN=...               required for YouTube ingest (free tier at apify.com)
#   DEEPLORE_API_KEY=your-secret      recommended — bearer token for the MCP server

# 1. Fetch transcripts + generate structured notes for each lecture (~$0.22 via Apify)
python scripts/ingest_youtube.py \
  --playlist "https://www.youtube.com/playlist?list=PLKdLUOouUfd5No6YtRq5X6IbrM1q8odKN" \
  --collection iliad \
  --series "Iliad"

# 2. Build the indexes (summary, hierarchical, sentence-window)
python scripts/ingest.py --collection iliad

# 3. Start the MCP server
docker compose up          # includes reranker sidecar (recommended)
# or: python mcp/deeplore_server.py

curl -s http://localhost:5673/health
# {"status":"ok","collections":["iliad"],"auth":false}

# Using mcpc (npm install -g @apify/mcpc) — universal CLI MCP client
mcpc connect -H "Authorization: Bearer your-secret" "http://localhost:5673/mcp/" @deeplore
mcpc @deeplore tools-list
mcpc @deeplore tools-call ask collection:=iliad question:="What is the significance of Achilles' withdrawal from battle?"

Achilles' withdrawal is the structural engine of the entire poem. Menis, the
opening word, signals this from line one: not ordinary anger, but a divine,
consuming wrath with consequences that cascade outward. The quarrel with
Agamemnon is a rupture in the honor economy that holds the Greek coalition
together. Agamemnon takes Briseis, the material proof of Achilles' valor, and
Achilles responds by removing the one force the Greeks cannot replace. What
follows is not a war with a gap in it. It is a war that bleeds at the gap.
Every Greek death after Book I carries his fingerprints.

Sources: [Book I (menis, quarrel with Agamemnon), Books VIII-XII (Hector's aristeia)]

projects/
  iliad.yaml       # tracked — the example collection
  aeneid.yaml      # your personal projects, gitignored
  ...

name: iliad
display_name: "Homer's Iliad: Alexander Schmid Lecture Series"

data_dir: data/iliad/raw        # where raw JSON source files live
storage_dir: storage/iliad      # where built indexes are persisted

# Retrieval tuning
chunk_sizes: [2048, 512, 128]   # hierarchical levels: lecture, section, sentence
sentence_window_size: 3          # context window around transcript chunks
similarity_top_k: 8              # vector search candidates
rerank_top_n: 4                  # kept after cross-encoder reranking

# Models
embedding_model: text-embedding-3-large
llm_answer_model: anthropic/claude-sonnet-4-5

# MCP
mcp_tool_name: ask_iliad

# Persona — shapes how the LLM composes the final answer
persona: |
  You are an expert on Homer's Iliad and this lecture series.
  Ground every answer in the lectures. Make the epic feel alive.

# 1. Copy the example config and edit it
cp projects/iliad.yaml projects/my_collection.yaml
# edit: name, display_name, data_dir, storage_dir, mcp_tool_name, persona

# 2. Ingest a YouTube playlist
python scripts/ingest_youtube.py \
  --playlist "YOUR_PLAYLIST_URL" \
  --collection my_collection \
  --series "My Series"

# 3. Build indexes
python scripts/ingest.py --collection my_collection

# 4. Restart the server — new tool is available immediately

# Re-run both steps — the ingestor skips videos already fetched
python scripts/ingest_youtube.py --playlist URL --collection my_collection --series "My Series"
python scripts/ingest.py --collection my_collection

# With Docker (recommended — includes reranker sidecar)
docker compose up

# Note: on first run the reranker container downloads the cross-encoder model from HuggingFace.
# If it fails with "Header etag is missing" (HF rate limit), pre-download and seed the volume:
#   python3 -c "from huggingface_hub import snapshot_download; snapshot_download('cross-encoder/ms-marco-MiniLM-L-6-v2', cache_dir='/tmp/hf-cache')"
#   docker run --rm -v deeplore_reranker-cache:/data -v /tmp/hf-cache:/src alpine cp -r /src/models--cross-encoder--ms-marco-MiniLM-L-6-v2 /data/
# Then re-run docker compose up.

# Without Docker
OPENROUTER_API_KEY=sk-or-... DEEPLORE_API_KEY=your-secret python mcp/deeplore_server.py

GET  /health   server status + available projects (no auth)
POST /mcp/     MCP Streamable HTTP (bearer token auth)

{
  "mcpServers": {
    "deeplore": {
      "url": "https://your-host/mcp/",
      "headers": { "Authorization": "Bearer your-secret" }
    }
  }
}

ask(collection, question)   query any project by name
list_projects()             list all indexed projects

  Source (YouTube playlist, Granola, PDF, ...)
    │
    ├── Structured notes (LLM-generated per lecture)
    │     └── Summary index + Hierarchical index (2048 / 512 / 128 token levels)
    │
    └── Verbatim transcript
          └── Sentence-window index (chunk + ±3 sentence context stored as metadata)

  Query
    │
    ├── 1. Theme retrieval      which lectures are relevant? (document summaries)
    │
    ├── 2. Section retrieval    what does the source say? (hierarchical, auto-merging)
    │        └── if multiple child chunks from the same section fire, promote to full section
    │
    └── 3. Verbatim retrieval   exact words? (sentence window from transcript)
             └── returns matched chunk + surrounding context, not just the fragment

  All results → cross-encoder reranker → composed answer (LLM + project persona) → MCP tool

python scripts/eval.py --collection iliad --questions eval/iliad_questions.yaml