# Lenny RAG MCP Server

An MCP server providing hierarchical RAG over 299 Lenny Rachitsky podcast transcripts. Enables product development brainstorming by retrieving relevant insights, real-world examples, and full transcript context.

## Quick Start

```bash
# Clone the repository (includes pre-built index via Git LFS)
git clone git@github.com:mpnikhil/lenny-rag-mcp.git
cd lenny-rag-mcp

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate

# Install the package
pip install -e .
```

### Claude Code

```bash
claude mcp add lenny --scope user -- /path/to/lenny-rag-mcp/venv/bin/python -m src.server
```

Or add to `~/.claude.json`:

```json
{
  "mcpServers": {
    "lenny": {
      "type": "stdio",
      "command": "/path/to/lenny-rag-mcp/venv/bin/python",
      "args": ["-m", "src.server"],
      "cwd": "/path/to/lenny-rag-mcp"
    }
  }
}
```

### Claude Desktop

Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "lenny": {
      "command": "/path/to/lenny-rag-mcp/venv/bin/python",
      "args": ["-m", "src.server"],
      "cwd": "/path/to/lenny-rag-mcp"
    }
  }
}
```

### Cursor

Add to `.cursor/mcp.json` in your project or `~/.cursor/mcp.json` globally:

```json
{
  "mcpServers": {
    "lenny": {
      "command": "/path/to/lenny-rag-mcp/venv/bin/python",
      "args": ["-m", "src.server"],
      "cwd": "/path/to/lenny-rag-mcp"
    }
  }
}
```

> Replace `/path/to/lenny-rag-mcp` with your actual clone location in all configs.

---

## MCP Tools

### `search_lenny`

Semantic search across the entire corpus. Returns pointers for progressive disclosure.

| Parameter | Type | Description |
|-----------|------|-------------|
| `query` | string | Search query (e.g., "pricing B2B products", "founder mode") |
| `top_k` | integer | Number of results (default: 5, max: 20) |
| `type_filter` | string | Filter by type: `insight`, `example`, `topic`, `episode` |

**Returns:** Ranked results with relevance scores, episode references, and topic IDs for drilling down.

### `get_chapter`

Load a specific topic with full context. Use after `search_lenny` to get details.

| Parameter | Type | Description |
|-----------|------|-------------|
| `episode` | string | Episode filename (e.g., "Brian Chesky.txt") |
| `topic_id` | string | Topic ID (e.g., "topic_3") |

**Returns:** Topic summary, all insights, all examples, and raw transcript segment.

### `get_full_transcript`

Load complete episode transcript with metadata.

| Parameter | Type | Description |
|-----------|------|-------------|
| `episode` | string | Episode filename (e.g., "Brian Chesky.txt") |

**Returns:** Full transcript (10-40K tokens), episode metadata, and topic list.

### `list_episodes`

Browse available episodes, optionally filtered by expertise.

| Parameter | Type | Description |
|-----------|------|-------------|
| `expertise_filter` | string | Filter by tag (e.g., "growth", "pricing", "AI") |

**Returns:** List of 299 episodes with guest names and expertise tags.

---

## Data Curation Approach

### Hierarchical Extraction

Each transcript is processed into a 4-level hierarchy enabling progressive disclosure:

```
Episode
├── Topics (10-20 per episode)
│   ├── Insights (2-4 per topic)
│   └── Examples (1-3 per topic)
```

This allows Claude to start with lightweight search results and drill down only when needed, keeping context windows efficient.

### Extraction Schema

```json
{
  "episode": {
    "guest": "Guest Name",
    "expertise_tags": ["growth", "pricing", "leadership"],
    "summary": "150-200 word episode summary",
    "key_frameworks": ["Framework 1", "Framework 2"]
  },
  "topics": [{
    "id": "topic_1",
    "title": "Searchable topic title",
    "summary": "Topic summary",
    "line_start": 1,
    "line_end": 150
  }],
  "insights": [{
    "id": "insight_1",
    "text": "Actionable insight or contrarian take",
    "context": "Additional context",
    "topic_id": "topic_1",
    "line_start": 45,
    "line_end": 52
  }],
  "examples": [{
    "id": "example_1",
    "explicit_text": "The story as told in the transcript",
    "inferred_identity": "Airbnb",
    "confidence": "high",
    "tags": ["marketplace", "growth", "launch strategy"],
    "lesson": "Specific lesson from this example",
    "topic_id": "topic_1",
    "line_start": 60,
    "line_end": 85
  }]
}
```

### Implicit Anchor Detection

Many guests reference companies without naming them ("at my previous company..."). The extraction prompt instructs the model to infer identities based on the guest's background:

- Brian Chesky saying "when we started" → Airbnb (high confidence)
- A marketplace expert saying "one ride-sharing company" → likely Uber/Lyft (medium confidence)

This surfaces examples that wouldn't be found by keyword search alone.

### Quality Thresholds

Each transcript extraction is validated against minimum thresholds:

| Element | Minimum | Typical |
|---------|---------|---------|
| Topics | 10 | 15-20 |
| Insights | 15 | 25-35 |
| Examples | 10 | 18-25 |

Extractions below thresholds trigger warnings for manual review.

---

## Models & Tech Stack

| Component | Model/Tool | Purpose |
|-----------|------------|---------|
| Preprocessing | **Claude Haiku** (via Claude CLI) | Extract structured hierarchy from transcripts |
| Embeddings | **bge-small-en-v1.5** | Semantic similarity for search |
| Vector DB | **ChromaDB** | Persistent vector storage |
| MCP Framework | **mcp** (Python SDK) | Tool interface for Claude |

### Why Claude Haiku for Preprocessing?

- **Quality**: Haiku follows complex extraction prompts reliably
- **Cost**: ~$0.02-0.03 per transcript (~$6-9 total for 299 episodes)
- **Speed**: ~30 seconds per transcript

### Why bge-small-en-v1.5 for Embeddings?

- **Performance**: Top-tier retrieval quality for its size
- **Efficiency**: 384 dimensions, fast inference
- **Local**: Runs entirely on CPU, no API calls needed

---

## Corpus Statistics

| Metric | Count |
|--------|-------|
| Episodes | 299 |
| Topics | 6,183 |
| Insights | 8,840 |
| Examples | 6,502 |
| Avg topics/episode | 20.7 |
| Avg insights/episode | 29.6 |
| Avg examples/episode | 21.7 |

---

## Rebuilding the Index

The repo includes a pre-built ChromaDB index. To rebuild from scratch:

### Reprocess Transcripts (requires Claude CLI)

```bash
# Process all unprocessed transcripts
python scripts/preprocess_haiku.py

# Process specific file
python scripts/preprocess_haiku.py --file "Brian Chesky.txt"

# Parallel processing (4 batches of 50)
python scripts/preprocess_haiku.py --limit 50 --offset 0 &
python scripts/preprocess_haiku.py --limit 50 --offset 50 &
python scripts/preprocess_haiku.py --limit 50 --offset 100 &
python scripts/preprocess_haiku.py --limit 50 --offset 150 &
```

### Rebuild Embeddings

```bash
# Incremental (only new files)
python scripts/embed.py

# Full rebuild
python scripts/embed.py --rebuild
```

---

## Project Structure

```
lenny-rag-mcp/
├── transcripts/           # 299 raw .txt podcast transcripts
├── preprocessed/          # Extracted JSON hierarchy (one per episode)
├── chroma_db/             # Vector embeddings (Git LFS)
├── prompts/
│   └── extraction.md      # Haiku extraction prompt
├── src/
│   ├── server.py          # MCP server & tool definitions
│   ├── retrieval.py       # LennyRetriever class (ChromaDB wrapper)
│   └── utils.py           # File loading utilities
├── scripts/
│   ├── preprocess_haiku.py  # Claude CLI preprocessing
│   └── embed.py             # ChromaDB embedding pipeline
└── pyproject.toml
```

---

## License

MIT