🧠 Second Brain MCP

Remember everything you watch and listen to.

A Model Context Protocol server that turns YouTube videos and podcasts into a private, searchable memory — built on the brand-new stateless MCP spec (2026-07-28).

Paste a link once. Ask about it forever.

"What did that video I watched last month say about salary negotiation?"

🧠 → "At [12:43] in Never Split the Difference — Chris Voss, he says never to accept the first offer without anchoring high…" — with a deep link that jumps to the exact second.

Why

You watch hours of talks, tutorials, and podcasts — and a week later you can quote none of it. Browser history remembers that you watched something; nothing remembers what it said. Second Brain gives your AI assistant total recall over everything you've ever watched or listened to:

• 🎯 Ask across your whole watch history — answers come back with timestamps and deep links to the exact moment.

• 🔒 Private by design — one local SQLite file. No cloud, no accounts, no API keys. Podcast audio is transcribed locally.

• ⚡ Zero-key ingestion — YouTube captions are fetched directly; nothing to configure.

• 🗑️ Nothing is deleted without you — destructive operations use the spec's new multi round-trip (MRTR) approval flow.

Related MCP server: yt

Quickstart

1. Install

Option A — as a tool, straight from GitHub (recommended):

uv tool install git+https://github.com/ravishu5/second-brain-mcp

This installs the second-brain command into ~/.local/bin. Find its absolute path — you'll need it below:

which second-brain     # e.g. /Users/you/.local/bin/second-brain

Option B — from a local clone (for hacking on it):

git clone https://github.com/ravishu5/second-brain-mcp
cd second-brain-mcp
python3 -m venv .venv && .venv/bin/pip install -e .

Your binary is then at <clone-dir>/.venv/bin/second-brain.

2. Connect a client

Always use the absolute path to the binary. GUI apps (like Claude Desktop) don't inherit your shell's PATH, so a bare second-brain command often fails with "Failed to spawn process: No such file or directory".

Claude Code

claude mcp add second-brain -- /absolute/path/to/second-brain

Claude Desktop — Settings → Developer → Edit Config, then add:

{
  "mcpServers": {
    "second-brain": {
      "command": "/absolute/path/to/second-brain"
    }
  }
}

ex: "command": "/Users/ravi/Desktop/mcp/second-brain-mcp/.venv/bin/second-brain" in my case

Fully quit (⌘Q) and reopen Claude Desktop — the server should show as connected under Settings → Developer.

As a stateless HTTP service (deployable behind any load balancer — no sticky sessions, no shared state):

second-brain --http --port 8000

3. Use it

Talk to your assistant:

› Remember this: https://www.youtube.com/watch?v=8S0FDjFBj8o
› What have I saved about system design?
› What did Lex's guest say about AGI timelines? Link me to the moment.
› Give me a digest of everything I added this week.

Tools

Podcast transcription is optional (local faster-whisper):

pip install "second-brain-mcp[whisper]"

Built on the 2026-07-28 stateless spec

This server is a working showcase of MCP's biggest release since remote MCP launched:

sequenceDiagram
    participant U as User
    participant C as Client (Claude)
    participant S as Second Brain
    C->>S: tools/call forget("that crypto video")
    S-->>C: resultType: input_required ("Permanently delete? No undo.")
    C->>U: Asks for approval
    U-->>C: Yes
    C->>S: tools/call retry (inputResponses + requestState)
    S-->>C: "Forgot 'Crypto Explained' (213 chunks removed)."

• Stateless by construction. No initialize handshake, no Mcp-Session-Id. Every request is self-contained; protocol metadata rides in _meta per request. Run one instance or twenty behind a load balancer — nothing breaks, because the only durable state is your local SQLite file, addressed through explicit item ids that clients thread between calls (exactly the application-state pattern the spec prescribes).

• MRTR instead of server-push. forget returns resultType: "input_required" with an elicitation request; the client gathers your approval and retries with inputResponses. No open streams, no sessions — and nothing is ever deleted without a human saying yes.

• Header-routable. Under the streamable HTTP transport, gateways can rate-limit Mcp-Method: tools/call + Mcp-Name: remember (expensive ingestion) differently from cheap recall reads — without parsing a byte of JSON.

• SDK v2. Built on mcp v2.0.0, released alongside the spec. Type hints are the schema; Resolve() powers the MRTR flow.

Architecture

┌─────────────┐   remember(url)   ┌──────────────┐
│ Claude /    │ ────────────────► │ ingest.py    │  YouTube captions (no key)
│ any MCP     │                   │              │  RSS + local whisper
│ client      │   recall(query)   ├──────────────┤
│             │ ────────────────► │ store.py     │  SQLite + FTS5 (BM25)
└─────────────┘   ◄─ passages +   │ ~/.second-   │  timestamped chunks
                     deep links   │  brain/      │
                                  └──────────────┘

Transcripts are merged into ~450-character chunks, split on silence gaps (usually topic boundaries), each carrying its start/end timestamps — so search hits map back to a playable moment, not just a document.

Troubleshooting

• "Server disconnected" / "Failed to spawn process: No such file or directory" — the client can't find the binary. Use the absolute path from which second-brain (or your venv's .venv/bin/second-brain) in the config, then fully restart the client.

• macOS: PermissionError: Operation not permitted on a path under ~/Desktop, ~/Documents, or ~/Downloads — macOS blocks other apps from reading those folders. Don't point the client at a venv inside them; install outside instead: uv tool install git+https://github.com/ravishu5/second-brain-mcp (lands in ~/.local, which isn't protected). Alternatively grant the client Desktop access under System Settings → Privacy & Security → Files and Folders.

• "No transcript/captions available" — the video has captions disabled; there's nothing to index (yet — see roadmap).

• Podcast ingestion errors about whisper — install the optional extra: pip install "second-brain-mcp[whisper]".

• Where's my data? One SQLite file at ~/.second-brain/brain.db. Override with the SECOND_BRAIN_DB env var. Delete the file to wipe the brain.

Development

git clone https://github.com/ravishu5/second-brain-mcp
cd second-brain-mcp
uv sync --extra dev
uv run pytest
uv run mcp dev src/second_brain_mcp/server.py   # MCP inspector

Roadmap

• Client-side summaries on ingest via MRTR sampling/createMessage (the client's model writes the summary — still zero server keys)

• Semantic search (local embeddings) alongside BM25

• Browser extension: one-click "remember this"

• Whisper speaker diarization for podcasts

PRs welcome — especially ingestion sources (lectures, audiobooks, Twitch VODs).

License

MIT © Ravi Shankar