# obsidian-second-brain

An Obsidian vault that an AI assistant can file into and search — self-hosted,
plain Markdown on disk, no third party holding the notes.

Say *"file this under Work"* to Claude on your phone and a properly formed note
lands in a vault on your own server, front matter filled in. Ask *"what did I
work out about chunking strategies"* six months later and get it back, even
though no note contains that phrasing.

The clever part isn't the AI. It's that filing costs almost nothing, so you
actually do it. The vector index is what makes a pile of notes you never
re-read still worth having.

---

## Contents

- [How it works](#how-it-works)
- [The MCP tools](#the-mcp-tools)
- [The `/pub` read façade](#the-pub-read-façade)
- [`POST /ingest`](#post-ingest)
- [Requirements](#requirements)
- [Quick start](#quick-start)
- [Configuration](#configuration)
- [Vault conventions](#vault-conventions)
- [Design notes](#design-notes)
- [Security](#security)
- [Development](#development)

---

## How it works

Three containers share one vault directory. Two dependencies live outside:
a Postgres with `pgvector`, and any OpenAI-compatible embeddings endpoint.

```
  Claude apps                        ┌─────────────────────────────────────┐
  (desktop, mobile,                  │  host                               │
   Claude Code)                      │                                     │
        │                            │   ┌───────────────┐                 │
        │  MCP over HTTPS            │   │  cloudflared  │  optional       │
        │  + bearer token            │   └───────┬───────┘                 │
        ▼                            │           │ :7620                   │
  ┌──────────────┐   outbound tunnel │           ▼                         │
  │  Cloudflare  │◄──────────────────┼── ┌──────────────────┐              │
  │  (optional)  │   no inbound port │   │ second-brain-mcp │──┐ rw        │
  └──────────────┘                   │   └──────────────────┘  │           │
                                     │                         ▼           │
                                     │   ┌──────────────────┐ ┌──────────┐ │
                                     │   │     indexer      │─┤  vault/  │ │
                                     │   │  polls every     │ │  *.md    │ │
                                     │   │  300s            │ └──────────┘ │
                                     │   └────────┬─────────┘      ▲       │
                                     │            │                │ rw    │
                                     │            │        ┌───────┴─────┐ │
                                     │            │        │  Obsidian   │ │
                                     │            │        │  (your app, │ │
                                     │            │        │   synced)   │ │
                                     │            │        └─────────────┘ │
                                     └────────────┼──────────────────────  ┘
                                                  │
                            ┌─────────────────────┴──────────────┐
                            ▼                                    ▼
                  ┌───────────────────┐              ┌──────────────────────┐
                  │ Postgres+pgvector │              │  embeddings endpoint │
                  │  chunks + vectors │              │  OpenAI-compatible   │
                  └───────────────────┘              └──────────────────────┘
```

**Filing is decoupled from indexing.** `file_note` returns as soon as the file
is on disk. The note is findable by structured search immediately and by
semantic search within one indexer cycle. A wedged embedding backend never
slows down anything you are waiting on.

The indexer **scans** the vault rather than waiting to be told, so notes you
type by hand in Obsidian are indexed exactly like notes the assistant files.
That matters more than it sounds — most vaults end up mostly hand-written.

---

## The MCP tools

Five tools over MCP's streamable-HTTP transport.

| Tool | Does | Depends on |
|---|---|---|
| `file_note` | Writes a note into `<Segment>/Inbox/` with valid front matter | vault disk |
| `search_notes` | Filters by segment, tag, status, date range, filename, literal text | vault disk |
| `get_note` | Returns one note in full, by vault-relative path | vault disk |
| `semantic_search` | Cosine top-k over note chunks | Postgres + embeddings |
| `index_status` | Index freshness and embedding-model consistency | Postgres |

**Three of the five never leave the filesystem.** That is deliberate: filing
and keyword search keep working when the model server is down, which on a
homelab it regularly is.

Plus three plain HTTP routes for monitoring:

| Route | Answers |
|---|---|
| `GET /healthz` | process is alive |
| `GET /readyz` | vault mounted, segment inboxes present |
| `GET /index-health` | index fresh, and built with the model currently configured |

---

## The `/pub` read façade

Some Claude deployments — a managed work tenant, for instance — do not allow
custom connectors at all. The `/pub` routes are a read-only fallback that rides
on an ordinary web fetch instead.

All `GET`, all returning `text/markdown`, all under `/pub/<token>/`:

| Route | Backed by |
|---|---|
| `/` | self-describing index: lists routes, parameters, segments |
| `search?q=` | `search_notes` — filesystem only |
| `semantic?q=` | `semantic_search` — 503 with a pointer to `search` if the index is down |
| `note?path=` | `get_note`, paged via `offset` |

Markdown rather than JSON, because it survives a fetch tool intact and reads
correctly to a model on the far side. Responses carry `Cache-Control: no-store`
(a capability URL in a shared cache is a leaked vault) and
`X-Robots-Tag: noindex, nofollow`.

Set `PUBLIC_READ_TOKEN` to enable. **Leave it unset and these routes return
404** — a prober learns nothing about whether they exist.

The handlers call the same functions the MCP tools call. They are a second
transport, never a second implementation: a divergent search would show up as
two clients disagreeing about what is in the vault.

> **The token is in the URL.** That is what makes it work with a fetch tool,
> and it is a real cost: the token appears in proxy logs, in any reverse-proxy
> access log, and in browser history. Treat it as a read-only, rotatable
> credential and nothing more. The service redacts `/pub/<token>` from its own
> logs; it cannot redact anyone else's.

---

## `POST /ingest`

A write path for the same constrained clients, taking a `write`-scoped token in
an `Authorization` header rather than in a URL.

```http
POST /ingest
Authorization: Bearer sbt_...
Content-Type: application/json

{
  "title":   "What I learned about HNSW",
  "body":    "# ...markdown...",
  "segment": "Industry",
  "tags":    ["pgvector", "retrieval"],
  "idempotency_key": "some-stable-id"
}
```

The idempotency key means a poller that retries does not duplicate the note.

**Tokens** live in a SQLite store (`TOKEN_DB_PATH`), not in Postgres —
authentication has to keep working when the database does not. They are
prefixed `sbt_`, stored hashed, carry a scope (`read` < `write` < `admin`), and
can be individually expired or revoked with a recorded last-used time.

---

## Requirements

| Need | Notes |
|---|---|
| A Docker host | One small VM or LXC. 2 vCPU / 4 GB is comfortable |
| Postgres with `pgvector` | Any Postgres 14+. You need one schema, not a cluster |
| An embeddings endpoint | Anything OpenAI-compatible: Ollama, llama.cpp, LM Studio, Lemonade, or OpenAI |
| Obsidian | Optional but the point — any sync mechanism works |

**On syncing the vault to your devices:** Obsidian Sync has no CLI and no
headless mode, so if you use it, a real Obsidian has to run somewhere
always-on. The `linuxserver/obsidian` image works for this. Self-hosted
alternatives (LiveSync over CouchDB, Syncthing) avoid that entirely. Whatever
you choose, **run exactly one syncer** — two over one vault is how you get
conflict files and lost edits.

---

## Quick start

```sh
git clone https://github.com/gmoorevt/obsidian-second-brain.git
cd obsidian-second-brain
cp .env.example .env
```

**1. Apply the database schema.**

```sh
psql "$DATABASE_URL" -f schema.sql
```

> `schema.sql` declares `vector(1024)`, which matches
> `Qwen3-Embedding-0.6B`. **Change it to match your model or every insert
> fails.** `bge-m3` is 1024, OpenAI `text-embedding-3-small` is 1536,
> `nomic-embed-text` is 768.

**2. Verify your embeddings endpoint** — the dependency most likely to be
subtly wrong:

```sh
curl -s http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"YOUR-MODEL","input":"hello"}' \
  | python3 -c 'import json,sys; print(len(json.load(sys.stdin)["data"][0]["embedding"]))'
# → must equal the N in vector(N)
```

**3. Scaffold the vault.**

```sh
./vault/scaffold-vault.sh ./data/vault
```

Idempotent — safe to re-run, creates only what is missing, never overwrites a
note. Use `--dry-run` first if you like.

**4. Fill in `.env`** — at minimum `MCP_BEARER_TOKEN` (`openssl rand -hex 32`),
`DATABASE_URL`, and `EMBEDDING_ENDPOINT`. Delete the `cloudflared` service from
`docker-compose.yml` if you are not publishing the endpoint.

**5. Start it.**

```sh
docker compose up -d --build
curl -s http://127.0.0.1:7620/healthz     # → ok
```

**6. Connect your client** to `https://your-host/mcp` with the bearer token as
`Authorization: Bearer <MCP_BEARER_TOKEN>`.

Round-trip test: *"File a note in Projects titled Connector Test"*, then
*"search my Projects inbox"*.

---

## Configuration

| Variable | Default | Meaning |
|---|---|---|
| `MCP_BEARER_TOKEN` | *required* | Shared secret for the MCP endpoint. Min 32 chars |
| `PUBLIC_READ_TOKEN` | *unset* | Enables `/pub`. Unset ⇒ those routes 404 |
| `TOKEN_DB_PATH` | `/state/tokens.db` | SQLite token store |
| `VAULT_ROOT` | `/vault` | Vault path inside the container |
| `MCP_HOST` | `0.0.0.0` | Bind address *inside* the container |
| `MCP_PORT` | `7620` | Listen port |
| `BIND_ADDR` | `127.0.0.1` | Host address Compose publishes on |
| `VAULT_DIR` | `./data/vault` | Host path to the vault |
| `STATE_DIR` | `./data/state` | Host path for the token store |
| `DATABASE_URL` | *required for semantic* | `postgres://user:pw@host:5432/db` |
| `EMBEDDING_ENDPOINT` | *required for semantic* | OpenAI-compatible base URL |
| `EMBEDDING_MODEL` | — | Also recorded per chunk, to detect model drift |
| `EMBEDDING_TIMEOUT` | `30` | Seconds before an embedding call fails |
| `INDEX_INTERVAL_SECONDS` | `300` | Indexer poll interval |
| `INDEX_STALENESS_SECONDS` | `1800` | Age at which `/index-health` reports unhealthy |
| `LOG_LEVEL` | `INFO` | Standard Python levels |
| `TUNNEL_TOKEN` | *unset* | cloudflared run-token, if publishing |

---

## Vault conventions

Deliberately shallow — four segments, two subfolders each, nothing deeper.
Depth is where vaults like this die.

```
Work/  Industry/  Projects/  Life/
  Inbox/       ← everything new lands here
  Notes/       ← reviewed and kept

Attachments/   images, PDFs, binaries
Daily/         quick capture; no front matter required
_templates/    capture templates
```

Every note in a segment carries exactly six fields. `Daily/` is exempt — that
folder is a zero-friction scratchpad and requiring structure there defeats it.

```yaml
---
title: Human readable title
created: 2026-09-07
segment: Industry          # Work · Industry · Projects · Life
tags: [rag, obsidian]      # flat list, no parent/child hierarchies
source: claude-desktop     # which channel wrote it
status: inbox              # inbox · filed
---
```

Filenames are `YYYY-MM-DD-slugged-title.md`. Collisions take a numeric suffix;
nothing is ever overwritten.

`vault/conventions.yaml` is the single source of truth for all of this, read by
both the scaffolding script and the service so they cannot disagree about the
segment list. It is duplicated at `src/second_brain_mcp/conventions.yaml`
because the package ships with it — **keep the two identical.**

> **Review is optional, and saying so out loud is what makes this survive.**
> `status: inbox` exists so a review pass is *possible*, not owed. A note that
> sits at `inbox` for a year is still fully searchable. A system that quietly
> guilts you is a system you abandon.

---

## Design notes

The decisions that were not obvious, and why.

**The indexer is a separate container from the server.** Same image, different
entrypoint. A slow or wedged embedding backend must not make filing slow, and
the two failures need to be separately visible to monitoring.

**The Postgres connection is lazy.** Built on first use, not at import. If it
were eager, a database outage would stop the whole service from starting —
including `file_note`, which needs no database at all.

**Auth runs before parameter validation and any filesystem access.** FastMCP
ships a `StaticTokenVerifier` whose own docstring says not to use it in
production. The replacement compares with `hmac.compare_digest` so the check
does not leak the token by timing, rejects anything under 32 characters at
startup, and never logs the supplied value — logging a near-miss token puts a
credential in the log file.

**Chunking splits on Markdown headings first**, then windows anything still too
long at 2,000 characters with 200 of overlap. Each chunk keeps its heading, so
a hit can say which section it came from.

**`embedding_model` is recorded on every chunk.** Swap models and your existing
vectors become meaningless — cosine distance across two embedding spaces
returns confident nonsense rather than an error. Recording the model turns a
silent corruption into a visible mismatch in `index_status`.

**Don't share the tunnel's network namespace.** Giving cloudflared
`network_mode: service:second-brain-mcp` looks tidier and lets you route to
`localhost`. It also couples the lifecycles: restarting the service gives it a
new namespace and leaves cloudflared attached to the dead one. Service reports
healthy, cloudflared reports healthy, public endpoint returns `530`. Reach it
by container name over the compose network instead.

**A stalled sync driver looks exactly like nothing being wrong.** Filing
succeeds, the assistant reports a path, and the note simply never reaches your
phone. Monitor sync separately from the endpoint.

---

## Security

Read this before exposing the service to the internet.

**The endpoint writes files to disk from network input.** Treat it accordingly.

1. **Put an allowlist in front of it.** If only one client should ever reach
   this hostname, restrict it at the edge to that client's egress range — a
   Cloudflare WAF custom rule, or equivalent. A bearer token alone is one
   layer, and one layer on a public write endpoint is thin. Anthropic publishes
   its egress range at
   <https://platform.claude.com/docs/en/api/ip-addresses>; verify it rather
   than copying a range out of a blog post, this one included.

2. **Do not route `/ingest` publicly.** If the client that posts to it runs on
   your own network, reach it over the LAN and block the path at the edge. A
   route that is not routed cannot be misconfigured into being public.

3. **`/pub` tokens are capability URLs.** Anyone holding one can read the whole
   vault. Rotate them, scope them read-only, and remember that your reverse
   proxy and CDN log full request paths even though this service does not.

4. **Bind to loopback unless you need otherwise.** `BIND_ADDR` defaults to
   `127.0.0.1`. Setting a LAN address publishes *every* route on the LAN,
   `/pub` included.

5. **The token store is not in the vault.** Keep it that way — credentials must
   not sync to every device Obsidian is open on.

This project has had no external security review. It is a personal homelab
tool that is useful enough to share, not a hardened product.

---

## Development

```sh
pip install -e ".[dev]"
pytest
```

96 tests, no network or database required — the suite covers the token store,
the public façade rendering, redaction, and clamping.

Layout:

| Path | Responsibility |
|---|---|
| `src/second_brain_mcp/server.py` | FastMCP app, tools, HTTP routes |
| `src/second_brain_mcp/index.py` | pgvector reads/writes, run bookkeeping |
| `src/second_brain_mcp/notes.py` | Front matter parsing, structured search |
| `src/second_brain_mcp/vault.py` | Path resolution, traversal refusal, safe writes |
| `src/second_brain_mcp/public.py` | `/pub` rendering, log redaction |
| `src/second_brain_mcp/tokens.py` | SQLite token store, scopes |
| `src/second_brain_mcp/chunking.py` | Heading-aware splitting, hashing |
| `src/second_brain_mcp/embedding.py` | Embedding client, typed failures |
| `src/second_brain_mcp/conventions.py` | Loads and validates `conventions.yaml` |
| `src/second_brain_mcp/auth.py` | Constant-time bearer verification |
| `src/second_brain_mcp/indexer.py` | The polling loop |

---

## License

MIT — see [LICENSE](LICENSE).