Second Brain Control Plane

A local-first automation engine that turns an Obsidian vault into a semantically searchable, governed "second brain" — using only models that run on your own machine via Ollama. No document content, embeddings, or queries are ever sent to a cloud AI provider.

It watches your vault in real time, ingests external documents (PDF, Word, PowerPoint, plain text, audio), extracts and structures their content, runs every candidate document through an ingestion policy (path/size/MIME checks, secret and PII detection, cloud-sync detection), embeds and indexes what's allowed into a local vector store, and proposes tags and links between related notes. Every generated note carries an extended manifest (content hash, resolved path, MIME type, extractor version, the model that embedded it, and the policy decision that let it through). A companion Obsidian plugin adds an in-app search box and a RAG chat, both showing provenance badges from that manifest; an MCP server exposes the same search/chat to Claude Desktop, Claude Code, Cursor, and other MCP-compatible clients.

This repository accompanies a single paper documenting the whole project — how it was built, what broke, and (after a security-focused review argued that "runs locally" and "is actually governed" are two different claims) how it was hardened into something closer to a reference architecture:

📄 Building a Verifiable Local AI Knowledge System (also available as Markdown) — covering the original failure modes, the control-plane work below in full, two adversarial findings (a real symlink-based ingestion vulnerability, and inconsistent prompt-injection resistance), and a mapping of these controls to NIST's AI Risk Management Framework and OWASP's LLM Top 10 — offered as a shared vocabulary, not a compliance claim.

An earlier, simpler snapshot of this codebase, from before the control-plane work, remains available at github.com/alebellotta/local-second-brain for anyone who wants to see the system before that phase of the work.

It's meant to be read and adapted, not run as-is out of the box on someone else's machine — file paths, model names, and the daily-schedule mechanism are all things you'll want to tune to your own setup.

Why local?

Three reasons drove every design decision here:

1. Privacy by construction. Personal and work notes often contain sensitive material. Routing them through a third-party AI API means trusting that provider's retention policy. Running embeddings and language models locally removes that dependency entirely — the trust boundary is your own laptop.

2. Cost. Indexing and re-indexing a growing vault, plus generating tags and summaries continuously, adds up fast on metered APIs. Local inference is effectively free after the one-time cost of downloading the models.

3. No rate limits, no outages. The system works offline, and doesn't depend on a third party's uptime.

The trade-off is hardware-bound quality and speed (see "Lessons learned" below) — this project treats that trade-off as a first-class design constraint, not an afterthought.

Related MCP server: Obsidian Index MCP Server

Architecture

                     ┌──────────────────────────┐
  cloud drives  ───▶ │ index_external_folders.py│──▶ policy.py ──▶ Notes/ (extended
  (shared, no        │  (daily, no copying)     │    (allow/           manifest in
  cloning wanted)    └──────────────────────────┘     quarantine/       frontmatter)
                                                        deny)                │
  manually dropped                                        │                 ▼
  documents      ───▶ Sources/ ──▶ watcher.py ──▶ text extraction ──▶ policy.py
  (PDF/DOCX/PPTX/                     │            (PDF/DOCX/PPTX/          │
   audio)                             │             audio)                 ▼
                                      │                            Notes/*.md (skips
                                      │                            embedding if
                                      ▼                            quarantined)
                              chunking + Ollama      tag/link suggestions
                              embeddings  ──▶  Chroma  (Ollama /api/chat, JSON
                                (nomic-embed-text)      schema + sanitizer,
                                      │                 system/user roles)
                                      ▼
                        search.py / chat.py / mcp_server.py / Obsidian plugin
                          (semantic search, RAG chat, MCP tools, provenance badges)

  digest.py (daily) ──▶ Reviews/YYYY-MM-DD.md   (LLM-written summary of the day's changes)
  policy_decisions.jsonl / events.jsonl ──▶ structured, content-free audit trail

Components

• watcher.py — a long-running process that watches the vault, converts source documents dropped into Sources/<project>/ into notes, keeps the vector index in sync, and generates tag/link suggestions.

• index_external_folders.py — indexes external folders (cloud drives, shared libraries) without cloning them: it reads files in place, extracts text, and writes only the resulting note. Meant for scheduled (e.g. daily) runs against folders you don't want to duplicate or watch in real time.

• digest.py — writes a daily summary note of what changed in the vault.

• search.py — one-shot semantic search CLI, also used by the Obsidian plugin.

• common.py — shared helpers: chunking, Ollama calls, document extraction, version/format deduplication.

• obsidian-plugin/ — a minimal Obsidian plugin exposing an in-app search box backed by search.py, and a chat modal backed by chat.py --json (same one-shot-subprocess pattern; the plugin keeps conversation history between messages so chat.py itself stays stateless per call). Both surfaces show small provenance badges (quarantined / external-readonly / ocr / low-confidence) read directly from each note's manifest via Obsidian's own frontmatter cache.

• eval/eval_search.py — a small retrieval-quality harness: runs a set of (query, expected note) pairs through the same search path a user experiences, and reports Precision@1/@3/@5 and mean reciprocal rank. Copy eval/golden_queries.example.json to eval/golden_queries.json (gitignored) and fill it in with real questions and notes from your own vault.

• chat.py — conversational RAG on top of the same index: ask a question, get a synthesized answer with cited sources (the sources are determined by code from what was actually retrieved, not by the model), and ask follow-ups that keep the conversation's context. Complements search.py, which returns raw passages instead.

• benchmarks/quantization_bench.py — compares speed/quality of the same model at different quantization levels (Q4/Q8/fp16) on the same real task, on your own hardware. See "Lessons learned" for what this found on a CPU-only laptop.

• redteam/prompt_injection_test.py — checks that a "malicious" source document (text aimed at the model rather than a human) can't make the tag/link pipeline write something misleading into a note. See "Lessons learned" for what it found.

• finetune/ — prepares your own vault's notes into a dataset and runs a light LoRA fine-tune (MLX, Apple Silicon only) to adapt a small local model's writing style to your corpus. Ships with a synthetic example dataset (finetune/data.example/) so the mechanism can be tried without a real vault. See "Lessons learned" for what this found on a small, real note collection.

• policy.py — the ingestion policy module: path/size checks and MIME detection before a file is read, secret/PII pattern detection on the extracted text, and cloud-sync detection for the vault's own storage location. Returns allow/quarantine/ deny and writes every decision to logs/policy_decisions.jsonl (path and hash only, never content). See "The control plane" below.

• mcp_server.py — exposes search_notes, retrieve_note, explain_sources, daily_digest, and healthcheck over the Model Context Protocol, so Claude Desktop, Claude Code, Cursor, and other MCP clients can query this second brain directly. Reuses search.py/chat.py/digest.py rather than reimplementing retrieval or generation.

• benchmarks/model_registry.py — records the Ollama models this project actually depends on (role, digest, size, install status) in models_registry.json, refreshable without losing manually-added notes. Complements quantization_bench.py, which measures performance rather than recording which models are approved.

• redteam/canary_check.py — re-runs a fixed document through tag/link generation and flags significant drift from a saved baseline (by tag-set similarity, not exact match — a local model's sampling isn't fully deterministic). Catches silent regressions after a model, prompt, or chunking change. Needs Ollama; run locally.

• connectors.py — a minimal Connector protocol (interface only) describing what a future read-only external source would need to implement to plug into this project's ingestion model, with FilesystemConnector (matching what index_external_folders.py already does) as the one concrete implementation. See "What this repository deliberately does not include."

• tests/ — a deterministic, Ollama-free pytest suite (policy decisions, the extended manifest, the tag sanitizer, the connector interface) wired into .github/workflows/test.yml, plus a small synthetic fixture vault (tests/fixtures/) for local, on-demand retrieval-quality checks without needing your own vault.

Models used (all via Ollama, all local)

• nomic-embed-text — embeddings

• llama3.2 (Q4_K_M) — tag/link suggestions, the daily digest, and the RAG chat

Both are small enough to run comfortably on a laptop CPU. See "Lessons learned" for why larger/multimodal models were tried and abandoned for parts of this pipeline, and why Q4_K_M specifically was kept as the default rather than a higher-precision variant.

Scanned PDFs (OCR)

PDFs with no text layer (scanned documents, photographed pages) fall back to local OCR automatically via pytesseract. This needs the tesseract binary installed separately (brew install tesseract tesseract-lang on macOS, apt install tesseract-ocr tesseract-ocr-<lang> on Linux) — if it's missing, those pages are simply left without text, no hard failure. Set OCR_LANGUAGES in common.py (e.g. "eng+ita") to match the languages you actually need; each language requires its own installed tessdata file.

Audio recordings (meeting transcripts)

Supported formats: .mp3, .m4a, .wav, .mp4, .aac, .flac — same Sources/ flow as any other document, no extra configuration. Transcription uses local Whisper (faster-whisper, small model, CPU with int8 quantization — see WHISPER_MODEL_SIZE/AUDIO_LANGUAGE in common.py); each segment becomes a timestamped line (**[MM:SS]**) so you can navigate a long recording without re-listening to it. AUDIO_LANGUAGE defaults to None (auto-detect); set it to a fixed language code to skip detection and speed things up when you already know what to expect.

Tested end-to-end with synthetic speech (macOS say, meeting-style sentences): transcription, indexing, and tag/link suggestions all work exactly as they would for a PDF or DOCX. Not tested against a real meeting recording (multiple speakers, background noise, overlapping speech) — expect lower transcription quality there than with clean synthetic audio.

The control plane

The piece that turns this from "a local RAG pipeline" into something closer to governed infrastructure, per the accompanying paper's Section 7.

The manifest

Every generated note's frontmatter now includes, beyond the original source_file/source_type/source_mtime: content_hash (SHA-256 of the extracted text), resolved_path (the real, symlink-resolved filesystem path the content was actually read from), mime_type, extractor_version (bumped when an extractor's logic changes materially), embedding_model, chunking_version, indexed_at, policy_decision, and sensitivity_flags. PDFs that fell back to OCR also carry ocr_confidence; audio transcripts carry audio_language/audio_language_confidence. None of this needs a database — it's ordinary YAML frontmatter, so a note's own provenance travels with the note.

The policy decision: allow, quarantine, deny

policy.py evaluates every candidate document twice: on its path and size before any content is read (denying anything whose resolved path escapes the trusted source folder — the symlink case below — or that exceeds MAX_FILE_SIZE), and on its extracted text (quarantining anything matching a secret- or PII-like pattern — an API token, an email address, a card-like digit run). A quarantined document is still written as a note and recorded by path and content hash, but the watcher (watcher.py → index_note()) skips semantic embedding for it entirely — it becomes findable by exact reference, never by semantic search or chat. All three outcomes are appended to logs/policy_decisions.jsonl.

Cloud-sync detection

policy.detect_cloud_sync() checks, once at watcher startup, whether the vault's own storage path resolves through a known cloud-sync mount (OneDrive, iCloud, Google Drive, Dropbox) and logs a warning if so — turning the "local processing isn't the same as local storage" finding below from a one-time manual discovery into a standing, automatic check.

Structured event logging

common.log_event() appends operation name, duration, model used, and chunk/source counts — never note content — to logs/events.jsonl, for indexing (watcher.py), tag generation, and chat (chat.py). Adopts OpenTelemetry's span/attribute vocabulary without the OTel SDK or a collector, which would be disproportionate for a single-machine tool; the data shape is compatible if you want to export it elsewhere later.

Continuous integration

.github/workflows/test.yml runs tests/ (policy, manifest, tag sanitizer, connector interface — all deterministic, no Ollama needed) on every push. It deliberately does not run retrieval-quality or model-level red-team checks: hosted runners have no local model to call. Run those yourself:

./venv/bin/python eval/eval_search.py --queries tests/fixtures/golden_queries.json  # against the bundled fixture vault
./venv/bin/python redteam/prompt_injection_test.py
./venv/bin/python redteam/canary_check.py

(Point SECOND_BRAIN_VAULT/SECOND_BRAIN_HOME at tests/fixtures/vault and a scratch directory first if you want to try the fixture vault without touching your own.)

MCP server

./venv/bin/python mcp_server.py

Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS) — use absolute paths, since MCP servers launch without your shell's PATH/venv:

{
  "mcpServers": {
    "second-brain": {
      "command": "/absolute/path/to/venv/bin/python",
      "args": ["/absolute/path/to/mcp_server.py"]
    }
  }
}

Setup

python3 -m venv venv
./venv/bin/pip install -r requirements.txt
# ./venv/bin/pip install -r requirements-dev.txt  # instead, if you also want to run tests/
cp external_folders_config.example.py external_folders_config.py  # if you want external-folder indexing

Set SECOND_BRAIN_VAULT (defaults to ~/Documents/SecondBrain) and SECOND_BRAIN_HOME (defaults to ~/.second-brain) if you want them elsewhere. OLLAMA_URL defaults to http://localhost:11434.

Run the watcher in the foreground to try it out:

./venv/bin/python watcher.py

For always-on background operation, wrap watcher.py, digest.py, and (if used) index_external_folders.py in your OS's service manager (launchd on macOS, systemd on Linux, Task Scheduler on Windows) — this repo doesn't ship service unit files since paths and scheduling preferences are inherently personal.

To build the Obsidian plugin:

cd obsidian-plugin
npm install
npm run build   # produces main.js

Copy the obsidian-plugin/ folder into <your-vault>/.obsidian/plugins/second-brain-search/, then enable it from Obsidian's Community Plugins settings.

Lessons learned

These are the design decisions this project got wrong on the first try, kept here because the failure mode is more instructive than the fix.

• Local vision models are not ready for slide/diagram captioning. An early version extracted embedded images from decks and captioned them with a small local vision model (moondream) to make diagrams searchable. In practice the model returned empty output for non-English prompts, and even in English it confidently hallucinated content on business diagrams (describing an architecture diagram as "a close-up of nerves in a brain"). A wrong caption silently indexed as fact is worse than no caption — the feature was removed entirely rather than shipped as "mostly working." Images are now left out of the index; the diagram itself is simply not represented as text.

• Never trust a model to reproduce a file path. An earlier version asked the LLM to write out full Obsidian wikilinks as free text. It occasionally invented paths that didn't exist, or merged two real paths into one that looked plausible but pointed nowhere. The fix: show the model a numbered list of candidate related notes and ask it to pick indices, then have the code build the actual link from the known, correct path. The model chooses; the code writes.

• Two notes can legitimately share a filename. Documents from different projects sometimes carry the same filename in different folders. A short-form wikilink is ambiguous in that case — Obsidian can't tell which note you mean. The system now detects this collision and automatically falls back to a fully-qualified path only when needed, keeping links short everywhere else.

• A parallel "metadata" folder tree doubles the file count and the cognitive load. An earlier design mirrored every note with a matching file in a separate _Suggestions/ folder, so tags and links never touched the original. For hand-written notes that caution is warranted. But for notes the pipeline itself generated from a source document, there's no user content to protect — so those suggestions are now appended directly to the note, guarded by a content hash to avoid a rewrite loop, cutting the visible file count roughly in half.

• Removing redundant duplicates needs a policy, not vigilance. A folder synced from multiple contributors accumulates near-duplicates: report_v01.pptx, report_v03.pptx, report_vFINAL.pptx, plus a lighter report_v03.pdf export of the same deck. Two simple, composable rules — keep the highest version number (vFINAL always wins), then prefer a PDF over an Office file with the same name — turned out to cover the overwhelming majority of real-world clutter, and are cheap to apply automatically every time new files appear.

• "Local" storage claims need to be checked, not assumed. Partway through this project, checking ~/Documents revealed it was a symlink into a cloud-drive sync folder set up by the OS-level "back up your folders" feature — meaning every file written to the vault had been silently syncing to a cloud account the whole time. The AI processing was local; the storage wasn't, and those are two different guarantees that are easy to conflate. Worth an explicit check before claiming "nothing leaves this machine."

• A daily-cadence, read-only ingestion pass is often the right answer for shared/external content — not because real-time file watching is technically hard, but because a folder you don't own (a shared drive, a colleague's export) doesn't need instant reactivity, and treating it read-only (no copying, no deletion of the archive if a file disappears upstream) is the conservative default that avoids surprises.

• A retrieval-quality harness finds real ambiguity, not just bugs. The first run of eval/eval_search.py against a real note collection scored a modest Precision@1 (33%), and the interesting part wasn't the number — it was which queries failed: two pairs of genuinely different documents about the same underlying event sometimes outrank each other. That's a property of the embedding model and the data, not something to "fix," but it's much better to know it's there than to find out anecdotally.

• A second failed vision model is more useful than a successful one would have been. The image-captioning experiment (see above) was deliberately repeated against a different, similarly-sized local vision model (llava-phi3) on the same real images. It didn't do better — it hallucinated a different but equally confident misreading of the same abstract diagram, and produced degenerate output (not just an empty string) when prompted in Italian. Two independent models failing in related ways is stronger evidence of a genuine capability boundary than one model failing once.

• Higher precision isn't automatically better precision for a small structured task. Benchmarking llama3.2 at Q4_K_M/Q8_0/fp16 on this hardware showed the expected, large speed gap (Q4 roughly 3x faster than fp16 in tokens/sec) but no clearly monotonic quality improvement on the tag/link-suggestion task specifically — fp16 was not obviously better than Q4 on that one job. The default stayed at Q4_K_M: it was already the fastest, and the benchmark didn't surface a quality reason to trade that away.

• A JSON schema is a cheaper fix than a better prompt. The tag/link prompt originally asked the model to format its answer as two labeled free-text lines, parsed with string matching. Ollama's native structured-output support (format as a JSON schema) removes that parsing step entirely — the model fills in fields, the code reads them directly. This doesn't replace the numbered-candidate-list safeguard above (the model still can't invent a path, it can only pick a valid index), but it does remove a second, smaller class of failure: the model almost-but-not-quite following a free-text format.

• A schema constrains shape, not content — and the gap is exploitable. The JSON schema above stops the model from inventing a malformed response, but nothing stopped a tag string itself from containing [[a wikilink]] or an embedded newline. redteam/prompt_injection_test.py planted an injected instruction inside a source document asking the model to emit exactly such a tag. Tested in English against llama3.2, the model complied — it produced the literal string [[Confidential CEO Data]] as a tag, which would have rendered as a real, clickable Obsidian link had it reached the note. The same test in Italian didn't convince the model, which is itself the point: model compliance with an injection is probabilistic and inconsistent across languages, so it can't be the only defense. _sanitize_tag() in watcher.py now rejects any tag containing wikilink brackets, newlines, or the suggestions-marker sequence before it's ever written — a content-level check the schema alone couldn't provide. Its tag count is also capped (maxItems: 10 in the schema, plus the identical cap in code — again, never trust the schema alone) so a misbehaving model can't bloat a note with hundreds of tags. The same two injected documents were also tested against the RAG chat (chat.py): in both cases the model retrieved the malicious note as context but explicitly recognized the embedded instruction as something to ignore rather than follow — encouraging, but observed behavior, not a structural guarantee the way tag sanitization is (chat output is free text shown to the user, not something the code parses and writes into a file, so there's no equivalent code-level backstop here).

• A symlink inside Sources/ silently read whatever it pointed to. A concrete test (a symlink placed inside Sources/ pointing at a file with fake "secret" content outside the vault entirely) showed the pipeline extracting, indexing, and making that content searchable and chattable — exactly as if the user had deliberately dropped it in. Path.read_text() and the other extraction functions follow symlinks transparently by default; nothing in the code checked for that. convert_source() now rejects any path whose resolved location doesn't actually fall under Sources/ before reading it, closing the gap for both single-file and whole-directory symlinks — verified again after the fix. This matters most for synced/shared folders (cloud drives, shared libraries) where a collaborator, or a malformed archive extraction, could introduce a symlink like this without you noticing.

• A LoRA fine-tune mostly learns the pipeline's formatting conventions before it learns a personal voice. finetune/ fine-tuned a small local model (LoRA, MLX, on Apple Silicon) on a real note collection — 13 notes, about 165KB of text, split into 214/26/26 train/valid/test records. Training genuinely worked: validation loss dropped consistently at every checkpoint (from 3.60 to 2.10 over 200 iterations), only 0.216% of the model's parameters were touched (a 28MB adapter file, not a duplicated model), and the whole run took about 13 minutes on a consumer laptop. But the most consistent qualitative change wasn't a personal "writing style" — it was the model reproducing the ingestion pipeline's own markdown conventions (numbered slide headers, short bullet-style fragments) more than any authorial voice, which makes sense once you notice most of the corpus is auto-extracted business documents, not personal reflective writing. The effect was also inconsistent depending on how the model was prompted: it showed up clearly with raw text completion (the format actually used for training) and mostly disappeared when prompted through the instruct model's normal chat template — a training/inference format mismatch that limits how much the adapter generalizes to everyday chat use. None of this is a failure of the technique; it's a direct consequence of fine-tuning on a genuinely small, auto-extracted corpus, which is precisely the caveat worth recording rather than glossing over.

• A mitigation is not a guarantee, and re-testing after a fix is what tells the difference. Moving every Ollama call from a single concatenated prompt to explicit system/user message roles (/api/chat) is a real, defensible hardening step — most instruct models weight the two differently. Re-running the exact same English-language tag-injection test from the finding above after this change produced the same result: the model still complied. The role separation likely raises the cost of some injection attempts without eliminating the risk category — worth knowing precisely because it would have been easy to assume the fix "worked" without re-testing it against the same adversarial input that motivated it.

• A regression check for a non-deterministic system needs a different notion of "pass." redteam/canary_check.py re-runs the same document through tag generation and compares tags against a saved baseline. An exact-match check would fail on almost every run — Ollama's default sampling isn't fully deterministic, confirmed in practice by two consecutive runs of the same canary producing "infrastructure" vs. "infrastructure review" as one of three tags. The check instead measures tag-set similarity (Jaccard) against a threshold, flagging significant drift for a human to look at rather than either accepting any output or demanding bit-for-bit reproducibility a probabilistic system can't honestly offer.

• CI that runs is more valuable than CI that overpromises. The test suite wired into GitHub Actions covers exactly what's genuinely deterministic (policy decisions, manifest fields, tag sanitization) and stops there — hosted runners have no local Ollama instance, so retrieval quality and model-level red-teaming stay local, on-demand checks. Scoping the CI honestly to what it can actually verify was a more useful design decision than adding a job that would either need a self-hosted GPU runner or silently mock the one thing (a real local model's behavior) this project exists to test.

What this repository deliberately does not include

• Any actual vault content, extracted document text, or personal configuration — this is the engine, not a dataset.

• OS-level service files (launchd/systemd) — these encode personal paths and scheduling preferences.

• A hosted or one-click deployment — this is meant to be read, adapted, and run on your own machine, not operated as a service.

• API-based connectors for SharePoint/OneDrive-online, Google Drive, Notion, or Slack/Teams exports — connectors.py defines the interface a future connector like this would need to satisfy, but each of these needs its own authentication, pagination, and rate-limit handling, which is real, separate engineering left as future work rather than claimed as done.

• A machine-learning-based sensitivity classifier — policy.py's secret/PII detection is regex-based, catching the obvious, common cases; a real classifier for legal privilege, board material, or subtler PII would need labeled training data and ongoing maintenance disproportionate to this project's scale.

• A full OpenTelemetry SDK, collector, or dashboard — common.log_event() adopts OTel's vocabulary (named operations, attributes, no content) in plain JSON Lines; wiring an actual exporter is a reasonable next step for someone who wants to aggregate this elsewhere, not something this repository ships.

• Any claim of compliance with NIST's AI RMF or OWASP's LLM Top 10 — the paper maps this project's controls to both frameworks as a shared descriptive vocabulary. A single-maintainer prototype tested by its own author is not an audited system.

License

MIT — see LICENSE.