Skip to main content
Glama
vault-search

vault-search

by JoshTeperman

vault-search

Read-only search interface for a private knowledge vault, with a bundled MCP server. It is intended for coding agents: find a page or section, read only what is needed, and keep context small.

This repository holds the search interface, the shared secret detector, a local FTS5 indexer with optional hybrid search, and a Streamable HTTP MCP server. The vault itself is a separate private checkout and is never committed here. Milestones 4–6 are implemented in this tree. The v1 deploy is keyword-only (full-text) search: install without the [semantic] extra. Meaning-based search can be turned on later by installing .[semantic] and reindexing; see DEPLOY.md.

Non-goals

  • Writing back into the vault (no ingest or "add this note" tool).

  • Exposing the service on the public internet.

  • Hiding or filtering pages by access control. Project names and tags are optional search filters, not permissions. Every authorised caller can search the full vault.

  • Building a wider context gateway. Tool names and result shapes are kept stable so a gateway can front this service later.

  • Shipping vault notes, hostnames, overlay-network details, addresses, or credentials in this repository.

Related MCP server: Yaps MCP

Tool contract

All tools are read-only. Every response is run through the secret masker. Results are compact markdown (path, title, heading breadcrumb, projects/tags, snippet) rather than raw JSON. There are no ids, raw scores, or UUIDs.

The Python API (Client) is what the MCP server wraps. The v1 deploy does not install embeddings, so search is keyword (full-text) only. The API default mode=hybrid fuses full-text with local semantic search (RRF, k=60) when embeddings exist; if the [semantic] extra is not installed, or the index has no vectors, hybrid and semantic fall back to full-text. mode=fulltext and mode=semantic remain explicit overrides. Filters apply to every mode. Do not treat a hybrid fallback as a semantic result.

Tool

Arguments

Result

search

query: string, mode: "hybrid" | "fulltext" | "semantic" = "hybrid", project?: string, tags?: string[], type?: string, limit: int = 8

Markdown hit list. Default limit is 8, hard cap 20. hybrid falls back to full-text when embeddings are not installed.

read_note

path: string, section?: string

Markdown header, links, backlinks, and the named section, the body, or a heading outline if the body is long. not_found only when the path is genuinely missing.

list_notes

project?: string, tags?: string[], type?: string, limit: int = 20, after?: string

Markdown list. after is the last path, not an opaque cursor.

list_projects

(none)

Markdown list of every project (name and one-line description).

path is a vault-relative wiki path such as wiki/concepts/example.md.

On a v1 (keyword-only) install, search returns full-text hits even if the caller passes mode=hybrid. mode is otherwise for debugging once semantic search is enabled. Optional project / tags / type filters narrow results; they do not hide pages from an unfiltered call.

read_note of a long page (over about 8,000 characters) without section returns a heading outline instead of the body.

Configuration

Runtime configuration comes from the environment (or a gitignored .env). This repository contains placeholders only. On the VPS, the filled-in file lives at /etc/vault-search/env and is not committed. See DEPLOY.md.

Variable

Purpose

VAULT_ROOT

Path to the local vault checkout. That checkout is not part of this repo. VAULT_PATH is accepted as an alias if VAULT_ROOT is unset.

INDEX_PATH

Path to the local SQLite FTS5 index. Defaults to index/vault.db under the current directory if unset (gitignored).

INDEX_INCLUDE_RAW

Set to 1 to also index raw/. Off by default.

EMBEDDING_MODEL

Unused in the v1 keyword-only deploy. Default BAAI/bge-small-en-v1.5 when semantic search is enabled later.

EMBEDDING_CACHE_DIR

Unused in v1. Directory for downloaded ONNX model files (gitignored) once semantic search is enabled.

EMBEDDING_MODEL_PATH

Unused in v1. Optional path to a pre-fetched model directory. Vault text is not sent to a hosted embedding API.

INDEX_INCLUDE_EXAMPLES

Set to 1 to index wiki/_examples and pages marked example: true. Off by default.

DENYLIST

Comma-separated device names, tags, or logins to reject.

AUDIT_LOG_PATH

Path to the append-only audit log.

DENYLIST_PATH

Optional path to a caller denylist (defence in depth, on top of overlay-network access rules).

LISTEN_HOST

Bind address. Must be 127.0.0.1 in production; the reverse proxy listens on the tailnet. Binding 0.0.0.0 or :: is refused unless LISTEN_ALLOW_NONLOCAL=1.

LISTEN_ALLOW_NONLOCAL

Set to 1 only if you really mean to bind a non-loopback address. Leave unset in production.

LISTEN_PORT

Bind port on loopback.

MCP_PATH

URL path the proxy mounts, default /vault/mcp. The public endpoint is https://<vps>.<tailnet>.ts.net/vault/mcp (placeholders only).

PUBLIC_BASE_URL

Tailnet-only base URL, for example https://<vps>.<tailnet>.ts.net. Use the real MagicDNS name only in the on-box env file.

IDENTITY_SOURCE

How the app identifies the caller. Production is whois (nginx does not inject Tailscale identity). See Caller identity.

TAILSCALED_SOCKET

Path to the local tailscaled Unix socket used for WhoIs, usually /var/run/tailscale/tailscaled.sock.

Copy .env.example to .env when running locally. Never commit .env or any filled-in values.

Security

No vault content in this repo. Notes, search indexes, model caches, and .env files are gitignored. Tests use synthetic fixtures, not vault pages. Documentation uses placeholder names only.

Secrets must never be stored in the vault. This repo is the home of the shared detector. Vault ingest lint should call the same scan API (pin a release, or vendor a copy). The service also masks every read and search result, which is defence in depth, not a licence to store secrets.

Detector API and CLI

from vault_search.secrets import scan, mask

findings = scan(text)          # rule, start, end, line, column; never the secret value
safe = mask(text)              # replaces spans with [REDACTED:<rule>]
python -m vault_search.secrets scan <paths>

The command exits 1 when it reports a finding, 0 when the paths are clean, and 2 on usage or missing-path errors. Output is path:line:column: rule. Matched values are never printed. A mask subcommand reads stdin and writes redacted text to stdout.

Rules cover common token formats (GitHub, OpenAI, Anthropic, AWS, Slack, Google), JWTs, private-key blocks, and high-entropy key= / token= (and similar) assignments.

Audit log. Every tool call is appended to the log at AUDIT_LOG_PATH: timestamp, caller identity, tool, arguments, result_count, and whether a secret-like span was masked (rule, path, span — not the secret value). Search calls that return no hits are logged with result_count 0 and zero_hits true. Vault Keeper uses those result counts to decide whether to enable semantic search later. Do not log secret values or vault body text. The log stays on the host; it is not committed here.

Caller identity and exposure. The service is reachable only on a private tailnet, not the public internet. nginx on the VPS listens on the node's MagicDNS name only and serves this app at https://<vps>.<tailnet>.ts.net/vault/mcp (placeholders; never commit real names). TLS certificates come from tailscale cert. The nginx config lives in a separate git on the VPS, not in this repo. The systemd unit binds to 127.0.0.1 only. Agents connect over HTTPS, not SSH tunnels. Admin access is Tailscale SSH, governed by ACLs, not SSH keys. Funnel stays off.

Caller identity is recorded on every call and used for the denylist. IDENTITY_SOURCE=whois is required under nginx: nginx does not inject Tailscale identity. It sets X-Real-IP and X-Forwarded-For from $remote_addr. The app trusts those headers only when the immediate TCP peer is loopback, then looks up the client address via the local tailscaled socket (GET /localapi/v0/whois?addr=... on TAILSCALED_SOCKET). That returns node name and tags, and a user profile when the node is not tagged. Reject requests that carry Tailscale-Funnel-Request.

A caller that is not on the tailnet, or that is on the denylist, is rejected.

Identity v1 (as of 17 Nov 2026). The shared Grok Bot box used by all of Josh's agents is on the tailnet through the Tailscale connector. It is a single untagged device carrying Josh's identity, meant for read and verify only, and it can reach tailnet hosts over HTTP(S)/TCP. WhoIs therefore cannot tell agents on that box apart: every call looks like the same user-owned node. Identity v1 is the device (WhoIs), plus a self-reported X-Vault-Agent-Name header. The audit log records the header when present and marks it unverified. The header is not used for authorisation or revocation. Revocation is per device (overlay ACL, device removal, or the service denylist). True per-agent identity comes later, from separate tagged devices or MCP OAuth through the gateway. Do not write real host or tailnet names in this repo.

This same shared box is Vault Keeper's verification node for the DEPLOY.md completion criteria. A report from the VPS deploy executor is not acceptance.

Deploy

Vault Keeper owns the code in this repository. Deployment on the VPS is done by pulling a release and following DEPLOY.md: service user, pull-only vault clone, loopback systemd unit, and an nginx location /vault/mcp in the on-box nginx git. The MCP server in this repo binds to loopback and is meant to sit behind the VPS nginx location. The runbook has checkable completion criteria for the VPS steps, which are still blocked on access.

Vault Keeper independently verifies a deploy from the shared Grok Bot box (the verification node; see Caller identity). A report from the VPS agent is not acceptance. For the externally visible checks (MCP initialize / tools/list, masked fixture query, public IP must fail):

# Real values only in the shell, never committed.
export PUBLIC_BASE_URL='https://<vps>.<tailnet>.ts.net'
export MCP_PATH=/vault/mcp
# optional:
# export PUBLIC_IP='<public-ip>'
# export VERIFY_FIXTURE_PATH='wiki/example.md'
# export VERIFY_SYNTHETIC_SECRET='...'   # assembled at runtime; never commit
python3 scripts/verify-deploy

Local index and CLI

Requires a vault checkout path and an index path (see configuration). wiki/ is indexed first; raw/ is off unless INDEX_INCLUDE_RAW=1 or --include-raw. Reindex is incremental (git SHA plus per-file mtimes and SHA-256). Tests use only synthetic fixtures under tests/fixtures; never point the test suite at a real vault.

from vault_search import Client

with Client.from_env() as vault:
    vault.reindex()
    print(vault.search("evals", project="alpha"))
    print(vault.read_note("wiki/concepts/evals.md", section="Scoring"))
    print(vault.list_notes(type="concept", after="wiki/concepts/example.md"))
    print(vault.list_projects())
python -m vault_search index
python -m vault_search search "evals" --project alpha --limit 8
python -m vault_search search "evals" --mode fulltext
python -m vault_search read wiki/concepts/evals.md --section Scoring
python -m vault_search list-notes --type concept
python -m vault_search list-projects
python -m vault_search sync          # git pull --ff-only at VAULT_ROOT, then reindex
python -m vault_search.mcp           # Streamable HTTP on 127.0.0.1 (or `vault_search serve`)
python -m vault_search.secrets scan .

Milestones

  1. Public repo and secret scanner — done. Detector, tests, CI, pre-commit, deploy runbook.

  2. Access and inventory — blocked on VPS access.

  3. Pull-only vault sync on the VPS — blocked on VPS access.

  4. Index and CLI — done in this tree. SQLite FTS5 over wiki/, heading chunks, backlinks, compact masked markdown, incremental reindex.

  5. Hybrid search — done in this tree, optional. The v1 deploy is keyword-only (no [semantic] extra). Installing fastembed (BAAI/bge-small-en-v1.5, ONNX, CPU) and reindexing attaches vectors to chunk_embeddings and turns on RRF hybrid (k=60).

  6. MCP server — done in this tree (loopback Streamable HTTP, WhoIs identity, denylist, audit log, scripts/verify-deploy). Binding it to the VPS tailnet is still blocked on VPS access; see DEPLOY.md.

  7. Adapter-ready — tool contract documented for a future gateway; no other data sources.

Development

Requires Python 3.11 or later. The core indexer is stdlib plus the official mcp SDK (also listed in the dev extra). Local embeddings are optional: pip install '.[semantic]' (fastembed/ONNX, not torch). The v1 deploy does not install that extra.

python -m pip install -e ".[dev]"
python -m pip install -e ".[semantic]"   # optional; not part of the v1 deploy
python -m pytest -q
python -m vault_search.secrets scan .

A GitHub Actions workflow runs the tests and gitleaks on push and pull request. To run the same hook locally:

pre-commit install
pre-commit run --all-files

This project is licensed under the MIT License.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Provides semantic search and keyword search over Obsidian notes, along with direct note retrieval, allowing external AI agents to query and access the vault.
      19
      BSD Zero Clause
    • A
      license
      Not graded
      quality
      C
      maintenance
      Provides controlled, read-only or write-enabled access to a private local Yaps Markdown vault, enabling AI clients to search, read, and manage notes securely.
      MIT
    • F
      license
      Not graded
      quality
      B
      maintenance
      Provides read-only access to a personal RAG knowledge base, enabling hybrid search, evidence-grounded retrieval with citations, and knowledge gap tracking for LLM agents.
      -
    • A
      license
      B
      quality
      B
      maintenance
      Enables AI agents to capture, search, relate, review, and safely update a local-first Markdown knowledge vault through typed operations such as inventory, retrieve, run_view, update, add_attachment, trash, and restore. Agents get validated access to personal notes and attachments without a generic shell or arbitrary filesystem access.
      28
      Apache 2.0