# Semantic Search

[![CI](https://github.com/bborbe/semantic-search/actions/workflows/ci.yml/badge.svg)](https://github.com/bborbe/semantic-search/actions/workflows/ci.yml)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/bborbe/semantic-search)

Semantic search over markdown files. Find related notes by meaning, not just keywords. Detect duplicates before creating new notes.

Supports two server transports:
- **stdio MCP** — For Claude Code integration (one process per session)
- **HTTP** — Combined MCP-over-HTTP + REST on one port; one warm process shared by all clients

## Features

- Semantic search using sentence-transformers
- Duplicate/similar note detection
- Auto-updating index with file watcher
- Multi-directory support
- Inline tag extraction (`#tag-name`)

## Install

**CPU-only install** — recommended for **macOS** (any Mac, Apple Silicon or Intel) and **Linux/Windows without an NVIDIA GPU**. Saves ~5GB of CUDA binaries. On macOS, Apple GPU (MPS) is still auto-detected and used via PyTorch's built-in MPS backend — the "CPU" label refers only to the absence of CUDA, not to the compute device at runtime.

```bash
uv tool install --index https://download.pytorch.org/whl/cpu \
  git+https://github.com/bborbe/semantic-search
```

**CUDA install** — only for **Linux/Windows with a dedicated NVIDIA GPU**. Not applicable to macOS (NVIDIA CUDA is not supported on Mac).

```bash
uv tool install git+https://github.com/bborbe/semantic-search
```

## Upgrade

```bash
uv tool upgrade semantic-search
```

## Server Modes

### stdio MCP (per-session Claude Code)

Spawns one process per Claude Code session. Simple, but each session loads its own ~400 MB–1 GB model copy.

```bash
claude mcp add -s project semantic-search \
  --env CONTENT_PATH=/path/to/vault \
  -- \
  uvx --from git+https://github.com/bborbe/semantic-search semantic-search-mcp serve
```

**Tools available:**
- `search_related(query, top_k=5)` — Find semantically related notes
- `get_content(path, snippet, query, context_lines)` — Retrieve file content from indexed vaults
- `check_duplicates(file_path)` — Detect duplicate/similar notes

### HTTP (shared across all clients)

Single long-running process serves MCP-over-HTTP at `/mcp` plus REST at `/search`, `/duplicates`, `/content`, `/health`, `/reindex`. All Claude Code sessions and REST clients share **one warm indexer**, and each request names the **scope** it wants — so one process gives every client its own view without a second index.

```bash
SEMANTIC_SCOPE_MAP=scopes.yaml semantic-search-http --host 127.0.0.1 --port 8321
```

The variable may be omitted, in which case the scope map is read from `~/.config/semantic-search/config.yaml`.

Scopes are declared in `scopes.yaml` — see [design/per-vault-scoping.md](docs/design/per-vault-scoping.md) for the contract and the fail-closed default. Point Claude Code at it via MCP config; clients differ only by the scope in their URL:

```json
{
  "mcpServers": {
    "semantic-search": {
      "type": "http",
      "url": "http://127.0.0.1:8321/mcp?scope=personal"
    }
  }
}
```

**REST endpoints:**

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/mcp` | POST | MCP-over-HTTP (Claude Code); scope comes from the URL |
| `/search?q=...&top_k=5&scope=...` | GET | Semantic search within a scope |
| `/duplicates?file=...&threshold=0.85&scope=...` | GET | Find duplicate notes within a scope |
| `/content?path=...&snippet=...&query=...&context_lines=...&scope=...` | GET | Retrieve file content within a scope |
| `/health` | GET | Health check with index stats (scopeless) |
| `/reindex` | GET/POST | Force index rebuild (scopeless) |

`scope` is **required** on `/search`, `/duplicates`, and `/content`. Omitting it returns HTTP 400 `MISSING_SCOPE`; an unknown name returns 400 `UNKNOWN_SCOPE`. Repeating the parameter resolves to the **union** of the named scopes' roots — `/search?q=...&scope=personal&scope=work` searches both scopes in one request, and the order the scopes are named in does not change the result set. There is deliberately no union-wide fallback: naming no scope, or any undeclared name, fails the whole request closed.


**Example queries:**
```bash
# Search
curl 'http://127.0.0.1:8321/search?q=kubernetes+deployment'

# Find duplicates
curl 'http://127.0.0.1:8321/duplicates?file=notes/my-note.md'

# Health check
curl 'http://127.0.0.1:8321/health'
```

### Two-Step Flow

Search for related notes, then fetch the full content of any result:

```bash
# Step 1: Search for related notes
curl 'http://127.0.0.1:8321/search?q=kubernetes+deployment'
# Returns: [{"path": "notes/k8s-guide.md", "score": 0.92}, ...]

# Step 2: Fetch the content of a result
curl 'http://127.0.0.1:8321/content?path=notes/k8s-guide.md'
# Returns: {"path": "/full/resolved/path.md", "content": "# Kubernetes Guide\n...", "mode": "full"}
```

### Snippet Mode

Retrieve a focused snippet around a specific query term within a file:

```bash
# Get a focused snippet around "service mesh" in the file
curl 'http://127.0.0.1:8321/content?path=notes/k8s-guide.md&snippet=true&query=service+mesh&context_lines=10'
# Returns: {"path": "...", "content": "...\n## Service Mesh\n...", "mode": "snippet"}
```

### Remote Deployment

`get_content` and `GET /content` enable remote deployment of semantic-search clients. Callers no longer need filesystem access to the vault directory — all content retrieval happens over HTTP/MCP from any network location. The server enforces path validation: files outside indexed roots are never served.

## Claude Code Plugin

This repo also ships as a Claude Code marketplace plugin with commands for setup, search, and research.

### Install

```bash
claude plugin marketplace add bborbe/semantic-search
claude plugin install semantic-search
```

### Update

```bash
claude plugin marketplace update semantic-search
claude plugin update semantic-search@semantic-search
```

### Quick Start

```bash
# One-shot interactive setup: installs the binary, writes the launchd/systemd
# unit, registers the MCP server in your Claude config.
/semantic-search:configure

# Search indexed markdown
/semantic-search:search kubernetes deployment

# Multi-step research across results
/semantic-search:research kafka backup strategy
```

### Commands

| Command | Description |
|---------|-------------|
| `/semantic-search:configure` | Install `semantic-search-http` as a launchd (macOS) or systemd-user (Linux) service and register the MCP server in Claude Code |
| `/semantic-search:search <query> [top_k]` | Semantic search via the running MCP server |
| `/semantic-search:research <topic>` | Multi-step research — search, categorize, read top sources, synthesize |

## Run in Background

For production-style usage, run `semantic-search-http` as a background service so every Claude Code session (and any REST client) shares one warm process.

| Platform | Guide |
|----------|-------|
| macOS (launchd) | [`docs/launchd-service.md`](docs/launchd-service.md) |
| Linux (systemd) | [`docs/systemd-user-service.md`](docs/systemd-user-service.md) |

Quick example (macOS):

```bash
launchctl load ~/Library/LaunchAgents/com.github.bborbe.semantic-search-http.plist
```

Quick example (Linux):

```bash
systemctl --user enable --now semantic-search-http.service
```

## CLI Commands

One-shot commands without running a server:

```bash
# Search
CONTENT_PATH=/path/to/vault semantic-search search "kubernetes deployment"

# Find duplicates
CONTENT_PATH=/path/to/vault semantic-search duplicates path/to/note.md
```

## Binaries

| Binary | Purpose |
|--------|---------|
| `semantic-search-http` | Combined HTTP server — MCP at `/mcp` + REST endpoints. Run once, share across clients. |
| `semantic-search-mcp` | stdio MCP server — one per Claude Code session. Use when HTTP service is not set up. |
| `semantic-search` | CLI only — `search` and `duplicates` one-shot commands. |

## Configuration

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `CONTENT_PATH` | Directory to index (comma-separated for multiple) | `./content` |
| `LOG_LEVEL` | Logging level (DEBUG, INFO, WARNING, ERROR) | `INFO` |

### Multiple Directories

Index multiple directories by separating paths with commas:

```bash
CONTENT_PATH=/path/to/vault1,/path/to/vault2,/path/to/docs
```

All directories are indexed together and searched as one unified index.

### Excluding Files with `.semanticignore`

Place a `.semanticignore` file at the root of any vault to exclude paths from indexing. Each vault uses its own rules independently; a missing `.semanticignore` means "index everything" (no change for existing vaults).

**Syntax** — gitignore-style patterns powered by the `pathspec` library's `gitwildmatch` dialect. The full gitignore rule set applies:
- Patterns are matched relative to the vault root.
- Trailing `/` matches directories (and everything inside them): `archive/`
- `**` matches across directory boundaries: `**/draft.md`
- Leading `/` anchors a pattern to the vault root: `/scratch.md`
- `!` negates a pattern (re-includes a previously excluded path): `!archive/keep.md`

**Behaviour** — matching paths are excluded from:
- Full index rebuilds (`rebuild_index`)
- Single-file add/update events (`add_file_to_index`)
- File-watcher `created`/`modified`/`moved` events

The `.semanticignore` file itself is never indexed.

**Runtime reload** — editing `.semanticignore` while the watcher is running triggers an atomic reload of that vault's rules. Subsequent file events immediately use the new patterns; no restart is required.

**Example `.semanticignore`:**

```gitignore
# Exclude an entire directory
archive/

# Exclude all draft files in any subdirectory
**/draft.md

# Re-include one specific file from the excluded directory
!archive/keep.md

# Anchor a pattern to the vault root only
/scratch.md
```

## How It Works

First run downloads a small embedding model (~90MB) and indexes your markdown files (<1s for typical vaults). The index auto-updates when files change via filesystem watcher.

### Indexed Content

Each markdown file is indexed with weighted components:

| Component | Weight | Notes |
|-----------|--------|-------|
| Filename | 3x | |
| Frontmatter `title` | 3x | |
| Frontmatter `tags` | 2x | Merged with inline tags |
| Frontmatter `aliases` | 2x | |
| Inline tags (`#tag`) | 2x | Extracted from body |
| First H1 heading | 2x | |
| Body content | 1x | First 500 words |

## Development

```bash
# Clone
git clone https://github.com/bborbe/semantic-search
cd semantic-search

# Install dev dependencies
make install

# Run checks
make check

# Run tests
make test
```

## License

BSD 2-Clause License — see [LICENSE](LICENSE).