# workspace-search-mcp

Incremental, private full-text search for a local workspace. Index Markdown, text,
code, PDF, DOCX, XLSX, CSV and HTML with SQLite FTS5, then search from the `wsearch`
CLI or an AI assistant using the official MCP Python SDK.

- Multiple explicitly configured roots; `.gitignore` and `.searchignore` support.
- BM25 ranking, Chinese bigrams, phrases, prefixes, path/type/date filters.
- Source line ranges, PDF pages and spreadsheet sheet/row locations.
- Incremental additions, content changes, deletions and configuration changes.
- Root-confined reads, no symlink following, no network or account required.
- JSON responses, JSON/CSV manifests, consistent SQLite ZIP backups and optional watching.

All bundled documents are fictional. Chinese text is included intentionally to test
multilingual search. Documentation and command output are English.

## Installation

Python 3.11 or 3.12 on Linux or macOS, with SQLite FTS5 enabled:

```sh
python3 -m venv .venv
. .venv/bin/activate
pip install -e .
```

Install `pip install -e '.[dev,watch]'` for tests, fixture regeneration and live
watching. `requirements.txt` installs the same base application. Runtime operations
use local files only; installation downloads open-source dependencies.

## Quick start: three commands

From a clone, with your chosen Python environment active:

```sh
pip install -e .
wsearch index --root examples/corpus --json
wsearch search '"golden bearing"' --json
```

The first index contains **12 files** and no parser errors. The query finds
`inspection.pdf`, page **2**, extracted line **1**. A representative response is:

```json
{"ok":true,"data":[{"path":"/workspace/apps/workspace-search-mcp/examples/corpus/inspection.pdf","ext":"pdf","line_start":1,"line_end":2,"page":2,"sheet":null,"location_kind":"page_text_line","snippet":"[[golden bearing]]\nThe fictional second-page inspection is complete.\n"}],"warnings":[]}
```

The actual response also contains modification time, chunk order and BM25 score.
Try `wsearch search '水化学' --json` to find the Markdown fixture.

## Configuration and data

Default data directory: `./data`, overridden by `WSEARCH_DATA_DIR` or `--data-dir`.
The SQLite database is `index.db`; saved roots/settings are `config.json` in that
directory. An explicit `--config settings.toml` or `WSEARCH_CONFIG` replaces the
saved settings. Relative roots in TOML are relative to that TOML file.

```toml
roots = ["/workspace/documents", "/workspace/notes"]
include = ["**"]
exclude = [".git/", ".venv/", "node_modules/", "private/"]
max_size = 20971520
chunk_size = 800
overlap = 100
timezone = "Asia/Shanghai"
```

See [examples/config.toml](examples/config.toml). `index --root DIR` replaces the
saved root list; repeat `--root` to authorize multiple roots. No root is implicitly
authorized. Ignore patterns use Git semantics, including nested rules and `!`
negations; excluded parent directories cannot be reopened by a child ignore file.
`.searchignore` is applied after `.gitignore` in each directory. The data directory,
`.git`, `.venv`, `node_modules` and `__pycache__` are always excluded.

UTF-8 text is supported, including a BOM; UTF-16 with a BOM is supported for text.
NUL-containing and undecodable text is recorded as a per-file error. Oversized or
unparseable files cannot retain stale searchable text. Roots must remain available
throughout indexing; a missing/inaccessible root aborts reconciliation.

## Commands

Global options work before or after the command: `--data-dir`, `--config`, `--json`,
`--quiet`, `--progress`, `--version`, `--yes`. Commands never require a TTY.
`--quiet` suppresses progress, not the machine-readable result. `--progress` writes
to stderr; stdout contains one JSON object. `--yes` is accepted for automation
compatibility; no command deletes source files or overwrites an existing export.

| Command | Purpose and example |
|---|---|
| `index` | `wsearch index --root ./notes --root ./reports --progress --json` |
| `index --full` | Reparse all eligible files, retaining source files |
| `search` | `wsearch search '"inspection complete" ext:pdf after:2026-01-01' --limit 10 --json` |
| `read` | `wsearch read ./notes/example.md --lines 120-180 --json` |
| `read --page` | `wsearch read ./reports/example.pdf --page 3 --json` |
| `read --sheet` | `wsearch read ./reports/example.xlsx --sheet Samples --lines 2-8 --json` |
| `recent` | `wsearch recent --since 24h --json` |
| `stats` | `wsearch stats --json` |
| `watch` | `wsearch watch --debounce 1 --progress`; Ctrl-C returns one final result |
| `watch --once` | One scan without requiring the optional watchdog package |
| `export` | `wsearch export --format json --out ./manifest.json --json` |
| `export` | `wsearch export --format csv --out ./manifest.csv --no-bom --json` |
| `import` | `wsearch import ./manifest.json --json` |
| `backup` | `wsearch backup --json`; optionally `--out ./unique-backup.zip` |
| `mcp` | `wsearch --data-dir /workspace/data/workspace-search-mcp mcp` |

### Query language

Whitespace joins terms with AND. Double quotes create a phrase; suffix `*` matches
prefixes (`inspect*`). Chinese-only phrases use ordered bigrams, with single-character
support. Latin text uses SQLite's `unicode61` tokenizer. Boolean FTS syntax is not
accepted as raw SQL or raw FTS expressions; words are safely quoted.

- `path:reports/`: literal path substring, case-sensitive.
- `ext:pdf`: normalized extension; also available as `--ext pdf`.
- `after:2026-01-01`: source modification time, not content publication date.
- `--path-prefix /workspace/documents/`: literal absolute path prefix.
- Filter-only searches such as `ext:pdf` are valid.
- Times without offsets use the configured timezone; `recent --since` also accepts
  `30m`, `24h`, `7d`, `2w` and ISO 8601 timestamps.

Lower BM25 scores rank first. Results are chunks, so a file can appear more than
once. `[[...]]` highlights are plain text; escape them if displaying in HTML.
An index is a snapshot: run `index` or `watch` after editing content. Search and
read apply current ignore/root restrictions even before a fresh index.

### Location semantics

| Format | `location_kind` and position meaning |
|---|---|
| Text/Markdown/code | `source_line`: actual one-based inclusive source lines |
| PDF | `page_text_line`: PDF page plus line in extracted text on that page |
| DOCX/HTML | `extracted_line`: line in parser output, not Word page or HTML source |
| XLSX | `sheet_row`: sheet name and original worksheet row; embedded cell newlines are escaped |
| CSV | `csv_record`: one-based logical CSV record including header; multiline cells are escaped |

Chunks target 800 characters with 100-character overlap, preferring paragraph,
heading and line boundaries. Very long lines are split with their original line
number. `read` returns the same parser representation. `--page` and `--lines` are
mutually exclusive; `--sheet` selects an XLSX sheet. Lines on a PDF without `--page`
are selected separately within each page. PDF extraction order is library-dependent;
no visual column-layout reconstruction or OCR is promised.

### Import, export and backups

Exports are **source manifests**, not copies of source documents. JSON uses
`{"schema_version":1,"files":[...]}`; CSV columns are
`path,ext,size,mtime,sha1,indexed_at,error`, UTF-8 with BOM by default.
Import validates every path against current roots and ignore rules, then re-reads
the original files. It does not trust imported timestamps, hashes or indexed text.
This makes re-import deterministic against current sources and avoids text injection
from a stale manifest. Original source files must still exist; move/remap manifest
paths deliberately when migrating machines.

`backup` uses SQLite's backup API and writes a dated ZIP containing `index.db`,
`config.json` and a restore note. It includes cached extracted content but **does
not include source documents**. Keep backups private. To restore, stop writers,
back up the current data directory, then extract a trusted backup into a new data
directory and update its roots before indexing. Existing export/backup paths are
refused rather than overwritten.

## MCP integration

The official `mcp` SDK is constrained to `>=1.20,<2` for its FastMCP API. Stdio only:

```json
{
  "mcpServers": {
    "workspace-search": {
      "command": "/workspace/apps/workspace-search-mcp/.venv/bin/wsearch",
      "args": ["--data-dir", "/workspace/data/workspace-search-mcp", "mcp"]
    }
  }
}
```

Initialize roots first using `index --root ...`. Tools:
`search_files(query, limit=10, ext=null, path_prefix=null)`,
`read_file(path, start_line=null, end_line=null, page=null, sheet=null)`,
`recent_files(since="24h")`, `reindex(path=null)`.
`reindex` modifies the derived index and may remove stale entries; it never changes
source files or adds authorized roots. Restart the server after changing configuration.
Full generated schemas are in [docs/mcp-tools.json](docs/mcp-tools.json).

## Validation and benchmark

```sh
pip install -e '.[dev,watch]'
ruff check .
ruff format --check .
pytest -q --cov=workspace_search_mcp --cov-report=term-missing
python docs/benchmark.py
```

The suite includes 10 fixed relevance/location queries in `examples/queries.json`,
all MCP tools plus a real SDK stdio session, live watching, snapshot integrity,
configuration changes, same-size/same-mtime modifications, and traversal/symlink
rejection. GitHub Actions runs Python 3.11 and 3.12 on Ubuntu.

Measured on Darwin arm64, Python 3.12.13, 10,000 synthetic small UTF-8 files
(797,780 bytes total): initial index **4.802 s**, unchanged rescan **4.511 s**,
Chinese top-10 query **0.026 s**. These local timings are not a performance guarantee;
see [docs/benchmark.json](docs/benchmark.json) for timestamp and raw counters.
Every rescan hashes file bytes, even when mtime/size match, to catch metadata-preserving
edits. Only changed files are reparsed. Large PDF/Office files have different costs.

## FAQ and limitations

- **Will this send my files anywhere?** No. Runtime code has no networking path.
- **Can an assistant read outside the roots?** No. Parent traversal, symlinks,
  excluded files, non-regular files and oversized files are denied. POSIX
  descriptor-relative opens prevent symlink replacement during file opening.
- **Windows?** Not supported in this version: secure file access requires POSIX
  `dir_fd` and `O_NOFOLLOW`. Linux and macOS are the tested/designated platforms.
- **Scanned/encrypted PDFs?** No OCR and no password handling. Empty scanned PDFs
  are recorded as errors. Use a separate offline OCR workflow before indexing.
- **Word page numbers?** DOCX has no stable page model here; extracted lines are labeled explicitly.
- **Spreadsheet formulas?** Stored formula text is indexed; formulas are never evaluated.
- **Other encodings?** Convert them to UTF-8 first. UTF-16 requires a BOM.
- **Parser extensions?** Register a byte parser using `core.parsers.register_parser`.
  See [docs/design.md](docs/design.md). Plugins execute only when explicitly imported in-process.
- **macOS editable import failure?** If Python reports `Skipping hidden .pth file`
  under `python -v`, clear the hidden flag only on your environment's `.pth` files:
  `chflags nohidden .venv/lib/python3.12/site-packages/*.pth`.

## Roadmap

Improve CJK/Latin mixed-phrase matching, add richer location-aware previews, and
measure large mixed-format corpora. Cloud sync and OCR remain outside this project.

## References and license

Implementation references: [SQLite FTS5](https://www.sqlite.org/fts5.html),
[official MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk),
[PathSpec](https://python-path-specification.readthedocs.io/en/latest/readme.html).
See [AGENTS.md](AGENTS.md) for the operator handbook and [LICENSE](LICENSE) for MIT terms.