workspace-search-mcp

Incremental, private full-text search for a local workspace. Index Markdown, text, code, PDF, DOCX, XLSX, CSV and HTML with SQLite FTS5, then search from the wsearch CLI or an AI assistant using the official MCP Python SDK.

• Multiple explicitly configured roots; .gitignore and .searchignore support.

• BM25 ranking, Chinese bigrams, phrases, prefixes, path/type/date filters.

• Source line ranges, PDF pages and spreadsheet sheet/row locations.

• Incremental additions, content changes, deletions and configuration changes.

• Root-confined reads, no symlink following, no network or account required.

• JSON responses, JSON/CSV manifests, consistent SQLite ZIP backups and optional watching.

All bundled documents are fictional. Chinese text is included intentionally to test multilingual search. Documentation and command output are English.

Installation

Python 3.11 or 3.12 on Linux or macOS, with SQLite FTS5 enabled:

python3 -m venv .venv
. .venv/bin/activate
pip install -e .

Install pip install -e '.[dev,watch]' for tests, fixture regeneration and live watching. requirements.txt installs the same base application. Runtime operations use local files only; installation downloads open-source dependencies.

Related MCP server: litedocs-mcp

Quick start: three commands

From a clone, with your chosen Python environment active:

pip install -e .
wsearch index --root examples/corpus --json
wsearch search '"golden bearing"' --json

The first index contains 12 files and no parser errors. The query finds inspection.pdf, page 2, extracted line 1. A representative response is:

{"ok":true,"data":[{"path":"/workspace/apps/workspace-search-mcp/examples/corpus/inspection.pdf","ext":"pdf","line_start":1,"line_end":2,"page":2,"sheet":null,"location_kind":"page_text_line","snippet":"[[golden bearing]]\nThe fictional second-page inspection is complete.\n"}],"warnings":[]}

The actual response also contains modification time, chunk order and BM25 score. Try wsearch search '水化学' --json to find the Markdown fixture.

Configuration and data

Default data directory: ./data, overridden by WSEARCH_DATA_DIR or --data-dir. The SQLite database is index.db; saved roots/settings are config.json in that directory. An explicit --config settings.toml or WSEARCH_CONFIG replaces the saved settings. Relative roots in TOML are relative to that TOML file.

roots = ["/workspace/documents", "/workspace/notes"]
include = ["**"]
exclude = [".git/", ".venv/", "node_modules/", "private/"]
max_size = 20971520
chunk_size = 800
overlap = 100
timezone = "Asia/Shanghai"

See examples/config.toml. index --root DIR replaces the saved root list; repeat --root to authorize multiple roots. No root is implicitly authorized. Ignore patterns use Git semantics, including nested rules and ! negations; excluded parent directories cannot be reopened by a child ignore file. .searchignore is applied after .gitignore in each directory. The data directory, .git, .venv, node_modules and __pycache__ are always excluded.

UTF-8 text is supported, including a BOM; UTF-16 with a BOM is supported for text. NUL-containing and undecodable text is recorded as a per-file error. Oversized or unparseable files cannot retain stale searchable text. Roots must remain available throughout indexing; a missing/inaccessible root aborts reconciliation.

Commands

Global options work before or after the command: --data-dir, --config, --json, --quiet, --progress, --version, --yes. Commands never require a TTY. --quiet suppresses progress, not the machine-readable result. --progress writes to stderr; stdout contains one JSON object. --yes is accepted for automation compatibility; no command deletes source files or overwrites an existing export.

Query language

Whitespace joins terms with AND. Double quotes create a phrase; suffix * matches prefixes (inspect*). Chinese-only phrases use ordered bigrams, with single-character support. Latin text uses SQLite's unicode61 tokenizer. Boolean FTS syntax is not accepted as raw SQL or raw FTS expressions; words are safely quoted.

• path:reports/: literal path substring, case-sensitive.

• ext:pdf: normalized extension; also available as --ext pdf.

• after:2026-01-01: source modification time, not content publication date.

• --path-prefix /workspace/documents/: literal absolute path prefix.

• Filter-only searches such as ext:pdf are valid.

• Times without offsets use the configured timezone; recent --since also accepts 30m, 24h, 7d, 2w and ISO 8601 timestamps.

Lower BM25 scores rank first. Results are chunks, so a file can appear more than once. [[...]] highlights are plain text; escape them if displaying in HTML. An index is a snapshot: run index or watch after editing content. Search and read apply current ignore/root restrictions even before a fresh index.

Location semantics

Chunks target 800 characters with 100-character overlap, preferring paragraph, heading and line boundaries. Very long lines are split with their original line number. read returns the same parser representation. --page and --lines are mutually exclusive; --sheet selects an XLSX sheet. Lines on a PDF without --page are selected separately within each page. PDF extraction order is library-dependent; no visual column-layout reconstruction or OCR is promised.

Import, export and backups

Exports are source manifests, not copies of source documents. JSON uses {"schema_version":1,"files":[...]}; CSV columns are path,ext,size,mtime,sha1,indexed_at,error, UTF-8 with BOM by default. Import validates every path against current roots and ignore rules, then re-reads the original files. It does not trust imported timestamps, hashes or indexed text. This makes re-import deterministic against current sources and avoids text injection from a stale manifest. Original source files must still exist; move/remap manifest paths deliberately when migrating machines.

backup uses SQLite's backup API and writes a dated ZIP containing index.db, config.json and a restore note. It includes cached extracted content but does not include source documents. Keep backups private. To restore, stop writers, back up the current data directory, then extract a trusted backup into a new data directory and update its roots before indexing. Existing export/backup paths are refused rather than overwritten.

MCP integration

The official mcp SDK is constrained to >=1.20,<2 for its FastMCP API. Stdio only:

{
  "mcpServers": {
    "workspace-search": {
      "command": "/workspace/apps/workspace-search-mcp/.venv/bin/wsearch",
      "args": ["--data-dir", "/workspace/data/workspace-search-mcp", "mcp"]
    }
  }
}

Initialize roots first using index --root .... Tools: search_files(query, limit=10, ext=null, path_prefix=null), read_file(path, start_line=null, end_line=null, page=null, sheet=null), recent_files(since="24h"), reindex(path=null). reindex modifies the derived index and may remove stale entries; it never changes source files or adds authorized roots. Restart the server after changing configuration. Full generated schemas are in docs/mcp-tools.json.

Validation and benchmark

pip install -e '.[dev,watch]'
ruff check .
ruff format --check .
pytest -q --cov=workspace_search_mcp --cov-report=term-missing
python docs/benchmark.py

The suite includes 10 fixed relevance/location queries in examples/queries.json, all MCP tools plus a real SDK stdio session, live watching, snapshot integrity, configuration changes, same-size/same-mtime modifications, and traversal/symlink rejection. GitHub Actions runs Python 3.11 and 3.12 on Ubuntu.

Measured on Darwin arm64, Python 3.12.13, 10,000 synthetic small UTF-8 files (797,780 bytes total): initial index 4.802 s, unchanged rescan 4.511 s, Chinese top-10 query 0.026 s. These local timings are not a performance guarantee; see docs/benchmark.json for timestamp and raw counters. Every rescan hashes file bytes, even when mtime/size match, to catch metadata-preserving edits. Only changed files are reparsed. Large PDF/Office files have different costs.

FAQ and limitations

• Will this send my files anywhere? No. Runtime code has no networking path.

• Can an assistant read outside the roots? No. Parent traversal, symlinks, excluded files, non-regular files and oversized files are denied. POSIX descriptor-relative opens prevent symlink replacement during file opening.

• Windows? Not supported in this version: secure file access requires POSIX dir_fd and O_NOFOLLOW. Linux and macOS are the tested/designated platforms.

• Scanned/encrypted PDFs? No OCR and no password handling. Empty scanned PDFs are recorded as errors. Use a separate offline OCR workflow before indexing.

• Word page numbers? DOCX has no stable page model here; extracted lines are labeled explicitly.

• Spreadsheet formulas? Stored formula text is indexed; formulas are never evaluated.

• Other encodings? Convert them to UTF-8 first. UTF-16 requires a BOM.

• Parser extensions? Register a byte parser using core.parsers.register_parser. See docs/design.md. Plugins execute only when explicitly imported in-process.

• macOS editable import failure? If Python reports Skipping hidden .pth file under python -v, clear the hidden flag only on your environment's .pth files: chflags nohidden .venv/lib/python3.12/site-packages/*.pth.

Roadmap

Improve CJK/Latin mixed-phrase matching, add richer location-aware previews, and measure large mixed-format corpora. Cloud sync and OCR remain outside this project.

References and license

Implementation references: SQLite FTS5, official MCP Python SDK, PathSpec. See AGENTS.md for the operator handbook and LICENSE for MIT terms.