Skip to main content
Glama
romainsc

lore-mcp

by romainsc
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

lore-mcp

LORE — Local Offline Retrieval Engine for MCP

An MCP server for semantic search over your local documents. Preprocesses, indexes, and serves any format — PDF, HTML, DOCX, markdown, and more. No cloud, no external database — just a single .db file on your workstation.

What it does

  • Preprocesses source documents in any format (PDF, HTML, DOCX, PPTX, XLSX, EPUB, images, CSV, JSON, XML, markdown) into clean markdown

  • Indexes with vector embeddings and full-text search (hybrid FTS5 + vector with RRF fusion)

  • Serves three MCP tools (search_docs, list_indexed_sources, list_collections) for any MCP client

  • Evaluates retrieval quality with built-in RAG evaluation and parameter optimization

  • Runs locally with automatic GPU/API/CPU fallback for embedding generation

Related MCP server: docs-mcp

Quickstart

1. Install

git clone https://github.com/romainsc/lore-mcp.git
cd lore-mcp
python -m venv .venv
source .venv/bin/activate
pip install -e ".[parse]"

Install extras by need:

  • pip install -e . — core (markdown only)

  • pip install -e ".[html]" — add HTML support (trafilatura)

  • pip install -e ".[pdf]" — add PDF/DOCX/PPTX support (Docling)

  • pip install -e ".[parse]" — all format support

  • pip install -e ".[eval]" — RAG evaluation (RAGAS)

2. Create a config file

# config.yaml
database:
  path: ./lore.db

embedding:
  model: nomic-ai/nomic-embed-text-v2-moe
  mode: builtin    # builtin, builtin:gpu, builtin:cpu, api

chunking:
  chunk_size: 1024
  chunk_overlap: 128

See docs/configuration.md for all options.

3. Create a manifest

# manifest.yaml
collection: my-docs
level: libre

sources:
  - title: Architecture Guide
    license: Apache-2.0
    orig: architecture.pdf

  - title: API Reference
    orig: api-ref.html

  - url: https://example.com/guide.md

The manifest declares sources abstractly. orig is the source file in its native format. path (output filename) is generated automatically.

4. Preprocess and build

# Preprocess: convert + clean sources
lore-mcp preprocess manifest.yaml \
  --config config.yaml \
  --docs-base-dir /path/to/corpus/ \
  --orig-subdir orig \
  --prep-subdir prep

# Build: index preprocessed sources
lore-mcp build manifest-prep.yaml \
  --config config.yaml \
  --docs-dir /path/to/corpus/prep/ \
  --output-dir /path/to/output/ \
  --skip-optimize

Or combine both in one step:

lore-mcp build manifest.yaml \
  --config config.yaml \
  --docs-dir /path/to/corpus/orig/ \
  --output-dir /path/to/output/ \
  --preprocess --skip-optimize

5. Start the MCP server

Option A: HTTP server (recommended)

lore-mcp --config config.yaml --transport sse
{
  "mcpServers": {
    "lore": {
      "url": "http://localhost:8000/sse"
    }
  }
}

Option B: subprocess (stdio)

{
  "mcpServers": {
    "lore": {
      "command": "/path/to/.venv/bin/lore-mcp",
      "args": ["--config", "/path/to/config.yaml"]
    }
  }
}

6. Use from your MCP client

search_docs("how to configure authentication")
search_docs("deployment", top_k=10, collection="docs-libre")
list_indexed_sources()
list_collections()

Search uses hybrid retrieval (vector + FTS5 full-text) with Reciprocal Rank Fusion for better keyword matching alongside semantic similarity.

CLI commands

Command

Purpose

lore-mcp

Start MCP server

lore-mcp preprocess

Convert and clean sources

lore-mcp build

Index sources into .db

lore-mcp lint

Analyze source quality

lore-mcp eval

Evaluate retrieval quality

lore-mcp optimize

Auto-optimize parameters

lore-mcp enrich

LLM enrichment (context, Q&A)

All commands accept --config config.yaml.

Architecture

Uses nomic-ai/nomic-embed-text-v2-moe for embeddings (768 dimensions, multilingual, Apache 2.0) and sqlite-vec for vector storage. Hybrid search combines vector KNN with FTS5 full-text via RRF fusion.

Preprocessing uses trafilatura (HTML, Apache 2.0), Docling (PDF/DOCX, MIT), and markitdown (CSV/JSON/XML, MIT).

See docs/architecture.md for the full design.

Roadmap

Done

  • SQLite + sqlite-vec storage with model validation

  • Embedding with GPU/API/CPU fallback

  • MCP server (search_docs, list_indexed_sources, list_collections)

  • Multi-collection support with license classification

  • Preprocessing tool (multi-format parsing, clean, dedup, PII detection, quality gate)

  • LLM enrichment (contextual retrieval, Q&A mode)

  • Hybrid search (FTS5 + vector + RRF fusion)

  • RAG evaluation and parameter optimization

  • Build workflow (manifest + config → optimized .db + metadata)

  • Unified config.yaml (no env vars)

  • MarkdownTextSplitter (structure-aware chunking)

  • 379+ tests (TDD)

Next

  • Documentation reorganization

  • Reranking (cross-encoder)

  • Adjacent-chunk / parent-child retrieval

  • End-to-end parameter optimization

  • CI/CD with GitHub Actions

  • pip install lore-mcp (PyPI)

  • Docker image

AI-assisted development

This project is developed with AI assistance (Claude, Anthropic). All AI-assisted content is marked with Assisted-by and Co-Authored-By trailers in commits. Every contribution is reviewed, tested, and validated by a human.

See docs/ai-guidelines.md.

License

AGPL-3.0-or-later — see docs/adr/001-license-gpl-v3.md for the rationale.

Copyright (C) 2026 Romain Chantereau

Available Tools

3 tools
list_collectionsA

List available collections with chunk and file counts.

Only available in multi-collection mode (LORE_DB_DIR set).

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden. It implies a read-only listing operation and mentions the availability condition, but it does not explicitly state that it has no side effects, nor does it describe error behavior or output format details beyond 'chunk and file counts'.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise, consisting of two short sentences. It front-loads the primary purpose and then adds a necessary condition. There is no fluff or redundant information, making it highly efficient for an agent to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with no parameters, the description is nearly complete. It states what is returned (chunk and file counts) and when it is available. It lacks explicit notes on edge cases or errors, but for a listing operation, the provided context is sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, and the schema coverage is trivially 100%. According to the rubric, the baseline for high coverage is 3, and there is no parameter information to add. The description does not need to explain parameters, but it also does not enhance anything beyond the baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: listing available collections and providing chunk and file counts. It uses a specific verb ('List') and object ('collections'), making the purpose unambiguous even without comparing to siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides an explicit precondition for use: 'Only available in multi-collection mode (LORE_DB_DIR set)'. This guides the agent on when this tool is applicable. However, it does not contrast with alternatives like search_docs or list_indexed_sources, so the guidance is not fully comprehensive.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_indexed_sourcesA

List all indexed files with chunk counts.

In multi-collection mode, specify a collection name or leave empty to list sources across all collections.

ParametersJSON Schema
NameRequiredDescriptionDefault
collectionNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. 'List' clearly implies a read-only operation, and the description states what is returned (indexed files with chunk counts). It does not mention edge cases like pagination or errors, but for a simple listing tool this is reasonably transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two concise sentences with no filler or redundancy. The core purpose is front-loaded, and the parameter behavior is explained efficiently in the second sentence.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the low complexity (one optional parameter) and the presence of an output schema, the description covers the essential usage scenarios. It could be slightly more explicit about behavior in single-collection mode, but overall it is sufficient for an agent to call the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Although the input schema has 0% description coverage, the tool description directly explains the only parameter: 'specify a collection name or leave empty to list sources across all collections.' This fully clarifies the meaning and default behavior of the collection parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action ('List'), a specific resource ('indexed files'), and the output ('with chunk counts'). It is clearly distinct from sibling tools like search_docs and list_collections by describing an inventory-style listing.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains the conditional behavior of the collection parameter ('In multi-collection mode... leave empty to list sources across all collections'), but it does not explicitly compare this tool with sibling tools or state when to prefer this over search_docs or list_collections. Usage is implied rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_docsA

Semantic search over indexed documents.

Returns the most relevant passages for the given query, with similarity scores and source files. In multi-collection mode, specify a collection name or leave empty to search across all collections.

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYes
top_kNo
collectionNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of disclosing behavior. It explains the search behavior and output contents, including similarity scores and source files, but does not explicitly state side effects, read-only guarantees, or error handling. This leaves some behavioral details unstated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well structured. It states the main purpose, summarizes the return content, and provides the key conditional usage note without unnecessary detail or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is mostly complete for a semantic search tool: it covers the query, collection behavior, and result content. However, the lack of explanation for top_k leaves a meaningful gap, since an agent cannot confidently know how many results to expect without inferring it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema provides no parameter descriptions, so the description must compensate. It explains the query and collection parameters reasonably, including multi-collection behavior, but does not explain top_k or its default meaning. This is a notable gap in parameter semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool performs semantic search over indexed documents and returns relevant passages with similarity scores and source files. It is distinct from the sibling listing tools, which focus on enumerating sources and collections rather than searching content.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives practical guidance for multi-collection mode, telling users to specify a collection or leave it empty to search all collections. It does not explicitly name sibling tools as alternatives, but the conditional usage instruction is clear enough for typical search scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 3 tool updatesv0.1.0
    • First observedlist_collections
    • First observedlist_indexed_sources
    • First observedsearch_docs
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.2/5.0

Scored across 3 tools

Disambiguation5/5

Each tool has a clear, distinct purpose: search_docs performs semantic search, list_indexed_sources lists indexed files, and list_collections lists collections. No functional overlap or ambiguity.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern in snake_case: search_docs, list_indexed_sources, list_collections. The naming is uniform and predictable.

Tool Count5/5

Three tools is well-scoped for a documentation retrieval server, covering search and listing operations without unnecessary bloat.

Completeness4/5

The core retrieval workflows are covered: searching documents, listing sources, and listing collections. Minor gaps exist, such as retrieving a full document by ID, but the surface is reasonably complete for a search-focused server.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • docs-mcp

    kage1020
    F
    license
    Not graded
    quality
    D
    maintenance
    Indexes documentation sites by base URL and serves keyword search, optional semantic search, and Markdown page retrieval as MCP tools, all from a single SQLite file.
    -

  • ragi

    SusuTawar
    A
    license
    A
    quality
    D
    maintenance
    Local-first RAG indexing and semantic search MCP server. Enables document retrieval and context-aware queries using local embedding models.
    3
    6 npm
    MIT