doc-lib-mcp

# doc-lib-mcp MCP server A Model Context Protocol (MCP) server for document ingestion, chunking, semantic search, and note management. ## Components ### Resources - Implements a simple note storage system with: - Custom `note://` URI scheme for accessing individual notes - Each note resource has a name, description, and `text/plain` mimetype ### Prompts - Provides a prompt: - **summarize-notes**: Creates summaries of all stored notes - Optional "style" argument to control detail level (brief/detailed) - Generates prompt combining all current notes with style preference ### Tools The server implements a wide range of tools: - **add-note**: Add a new note to the in-memory note store - Arguments: `name` (string), `content` (string) - **ingest-string**: Ingest and chunk a markdown or plain text string provided via message - Arguments: `content` (string, required), `source` (string, optional), `tags` (list of strings, optional) - **ingest-markdown**: Ingest and chunk a markdown (.md) file - Arguments: `path` (string) - **ingest-python**: Ingest and chunk a Python (.py) file - Arguments: `path` (string) - **ingest-openapi**: Ingest and chunk an OpenAPI JSON file - Arguments: `path` (string) - **ingest-html**: Ingest and chunk an HTML file - Arguments: `path` (string) - **ingest-html-url**: Ingest and chunk HTML content from a URL (optionally using Playwright for dynamic content) - Arguments: `url` (string), `dynamic` (boolean, optional) - **smart_ingestion**: Extracts all technically relevant content from a file using Gemini, then chunks it using robust markdown logic. - Arguments: - `path` (string, required): File path to ingest. - `prompt` (string, optional): Custom prompt to use for Gemini. - `tags` (list of strings, optional): Optional list of tags for classification. - Uses Gemini 2.0 Flash 001 to extract only code, configuration, markdown structure, and technical definitions (no summaries or commentary). - Passes the extracted content to a mistune 3.x-based chunker that preserves both code blocks and markdown/narrative content as separate chunks. - Each chunk is embedded and stored for semantic search and retrieval. - **search-chunks**: Semantic search over ingested content - Arguments: - `query` (string): The semantic search query. - `top_k` (integer, optional, default 3): Number of top results to return. - `type` (string, optional): Filter results by chunk type (e.g., `code`, `html`, `markdown`). - `tag` (string, optional): Filter results by tag in chunk metadata. - Returns the most relevant chunks for a given query, optionally filtered by type and/or tag. - **delete-source**: Delete all chunks from a given source - Arguments: `source` (string) - **delete-chunk-by-id**: Delete one or more chunks by id - Arguments: `id` (integer, optional), `ids` (list of integers, optional) - You can delete a single chunk by specifying `id`, or delete multiple chunks at once by specifying `ids`. - **update-chunk-type**: Update the type attribute for a chunk by id - Arguments: `id` (integer, required), `type` (string, required) - **ingest-batch**: Ingest and chunk multiple documentation files (markdown, OpenAPI JSON, Python) in batch - Arguments: `paths` (list of strings) - **list-sources**: List all unique sources (file paths) that have been ingested and stored in memory, with optional filtering by tag or semantic search. - Arguments: - `tag` (string, optional): Filter sources by tag in chunk metadata. - `query` (string, optional): Semantic search query to find relevant sources. - `top_k` (integer, optional, default 10): Number of top sources to return when using query. - **get-context**: Retrieve relevant content chunks (content only) for use as AI context, with filtering by tag, type, and semantic similarity. - Arguments: - `query` (string, optional): The semantic search query. - `tag` (string, optional): Filter results by a specific tag in chunk metadata. - `type` (string, optional): Filter results by chunk type (e.g., 'code', 'markdown'). - `top_k` (integer, optional, default 5): The number of top relevant chunks to retrieve. - **update-chunk-metadata**: Update the metadata field for a chunk by id - Arguments: `id` (integer), `metadata` (object) - **tag-chunks-by-source**: Adds specified tags to the metadata of all chunks associated with a given source (URL or file path). Merges with existing tags. - Arguments: `source` (string), `tags` (list of strings) - **list-notes**: List all currently stored notes and their content. #### Chunking and Code Extraction - Markdown, Python, OpenAPI, and HTML files are split into logical chunks for efficient retrieval and search. - The markdown chunker uses mistune 3.x's AST API and regex to robustly split content by code blocks and narrative, preserving all original formatting. - Both code blocks and markdown/narrative content are preserved as separate chunks. - The HTML chunker uses the `readability-lxml` library to extract main content first, then extracts block code snippets from `<pre>` tags as dedicated "code" chunks. Inline `<code>` content remains part of the narrative chunks. #### Semantic Search - The `search-chunks` tool performs vector-based semantic search over all ingested content, returning the most relevant chunks for a given query. - Supports optional `type` and `tag` arguments to filter results by chunk type (e.g., `code`, `html`, `markdown`) and/or by tag in chunk metadata, before semantic ranking. - This enables highly targeted retrieval, such as "all code chunks tagged with 'langfuse' relevant to 'cost and usage'". #### Metadata Management - Chunks include a `metadata` field for categorization and tagging. - The `update-chunk-metadata` tool allows updating metadata for any chunk by its id. - The `tag-chunks-by-source` tool allows adding tags to all chunks from a specific source in one operation. Tagging merges new tags with existing ones, preserving previous tags. ## Configuration The server requires the following environment variables (can be set in a .env file): ### Ollama Configuration - OLLAMA_HOST: Hostname for Ollama API (default: localhost) - OLLAMA_PORT: Port for Ollama API (default: 11434) - RAG_AGENT: Ollama model to use for RAG responses (default: llama3) - OLLAMA_MODEL: Ollama model to use for embeddings (default: nomic-embed-text-v2-moe) ### Database Configuration - HOST: PostgreSQL database host (default: localhost) - DB_PORT: PostgreSQL database port (default: 5432) - DB_NAME: PostgreSQL database name (default: doclibdb) - DB_USER: PostgreSQL database user (default: doclibdb_user) - DB_PASSWORD: PostgreSQL database password (default: doclibdb_password) ### Reranker Configuration - RERANKER_MODEL_PATH: Path to the reranker model (default: /srv/samba/fileshare2/AI/models/bge-reranker-v2-m3) - RERANKER_USE_FP16: Whether to use FP16 for reranker (default: True) ## Quickstart ### Install #### Claude Desktop On MacOS: `~/Library/Application\ Support/Claude/claude_desktop_config.json` On Windows: `%APPDATA%/Claude/claude_desktop_config.json` <details> <summary>Development/Unpublished Servers Configuration</summary> ``` "mcpServers": { "doc-lib-mcp": { "command": "uv", "args": [ "--directory", "/home/administrator/python-share/doc-lib-mcp", "run", "doc-lib-mcp" ] } } ``` </details> <details> <summary>Published Servers Configuration</summary> ``` "mcpServers": { "doc-lib-mcp": { "command": "uvx", "args": [ "doc-lib-mcp" ] } } ``` </details> ## Development ### Building and Publishing To prepare the package for distribution: 1. Sync dependencies and update lockfile: ```bash uv sync ``` 2. Build package distributions: ```bash uv build ``` This will create source and wheel distributions in the `dist/` directory. 3. Publish to PyPI: ```bash uv publish ``` Note: You'll need to set PyPI credentials via environment variables or command flags: - Token: `--token` or `UV_PUBLISH_TOKEN` - Or username/password: `--username`/`UV_PUBLISH_USERNAME` and `--password`/`UV_PUBLISH_PASSWORD` ### Debugging Since MCP servers run over stdio, debugging can be challenging. For the best debugging experience, we strongly recommend using the [MCP Inspector](https://github.com/modelcontextprotocol/inspector). You can launch the MCP Inspector via [`npm`](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) with this command: ```bash npx @modelcontextprotocol/inspector uv --directory /home/administrator/python-share/doc-lib-mcp run doc-lib-mcp ``` Upon launching, the Inspector will display a URL that you can access in your browser to begin debugging.