Skip to main content
Glama
Sign In
Corpus-KB

Corpus-KB

by moliver28
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

Corpus-KB

A local NLP engine for files, transcripts, and code. Ingest, search, ask, extract, and code. No cloud.

Corpus-KB serves three kinds of users. Non-coders get a browser UI. Novice coders get a CLI with a guided quickstart. Expert researchers get a qualitative-coding pipeline, an OpenAI-compatible API, and MCP tools for coding agents. Everything runs on your machine. Postgres stores the data by default; DuckDB is the single-file local engine (Storage backends). Ollama or PostgresML generates embeddings. One server exposes the catalogue over MCP stdio, HTTP, and a JSON-RPC socket. An OpenAI-compatible adapter serves /v1 chat and embeddings. See Editor integration.

Two ways to start

Pick the path that fits you. Both end at the same working system.

The novice path: the browser

No terminal work beyond a few commands. First install the package:

pip install -e .

Then build the browser app once. You need Node 24 and pnpm for this step. Node is a build tool only. The running server stays pure Python:

pnpm -C ui_src install
pnpm -C ui_src build

Then start the stack. corpus-kb setup starts Postgres and loads the schema. corpus-kb start starts the server and opens your browser:

corpus-kb setup
corpus-kb start

The browser opens the lifecycle app at http://localhost:8010/app/. The Setup stage shows the system's health in plain language. The other stages follow the research flow: Ingest, Review, Scale, Ask, Analyze, Codebook, Release.

Details: docs/UI.md.

The expert path: terminals and APIs

Every operation is reachable programmatically. One line per surface:

First time on a terminal? corpus-kb quickstart walks you from zero to a first search. The step-by-step quickstart is in docs/getting-started.md.

Related MCP server: ragi

What you get

  • Hybrid search that blends vector similarity, full-text search, and rank fusion, with honest handling of degraded (zero-vector) chunks

  • Reranking — a local judgment scorer by default, or a true cross-encoder through llama.cpp

  • Answer generation with per-sentence citations, plus assertion-level verification with honest abstention

  • Query routing — each question goes to vector search, SQL aggregates, or graph traversal

  • Self-query filters — natural-language filters parsed into parameterized SQL predicates

  • Knowledge graph with entities, relations, and BFS traversal

  • Ontology-aware extraction with configurable entity/relation types and pluggable backends (regex, LangExtract, PostgresML, plus opt-in GLiNER)

  • Pluggable ingestion strategies — eight built-ins (default, transcript, slides, web, survey, markdown, code, textbook), extensible three ways (docs/INGESTION.md)

  • Website ingestion — a robots-respecting fetcher with same-domain crawl and heading-section chunks

  • Late chunking — one model pass per document; chunk vectors pool from their own token spans

  • Qualitative coding — transcript ingestion, codebooks, deductive and inductive runs, human review, governance reports, and a gated release lifecycle

  • Topics — offline BERTopic clustering over stored embeddings (docs/ANALYSIS.md)

  • LlamaIndex RAG backend with PGVectorStore and Ollama for local vector search

  • Event sourcing for audit trails and rebuildable projections

  • Multi-tenant Postgres with row-level security on every table

  • Full-stack installer with hardware detection, profile-based recommendations, and guided setup

  • MCP, HTTP, socket, and CLI surfaces so any editor or script can talk to it

  • Browser UI with drag-and-drop ingest, cited answers, and plain-language health checks (docs/UI.md)

  • Notebooks console — marimo notebooks served by the running server (docs/NOTEBOOKS.md)

Architecture

graph TB
    Editor[AI Code Editor] -->|"MCP stdio"| MCP[MCP Server]
    Editor -->|HTTP :8010| HTTP[Starlette API]
    Editor -->|JSON-RPC socket| Socket[Unix socket / named pipe]
    MCP --> Handlers[Command / Query Handlers]
    HTTP --> Handlers
    Socket --> Handlers
    Handlers --> Domain[Domain Layer<br/>Aggregates + Events]
    Domain --> ES[Event Store<br/>append-only]
    ES --> Projections[Async Projections]
    Projections --> PG[Postgres 17]
    Projections --> DUCK[DuckDB<br/>local single-user engine]
    PG --> VEC[pgvector<br/>vector search]
    PG --> FTS[Postgres FTS<br/>to_tsvector]
    PG --> AGE[Apache AGE<br/>Cypher graphs]
    PG --> RLS[RLS on 12 tables]
    Projections --> Ollama[Ollama<br/>embedding service]

  1. Ingest a file, directory, or raw text.

  2. The pipeline partitions it into chunks and embeds each chunk through Ollama or PostgresML. It extracts entities and relations, then stores the result.

  3. Commands append events to the event store; async projections write the read models into Postgres or DuckDB.

  4. Your editor queries the read models through search, SQL, or graph traversal.

Event sourcing flow

sequenceDiagram
    participant C as Client
    participant H as CommandHandler
    participant A as Document Aggregate
    participant ES as Event Store
    participant P as Projection
    participant DB as Postgres Tables

    C->>H: ingest_file(file_path)
    H->>A: create Document
    A->>A: apply Ingested event
    A->>A: apply ChunksAdded event
    H->>ES: app.save(aggregate)
    ES-->>H: event_id, version
    H-->>C: {status: "success"}
    ES->>P: subscribe(ChunksAdded)
    P->>P: embed chunks via Ollama
    P->>DB: INSERT chunks, chunks_vectors
    P->>DB: UPDATE projection_checkpoints

Events are the source of truth. Projections are derived and can be rebuilt by replaying the event log. Vectors live in the chunks_vectors table and are treated as derived data, not event payload.

Documentation

Page

What it covers

Getting started

Both paths: the browser for novices, the terminal quickstart for everyone else. Then qualitative coding from zero: the demo, your first transcript, first codebook

Install

Full setup from scratch: Postgres, Python, Ollama, schema, first query

Browser UI

The lifecycle app: stage-by-stage user guide, config, build, gates, troubleshooting

Features

Ingest, search, quality controls, graph, tags, research subsystem, embedding models, MCP tool reference

Admin

Configuration, schema, multi-tenancy, backups, monitoring, CI/CD

Storage backends

DuckDB vs Postgres: quickstart, engine selection, migrate-data, runtime tuning

Operations

The operator's contract: exit codes, ports, vocabularies, tenancy, the MCP surface

API

HTTP routes, the /v1 OpenAI adapter, curl examples, MCP tool reference

Observability

OpenTelemetry spans, exporters, zero-cost when off

Evaluation

Golden sets, recall/MRR/nDCG, answer metrics

Notebooks

The marimo console and the three example notebooks

Development

Architecture deep dive, testing, PR workflow, conventions

Research

Research command reference, config keys, embedding requirements

Understanding your report

Plain-language guide to every governance-report section

FAQ

Common questions

Ingestion

Full pipeline: partition, chunk, embed, extract, store, plus the strategy reference and extension ladders

Analysis

The analysis layer: analyze-run report, labeling, classification, topics, trends, method register

Analysis methods

GENERATED reference: the 30-method register with required validation and permitted claims

Rollback drill

Practiced procedure for backing out a bad deployment

Internal program docs

Archive: playbooks, research math, benchmarks, harness distribution, program plans

Editor integration

Editors connect over MCP stdio (native, no bridge), the HTTP API, or the JSON-RPC socket:

  • MCP stdio: corpus-kb --transport stdio — the mcp-configs/ files (Claude Code, Cursor, Codex) and the root opencode.json (OpenCode) launch exactly this command. Generic MCP clients such as Hermes or Cline need no config file: they point at the same stdio server. See docs/research/harness-distribution.md. The server registers every catalogue operation as an MCP tool; see docs/FEATURES.md for the tool reference.

  • HTTP: corpus-kb start --transport http --port 8010 — every catalogue operation has a route, plus the /v1 OpenAI adapter (see docs/API.md).

  • JSON-RPC socket: same command, for local automation.

  • The research and coding surfaces are first-class CLI commands (corpus-kb research ... and corpus-kb coding ...). Ingestion has a top-level front door: corpus-kb ingest with --watch auto-heal, plus corpus-kb ingest-status and corpus-kb ingest-reembed.

Every catalogue operation is reachable on all four surfaces (MCP, CLI, HTTP, socket) and returns the same envelope on each. The parity gate in tests/test_surface_parity_deep.py enforces this.

License

MIT License. See pyproject.toml for the full text.

Available Tools

33 tools
add_entityA

Add an entity to the knowledge graph.

Args: name: Entity name (e.g., "MyClass", "Authentication", "Paris"). type: Entity type (class, function, concept, person, place, etc.). metadata: Optional metadata dict.

Returns: Created entity details including entity_id.

Inputs: name (required), type, metadata.

add_relationA

Add a directed relation between two entities.

Args: source_id: Source entity ID. target_id: Target entity ID. rel_type: Type of relation (CALLS, DEPENDS_ON, CONTAINS, etc.). weight: Relation strength (0.0 to 1.0).

Returns: Created relation details.

Inputs: source_id (required), target_id (required), weight, rel_type.

add_tagC

Create a new tag for categorizing documents.

Inputs: name (required), color, description.

bfsC

BFS traversal from a starting entity.

Args: start_entity_id: Entity ID to start from. max_depth: Max traversal depth (1-10).

Returns: List of (entity_id, name, type, depth) entries.

Inputs: start_entity_id (required), max_depth. Returns: result

checkout_versionC

Check out a specific table version for time-travel queries.

Inputs: version (required).

create_branchC

Create a new branch from an optional version.

Inputs: branch_name (required), from_version.

create_tagB

Tag a specific version for reference.

Args: version: Version number to tag. tag_name: Human-readable tag name (e.g., "v1.0", "before-refactor").

Returns: Confirmation with version and tag name.

Inputs: version (required), tag_name (required).

delete_documentC

Delete an ingested document by its doc_id.

Inputs: doc_id (required).

get_document_tagsA

Get all tags applied to a document.

Inputs: doc_id (required). Returns: result

get_entity_relationsC

Get all relations for an entity (incoming and outgoing).

Args: entity_id: Entity ID.

Returns: List of relations with source/target info.

Inputs: entity_id (required). Returns: result

get_metadataA

Retrieve metadata entries, optionally filtered by key and/or doc_id.

Omit both to get all metadata. Filter by key to find all values for a key. Filter by doc_id to find all metadata for a document.

Inputs: key, doc_id. Returns: result

get_statsC

Get database statistics.

Returns: Stats object with counts and storage info.

Inputs: no named parameters.

ingest_directoryC

Ingest all supported files in a directory.

Supports all extensions in CodeChunker.LANGUAGE_MAP plus .md, .rst, .txt.

Inputs: directory_path (required), recursive. Returns: result

ingest_fileC

Ingest a single file. Auto-detects code/markdown/text.

Inputs: file_path (required).

ingest_textB

Ingest raw text with optional type hint (code/markdown/text).

Inputs: text (required), source, file_type.

list_branchesA

List all branches.

Inputs: no named parameters. Returns: result

list_documentsB

List all ingested documents with their metadata.

Inputs: no named parameters. Returns: result

list_versionsA

List all versions of the chunks table for time-travel.

Returns: List of version entries with version number, timestamp, and tag.

Inputs: no named parameters. Returns: result

query_document_statsB

Get aggregate statistics about the document corpus via SQL.

Returns: total documents, total chunks, docs by type, chunks by type, average chunks per document, date range.

Inputs: no named parameters.

restore_versionC

Restore the table to a specific version.

Inputs: version (required).

retrieve_contextB

Search and return results formatted for LLM context building.

Args: query: Natural language query. k: Number of results (max 20). filters: Optional JSON string of filter conditions.

Returns: Formatted string with source citations.

Inputs: query (required), k, filters. Returns: result

searchA

Hybrid search (vector + full-text + RRF) across all chunks.

Args: query: Natural language query. k: Number of results (max 50). source_type: Optional filter: "code", "markdown", or "text".

Returns: List of search results with text, source, score, and metadata.

Inputs: query (required), k, source_type. Returns: result

search_contextA

Search with parent/sibling/child context expansion.

For each result, includes surrounding chunks (context_chunks before and after) so the LLM has full context.

Args: query: Natural language query. k: Number of primary results (max 20). context_chunks: Number of adjacent chunks to include (0-5). source_type: Optional filter: "code", "markdown", or "text".

Returns: List of results, each with a "context" field containing surrounding chunks.

Inputs: query (required), k, source_type, context_chunks. Returns: result

search_graphA

Search entities in the knowledge graph by name or type.

Args: query: Search term for entity name. type: Optional entity type filter. limit: Max results (max 100).

Returns: List of matching entities.

Inputs: query (required), type, limit. Returns: result

search_similarA

Find chunks similar to a given chunk by ID.

Args: chunk_id: ID of the source chunk. k: Number of similar results to return (max 20).

Returns: List of similar chunks with scores and text snippets.

Inputs: chunk_id (required), k. Returns: result

set_metadataA

Set a metadata key-value pair, optionally scoped to a document.

Overwrites any existing value for the same key+doc_id combination.

Inputs: key (required), value (required), doc_id.

sql_executeA

Execute a write SQL statement (INSERT, UPDATE, DELETE) with safety rails.

Safety rules enforced by the engine:

  • DELETE/UPDATE without WHERE clause is BLOCKED

  • DROP TABLE/DATABASE/SCHEMA is BLOCKED

Use parameterized ? placeholders for values to prevent injection. For SELECT queries, use sql_query instead.

Examples: INSERT INTO tags (name, color) VALUES ('urgent', 'red') UPDATE documents SET source_type = 'markdown' WHERE doc_id = 'abc-123' DELETE FROM document_tags WHERE doc_id = 'abc-123' INSERT INTO metadata (key, value, doc_id) VALUES ('reviewer', 'alice', 'abc-123')

Args: statement: SQL write statement (INSERT, UPDATE, DELETE).

Returns: Dict with "affected_rows" count, or "error" if blocked.

Inputs: statement (required).

sql_queryA

Run a SQL SELECT query over relational tables.

Full SQL supported: SELECT, JOIN, CTE, GROUP BY, window functions, subqueries, UNION, etc.

Available tables:

  • documents: Document-level metadata (doc_id, source, source_type, chunk_count, file_size, file_hash, language, created_at)

  • chunks: Individual text chunks (chunk_id, doc_id, chunk_index, source_type, chunk_type, entity_name, file_path, start_line, end_line, char_count)

  • tags: Defined tags (tag_id, name, color, description)

  • document_tags: Many-to-many mapping (doc_id, tag_id)

  • metadata: Flexible key-value store (key, value, doc_id)

Examples: SELECT source_type, COUNT(*) as cnt FROM documents GROUP BY source_type SELECT * FROM chunks WHERE source_type = 'code' LIMIT 10 SELECT d.source, COUNT(c.chunk_id) as chunks FROM documents d JOIN chunks c ON d.doc_id = c.doc_id GROUP BY d.source ORDER BY chunks DESC SELECT d.source FROM documents d JOIN document_tags dt ON d.doc_id = dt.doc_id JOIN tags t ON dt.tag_id = t.tag_id WHERE t.name = 'important'

Args: query: SQL SELECT query string. limit: Max rows to return (default 100, max 5000).

Returns: Dict with "columns", "rows", and "row_count".

Inputs: query (required), limit.

sql_tablesA

List all available relational tables with their schema.

Returns table name, column name, column type, and nullability for each column in every user table.

Inputs: no named parameters. Returns: result

switch_branchC

Switch to a specified branch.

Inputs: branch_name (required).

sync_databaseA

Sync data from LanceDB vector store into relational tables.

Call this after ingesting documents to make relational queries up to date. The sync is idempotent — call it anytime.

Inputs: no named parameters.

tag_documentB

Apply a tag to a document. Creates the tag if it doesn't exist.

Inputs: tag (required), doc_id (required).

untag_documentC

Remove a tag from a document.

Inputs: tag (required), doc_id (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Deploy Server
    One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

    TDQS

    B3.4/5.0

    Scored across 33 tools

    Disambiguation4/5

    Most tools have distinct purposes, but the high number of search tools (search, search_context, search_similar, search_graph) and versioning tools could cause minor ambiguity. However, descriptions are detailed enough to differentiate them.

    Naming Consistency5/5

    All tools consistently use snake_case with a verb_noun pattern, e.g., add_entity, create_branch, search_similar. Even 'bfs' is a clear abbreviation following the same style.

    Tool Count4/5

    33 tools is on the higher side, but each serves a distinct purpose in ingestion, search, graph, versioning, and SQL querying. The count is slightly heavy but still well-scoped for a knowledge base server.

    Completeness5/5

    The tool set covers the full lifecycle: ingestion, search with multiple modes, knowledge graph management, version control, SQL querying, and metadata handling. No obvious gaps for the domain.

    Maintenance

    ActivityActive
    ResponsivenessOutdated data

    Resources

    Related MCP Connectors

    Related MCP Servers

    • mcp-context

      ericlimabr
      F
      license
      Not graded
      quality
      D
      maintenance
      Local MCP server that provides semantic search (RAG) over code repositories, enabling AI clients like Claude and Gemini to access project context without manual re-upload.
      -

    • ragi

      SusuTawar
      A
      license
      A
      quality
      D
      maintenance
      Local-first RAG indexing and semantic search MCP server. Enables document retrieval and context-aware queries using local embedding models.
      3
      14 npm
      MIT