Skip to main content
Glama
solid-knowledge-ai

solid-knowledge-ai

by hailampy123

Solid Knowledge AI

A multi-source document knowledge assistant driven by a self-reflective LangGraph agent. It ingests PDF + Markdown + web pages into one vector store, then answers questions through an agent that grades its own retrieval and verifies its own answer for grounding — retrying with a rewritten query when either check fails, and refusing to fabricate when it can't ground an answer. Traced with Langfuse, quality-tested with DeepEval, and exposed over MCP.

Built to showcase agentic development: LangGraph · LiteLLM · ChromaDB · MCP · Langfuse · DeepEval.

Why this is not "just RAG"

The agent is a corrective / self-reflective RAG loop, not a linear retrieve → generate chain:

question
   │
   ▼
 route ──chitchat/out_of_scope──▶ generate ──▶ END
   │ kb
   ▼
retrieve ──▶ grade_docs ──irrelevant (rewrite query, retry)──▶ retrieve
                 │ relevant
                 ▼
             generate ──▶ self_check ──ungrounded (retry)──▶ retrieve
                              │ grounded / budget spent
                              ▼
                     answer + citations  (or an honest "I don't know")
  • route — skips retrieval on small talk / out-of-scope questions.

  • grade_docs — an LLM relevance gate; on failure it rewrites the query and retries.

  • self_check — verifies the drafted answer is entailed by the retrieved context; if not, it retries or hedges instead of hallucinating.

  • A shared retry budget (max_retries, default 2) bounds both loops.

  • Memory — a SQLite checkpointer keeps multi-turn conversation state per thread_id.

Related MCP server: pdf-analyzer

Quickstart

# 1. Install (Python 3.11+, uv)
uv sync

# 2. Configure — only ANTHROPIC_API_KEY is required
cp .env.example .env      # then edit .env

# 3. Ingest the sample corpus (2 Markdown + 1 PDF + 1 Wikipedia article)
uv run skai ingest        # -> builds ./.chroma  (local MiniLM embeddings, no API)

# 4. Ask (defaults to Haiku 4.5; switch per-call with --model)
uv run skai ask "What do orcas eat?"
uv run skai ask "How do orcas communicate?" --source md
uv run skai ask "Summarize orca threats" --model sonnet   # haiku | sonnet (Opus blocked)

# 5. Multi-turn chat (remembers the conversation)
uv run skai chat

# 6. Web UI (chat + feedback + live data ingestion)
uv run skai ui        # http://localhost:7860

# 7. Serve over MCP (stdio) for Claude Desktop / an IDE
uv run skai mcp

Web UI

skai ui launches a Gradio app with the features a live demo needs:

  • Chat with per-session memory; every answer shows its sources, route, and model.

  • Feedback after every response — 👍/👎 + an optional comment, stored to SQLite (.skai/feedback.sqlite) and pushed as a Langfuse score on that turn's trace when tracing is on. That's the closed loop: real usage becomes an eval signal.

  • Example prompts to guide the first interaction.

  • Grow the knowledge base live — upload a .md/.txt/.pdf or paste a URL and it's ingested into Chroma on the spot, so the demo isn't limited to the seed corpus.

  • Model (haiku/sonnet) and source filter (all/pdf/md/web) selectors.

Feedback is exportable to a JSONL eval seed via skai.feedback.export_jsonl.

Commands

Command

What it does

skai ingest [--path data/docs --urls data/urls.txt --reset]

Load → chunk → embed → persist to Chroma

skai ask "..." [--source pdf|md|web] [--thread-id X]

One-shot question with citations

skai chat

Interactive multi-turn chat with memory

skai ui [--port 7860 --share]

Gradio web UI: chat, feedback, live ingestion

skai mcp

Run the MCP server exposing search_kb and ask

skai eval

Run the DeepEval quality suite (needs --group eval + key)

MCP client config

The server exposes two tools — search_kb(query, source_type?) (raw retrieval) and ask(question) (full agent). Point an MCP client at it:

{
  "mcpServers": {
    "solid-knowledge-ai": {
      "command": "uv",
      "args": ["run", "skai", "mcp"],
      "cwd": "/absolute/path/to/solid-knowledge-ai"
    }
  }
}

Model selection

Default Haiku 4.5 (fast, cheap — good for a Q&A router+grader+generator loop). Switch per call with --model, or globally via SKAI_MODEL in .env:

Value

Resolves to

haiku (default)

anthropic/claude-haiku-4-5

sonnet

anthropic/claude-sonnet-4-5

any LiteLLM id

passed through (e.g. openai/gpt-4o-mini)

Opus is intentionally blocked (resolve_model raises), so the assistant can't be pointed at the most expensive tier by accident.

Observability

Set LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY (and optionally LANGFUSE_HOST) in .env. Every graph run then produces one trace with a span per node and per LLM call. Without keys, tracing is a clean no-op — nothing else changes.

Quality evaluation (DeepEval)

uv sync --group eval
export ANTHROPIC_API_KEY=...
uv run skai ingest
uv run --group eval pytest evals -v      # or: skai eval

The judge is Claude via LiteLLM, so no OpenAI key is needed. Metrics: faithfulness, answer relevancy, contextual relevancy — plus a cheap keyword gate.

Tests

uv run pytest        # 39 tests, fully offline: no network, no API keys

The LLM is dependency-injected, so the whole graph runs in tests against a deterministic stub, and Chroma uses a deterministic in-process embedding function.

How it's put together

src/skai/
  config.py            settings (.env)              agent/llm.py     ChatLiteLLM -> Claude
  models.py            Document/Chunk/AgentState    agent/nodes.py   route/retrieve/grade/generate/self_check
  ingest/loaders.py    pdf | md | web  -> Document  agent/prompts.py node prompts
  ingest/chunk.py      source-aware splitting       agent/graph.py   StateGraph + SQLite memory
  ingest/store.py      Chroma add/query             observability.py Langfuse handler (or no-op)
  cli.py               ingest|ask|chat|mcp|eval     mcp_server.py    search_kb / ask as MCP tools
evals/                 DeepEval suite               tests/           offline unit + graph tests

Agent graph & component diagrams (Mermaid): see docs/ARCHITECTURE.md Design rationale and tech trade-offs: see docs/DECISIONS.md Where it goes next (capability & use cases): see docs/CAPABILITY-ROADMAP.md Running it on Gemini Enterprise CX / Google Cloud: see docs/GEMINI-ENTERPRISE-PORT.md

Status

Verified: uv run skai ingest loads all three source types (2 md + 1 pdf + 1 web → 170 chunks) and real semantic retrieval returns relevant passages. 39 offline tests pass. ask/chat/eval require an ANTHROPIC_API_KEY

Available Tools

2 tools
askB

Ask the self-reflective knowledge agent. Returns a grounded, cited answer.

Inputs: question (required). Returns: result

search_kbC

Semantic search over the ingested documents. Optional source_type: pdf|md|web.

Inputs: query (required), source_type. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.2/5.0

    Scored across 2 tools

    Disambiguation4/5

    The two tools are distinct in purpose: search_kb returns semantic search results over sources, while ask provides a grounded, cited answer through an agent. There is some overlap in that both retrieve information from the same knowledge base, but the descriptions clearly differentiate a low-level search from a high-level Q&A interaction, making misselection unlikely.

    Naming Consistency4/5

    Both tool names use lowercase snake_case and are verb-based. 'search_kb' follows a verb_noun pattern while 'ask' is a bare verb, creating a minor inconsistency, but the pattern is still simple and predictable given the small set.

    Tool Count3/5

    With only two tools, the server feels thin but not unreasonable. Search and ask cover the core knowledge-access functions, though additional tools like listing sources or managing documents might be expected in a fuller knowledge-management server.

    Completeness4/5

    The tool surface covers the core domain of querying an ingested knowledge base: search_kb for retrieval and ask for synthesized answers. Minor gaps exist, such as no way to enumerate available sources or inspect document metadata, but these are not critical for the stated purpose.

    Maintenance

    ActivitySlowing
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers