Skip to main content
Glama
deenesjakoruzh
hailampy123

solid-knowledge-ai

by hailampy123
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

Solid Knowledge AI

Ein Multi-Quellen-Dokument-Wissensassistent, gesteuert von einem selbstreflektiven LangGraph-Agenten. Er nimmt PDF + Markdown + Webseiten in einen einzigen Vektor-Store auf und beantwortet dann Fragen über einen Agenten, der seine eigene Abfrage bewertet und seine eigene Antwort auf Fundierung prüft – mit erneutem Versuch bei umgeschriebener Abfrage, wenn eine der Prüfungen fehlschlägt, und der sich weigert, zu erfinden, wenn er keine Antwort fundieren kann. Mit Langfuse getracet, mit DeepEval qualitätsgetestet und über MCP bereitgestellt.

Entwickelt, um agentische Entwicklung zu demonstrieren: LangGraph · LiteLLM · ChromaDB · MCP · Langfuse · DeepEval.

Warum das nicht „nur RAG“ ist

Der Agent ist eine korrektive / selbstreflektive RAG-Schleife, keine lineare retrieve → generate-Kette:

question
   │
   ▼
 route ──chitchat/out_of_scope──▶ generate ──▶ END
   │ kb
   ▼
retrieve ──▶ grade_docs ──irrelevant (rewrite query, retry)──▶ retrieve
                 │ relevant
                 ▼
             generate ──▶ self_check ──ungrounded (retry)──▶ retrieve
                              │ grounded / budget spent
                              ▼
                     answer + citations  (or an honest "I don't know")

  • route – überspringt die Abfrage bei Smalltalk / Fragen außerhalb des Themenbereichs.

  • grade_docs – ein LLM-Relevanz-Gate; bei Fehlschlag wird die Abfrage umgeschrieben und erneut versucht.

  • self_check – verifiziert, dass der entworfene Antwortentwurf durch den abgerufenen Kontext gedeckt ist; wenn nicht, wird erneut versucht oder abgeschwächt, anstatt zu halluzinieren.

  • Ein gemeinsames Wiederholungsbudget (max_retries, Standard 2) begrenzt beide Schleifen.

  • Memory – ein SQLite-Checkpointer hält den Zustand von Multi-Turn-Gesprächen pro thread_id fest.

Related MCP server: pdf-analyzer

Schnellstart

# 1. Install (Python 3.11+, uv)
uv sync

# 2. Configure — only ANTHROPIC_API_KEY is required
cp .env.example .env      # then edit .env

# 3. Ingest the sample corpus (2 Markdown + 1 PDF + 1 Wikipedia article)
uv run skai ingest        # -> builds ./.chroma  (local MiniLM embeddings, no API)

# 4. Ask (defaults to Haiku 4.5; switch per-call with --model)
uv run skai ask "What do orcas eat?"
uv run skai ask "How do orcas communicate?" --source md
uv run skai ask "Summarize orca threats" --model sonnet   # haiku | sonnet (Opus blocked)

# 5. Multi-turn chat (remembers the conversation)
uv run skai chat

# 6. Web UI (chat + feedback + live data ingestion)
uv run skai ui        # http://localhost:7860

# 7. Serve over MCP (stdio) for Claude Desktop / an IDE
uv run skai mcp

Web-UI

skai ui startet eine Gradio-App mit den Funktionen, die eine Live-Demo benötigt:

  • Chat mit Sitzungsspeicher; jede Antwort zeigt ihre Quellen, Route und Modell.

  • Feedback nach jeder Antwort – 👍/👎 + ein optionaler Kommentar, gespeichert in SQLite (.skai/feedback.sqlite) und als Langfuse-Score auf dem Trace dieser Runde übertragen, wenn Tracing aktiviert ist. Das ist die geschlossene Schleife: echte Nutzung wird zu einem Eval-Signal.

  • Beispiel-Prompts zur Orientierung für die erste Interaktion.

  • Wissensbasis live erweitern – eine .md/.txt/.pdf-Datei hochladen oder eine URL einfügen, und sie wird sofort in Chroma aufgenommen, sodass die Demo nicht auf das Startkorpus beschränkt ist.

  • Modell (haiku/sonnet) und Quellenfilter (all/pdf/md/web) Auswahlmöglichkeiten.

Feedback ist über skai.feedback.export_jsonl als JSONL-Eval-Seed exportierbar.

Befehle

Befehl

Was er tut

skai ingest [--path data/docs --urls data/urls.txt --reset]

Laden → Chunken → Einbetten → in Chroma speichern

skai ask "..." [--source pdf|md|web] [--thread-id X]

Einmalige Frage mit Zitaten

skai chat

Interaktiver Multi-Turn-Chat mit Speicher

skai ui [--port 7860 --share]

Gradio-Web-UI: Chat, Feedback, Live-Aufnahme

skai mcp

MCP-Server ausführen, der search_kb und ask bereitstellt

skai eval

DeepEval-Qualitätssuite ausführen (benötigt --group eval + Schlüssel)

MCP-Client-Konfiguration

Der Server stellt zwei Werkzeuge bereit – search_kb(query, source_type?) (rohe Abfrage) und ask(question) (voller Agent). Richten Sie einen MCP-Client darauf aus:

{
  "mcpServers": {
    "solid-knowledge-ai": {
      "command": "uv",
      "args": ["run", "skai", "mcp"],
      "cwd": "/absolute/path/to/solid-knowledge-ai"
    }
  }
}

Modellauswahl

Standard ist Haiku 4.5 (schnell, günstig – gut für eine Q&A-Router+Grader+Generator-Schleife). Wechseln Sie pro Aufruf mit --model oder global über SKAI_MODEL in .env:

Wert

Wird aufgelöst zu

haiku (Standard)

anthropic/claude-haiku-4-5

sonnet

anthropic/claude-sonnet-4-5

beliebige LiteLLM-ID

durchgereicht (z. B. openai/gpt-4o-mini)

Opus ist absichtlich blockiert (resolve_model wirft eine Ausnahme), sodass der Assistent nicht versehentlich auf die teuerste Stufe zeigen kann.

Beobachtbarkeit

Setzen Sie LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY (und optional LANGFUSE_HOST) in .env. Jeder Graph-Lauf erzeugt dann einen Trace mit einem Span pro Knoten und pro LLM-Aufruf. Ohne Schlüssel ist Tracing ein sauberer No-op – sonst ändert sich nichts.

Qualitätsbewertung (DeepEval)

uv sync --group eval
export ANTHROPIC_API_KEY=...
uv run skai ingest
uv run --group eval pytest evals -v      # or: skai eval

Der Bewerter ist Claude über LiteLLM, daher ist kein OpenAI-Schlüssel erforderlich. Metriken: Faithfulness, Antwortrelevanz, kontextuelle Relevanz – plus ein günstiges Keyword-Gate.

Tests

uv run pytest        # 39 tests, fully offline: no network, no API keys

Das LLM ist dependency-injiziert, sodass der gesamte Graph in Tests gegen einen deterministischen Stub läuft, und Chroma verwendet eine deterministische In-Process-Embedding-Funktion.

Wie es aufgebaut ist

src/skai/
  config.py            settings (.env)              agent/llm.py     ChatLiteLLM -> Claude
  models.py            Document/Chunk/AgentState    agent/nodes.py   route/retrieve/grade/generate/self_check
  ingest/loaders.py    pdf | md | web  -> Document  agent/prompts.py node prompts
  ingest/chunk.py      source-aware splitting       agent/graph.py   StateGraph + SQLite memory
  ingest/store.py      Chroma add/query             observability.py Langfuse handler (or no-op)
  cli.py               ingest|ask|chat|mcp|eval     mcp_server.py    search_kb / ask as MCP tools
evals/                 DeepEval suite               tests/           offline unit + graph tests

Agentengraph- und Komponentendiagramme (Mermaid): siehe docs/ARCHITECTURE.md. Designbegründung und technische Abwägungen: siehe docs/DECISIONS.md.

Status

Verifiziert: uv run skai ingest lädt alle drei Quellentypen (2 md + 1 pdf + 1 web → 170 Chunks) und echte semantische Abfrage liefert relevante Passagen. 39 Offline-Tests bestehen. ask/chat/eval erfordern einen ANTHROPIC_API_KEY.

Available Tools

2 tools
askB

Ask the self-reflective knowledge agent. Returns a grounded, cited answer.

NameRequiredDescriptionDefault
questionYes

Output Schema

NameRequiredDescription
resultYes

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry behavior disclosure; it does say outputs are grounded and cited, a meaningful trait. However, it does not mention limitations, confidence, citation format, or whether the agent can refuse/ask follow-ups, leaving the behavioral profile thin.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, with the core instruction and return behavior front-loaded. No filler or redundant restatement of the schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter tool with an output schema, the description covers the key behavior and return characteristic. It is mostly complete, though the lack of sibling differentiation and parameter detail keeps it from being fully self-sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has a single required `question` string with no field description, and the description does not elaborate on expected question format, length, or scope. The parameter name is self-explanatory, but the description adds no semantic detail to compensate for 0% schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific action ('Ask') on a distinct resource ('self-reflective knowledge agent') and its output ('grounded, cited answer'). It does not explicitly compare itself to search_kb, but the resource and output type give enough differentiation for a general sense.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to ask versus using search_kb, nor any exclusions or conditions. The name and description imply Q&A usage, but the agent is not told when to choose this tool over the sibling.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_kbC

Semantic search over the ingested documents. Optional source_type: pdf|md|web.

NameRequiredDescriptionDefault
queryYes
source_typeNo

Output Schema

NameRequiredDescription
resultYes

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are present, so the description carries the full burden of disclosing behavior. It reveals that the tool searches over ingested documents and accepts an optional source_type, but it does not disclose result limits, relevance behavior, authentication needs, or any side effects. For a read/search tool this is a notable but not severe gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single tight sentence followed by a compact optional-parameter note. It is front-loaded with the core action and resource, and every word adds information. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple two-parameter search tool, the description covers the essential invocation surface, and an output schema likely documents return values. However, it omits any comparison with 'ask' and does not explain result behavior or limitations, leaving the agent to infer when this tool is appropriate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does add meaning to source_type by enumerating allowed values ('pdf|md|web'), but it leaves the main 'query' parameter semantically undefined beyond the schema's bare type declaration. The compensation is only partial.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action ('Semantic search') on a clear resource ('ingested documents') and lists the optional source_type filter. It does not explicitly name the sibling tool 'ask' as the alternative, so some differentiation is left to inference, but the operation is unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to prefer search_kb over the sibling tool 'ask', nor any exclusions or prerequisites. Usage context is only implied by the phrase 'semantic search', which is not enough to route an agent reliably.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updatesv0.1.0
    • First observedask
    • First observedsearch_kb
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

B3.2/5.0

Scored across 2 tools

Disambiguation4/5

The two tools are distinct in purpose: search_kb returns semantic search results over sources, while ask provides a grounded, cited answer through an agent. There is some overlap in that both retrieve information from the same knowledge base, but the descriptions clearly differentiate a low-level search from a high-level Q&A interaction, making misselection unlikely.

Naming Consistency4/5

Both tool names use lowercase snake_case and are verb-based. 'search_kb' follows a verb_noun pattern while 'ask' is a bare verb, creating a minor inconsistency, but the pattern is still simple and predictable given the small set.

Tool Count3/5

With only two tools, the server feels thin but not unreasonable. Search and ask cover the core knowledge-access functions, though additional tools like listing sources or managing documents might be expected in a fuller knowledge-management server.

Completeness4/5

The tool surface covers the core domain of querying an ingested knowledge base: search_kb for retrieval and ask for synthesized answers. Minor gaps exist, such as no way to enumerate available sources or inspect document metadata, but these are not critical for the stated purpose.

Maintenance

ActivitySlowing
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers