Skip to main content
Glama
deenesjakoruzh
hailampy123

solid-knowledge-ai

by hailampy123
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

Solid Knowledge AI

다중 소스 문서 지식 어시스턴트로, 자기 반성적 LangGraph 에이전트에 의해 구동됩니다. PDF + Markdown + 웹 페이지를 하나의 벡터 저장소에 수집한 후, 자체 검색 결과를 평가하고 답변의 근거를 검증하는 에이전트를 통해 질문에 답합니다 — 두 검사 중 하나라도 실패하면 재작성된 쿼리로 재시도하고, 답변의 근거를 확보할 수 없을 때는 지어내기를 거부합니다. Langfuse로 추적되고, DeepEval로 품질 테스트되며, MCP로 노출됩니다.

에이전틱 개발을 보여주기 위해 제작되었습니다: LangGraph · LiteLLM · ChromaDB · MCP · Langfuse · DeepEval.

이것이 "단순 RAG"가 아닌 이유

이 에이전트는 교정적 / 자기 반영적 RAG 루프이지, 선형적인 retrieve → generate 체인이 아닙니다:

question
   │
   ▼
 route ──chitchat/out_of_scope──▶ generate ──▶ END
   │ kb
   ▼
retrieve ──▶ grade_docs ──irrelevant (rewrite query, retry)──▶ retrieve
                 │ relevant
                 ▼
             generate ──▶ self_check ──ungrounded (retry)──▶ retrieve
                              │ grounded / budget spent
                              ▼
                     answer + citations  (or an honest "I don't know")

  • route — 잡담 / 범위 외 질문에서는 검색을 건너뜁니다.

  • grade_docs — LLM 관련성 게이트; 실패 시 쿼리를 재작성하고 재시도합니다.

  • self_check — 초안 답변이 검색된 컨텍스트에 의해 함의되는지 검증합니다; 그렇지 않으면 재시도하거나 모호하게 답변하여 환각을 피합니다.

  • 공유 재시도 예산(max_retries, 기본값 2)이 두 루프를 모두 제한합니다.

  • 메모리 — SQLite 체크포인터가 thread_id별 다중 턴 대화 상태를 유지합니다.

Related MCP server: pdf-analyzer

빠른 시작

# 1. Install (Python 3.11+, uv)
uv sync

# 2. Configure — only ANTHROPIC_API_KEY is required
cp .env.example .env      # then edit .env

# 3. Ingest the sample corpus (2 Markdown + 1 PDF + 1 Wikipedia article)
uv run skai ingest        # -> builds ./.chroma  (local MiniLM embeddings, no API)

# 4. Ask (defaults to Haiku 4.5; switch per-call with --model)
uv run skai ask "What do orcas eat?"
uv run skai ask "How do orcas communicate?" --source md
uv run skai ask "Summarize orca threats" --model sonnet   # haiku | sonnet (Opus blocked)

# 5. Multi-turn chat (remembers the conversation)
uv run skai chat

# 6. Web UI (chat + feedback + live data ingestion)
uv run skai ui        # http://localhost:7860

# 7. Serve over MCP (stdio) for Claude Desktop / an IDE
uv run skai mcp

웹 UI

skai ui는 라이브 데모에 필요한 기능을 갖춘 Gradio 앱을 실행합니다:

  • 채팅 — 세션별 메모리 포함; 모든 답변은 출처, 경로, 모델을 표시합니다.

  • 모든 응답 후 피드백 — 👍/👎 + 선택적 코멘트, SQLite(.skai/feedback.sqlite)에 저장되고 *추적이 켜져 있을 때 해당 턴의 트레이스에 Langfuse 점수로 푸시됩니다*. 이것이 폐쇄 루프입니다: 실제 사용이 평가 신호가 됩니다.

  • 예시 프롬프트 — 첫 상호작용을 안내합니다.

  • 지식 베이스 실시간 확장 — .md/.txt/.pdf 업로드 또는 URL 붙여넣기로 즉시 Chroma에 수집되어, 데모가 시드 코퍼스에 국한되지 않습니다.

  • 모델(haiku/sonnet) 및 출처 필터(all/pdf/md/web) 선택기.

피드백은 skai.feedback.export_jsonl을 통해 JSONL 평가 시드로 내보낼 수 있습니다.

명령어

명령어

기능

skai ingest [--path data/docs --urls data/urls.txt --reset]

로드 → 청크 → 임베딩 → Chroma에 저장

skai ask "..." [--source pdf|md|web] [--thread-id X]

인용이 포함된 일회성 질문

skai chat

메모리가 있는 대화형 다중 턴 채팅

skai ui [--port 7860 --share]

Gradio 웹 UI: 채팅, 피드백, 실시간 수집

skai mcp

search_kb 및 ask를 노출하는 MCP 서버 실행

skai eval

DeepEval 품질 스위트 실행 (--group eval + 키 필요)

MCP 클라이언트 구성

서버는 두 가지 도구를 노출합니다 — search_kb(query, source_type?)(원시 검색) 및 ask(question)(전체 에이전트). MCP 클라이언트를 다음과 같이 지정합니다:

{
  "mcpServers": {
    "solid-knowledge-ai": {
      "command": "uv",
      "args": ["run", "skai", "mcp"],
      "cwd": "/absolute/path/to/solid-knowledge-ai"
    }
  }
}

모델 선택

기본값은 Haiku 4.5(빠르고 저렴 — Q&A 라우터+그레이더+생성기 루프에 적합)입니다. 호출별로 --model로 전환하거나 .env의 SKAI_MODEL로 전역 설정합니다:

값

해석

haiku (기본값)

anthropic/claude-haiku-4-5

sonnet

anthropic/claude-sonnet-4-5

모든 LiteLLM ID

통과 (예: openai/gpt-4o-mini)

Opus는 의도적으로 차단됩니다(resolve_model이 예외를 발생시킴), 따라서 어시스턴트가 실수로 가장 비싼 등급을 가리킬 수 없습니다.

관찰 가능성

.env에 LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY(및 선택적으로 LANGFUSE_HOST)를 설정하세요. 그러면 모든 그래프 실행이 노드별 및 LLM 호출별 스팬이 있는 하나의 트레이스를 생성합니다. 키가 없으면 추적은 완전한 no-op입니다 — 다른 것은 변경되지 않습니다.

품질 평가 (DeepEval)

uv sync --group eval
export ANTHROPIC_API_KEY=...
uv run skai ingest
uv run --group eval pytest evals -v      # or: skai eval

판정자는 LiteLLM을 통한 Claude이므로 OpenAI 키가 필요 없습니다. 지표: 충실도, 답변 관련성, 컨텍스트 관련성 — 추가로 저렴한 키워드 게이트.

테스트

uv run pytest        # 39 tests, fully offline: no network, no API keys

LLM은 의존성 주입되므로 전체 그래프가 결정적 스텁에 대해 테스트에서 실행되고, Chroma는 결정적 인프로세스 임베딩 함수를 사용합니다.

구성 방식

src/skai/
  config.py            settings (.env)              agent/llm.py     ChatLiteLLM -> Claude
  models.py            Document/Chunk/AgentState    agent/nodes.py   route/retrieve/grade/generate/self_check
  ingest/loaders.py    pdf | md | web  -> Document  agent/prompts.py node prompts
  ingest/chunk.py      source-aware splitting       agent/graph.py   StateGraph + SQLite memory
  ingest/store.py      Chroma add/query             observability.py Langfuse handler (or no-op)
  cli.py               ingest|ask|chat|mcp|eval     mcp_server.py    search_kb / ask as MCP tools
evals/                 DeepEval suite               tests/           offline unit + graph tests

에이전트 그래프 및 컴포넌트 다이어그램 (Mermaid): docs/ARCHITECTURE.md 참조. 설계 근거 및 기술 트레이드오프: docs/DECISIONS.md 참조.

상태

검증됨: uv run skai ingest가 세 가지 소스 유형 모두를 로드하고(2 md + 1 pdf + 1 web → 170 청크) 실제 의미 검색이 관련 구절을 반환합니다. 39개의 오프라인 테스트가 통과합니다. ask/chat/eval은 ANTHROPIC_API_KEY가 필요합니다.

Available Tools

2 tools
askB

Ask the self-reflective knowledge agent. Returns a grounded, cited answer.

NameRequiredDescriptionDefault
questionYes

Output Schema

NameRequiredDescription
resultYes

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry behavior disclosure; it does say outputs are grounded and cited, a meaningful trait. However, it does not mention limitations, confidence, citation format, or whether the agent can refuse/ask follow-ups, leaving the behavioral profile thin.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, with the core instruction and return behavior front-loaded. No filler or redundant restatement of the schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter tool with an output schema, the description covers the key behavior and return characteristic. It is mostly complete, though the lack of sibling differentiation and parameter detail keeps it from being fully self-sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has a single required `question` string with no field description, and the description does not elaborate on expected question format, length, or scope. The parameter name is self-explanatory, but the description adds no semantic detail to compensate for 0% schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific action ('Ask') on a distinct resource ('self-reflective knowledge agent') and its output ('grounded, cited answer'). It does not explicitly compare itself to search_kb, but the resource and output type give enough differentiation for a general sense.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to ask versus using search_kb, nor any exclusions or conditions. The name and description imply Q&A usage, but the agent is not told when to choose this tool over the sibling.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_kbC

Semantic search over the ingested documents. Optional source_type: pdf|md|web.

NameRequiredDescriptionDefault
queryYes
source_typeNo

Output Schema

NameRequiredDescription
resultYes

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are present, so the description carries the full burden of disclosing behavior. It reveals that the tool searches over ingested documents and accepts an optional source_type, but it does not disclose result limits, relevance behavior, authentication needs, or any side effects. For a read/search tool this is a notable but not severe gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single tight sentence followed by a compact optional-parameter note. It is front-loaded with the core action and resource, and every word adds information. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple two-parameter search tool, the description covers the essential invocation surface, and an output schema likely documents return values. However, it omits any comparison with 'ask' and does not explain result behavior or limitations, leaving the agent to infer when this tool is appropriate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does add meaning to source_type by enumerating allowed values ('pdf|md|web'), but it leaves the main 'query' parameter semantically undefined beyond the schema's bare type declaration. The compensation is only partial.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action ('Semantic search') on a clear resource ('ingested documents') and lists the optional source_type filter. It does not explicitly name the sibling tool 'ask' as the alternative, so some differentiation is left to inference, but the operation is unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to prefer search_kb over the sibling tool 'ask', nor any exclusions or prerequisites. Usage context is only implied by the phrase 'semantic search', which is not enough to route an agent reliably.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updatesv0.1.0
    • First observedask
    • First observedsearch_kb
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

B3.2/5.0

Scored across 2 tools

Disambiguation4/5

The two tools are distinct in purpose: search_kb returns semantic search results over sources, while ask provides a grounded, cited answer through an agent. There is some overlap in that both retrieve information from the same knowledge base, but the descriptions clearly differentiate a low-level search from a high-level Q&A interaction, making misselection unlikely.

Naming Consistency4/5

Both tool names use lowercase snake_case and are verb-based. 'search_kb' follows a verb_noun pattern while 'ask' is a bare verb, creating a minor inconsistency, but the pattern is still simple and predictable given the small set.

Tool Count3/5

With only two tools, the server feels thin but not unreasonable. Search and ask cover the core knowledge-access functions, though additional tools like listing sources or managing documents might be expected in a fuller knowledge-management server.

Completeness4/5

The tool surface covers the core domain of querying an ingested knowledge base: search_kb for retrieval and ask for synthesized answers. Minor gaps exist, such as no way to enumerate available sources or inspect document metadata, but these are not critical for the stated purpose.

Maintenance

ActivitySlowing
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers