Skip to main content
Glama

KnowledgeMCP

CI Python AWS CDK MCP License: MIT

A team-scoped RAG knowledge base on AWS, served as an MCP server and a REST API. Upload PDFs, Markdown or text files to S3. Ask questions from Claude Desktop, Claude Code or any app, and get answers with citations (document + page/section). Each user only sees answers from the documents their team may read. If the answer isn't in those documents, it says so instead of guessing.

KnowledgeMCP preview

What it does

Ingestion

S3 upload → Lambda parses PDF/Markdown/text → section-aware chunks (page or heading kept) → Amazon Titan Text Embeddings V2 → Aurora PostgreSQL + pgvector. Re-uploads of unchanged files are skipped; deletes remove the document.

Hybrid search

Vector search (pgvector HNSW, cosine) + keyword search (Postgres full-text), merged with Reciprocal Rank Fusion, then re-ranked by Amazon Rerank 1.0 (falls back to a local re-ranker if the call fails).

Answers with citations

Claude on Bedrock answers only from the retrieved passages, citing [S1], [S2]… Citations are verified; uncited answers are refused.

Access control

The first S3 folder is the team (hr/, finance/, public/). API keys map to users and teams; team filtering happens in SQL, so other teams' text never reaches the model.

Guardrails

Input validation and prompt-injection checks on questions; retrieved text is sanitized and wrapped as untrusted data; relevance gate and NOT_FOUND path for refusals.

MCP server

Tools ask_question, search_documents, list_documents (typed results) and doc://{doc_id} resources. Talks to the deployed API, so access control stays server-side.

Evaluation

35-case test set: recall@k, MRR, answer correctness, citation accuracy, refusal accuracy, access control, injection blocking, optional LLM-judge faithfulness, p50/p95 latency, cost per query.

Observability

Structured JSON logs, CloudWatch metrics via Embedded Metric Format, X-Ray traces, per-step timings in every answer, and a CloudWatch dashboard.

Infrastructure

AWS CDK (Python): Aurora Serverless v2 (scales to 0), S3, three container Lambdas, HTTP API with throttling, Secrets Manager, dashboard, optional $10/month budget alert.

Related MCP server: senza-knowledge-mcp

Architecture

Architecture

Asking a question: API key → user and teams → input guardrail → embed question → vector + keyword search (both filtered to the user's teams) → RRF fusion → re-rank → relevance gate → sanitize sources → Claude → citation check → answer, or a refusal.

Design decisions

  • Aurora Serverless v2 with min 0 ACU instead of an always-on RDS instance. The database pauses when idle, so a portfolio project doesn't pay for compute 24/7. Trade-off: the first request after a pause waits while it resumes (about 15 s); the Data API client retries automatically.

  • RDS Data API instead of Lambda-in-VPC. Lambdas reach the database over HTTPS with IAM, so there is no NAT gateway or VPC endpoint to pay for, and no connection pooling to manage.

  • One container image, three handlers (API, ingest, migrate), selected with the Lambda cmd.

  • Schema migrations run after every deploy through a CDK Trigger (idempotent IF NOT EXISTS).

  • Same SQL locally and on AWS: PsycopgExecutor (local Postgres) and DataApiExecutor (Aurora) share one store, so integration tests run against a real pgvector database.

  • Honest naming: the keyword side uses Postgres full-text ranking (ts_rank_cd), which is similar in spirit to BM25 but is not BM25.

Results

Measured against the deployed stack (Bedrock models) with python -m evals.run_evals --target http --judge bedrock.

Metric

Value

Retrieval recall@5

to be measured after deployment

Answer correctness

to be measured

Citation accuracy

to be measured

Faithfulness (LLM judge)

to be measured

Refusal accuracy (unanswerable + other-team)

to be measured

Access control / injection blocking

to be measured

Latency p50 / p95 (warm)

to be measured

Model cost per question

to be measured

CI also runs the evals with offline stand-in models on every push. Those numbers only test the plumbing, but CI fails if access control or injection blocking drops below 100%.

Run it locally (no AWS needed)

docker compose up --build        # Postgres + pgvector and the API; sample docs are indexed on start
open http://localhost:8000/docs  # interactive API docs

Or without Docker, against any Postgres that has pgvector:

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,local]"
export DATABASE_URL=postgresql://postgres:postgres@localhost:5432/kmcp
python -m knowledgemcp.cli ingest sample_docs
python -m knowledgemcp.cli ask "How many vacation days can I carry over?" --teams hr
pytest -q                        # 50 tests (includes the pgvector integration test)

Local mode uses offline stand-ins (word-hash embeddings, an extractive "LLM", lexical re-ranker) so everything runs without credentials. Set EMBEDDER=bedrock LLM=bedrock RERANKER=bedrock to use the real models.

Deploy to AWS

Prerequisites: AWS CLI configured (aws sts get-caller-identity works), Docker running, Node.js 18+, Python 3.12. In the Bedrock console (us-west-2), make sure your account can use Titan Text Embeddings V2, Amazon Rerank 1.0 and Claude Haiku 4.5.

pip install -e ".[dev,local]" -r infra/requirements.txt
npm install -g aws-cdk
cd infra
cdk bootstrap                                    # once per account/region
cdk deploy -c alert_email=you@example.com        # prints ApiUrl, BucketName, ApiKeysSecretArn, ...

Then load documents, create a key and ask:

aws s3 sync ../sample_docs s3://<BucketName>/                      # team = first folder
python ../scripts/create_api_key.py --user teja --teams hr,engineering,finance --secret-id <ApiKeysSecretArn>
curl -s <ApiUrl>/ask -H "x-api-key: kmcp_..." -H "content-type: application/json" \
  -d '{"question": "Who approves expenses above 1,000 US dollars?"}'

Tear everything down with cdk destroy.

Use it from Claude Desktop (MCP)

pip install -e ".[mcp]"

Add this to claude_desktop_config.json:

{
  "mcpServers": {
    "knowledgemcp": {
      "command": "/absolute/path/to/.venv/bin/knowledgemcp-mcp",
      "env": { "KMCP_API_URL": "https://<id>.execute-api.us-west-2.amazonaws.com", "KMCP_API_KEY": "kmcp_..." }
    }
  }
}

For development without AWS, use "env": {"KMCP_LOCAL": "1", "KMCP_TEAMS": "hr,engineering", "DATABASE_URL": "..."}.

Evaluate

# one key per persona: hr, engineering, finance, none (public only)
python scripts/create_api_key.py --user eval-hr --teams hr --secret-id <ApiKeysSecretArn>   # repeat per persona
# evals/keys.json  ->  {"hr": "kmcp_...", "engineering": "kmcp_...", "finance": "kmcp_...", "none": "kmcp_..."}
python -m evals.run_evals --target http --url <ApiUrl> --keys evals/keys.json --judge bedrock

Results are written to evals/results/ as JSON and Markdown.

Project structure

src/knowledgemcp/
  loaders.py, chunking.py     PDF/Markdown/text → sections → chunks (page/section kept)
  embeddings.py               Titan V2 embedder (+ offline hash embedder)
  db.py, store.py             Data API / psycopg executors; pgvector store with team filtering
  search.py, rerank.py        hybrid retrieval, RRF, Bedrock re-ranker (+ lexical fallback)
  guardrails.py, llm.py       input/source/output guardrails; Claude via Bedrock Converse
  rag.py                      the question-answering pipeline, metrics and cost
  api.py, auth.py             FastAPI app + Lambda handler; hashed API keys
  mcp_server.py               MCP tools and resources
  ingest.py, handlers_ingest.py, cli.py
infra/                        AWS CDK app (Python) + synth tests
evals/                        test set and evaluation harness
tests/                        pytest suite (moto, botocore Stubber, real pgvector)
sample_docs/                  fictional company documents (hr, engineering, finance, public)

Limitations

  • Prompt-injection detection is pattern-based. It is one layer of several (system prompt, data wrapping, SQL team filtering, citation checks), not a complete defense.

  • HNSW search with a team filter can return fewer than k results when a team owns a small share of chunks; at larger scale, use pgvector iterative scans or per-team partitions.

  • The sample documents describe a fictional company.

License

MIT

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables MCP clients to ask plain-language questions and receive answers grounded only in documents the configured role is cleared to read, with the same access-controlled tools available across any client.
      MIT
    • F
      license
      Not graded
      quality
      B
      maintenance
      Enables retrieval-augmented question answering with source-grounded citations, document retrieval, and idempotent ingestion through MCP tools.
      -