Confluent Support RAG — MCP Server

An MCP server that gives any AI assistant (Claude, ChatGPT, Gemini) or any local LLM searchable access to a knowledge base of Confluent support articles, and lets it save solved problems back into that knowledge base.

It runs entirely in Docker, is reachable over HTTPS through an ngrok tunnel, and can require Microsoft (Entra) sign-in restriction.

1. Overview

What this does. It exposes a small set of tools over the Model Context Protocol (MCP), the standard interface AI assistants use to call external tools. An assistant can search the knowledge base, and with your explicit approval it can write a solved problem back into it.

Current state

The tools

List of tools and what it does

1. confluent_search_documents: Hybrid search across everything. The default.

2. confluent_search_official_articles: Same search, official articles only.

3. confluent_search_saved_conversations: Same search, user-saved fixes only.

4. confluent_save_solved_conversation: Step 1 of 2. Returns a preview. Writes nothing.

5. confluent_confirm_save_conversation: Step 2 of 2. Commits the draft. The only tool that writes.

6. confluent_collection_info: Collection stats, chunk counts and models in use.

MCP prompt (only supported by some AI assistants/clients.)

1. confluent_prepare_solved_conversation, for when a user wants to paste the excerpt themselves.

Related MCP server: context-repo-mcp

2. How it works

        MCP client  (Claude · ChatGPT · Gemini · rag/orchestrator.py)
                              │
                              │  HTTPS
                              │
                        ngrok tunnel  ── TLS terminates here
                              │
                              │  plain HTTP, internal Docker network
                              │
                    ┌─────────▼───────────┐
                    │  mcp_server :8000   │   mcp_server/qdrant_server.py
                    │    auth · tools     │
                    └──┬──────────────┬───┘
           embeddings  │              │  hybrid search / save
                ┌──────▼─────┐  ┌─────▼──────┐
                │  Ollama    │  │  Qdrant    │   collection:
                │  :11434    │  │  :6333     │   confluent_support
                └────────────┘  └────────────┘

Three flows share one collection.

Flow A — Ingest (offline, run manually)

Confluent Support Portal (Zendesk Help Center API)
   │
   │  fetch_articles_for_category()          (from ingest/scrape_support.py)
   │
   ▼  raw article HTML
   │
   │  extract_blocks()  → [(block_type, text), …]
   │     headings, paragraphs, code, tables (_table_to_markdown)
   │
   ▼
   │
   │  chunk_article()                        (from shared/chunking.py)
   │     ~300-word target, 1-block overlap
   │     each chunk is prepended with its section heading
   │
   ▼  ingest/fetched_docs.json
   │
   │  main()                                 (from ingest/embed_store.py)
   │     embed_dense_batch()  → Ollama       → 1024-d vector
   │     embed_sparse_batch() → BM25         → sparse vector
   │     point_id(article_id, chunk_index)   → deterministic UUID
   │
   ▼
Qdrant.upsert()   source_type="official_article"

(article_id, updated_at): Zendesk article IDs are immutable, so an edited article keeps its ID and only its updated_at moves. load_existing_articles() reads {article_id: updated_at} from the collection and each article is classified:

Updated articles are deleted before re-inserting rather than upserted in place. That is what clears orphans: if an edit shortens an article from 16 chunks to 15, upserting alone would leave chunk 15 behind, still returnable by search.

Only source_type="official_article" points are considered, so saved conversations are never refreshed or deleted by ingestion.

A typical run re-embeds 41 chunks out of 3,281 (1.2%), versus --reset which redoes all of them.

Articles missing from a scrape are deliberately left alone, never deleted. An expired SUPPORT_COOKIE makes every article look gone, and a sync that trusted absence would wipe the knowledge base. Observed in practice: article 48565010397972 vanished from the scrape but returns HTTP 403, not 404. It still exists, its permissions changed.

Flow B — Query (live)

All three search tools call one shared _run_search(); they differ only by which source_type they filter to.

query
   │
   │  MAX_TOP_K (50)
   │
   ▼  embed twice: Ollama dense + _embed_sparse_query() BM25
   │
STAGE 1 · recall ──  _client.query_points()
   │
   │  Prefetch(dense, filter) ┐
   │                          ├── FusionQuery(RRF) → candidate pool
   │  Prefetch(bm25,  filter) ┘    size = max(top_k × 4, 15)
   │
   │  The source_type filter is applied to each prefetch branch.
   │
   ▼
STAGE 2 · precision ──  _reranker.rerank()
   │
   │  cross-encoder reads query + each chunk together, one pass per candidate
   │  → sort by score → drop below RERANK_MIN_SCORE → keep top_k
   │
   ▼
formatted results (rerank score, title, source_type, source URL) → the LLM

Why two stages?: dense catches meaning, BM25 catches exact strings; RRF fuses them for recall. The cross-encoder then fixes the ordering, because rank-fusion scores tie often. The reranker is expensive (one model pass per candidate), so it only ever sees the small stage-1 pool, never all 3,283 chunks.

Flow C — Save (live, two steps)

The server never sees the conversation, only what the AI chooses to send. So it can't verify a summary is faithful; the human is the last check. The design makes the AI show its work and makes the human look before anything is written.

STEP 1  confluent_save_solved_conversation(title, description, resolution, source_excerpt, applies_to, cause, source_ai)
   │
   │  required fields present?
   │  source_excerpt >= 20 words?                        → else reject
   │  _grounding_overlap(summary, excerpt) < 0.15        → else reject
   │  _find_near_duplicate(text)                         → advisory only, never auto reject
   │  park a draft in memory (owner-bound, 30 min TTL)
   │
   ▼
   PREVIEW + draft_token.  NOTHING IS WRITTEN.
   Shows the quoted excerpt and any similarity warning.
   User checks before moving on to step 2.


STEP 2  confluent_confirm_save_conversation(draft_token)
   │
   │  token valid, unexpired, same owner?
   │  chunk_article() → embed dense + sparse (same path as ingestion)
   │
   ▼
Qdrant.upsert()   source_type="chat_history", + source_excerpt + saved_by

Four guards sit in front of a write, each catching what the others can't:

The duplicate check only warns, it does not block. It shows the matching entry's title, URL and existing text so you can compare.

Data model

Two named vectors per point, so one hybrid query can use both. Both writers emit an identical key set, so code reading the collection never branches on source.

3. Setup on a new machine

Prerequisites

• Docker Desktop (or Docker Engine) with Compose v2+.

• 8 GB RAM minimum, 16 GB recommended.

• ~4 GB free disk for the image, models and vectors.

Step 1 — Get the code and make a config file

git clone <your-repo-url> confluent-rag
cd confluent-rag
cp .env.example .env

The stack starts with .env left exactly as copied. Be aware that the knowledge base is not in the repository, so a fresh clone starts with an empty collection and every search returns nothing until Step 3 is done. Fill in what you need:

Step 2 — Start the stack

docker compose up -d

This starts five services: qdrant, ollama, ollama_init, mcp_server, ngrok.

The first boot will be downloading the embedding model (~640 MB) into the ollama_data volume and the BM25 + reranker weights (~1 GB) into model_cache. Both persist, so later starts take seconds.

docker compose ps
docker compose logs -f mcp_server

mcp_server should reach (healthy). ollama_init showing Exited (0) is correct.

ngrok restarts in a loop until NGROK_AUTHTOKEN is set, logging ERR_NGROK_4018. That is expected on a fresh clone and affects nothing else. Set the token (see "Remote access"), or run docker compose stop ngrok if you only want local access.

Step 3 — Load the knowledge base

The vectors live in the qdrant_storage Docker volume, which is local to one machine and not part of the repository. Neither is ingest/fetched_docs.json. A fresh clone therefore has to build the knowledge base before anything can be searched. If you're restoring a machine that already has that volume, skip this step.

Three values drive the scrape. .env.example ships the first two already filled in:

Those three category IDs are Getting Started, Confluent Cloud and Confluent Platform. Blanking SUPPORT_BASE_URL does not fall back to a default: an empty value in .env overrides the fallback inside the script, and every request then fails as a URL with no protocol.

To build it from scratch:

# 1. Scrape the support portal, **needs** a valid SUPPORT_COOKIE in .env
docker compose run --rm ingestion python ingest/scrape_support.py

# 2. Embed and store
docker compose run --rm ingestion python ingest/embed_store.py

Always pass --rm these are one-off commands, not services, and without it you accumulate stopped containers.

SUPPORT_COOKIE is a browser session cookie copied from a logged-in Confluent support portal session, and it expires. Whoever sets this up needs portal access. If the scrape returns 401s or empty results, re-copy it from a logged-in browser. This only affects re-scraping; the server keeps serving whatever is already in Qdrant.

Step 4 — Verify it worked

curl http://localhost:6333/collections/confluent_support

Inside the result object you should see "status": "green" and a non-zero points_count. A 404 saying the collection doesn't exist means Step 3 hasn't run yet, because the collection is created by embed_store.py and not at startup.

Then try a real question end to end:

docker compose run --rm -it orchestrator

This is an interactive terminal client. Ask something like "why does my consumer group keep rebalancing?", you should see retrieved chunks followed by a generated answer. It sends no auth headers, so it only works with MCP_AUTH_ENABLED=false.

Step 5 — Connect a real MCP client

Point your client at:

https://<your-ngrok-domain>/mcp    (remote)

4. Remote access

The server speaks plain HTTP on port 8000. Remote MCP clients need HTTPS, so something must terminate TLS in front of it. That is the tunnel's job.

A tunnel agent opens an outbound connection to the provider and holds it open, which is why this needs no port forwarding or firewall changes. A client hits https://your-domain/mcp, the request lands on the provider's edge which owns the certificate, so TLS terminates there, and the edge forwards it down the already-open connection to mcp_server:8000.

The agent must run continuously. That is normal for every tunnel, not an ngrok quirk. It runs as a compose service with restart: unless-stopped, so it comes back on its own after a reboot. The host machine must also be powered on with Docker running.

Docker alone cannot replace this. It can publish port 8000 on the host, but it cannot give you a public hostname or a TLS certificate.

Setup: put your authtoken and reserved static domain in .env (NGROK_DOMAIN is the hostname only, no https:// and no /mcp), then docker compose up -d. ngrok starts with the stack.

Verify. Don't trust the container status, because ngrok logs almost nothing on success:

curl -s -o /dev/null -w '%{http_code}\n' https://<your-domain>/mcp

406 is the correct answer: it's the MCP server saying the client must accept text/event-stream. That proves TLS terminated, the tunnel routed, and the server replied. 401 means auth is on and working. 502 means the tunnel is up but can't reach the server. HTML means something else answered.

Rate limiting. ngrok-policy.yml applies 120 requests/min per IP at the edge. It is the only edge control that costs nothing in client compatibility, because edge auth would break MCP clients entirely.

Cloudflare Tunnel is wired up as an alternative behind a profile (docker compose --profile cloudflare up -d), which reads CLOUDFLARE_TUNNEL_TOKEN from .env. It needs a domain on Cloudflare but has no request quota, so it is the escape hatch if ngrok's 20k/month becomes a problem. Running both just gives you two public URLs onto the same server, so docker compose stop ngrok once you've switched.

5. Authentication

Optional and off by default (MCP_AUTH_ENABLED=false), so the stack starts and serves for someone who hasn't been given credentials.

Auth must live inside the server, not at the tunnel edge. MCP clients authenticate by exchanging tokens, not by completing a browser login. An edge login page returns a redirect the client can't follow, so it doesn't add auth, it makes the server unreachable.

The server is an OAuth 2.1 resource server: it never issues credentials, it only validates tokens Microsoft issued.

client → /mcp with no token        → 401 + pointer to /.well-known metadata      → registers, sends user to Microsoft Entra sign-in
Entra  → user signs in             → signed JWT (audience = this server)
client → /mcp + Bearer JWT
             │  
             │
             │
   AzureProvider   verifies signature · issuer · audience
             │
             │  token OK
             │
   EmailDomainMiddleware.on_message()
       _current_user_email() reads email / preferred_username / upn
       ALLOWED_EMAILS set?  → address must be on that exact list
       else                 → domain must be in ALLOWED_EMAIL_DOMAINS
             │
             │  allowed
             │
          tool runs

Entra app registration (one time)

Azure portal → Microsoft Entra ID → App registrations → New registration.

1. Register it SINGLE-TENANT. This is a security requirement, not a preference. preferred_username is mutable, so with a multi-tenant registration a user in another Entra directory could set theirs to end in your domain and pass the email check. Single-tenant means only your own directory can obtain a token for this app.

2. Redirect URI, type Web, put in exactly <PUBLIC_MCP_URL>/auth/callback.

3. Expose an API → accept the default Application ID URI (api://<client-id>) → Add a scope named access, consentable by admins and users. This step is mandatory: Entra access tokens carry only custom API scopes in the scp claim, so the standard OIDC scopes can't be used and the server refuses to start without one.

4. Manifest → set "requestedAccessTokenVersion": 2. Without this Entra issues v1.0 tokens, whose issuer differs from what the server expects, and every request is rejected with an issuer mismatch. v2 also supplies the preferred_username claim the email check wants.

5. Certificates & secrets → New client secret. Copy the value, not the ID. It is shown only once.

The four credentials

Generate the signing key once with python -c "import secrets; print(secrets.token_urlsafe(32))".

Who may use the tools

Two modes. Set one in .env:

When ALLOWED_EMAILS is non-empty it wins and ALLOWED_EMAIL_DOMAINS is ignored entirely, so a colleague on an allowed domain who isn't listed is denied. The two are deliberately not combined, which also lets you name an outside contractor (dave@partner.co.th) without opening their whole domain. Matching is case-insensitive and exact.

Leaving both blank while auth is on accepts any authenticated account. The server logs a warning at startup if you do.

Changing either value needs a restart: docker compose up -d --force-recreate mcp_server.

Two layers, and the second is not redundant. Single-tenant blocks accounts outside the directory. The middleware additionally covers guest users invited into the tenant, contractors and partners whose address may be @gmail.com or another company's domain. Entra would authenticate those happily. The check runs as middleware rather than per-tool, so any tool added later is protected by default.

6. Day-to-day operations

docker compose up -d                        # start
docker compose down                         # stop (volumes and data survive)
docker compose ps                           # what's running
docker compose logs -f mcp_server           # tail logs
docker compose down -v                      # DESTRUCTIVE: also wipes the knowledge base

After changing code

docker compose up -d --build mcp_server

docker compose build alone is not enough. It rebuilds the image but leaves the old container running, so your changes silently don't take effect. To confirm the running container matches your source:

docker exec confluent-rag-mcp-server md5sum /app/mcp_server/qdrant_server.py
md5sum mcp_server/qdrant_server.py

Other tasks

# Interactive test client (needs MCP_AUTH_ENABLED=false)
docker compose run --rm -it orchestrator

# Re-scrape and re-embed
docker compose run --rm ingestion python ingest/scrape_support.py
docker compose run --rm ingestion python ingest/embed_store.py
docker compose run --rm ingestion python ingest/embed_store.py --reset   # full rebuild

# Collection stats
curl http://localhost:6333/collections/confluent_support

Volumes

7. Configuration reference

Values marked (docker) are overridden by docker-compose.yml's environment: block, so they only matter if you run the Python scripts directly on the host.

The Default column is what .env.example ships, which is what you get after Step 1. Note that a variable present but blank in .env overrides the fallback written into the Python, it does not fall back to it.

Environment variables

Tuning constants

In mcp_server/qdrant_server.py, not environment variables.

8. Known limits and risks

Memory is the real constraint. Measured on this stack:

A single real search at the default top_k=5 pushed one container to 6.2 GB on an 8 GB host, after which its network calls began failing, and OOMKilled=true has been observed. Growth is roughly linear in candidate count, and MAX_TOP_K=50 permits a 200-candidate pool which would extrapolate well past 8 GB. Consider a mem_limit on mcp_server so it fails predictably instead of starving the host, and lowering MAX_TOP_K if the full range isn't needed.

Practical consequence: don't run test scripts via docker exec into the running server. That loads a second copy of the models and gets OOM-killed, which surfaces as misleading "DNS resolution" errors. Use docker compose run --rm instead.

A fabricated excerpt cannot be detected. The grounding check compares the summary against the excerpt, so an invented pair is internally consistent and passes. The word floor only catches lazy padding. The human reviewing the preview is the last line of defence, which is why the preview shows the excerpt.

ngrok free tier: 20,000 requests/month, 1 GB transfer. MCP is chatty, each session costs several HTTP requests. That is roughly 660/day: fine for a couple of people, tight for a team. You get no warning, it just starts failing. The cloudflare profile is the way out.

orchestrator.py sends no auth headers, so it only works with MCP_AUTH_ENABLED=false. That is deliberate, since a bypass would be a second way in to secure and audit.

Ollama Cloud retires models with little notice, and a retired model fails at answer time with HTTP 410. Only orchestrator.py is affected because embeddings are local. List what's available and swap INFERENCE_MODEL:

curl -s -H "Authorization: Bearer $OLLAMA_API_KEY" https://ollama.com/api/tags

Being listed doesn't mean you can use it, many return 403 on the free tier.

There are no automated tests. Changes are verified by hand.

9. Troubleshooting

10. Handoff and ownership

Credentials that must be transferred

These are tied to individual accounts. If the current owner leaves, the service breaks or becomes unmanageable.

Also note ALLOWED_EMAILS: if it's set to one person, only that person can use the tools. Add the new owner before you hand over, or switch to ALLOWED_EMAIL_DOMAINS.

Auditing what's been saved

Every saved conversation stores source_excerpt (the verbatim conversation it came from, on chunk 0) and saved_by. Excerpts are never embedded and never appear in search results, so they exist purely so a bad entry can be traced back.

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue

q = QdrantClient(host="localhost", port=6333)
pts, _ = q.scroll("confluent_support", limit=100, with_payload=True,
    scroll_filter=Filter(must=[FieldCondition(
        key="source_type", match=MatchValue(value="chat_history"))]))
for p in pts:
    print(p.payload["article_id"], "|", p.payload["saved_by"], "|", p.payload["title"])

Removing a bad entry

A bad entry in a RAG store is self-perpetuating, it surfaces in future searches and shapes future answers. Delete by article_id:

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue, FilterSelector

q = QdrantClient(host="localhost", port=6333)
q.delete("confluent_support", points_selector=FilterSelector(filter=Filter(
    must=[FieldCondition(key="article_id", match=MatchValue(value="<article_id>"))])))

Repository layout

mcp_server/qdrant_server.py   the MCP server: tools, search, save, auth
shared/chunking.py            chunk_article(), shared by ingestion and save
ingest/scrape_support.py      step 1, scrape and chunk to fetched_docs.json
ingest/embed_store.py         step 2, embed and upsert into Qdrant
rag/orchestrator.py           interactive terminal test client
docker-compose.yml            the whole stack
Dockerfile                    image shared by server and ingestion
requirements.txt              pinned Python dependencies
ngrok-policy.yml              edge rate limit
.env.example                  config template, copy to .env
.gitignore                    keeps .env, fetched_docs.json and caches out of git
.dockerignore                 keeps the same out of the image
README.md                     this file

ingest/fetched_docs.json and the Docker volumes are deliberately not in the repository, which is why Step 3 exists.