semoss-dev-mcp

This app IS the MCP server. It's the real SEMOSS project Semoss Dev MCP (65eb25bb-c981-48c3-bcef-5a8dd6ae4391), symlinked to this repo (see "Local setup" below) so content and code here are edited once and reflected in the running project with no copy/paste. Its tools call a separate real vector engine (semoss-dev-vector, c3cbeb32-98ee-475a-8f8b-7f7a47a9a84d) internally, scoped per-tool to specific source files, so an agent gets a small set of purpose-named, focused tools instead of one generic "search everything" tool it has to guess how to aim. It also serves a browsable portal (a real Vite + React app) so a person can read the same content in a web page.

Confirmed working end-to-end (2026-07-15): connected via Copilot CLI, called get_reactor_patterns for real, got real ranked results back from semoss-dev-vector.

What's here

semoss-dev-mcp/
  public/md/                       content — must be under public/ (see below)
    agents.md                       documentation map — read this one first
    architecture/
      monolith-semoss-boundary.md
    java/
      java-reactor-patterns.md
    python/
      python-java-tcp-bridge.md
      python-reactor-conventions.md
    security/
      security-checklist.md
  vector-upload/                   .txt copies for uploading into semoss-dev-vector
                                     (gitignored — regenerate with `cp x.md x.txt`)
  py/
    mcp_driver.py                  the MCP tools only — see "Why almost no logic
                                     lives in mcp_driver.py" below
    _vector_helpers.py             all real logic (pixel building, running, result
                                     formatting) — deliberately not in mcp_driver.py
  mcp/
    py_mcp.json                     manifest for mcp_driver.py's tools — MACHINE-
                                     GENERATED by SEMOSS's own MakePythonMCP, confirmed
                                     regenerating live on this exact repo (2026-07-15,
                                     picked up a docstring edit unprompted). Don't
                                     hand-edit; edit mcp_driver.py's docstrings instead.
    pixel_mcp.json                  BrowseAppAssets/GetAppAssets, wrapping real
                                     existing Semoss reactors (no custom code)
  client/                          real Vite + React + TypeScript app (doc portal)
    src/
      hooks/useDocs.ts               BrowseAppAssets/GetAppAssets via @semoss/sdk-react
      components/DocViewer.tsx       sidebar + markdown render (marked/dompurify/highlight.js)
  portals/                         `pnpm build` output — committed, not hand-edited
  package.json / vite.config.ts / tsconfig.json / index.html
  README.md

Related MCP server: RAG Documentation MCP Server

Why almost no logic lives in mcp_driver.py

Confirmed empirically (2026-07-15): SEMOSS's own MCP-manifest auto-generator re-scans py/mcp_driver.py and turns every top-level function into an MCP tool — including private, underscore-prefixed helpers that were never meant to be tools. It does not filter by leading underscore or by presence/absence of @mcp_metadata. Fix: all real logic (build_query_pixel, run_pixel, format_results, query_vector, the SOURCE_FILES registry) lives in _vector_helpers.py, a plain module mcp_driver.py imports from. Only the six real tool functions exist as top-level names in mcp_driver.py itself. If you add a helper function, put it in _vector_helpers.py, never directly in mcp_driver.py.

Why public/

Non-editor callers (which is what most MCP access/secret-key callers will be, from this project's own perspective) are restricted to the public/ folder by BrowseAppAssetsReactor/GetAppAssetsReactor — confirmed in Semoss's own source:

// src/prerna/util/FileSystemUtil.java:412-426
public static boolean restrictBrowseToPublicFolder(boolean canEdit, String relativeFilePath) {
    if (canEdit) {
        return false;
    }
    String p = stripAssetPathSlashes(relativeFilePath);
    if (p.isEmpty()) {
        return true; // view-only user at the assets root: only show the public folder node
    }
    if (p.equals(Constants.PUBLIC_ASSETS_FOLDER) || p.startsWith(Constants.PUBLIC_ASSETS_FOLDER + "/")) {
        return false;
    }
    throw new IllegalArgumentException("User only has read access to the '"
            + Constants.PUBLIC_ASSETS_FOLDER + "' folder within the assets folder.");
}

Local setup (already done for this checkout — documented for reproducing elsewhere)

The real project's assets directory is symlinked to this repo, per-folder, so editing either side is the same file:

ASSETS="<path to>/Semoss Dev MCP__65eb25bb-c981-48c3-bcef-5a8dd6ae4391/app_root/version/assets"
REPO="<path to>/semoss-dev-mcp"

ln -s "$REPO/public" "$ASSETS/public"
ln -s "$REPO/mcp" "$ASSETS/mcp"
ln -s "$REPO/py" "$ASSETS/py"
rm -rf "$ASSETS/portals"
ln -s "$REPO/portals" "$ASSETS/portals"   # symlinks the whole build-output dir

.notebooks/ and .admin/ in the assets folder are real SEMOSS scaffold — left alone, not symlinked. Adding a new file under public/md/ or editing py//mcp/ needs no new symlink — those are whole-directory symlinks, so new files inside them are picked up automatically.

The client app (portal)

Real Vite + React + TypeScript, modeled on vba-futures/apps/prompt-manager's real build setup (package.json/vite.config.ts shape), minus its @vba-futures/*/Tailwind workspace deps since this is a standalone repo. Build output goes to portals/ (vite.config.ts's outDir), which is what's symlinked into the SEMOSS project above — portals/ is build output, not something to hand-edit.

pnpm install
pnpm run build       # writes portals/ — commit the result, matching the real
                      # vba-futures convention of committing built portal output
pnpm run dev         # local dev server against a real instance — needs .env.local,
                      # copy from .env.example

Type-checked (npx tsc --noEmit) and build-verified as of 2026-07-15.

Viewing the portal — use the public_home URL, not BrowseAppAssets

Confirmed (2026-07-15) via reading the real serving path in both repos — the portal must be viewed at:

<applicationUrl>/public_home/<projectId>/portals/
# e.g. locally: http://localhost:9090/Monolith/public_home/65eb25bb-c981-48c3-bcef-5a8dd6ae4391/portals/

not via BrowseAppAssets/a raw asset path. Browsing raw assets serves portals/index.html byte-for-byte as committed — with no #semoss-env script tag — so Env.MODULE stays "" in the production build (this app's App.tsx, like vba-futures/apps/veteran-portal and apps/prompt-manager, only calls Env.update() in import.meta.env.DEV; it relies on the platform to inject the rest in production). With Env.MODULE empty, @semoss/sdk builds request URLs relative to the current origin with no /Monolith prefix — hence the http://localhost:9090/api/engine/runPixel 404 seen instead of the correct http://localhost:9090/Monolith/api/engine/runPixel.

The public_home URL is handled by a dedicated servlet filter (Monolith/src/prerna/web/conf/PublicHomeCheckFilter.java:110-124), which — on every request under that path — calls project.requirePublish(false) and, if true, lazily calls project.publish(...) before serving the file. Project.publish() (Semoss/src/prerna/project/impl/Project.java:1147-1211) calls rewritePortalIndexHtml() (same file, :1317-1349), which rewrites portals/index.html in place to inject:

<script id="semoss-env" type="application/json">{"APP":"<projectId>","MODULE":"<module>"}</script>

right before the first child of <head> — <module> here comes from Utility.getApplicationRouteAndContextPath(), which is /Monolith for this local setup. @semoss/sdk's InsightProvider (libs/sdk/src/stores/insight/insight.store.ts:301-321 in semoss-ui) reads that same #semoss-env tag and calls Env.update({APP, MODULE}) from it — this is the only thing that sets Env.MODULE correctly in a production build; no app-level code change is needed here, since this repo's App.tsx already matches the real veteran-portal/prompt-manager pattern exactly.

You do not need to explicitly call the PublishProject pixel first — just requesting the public_home URL triggers the lazy publish. (PublishProject(project=[...], release=[...]), Semoss/src/prerna/reactor/project/PublishProjectReactor.java:50-80, exists for explicitly forcing a republish/release and returns this same URL — useful after editing portals/ without waiting for the lazy check, but not required for normal viewing.) Note the filter also enforces userCanViewProject for non-global projects, so the browser needs an active, logged-in SEMOSS session (not just the .env.local API keys, which only cover the SDK's own runPixel calls).

The tools (py/mcp_driver.py, logic in py/_vector_helpers.py)

• health_check() — cross-checks the topic registry against what's actually uploaded in the vector engine right now.

• list_topics() — lists which topic-scoped tools exist. No vector call — pure local lookup.

• get_reactor_patterns(query, limit=5) — scoped to java-reactor-patterns.txt.

• get_monolith_semoss_boundary(query, limit=5) — scoped to monolith-semoss-boundary.txt.

• get_python_tcp_bridge(query, limit=5) — scoped to python-java-tcp-bridge.txt.

• get_python_reactor_conventions(query, limit=5) — scoped to python-reactor-conventions.txt.

• get_security_checklist(query, limit=5) — scoped to security-checklist.txt.

• get_vector_engine(query, limit=5) — scoped to vector-engine.txt.

• get_adding_new_engine_type(query, limit=5) — scoped to adding-new-engine-type.txt.

• get_adding_new_model_type(query, limit=5) — scoped to adding-new-model-type.txt.

• get_semoss_ui_fe_be_boundary(query, limit=5) — scoped to semoss-ui-fe-be-boundary.txt.

• get_testing_conventions(query, limit=5) — scoped to testing-conventions.txt.

• get_monolith_endpoint_groups(query, limit=5) — scoped to endpoint-groups.txt.

• get_database_engines_overview(query, limit=5) — scoped to database-engines-overview.txt.

• get_model_engines_overview(query, limit=5) — scoped to model-engines-overview.txt.

• get_storage_engines_overview(query, limit=5) — scoped to storage-engines-overview.txt.

• get_function_engines_overview(query, limit=5) — scoped to function-engines-overview.txt.

• search_all_architecture_docs(query, limit=5) — no source filter, the fallback when no topic-specific tool fits.

Each returns ranked [source] content excerpts only — confirmed and fixed (see "Output cleanup" below), no ranking-score/bookkeeping noise. Each new topic guide added under public/md/<category>/ should get its own scoped tool in mcp_driver.py + SOURCE_FILES entry in _vector_helpers.py, following this same pattern, rather than making search_all_architecture_docs do all the work.

Output cleanup (confirmed via a real live tool call, 2026-07-15)

The raw VectorDatabaseQuery pixel result is noisy — real confirmed shape:

{"insightID": "...", "pixelReturn": [{"output": [
  {"Score": 0.52, "idx": 1, "Source": "java-reactor-patterns.txt", "Modality": "text",
   "Divider": "1", "Part": "1", "Tokens": "146", "Content": "...", 
   "Weighted_RRF_Score": 0.011, "BM25_Score": 0.21}
]}]}

_vector_helpers.extract_chunks()/render_chunks() strip this down to just Source (for citation) and Content (the actual text) — none of Score/idx/Modality/Divider/Part/Tokens/Weighted_RRF_Score/BM25_Score/the pixelReturn/insightID envelope are useful to the calling agent. Tested locally against the exact real shape above before shipping. Also confirmed empirically that Insight().run_pixel() returns a list wrapping that dict, not the dict directly — an earlier version of this code assumed a bare dict and crashed on a live call.

metaFilters does not work — corrected after shipping it as "verified"

An earlier version of this README claimed metaFilters was "verified via a real live tool call" to narrow results. That was wrong — the original test's queries happened to be worded closely enough to their intended topic that unfiltered semantic ranking returned the right file by coincidence, not because filtering was real. A disambiguating live test on 2026-07-15 (asking get_reactor_patterns, scoped to java-reactor-patterns.txt, a question clearly about a different uploaded file) returned that other file's content — proving metaFilters had zero effect. Root cause, confirmed in Semoss's real source: FaissDatabaseEngine.nearestNeighborCall() (src/prerna/engine/impl/vector/FaissDatabaseEngine.java:537-631) only reads parameters.containsKey("filters") — there is no metaFilters/METADATA_FILTERS_KEY branch anywhere in that method, and VectorDatabaseQueryReactor.getFilters() likely never even parses the {"Source": [...]} dict-shorthand into a valid filter to begin with (it requires a real PixelDataType.QUERY_STRUCT Pixel noun). Fix: topic scoping is now done client-side in _vector_helpers.query_vector() — request a larger unfiltered candidate pool, filter by real Source in Python, then truncate to the requested limit. Re-verified live with the same disambiguating query after the fix — correctly scoped. See public/md/engines/vector-engine.md's FAISS section for the Java-source side of this same finding, confirmed independently by a separate research pass before the two were cross-referenced.

Confirmed, not assumed: the vector engine's upload UI rejects .md files. Workaround: upload the .txt copies in vector-upload/ instead (same content, renamed extension — .md files stay .md in public/md/ for git/portal legibility). Real Source values are therefore *.txt, which SOURCE_FILES in _vector_helpers.py already reflects.

The running Python process caches modules — file edits need a restart

Confirmed twice (2026-07-15): editing _vector_helpers.py on disk does not take effect on the next MCP tool call. The project's persistent Python worker process keeps _vector_helpers in sys.modules in memory and doesn't reimport it. Clearing py/__pycache__ does not help — that's disk-level bytecode caching, a different layer from the in-memory module cache that's actually stale. After editing any .py file under py/, the project's Python process needs to be restarted before testing again — there is no known way to trigger this from outside the SEMOSS UI/instance itself.

Tests (tests/)

Pure-stdlib unittest, deliberately kept outside py/ — see "Why almost no logic lives in mcp_driver.py" above for why: anything under py/ at the top level of mcp_driver.py gets scanned as an MCP tool, so test files must never live there.

python3 -m unittest discover -s tests -v

• tests/test_vector_helpers.py — extract_chunks/extract_source_list against both the list-wrapped and bare-dict Insight().run_pixel() shapes, render_chunks, query_vector's client-side filtering/candidate-pool sizing/scaling-risk warning (run_pixel is mocked — no real network/pixel call), health_check's ready/missing/orphan reporting.

• tests/test_registry_consistency.py — the guard against the exact drift the TOPICS refactor above was meant to prevent: every TOPICS entry has a matching get_<key> tool function in mcp_driver.py and vice versa, no duplicate keys, and scripts/generate_mcp_driver.py --check reports zero drift against the files as currently committed.

Run this after any edit to TOPICS, mcp_driver.py, or the generator — before trusting a regeneration.

Content policy

Every claim and code excerpt in public/md/**/*.md must trace to a real file in a real SEMOSS/Monolith/semoss-ui repo, cited by path (line numbers where reasonable). Unverified assumptions get an explicit "Open / not yet verified" section in the file itself, not confident-sounding prose. An agent will trust whatever these files say — stale-but-confident documentation is worse than an honest, visible gap.

Uploading content into the vector engine

Manual — via SEMOSS's own UI against semoss-dev-vector (c3cbeb32-98ee-475a-8f8b-7f7a47a9a84d). Upload the .txt files in vector-upload/ (regenerate with cp public/md/<category>/<file>.md vector-upload/<file>.txt whenever content changes — that folder is gitignored, not committed). No ingestion script — this is an occasional, deliberate action.

Connecting an agent to THIS app's MCP endpoint

<base_url>/Monolith/api/ext/mcp/65eb25bb-c981-48c3-bcef-5a8dd6ae4391/comms

(Not the vector engine's endpoint — this app's. The tools above call the vector engine internally.) Get your access/secret keys from SEMOSS Settings -> My Profile before starting any of the steps below. Substitute your own <base_url> (e.g. http://localhost:9090 for local dev, or the real cfg-ai-dev HTTPS URL once deployed there) and <accessKey>/<secretKey> everywhere below.

Header format note, applies to all three clients below: SEMOSS's Authorization header is Bearer<accessKey>:<secretKey> — no space after Bearer, access/secret joined by a colon. This is SEMOSS-specific, not the generic OAuth Bearer <token> convention (verified against SEMOSS/vibe_setup_vscode's real working .vscode/mcp.json, and confirmed working live against this exact endpoint via both Copilot CLI and Claude Code CLI, below).

GitHub Copilot CLI — confirmed working live, 2026-07-15

1. Don't use the interactive /mcp add form for a remote HTTP server against this endpoint — its own connectivity probe can report "failed to connect" even when the endpoint is fine (confirmed via direct curl: the endpoint returns a real MCP initialize response; a plain GET correctly 405s, which may be what trips up the form's own preflight check).

2. Use the CLI subcommand instead:

   copilot mcp add --transport http semoss-dev-mcp \
     "<base_url>/Monolith/api/ext/mcp/65eb25bb-c981-48c3-bcef-5a8dd6ae4391/comms" \
     --header "Authorization: Bearer<accessKey>:<secretKey>"

3. Start a fresh copilot session — MCP servers load at session start, an already-running session won't pick up the new server.

4. Verify: ask the fresh session to call list_topics or health_check. A real response (not a connection error) confirms it worked.

5. To remove: copilot mcp remove semoss-dev-mcp.

Claude Code CLI — confirmed working live, 2026-07-15

claude mcp add --transport http semoss-dev-mcp \
  "<base_url>/Monolith/api/ext/mcp/65eb25bb-c981-48c3-bcef-5a8dd6ae4391/comms" \
  --header "Authorization: Bearer<accessKey>:<secretKey>"

Unlike Copilot CLI, this one's own health check is reliable — run claude mcp list or claude mcp get semoss-dev-mcp right after adding, and you should see Status: OK Connected immediately (confirmed live against this exact endpoint — no restart needed, no false "failed to connect"). To remove: claude mcp remove semoss-dev-mcp -s local.

VS Code (.vscode/mcp.json) — not yet live-tested

Following the same real, working pattern from SEMOSS/vibe_setup_vscode's .vscode/mcp.json (mcp-remote bridging stdio<->HTTP), but not confirmed against this specific endpoint the way the two CLIs above were:

{
  "servers": {
    "semoss_dev_mcp": {
      "type": "stdio",
      "command": "npx",
      "tools": ["*"],
      "args": [
        "-y",
        "mcp-remote",
        "<base_url>/Monolith/api/ext/mcp/65eb25bb-c981-48c3-bcef-5a8dd6ae4391/comms",
        "--header",
        "Authorization:Bearer<accessKey>:<secretKey>"
      ]
    }
  },
  "inputs": []
}

Save this as .vscode/mcp.json in whatever repo you're working in, reload the VS Code window, then check the MCP servers panel for a connected status.

Known gaps / not yet verified

• Claude Code CLI connection — confirmed (2026-07-15): claude mcp add/ claude mcp list both showed ✔ Connected against the real remote endpoint, same as Copilot CLI.

• Portal (client/) 404 — root cause identified and resolved, real browser test still pending. The 404 (http://localhost:9090/api/engine/runPixel — missing /Monolith) was never a bug in this app or in BrowseAppAssets: it was viewed via the wrong URL. Portals must be viewed at <applicationUrl>/public_home/<projectId>/ portals/ (see "Viewing the portal" above), which triggers PublicHomeCheckFilter's lazy Project.publish() call and injects the #semoss-env script tag Env.MODULE needs in a production build. BrowseAppAssets/raw asset paths never get that injection, so Env.MODULE stays "" and the SDK falls back to origin-relative URLs. No code change was needed — this app's App.tsx already matches veteran-portal/prompt-manager's real pattern. Still open: an actual browser load at the correct public_home URL against the live local instance, since that needs a logged-in browser session (the .env.local API keys only cover the SDK's own runPixel calls, not the filter's userCanViewProject browser-session check).

• Registry dedup — fixed (2026-07-15). _vector_helpers.py's TOPICS list is now the single source of truth; mcp_driver.py's per-topic tool functions, agents.md's topic list, and this README's tool list are all generated from it by scripts/generate_mcp_driver.py (BEGIN/END marker blocks in each file — don't hand-edit inside those markers). scripts/generate_mcp_driver.py --check exits non-zero on drift; tests/test_registry_consistency.py asserts zero drift and that every TOPICS entry has a matching tool function and vice versa. Run python3 -m unittest discover tests after any TOPICS edit, then python3 scripts/generate_mcp_driver.py to regenerate, then re-run the tests.

• Logging added but not confirmed visible anywhere. _vector_helpers.py now logs via a named logging.getLogger("semoss_dev_mcp") at key points (query built, chunk counts, under-filled topic-scoped results). Whether these records actually surface anywhere (SEMOSS server logs, stdout capture, or nowhere) inside this hosted Python worker process was not verified — best-effort addition, not a confirmed observability channel yet.

• Content topics currently cover: monolith-semoss-boundary, java-reactor-patterns, python-java-tcp-bridge, python-reactor-conventions, security-checklist, vector-engine, adding-new-engine-type, adding-new-model-type, semoss-ui-fe-be-boundary, testing-conventions, monolith/endpoint-groups, database-engines-overview, model-engines-overview, storage-engines-overview, function-engines-overview — 15 topics, 16 tools including health_check and search_all_architecture_docs. Not yet uploaded to semoss-dev-vector: the 5 newest (endpoint-groups, database/model/storage/function-engines-overview) — check with health_check() before assuming they're queryable.

History

• v1 (replaced): hand-rolled Python MCP driver with a pure-stdlib BM25 search tool — reinvented something SEMOSS's own vector engine already does better.

• v2 (replaced): assumed the vector engine's generic MCP tools were sufficient on their own with no app of our own — missed that generic, unscoped tools aren't discoverable/focused enough, and that a portal needs its own app to serve from.

• v3 (replaced): static single-file HTML portal (exact copy of Semoss Builder MCP's portal, minimal edits) — worked, but wasn't "a proper app."

• v4 (this version): real Vite + React client app for the portal; mcp_driver.py split so private helpers stop leaking into the MCP manifest as fake tools; tool output cleaned of ranking/bookkeeping noise; five real content topics; confirmed working end-to-end via Copilot CLI.