# semoss-dev-mcp

**This app IS the MCP server.** It's the real SEMOSS project `Semoss Dev MCP`
(`65eb25bb-c981-48c3-bcef-5a8dd6ae4391`), symlinked to this repo (see "Local setup"
below) so content and code here are edited once and reflected in the running project
with no copy/paste. Its tools call a separate real vector engine (`semoss-dev-vector`,
`c3cbeb32-98ee-475a-8f8b-7f7a47a9a84d`) internally, scoped per-tool to specific source
files, so an agent gets a small set of purpose-named, focused tools instead of one
generic "search everything" tool it has to guess how to aim. It also serves a browsable
portal (a real Vite + React app) so a person can read the same content in a web page.

**Confirmed working end-to-end** (2026-07-15): connected via Copilot CLI, called
`get_reactor_patterns` for real, got real ranked results back from `semoss-dev-vector`.

## What's here

```
semoss-dev-mcp/
  public/md/                       content — must be under public/ (see below)
    agents.md                       documentation map — read this one first
    architecture/
      monolith-semoss-boundary.md
    java/
      java-reactor-patterns.md
    python/
      python-java-tcp-bridge.md
      python-reactor-conventions.md
    security/
      security-checklist.md
  vector-upload/                   .txt copies for uploading into semoss-dev-vector
                                     (gitignored — regenerate with `cp x.md x.txt`)
  py/
    mcp_driver.py                  the MCP tools only — see "Why almost no logic
                                     lives in mcp_driver.py" below
    _vector_helpers.py             all real logic (pixel building, running, result
                                     formatting) — deliberately not in mcp_driver.py
  mcp/
    py_mcp.json                     manifest for mcp_driver.py's tools — MACHINE-
                                     GENERATED by SEMOSS's own MakePythonMCP, confirmed
                                     regenerating live on this exact repo (2026-07-15,
                                     picked up a docstring edit unprompted). Don't
                                     hand-edit; edit mcp_driver.py's docstrings instead.
    pixel_mcp.json                  BrowseAppAssets/GetAppAssets, wrapping real
                                     existing Semoss reactors (no custom code)
  client/                          real Vite + React + TypeScript app (doc portal)
    src/
      hooks/useDocs.ts               BrowseAppAssets/GetAppAssets via @semoss/sdk-react
      components/DocViewer.tsx       sidebar + markdown render (marked/dompurify/highlight.js)
  portals/                         `pnpm build` output — committed, not hand-edited
  package.json / vite.config.ts / tsconfig.json / index.html
  README.md
```

## Why almost no logic lives in `mcp_driver.py`

**Confirmed empirically** (2026-07-15): SEMOSS's own MCP-manifest auto-generator
re-scans `py/mcp_driver.py` and turns *every* top-level function into an MCP tool —
including private, underscore-prefixed helpers that were never meant to be tools. It
does not filter by leading underscore or by presence/absence of `@mcp_metadata`.
Fix: all real logic (`build_query_pixel`, `run_pixel`, `format_results`,
`query_vector`, the `SOURCE_FILES` registry) lives in `_vector_helpers.py`, a plain
module `mcp_driver.py` imports from. Only the six real tool functions exist as
top-level names in `mcp_driver.py` itself. **If you add a helper function, put it in
`_vector_helpers.py`, never directly in `mcp_driver.py`.**

## Why `public/`

Non-editor callers (which is what most MCP access/secret-key callers will be, from this
project's own perspective) are restricted to the `public/` folder by
`BrowseAppAssetsReactor`/`GetAppAssetsReactor` — confirmed in Semoss's own source:

```java
// src/prerna/util/FileSystemUtil.java:412-426
public static boolean restrictBrowseToPublicFolder(boolean canEdit, String relativeFilePath) {
    if (canEdit) {
        return false;
    }
    String p = stripAssetPathSlashes(relativeFilePath);
    if (p.isEmpty()) {
        return true; // view-only user at the assets root: only show the public folder node
    }
    if (p.equals(Constants.PUBLIC_ASSETS_FOLDER) || p.startsWith(Constants.PUBLIC_ASSETS_FOLDER + "/")) {
        return false;
    }
    throw new IllegalArgumentException("User only has read access to the '"
            + Constants.PUBLIC_ASSETS_FOLDER + "' folder within the assets folder.");
}
```

## Local setup (already done for this checkout — documented for reproducing elsewhere)

The real project's assets directory is symlinked to this repo, per-folder, so editing
either side is the same file:

```bash
ASSETS="<path to>/Semoss Dev MCP__65eb25bb-c981-48c3-bcef-5a8dd6ae4391/app_root/version/assets"
REPO="<path to>/semoss-dev-mcp"

ln -s "$REPO/public" "$ASSETS/public"
ln -s "$REPO/mcp" "$ASSETS/mcp"
ln -s "$REPO/py" "$ASSETS/py"
rm -rf "$ASSETS/portals"
ln -s "$REPO/portals" "$ASSETS/portals"   # symlinks the whole build-output dir
```

`.notebooks/` and `.admin/` in the assets folder are real SEMOSS scaffold — left alone,
not symlinked. Adding a new file under `public/md/` or editing `py/`/`mcp/` needs no
new symlink — those are whole-directory symlinks, so new files inside them are picked
up automatically.

## The client app (portal)

Real Vite + React + TypeScript, modeled on `vba-futures/apps/prompt-manager`'s real
build setup (package.json/vite.config.ts shape), minus its `@vba-futures/*`/Tailwind
workspace deps since this is a standalone repo. Build output goes to `portals/`
(`vite.config.ts`'s `outDir`), which is what's symlinked into the SEMOSS project above
— **`portals/` is build output, not something to hand-edit.**

```bash
pnpm install
pnpm run build       # writes portals/ — commit the result, matching the real
                      # vba-futures convention of committing built portal output
pnpm run dev         # local dev server against a real instance — needs .env.local,
                      # copy from .env.example
```

Type-checked (`npx tsc --noEmit`) and build-verified as of 2026-07-15.

### Viewing the portal — use the `public_home` URL, not `BrowseAppAssets`

**Confirmed (2026-07-15) via reading the real serving path in both repos** — the portal
must be viewed at:

```
<applicationUrl>/public_home/<projectId>/portals/
# e.g. locally: http://localhost:9090/Monolith/public_home/65eb25bb-c981-48c3-bcef-5a8dd6ae4391/portals/
```

not via `BrowseAppAssets`/a raw asset path. Browsing raw assets serves `portals/index.html`
byte-for-byte as committed — with no `#semoss-env` script tag — so `Env.MODULE` stays `""`
in the production build (this app's `App.tsx`, like `vba-futures/apps/veteran-portal` and
`apps/prompt-manager`, only calls `Env.update()` in `import.meta.env.DEV`; it relies on the
platform to inject the rest in production). With `Env.MODULE` empty, `@semoss/sdk` builds
request URLs relative to the current origin with no `/Monolith` prefix — hence the
`http://localhost:9090/api/engine/runPixel` 404 seen instead of the correct
`http://localhost:9090/Monolith/api/engine/runPixel`.

The `public_home` URL is handled by a dedicated servlet filter
(`Monolith/src/prerna/web/conf/PublicHomeCheckFilter.java:110-124`), which — on every
request under that path — calls `project.requirePublish(false)` and, if true, lazily calls
`project.publish(...)` before serving the file. `Project.publish()`
(`Semoss/src/prerna/project/impl/Project.java:1147-1211`) calls `rewritePortalIndexHtml()`
(same file, `:1317-1349`), which rewrites `portals/index.html` **in place** to inject:

```html
<script id="semoss-env" type="application/json">{"APP":"<projectId>","MODULE":"<module>"}</script>
```

right before the first child of `<head>` — `<module>` here comes from
`Utility.getApplicationRouteAndContextPath()`, which is `/Monolith` for this local setup.
`@semoss/sdk`'s `InsightProvider` (`libs/sdk/src/stores/insight/insight.store.ts:301-321` in
`semoss-ui`) reads that same `#semoss-env` tag and calls `Env.update({APP, MODULE})` from it
— this is the only thing that sets `Env.MODULE` correctly in a production build; no app-level
code change is needed here, since this repo's `App.tsx` already matches the real
`veteran-portal`/`prompt-manager` pattern exactly.

You do not need to explicitly call the `PublishProject` pixel first — just requesting the
`public_home` URL triggers the lazy publish. (`PublishProject(project=[...], release=[...])`,
`Semoss/src/prerna/reactor/project/PublishProjectReactor.java:50-80`, exists for explicitly
forcing a republish/release and returns this same URL — useful after editing `portals/`
without waiting for the lazy check, but not required for normal viewing.) Note the filter also
enforces `userCanViewProject` for non-global projects, so the browser needs an active,
logged-in SEMOSS session (not just the `.env.local` API keys, which only cover the SDK's own
`runPixel` calls).

## The tools (`py/mcp_driver.py`, logic in `py/_vector_helpers.py`)

<!-- BEGIN GENERATED TOOL LIST (scripts/generate_mcp_driver.py) -->
- **`health_check()`** — cross-checks the topic registry against what's actually uploaded in the vector engine right now.
- **`list_topics()`** — lists which topic-scoped tools exist. No vector call — pure local lookup.
- **`get_reactor_patterns(query, limit=5)`** — scoped to `java-reactor-patterns.txt`.
- **`get_monolith_semoss_boundary(query, limit=5)`** — scoped to `monolith-semoss-boundary.txt`.
- **`get_python_tcp_bridge(query, limit=5)`** — scoped to `python-java-tcp-bridge.txt`.
- **`get_python_reactor_conventions(query, limit=5)`** — scoped to `python-reactor-conventions.txt`.
- **`get_security_checklist(query, limit=5)`** — scoped to `security-checklist.txt`.
- **`get_vector_engine(query, limit=5)`** — scoped to `vector-engine.txt`.
- **`get_adding_new_engine_type(query, limit=5)`** — scoped to `adding-new-engine-type.txt`.
- **`get_adding_new_model_type(query, limit=5)`** — scoped to `adding-new-model-type.txt`.
- **`get_semoss_ui_fe_be_boundary(query, limit=5)`** — scoped to `semoss-ui-fe-be-boundary.txt`.
- **`get_testing_conventions(query, limit=5)`** — scoped to `testing-conventions.txt`.
- **`get_monolith_endpoint_groups(query, limit=5)`** — scoped to `endpoint-groups.txt`.
- **`get_database_engines_overview(query, limit=5)`** — scoped to `database-engines-overview.txt`.
- **`get_model_engines_overview(query, limit=5)`** — scoped to `model-engines-overview.txt`.
- **`get_storage_engines_overview(query, limit=5)`** — scoped to `storage-engines-overview.txt`.
- **`get_function_engines_overview(query, limit=5)`** — scoped to `function-engines-overview.txt`.
- **`search_all_architecture_docs(query, limit=5)`** — no source filter, the fallback when no topic-specific tool fits.
<!-- END GENERATED TOOL LIST -->

Each returns ranked `[source] content` excerpts only — **confirmed and fixed** (see
"Output cleanup" below), no ranking-score/bookkeeping noise. Each new topic guide added
under `public/md/<category>/` should get its own scoped tool in `mcp_driver.py` +
`SOURCE_FILES` entry in `_vector_helpers.py`, following this same pattern, rather than
making `search_all_architecture_docs` do all the work.

### Output cleanup (confirmed via a real live tool call, 2026-07-15)

The raw `VectorDatabaseQuery` pixel result is noisy — real confirmed shape:

```json
{"insightID": "...", "pixelReturn": [{"output": [
  {"Score": 0.52, "idx": 1, "Source": "java-reactor-patterns.txt", "Modality": "text",
   "Divider": "1", "Part": "1", "Tokens": "146", "Content": "...", 
   "Weighted_RRF_Score": 0.011, "BM25_Score": 0.21}
]}]}
```

`_vector_helpers.extract_chunks()`/`render_chunks()` strip this down to just `Source`
(for citation) and `Content` (the actual text) — none of
`Score`/`idx`/`Modality`/`Divider`/`Part`/`Tokens`/`Weighted_RRF_Score`/`BM25_Score`/the
`pixelReturn`/`insightID` envelope are useful to the calling agent. Tested locally
against the exact real shape above before shipping. Also confirmed empirically that
`Insight().run_pixel()` returns a **list** wrapping that dict, not the dict directly —
an earlier version of this code assumed a bare dict and crashed on a live call.

### `metaFilters` does not work — corrected after shipping it as "verified"

An earlier version of this README claimed `metaFilters` was "verified via a real live
tool call" to narrow results. **That was wrong** — the original test's queries happened
to be worded closely enough to their intended topic that unfiltered semantic ranking
returned the right file by coincidence, not because filtering was real. A disambiguating
live test on 2026-07-15 (asking `get_reactor_patterns`, scoped to
`java-reactor-patterns.txt`, a question clearly about a *different* uploaded file)
returned that other file's content — proving `metaFilters` had zero effect. Root cause,
confirmed in Semoss's real source: `FaissDatabaseEngine.nearestNeighborCall()`
(`src/prerna/engine/impl/vector/FaissDatabaseEngine.java:537-631`) only reads
`parameters.containsKey("filters")` — there is no `metaFilters`/`METADATA_FILTERS_KEY`
branch anywhere in that method, and `VectorDatabaseQueryReactor.getFilters()` likely
never even parses the `{"Source": [...]}` dict-shorthand into a valid filter to begin
with (it requires a real `PixelDataType.QUERY_STRUCT` Pixel noun). **Fix**: topic
scoping is now done client-side in `_vector_helpers.query_vector()` — request a larger
unfiltered candidate pool, filter by real `Source` in Python, then truncate to the
requested limit. Re-verified live with the same disambiguating query after the fix —
correctly scoped. See `public/md/engines/vector-engine.md`'s FAISS section for the
Java-source side of this same finding, confirmed independently by a separate research
pass before the two were cross-referenced.

**Confirmed, not assumed**: the vector engine's upload UI rejects `.md` files.
Workaround: upload the `.txt` copies in `vector-upload/` instead (same content, renamed
extension — `.md` files stay `.md` in `public/md/` for git/portal legibility). Real
`Source` values are therefore `*.txt`, which `SOURCE_FILES` in `_vector_helpers.py`
already reflects.

### The running Python process caches modules — file edits need a restart

Confirmed twice (2026-07-15): editing `_vector_helpers.py` on disk does **not** take
effect on the next MCP tool call. The project's persistent Python worker process keeps
`_vector_helpers` in `sys.modules` in memory and doesn't reimport it. Clearing
`py/__pycache__` does not help — that's disk-level bytecode caching, a different layer
from the in-memory module cache that's actually stale. **After editing any `.py` file
under `py/`, the project's Python process needs to be restarted before testing again**
— there is no known way to trigger this from outside the SEMOSS UI/instance itself.

## Tests (`tests/`)

Pure-stdlib `unittest`, deliberately kept **outside** `py/` — see
"Why almost no logic lives in `mcp_driver.py`" above for why: anything under `py/` at
the top level of `mcp_driver.py` gets scanned as an MCP tool, so test files must never
live there.

```bash
python3 -m unittest discover -s tests -v
```

- `tests/test_vector_helpers.py` — `extract_chunks`/`extract_source_list` against both
  the list-wrapped and bare-dict `Insight().run_pixel()` shapes, `render_chunks`,
  `query_vector`'s client-side filtering/candidate-pool sizing/scaling-risk warning
  (`run_pixel` is mocked — no real network/pixel call), `health_check`'s
  ready/missing/orphan reporting.
- `tests/test_registry_consistency.py` — the guard against the exact drift the `TOPICS`
  refactor above was meant to prevent: every `TOPICS` entry has a matching `get_<key>`
  tool function in `mcp_driver.py` and vice versa, no duplicate keys, and
  `scripts/generate_mcp_driver.py --check` reports zero drift against the files as
  currently committed.

Run this after any edit to `TOPICS`, `mcp_driver.py`, or the generator — before trusting
a regeneration.

## Content policy

Every claim and code excerpt in `public/md/**/*.md` must trace to a real file in a real
SEMOSS/Monolith/semoss-ui repo, cited by path (line numbers where reasonable). Unverified
assumptions get an explicit "Open / not yet verified" section in the file itself, not
confident-sounding prose. An agent will trust whatever these files say — stale-but-confident
documentation is worse than an honest, visible gap.

## Uploading content into the vector engine

Manual — via SEMOSS's own UI against `semoss-dev-vector`
(`c3cbeb32-98ee-475a-8f8b-7f7a47a9a84d`). Upload the `.txt` files in `vector-upload/`
(regenerate with `cp public/md/<category>/<file>.md vector-upload/<file>.txt` whenever
content changes — that folder is gitignored, not committed). No ingestion script — this
is an occasional, deliberate action.

## Connecting an agent to THIS app's MCP endpoint

```
<base_url>/Monolith/api/ext/mcp/65eb25bb-c981-48c3-bcef-5a8dd6ae4391/comms
```

(Not the vector engine's endpoint — this app's. The tools above call the vector engine
internally.) Get your access/secret keys from SEMOSS Settings -> My Profile before
starting any of the steps below. Substitute your own `<base_url>` (e.g.
`http://localhost:9090` for local dev, or the real `cfg-ai-dev` HTTPS URL once deployed
there) and `<accessKey>`/`<secretKey>` everywhere below.

**Header format note, applies to all three clients below**: SEMOSS's `Authorization`
header is `Bearer<accessKey>:<secretKey>` — **no space** after `Bearer`, access/secret
joined by a colon. This is SEMOSS-specific, not the generic OAuth `Bearer <token>`
convention (verified against `SEMOSS/vibe_setup_vscode`'s real working
`.vscode/mcp.json`, and confirmed working live against this exact endpoint via both
Copilot CLI and Claude Code CLI, below).

### GitHub Copilot CLI — confirmed working live, 2026-07-15

1. **Don't use the interactive `/mcp add` form** for a remote HTTP server against this
   endpoint — its own connectivity probe can report "failed to connect" even when the
   endpoint is fine (confirmed via direct `curl`: the endpoint returns a real MCP
   `initialize` response; a plain `GET` correctly 405s, which may be what trips up the
   form's own preflight check).
2. **Use the CLI subcommand instead**:
   ```bash
   copilot mcp add --transport http semoss-dev-mcp \
     "<base_url>/Monolith/api/ext/mcp/65eb25bb-c981-48c3-bcef-5a8dd6ae4391/comms" \
     --header "Authorization: Bearer<accessKey>:<secretKey>"
   ```
3. **Start a fresh `copilot` session** — MCP servers load at session start, an
   already-running session won't pick up the new server.
4. **Verify**: ask the fresh session to call `list_topics` or `health_check`. A real
   response (not a connection error) confirms it worked.
5. **To remove**: `copilot mcp remove semoss-dev-mcp`.

### Claude Code CLI — confirmed working live, 2026-07-15

```bash
claude mcp add --transport http semoss-dev-mcp \
  "<base_url>/Monolith/api/ext/mcp/65eb25bb-c981-48c3-bcef-5a8dd6ae4391/comms" \
  --header "Authorization: Bearer<accessKey>:<secretKey>"
```

Unlike Copilot CLI, this one's own health check is reliable — run `claude mcp list` or
`claude mcp get semoss-dev-mcp` right after adding, and you should see
`Status: OK Connected` immediately (confirmed live against this exact endpoint — no
restart needed, no false "failed to connect"). To remove: `claude mcp remove
semoss-dev-mcp -s local`.

### VS Code (`.vscode/mcp.json`) — not yet live-tested

Following the same real, working pattern from `SEMOSS/vibe_setup_vscode`'s
`.vscode/mcp.json` (`mcp-remote` bridging stdio<->HTTP), but not confirmed against this
specific endpoint the way the two CLIs above were:

```json
{
  "servers": {
    "semoss_dev_mcp": {
      "type": "stdio",
      "command": "npx",
      "tools": ["*"],
      "args": [
        "-y",
        "mcp-remote",
        "<base_url>/Monolith/api/ext/mcp/65eb25bb-c981-48c3-bcef-5a8dd6ae4391/comms",
        "--header",
        "Authorization:Bearer<accessKey>:<secretKey>"
      ]
    }
  },
  "inputs": []
}
```

Save this as `.vscode/mcp.json` in whatever repo you're working in, reload the VS Code
window, then check the MCP servers panel for a connected status.

## Known gaps / not yet verified

- **Claude Code CLI connection** — confirmed (2026-07-15): `claude mcp add`/
  `claude mcp list` both showed `✔ Connected` against the real remote endpoint, same as
  Copilot CLI.
- **Portal (`client/`) 404 — root cause identified and resolved, real browser test
  still pending.** The 404 (`http://localhost:9090/api/engine/runPixel` — missing
  `/Monolith`) was never a bug in this app or in `BrowseAppAssets`: it was viewed via
  the wrong URL. Portals must be viewed at `<applicationUrl>/public_home/<projectId>/
  portals/` (see "Viewing the portal" above), which triggers
  `PublicHomeCheckFilter`'s lazy `Project.publish()` call and injects the
  `#semoss-env` script tag `Env.MODULE` needs in a production build.
  `BrowseAppAssets`/raw asset paths never get that injection, so `Env.MODULE` stays
  `""` and the SDK falls back to origin-relative URLs. No code change was needed —
  this app's `App.tsx` already matches `veteran-portal`/`prompt-manager`'s real
  pattern. Still open: an actual browser load at the correct `public_home` URL against
  the live local instance, since that needs a logged-in browser session (the
  `.env.local` API keys only cover the SDK's own `runPixel` calls, not the filter's
  `userCanViewProject` browser-session check).
- **Registry dedup — fixed (2026-07-15).** `_vector_helpers.py`'s `TOPICS` list is now
  the single source of truth; `mcp_driver.py`'s per-topic tool functions, `agents.md`'s
  topic list, and this README's tool list are all generated from it by
  `scripts/generate_mcp_driver.py` (BEGIN/END marker blocks in each file — don't hand-edit
  inside those markers). `scripts/generate_mcp_driver.py --check` exits non-zero on
  drift; `tests/test_registry_consistency.py` asserts zero drift and that every
  `TOPICS` entry has a matching tool function and vice versa. Run
  `python3 -m unittest discover tests` after any `TOPICS` edit, then
  `python3 scripts/generate_mcp_driver.py` to regenerate, then re-run the tests.
- **Logging added but not confirmed visible anywhere.** `_vector_helpers.py` now logs
  via a named `logging.getLogger("semoss_dev_mcp")` at key points (query built, chunk
  counts, under-filled topic-scoped results). Whether these records actually surface
  anywhere (SEMOSS server logs, stdout capture, or nowhere) inside this hosted Python
  worker process was not verified — best-effort addition, not a confirmed
  observability channel yet.
- Content topics currently cover: `monolith-semoss-boundary`, `java-reactor-patterns`,
  `python-java-tcp-bridge`, `python-reactor-conventions`, `security-checklist`,
  `vector-engine`, `adding-new-engine-type`, `adding-new-model-type`,
  `semoss-ui-fe-be-boundary`, `testing-conventions`, `monolith/endpoint-groups`,
  `database-engines-overview`, `model-engines-overview`, `storage-engines-overview`,
  `function-engines-overview` — 15 topics, 16 tools including `health_check` and
  `search_all_architecture_docs`. Not yet uploaded to `semoss-dev-vector`: the 5
  newest (`endpoint-groups`, `database/model/storage/function-engines-overview`) —
  check with `health_check()` before assuming they're queryable.

## History

- **v1** (replaced): hand-rolled Python MCP driver with a pure-stdlib BM25 search tool
  — reinvented something SEMOSS's own vector engine already does better.
- **v2** (replaced): assumed the vector engine's generic MCP tools were sufficient on
  their own with no app of our own — missed that generic, unscoped tools aren't
  discoverable/focused enough, and that a portal needs its own app to serve from.
- **v3** (replaced): static single-file HTML portal (exact copy of `Semoss Builder MCP`'s
  portal, minimal edits) — worked, but wasn't "a proper app."
- **v4** (this version): real Vite + React client app for the portal; `mcp_driver.py`
  split so private helpers stop leaking into the MCP manifest as fake tools; tool
  output cleaned of ranking/bookkeeping noise; five real content topics; confirmed
  working end-to-end via Copilot CLI.