lyra-mcp

Typed pipeline-stage abstraction + MCP tool surface for RAG/agent pipelines — lets Claude read/run/modify individual pipeline stages (retrieve, rerank, generate, eval) as structured, typed calls instead of tracing and re-reading raw code each time.

Status: Phase 3 done, plus a wiqas.eval stage. The abstraction is validated against a second, differently-built project (Orion, a FastAPI service vs. WiQAS's script/CLI shape) with zero changes to any shared code — only new adapter files. A third stage type (wiqas.eval, wrapping RAGAS's answer_similarity metric) proves the same abstraction also extends to a stage with a real live-LLM dependency, not just local-inference stages like retrieval/reranking.

Running the MCP server

uv run lyra-mcp

Registered with this session's Claude Code project (.mcp.json, generated via claude mcp add):

claude mcp add lyra -s project -e WIQAS_REPO_ROOT="D:\GitHub\WiQAS" -e WIQAS_VENV_PYTHON="D:\GitHub\WiQAS\.venv\Scripts\python.exe" -- uv run --project D:\GitHub\lyra-mcp lyra-mcp

New MCP servers load at Claude Code session start — restart/reconnect the session for lyra's tools to appear.

Tools

• list_stages() — every stage in the loaded graph (graphs/wiqas_query_pipeline.json by default, override with LYRA_GRAPH_PATH): id, type, current config, and full config/input/output JSON schema. Enough to call the other tools without opening any adapter source file.

• get_stage_config(stage_id) / set_stage_config(stage_id, config_overrides) — read/persistently mutate a stage's baseline config (validated before committing).

• run_stage(stage_id, input, config_override=None) — run one stage. config_override merges over the baseline for that call only, without persisting — the way to compare configs on the same input (call repeatedly with a different override each time, nothing to reset).

• run_pipeline(external_inputs) — run the whole loaded graph end to end via pipelines/executor.py's topological-order executor.

Real bugs found building this (Phase 2)

Two contract mismatches only surfaced through live protocol testing, not unit tests — both are why the project's test-first-workflow rule treats "touching a real external system" as its own verification step:

• Client-visible errors were silently generic. The installed MCP SDK (mcp==2.1.1) only forwards a raised exception's own message to the client when it's the SDK's own ToolError — anything else (including Lyra's own LyraError hierarchy) becomes a bare "Error executing tool <name>", discarding every hand-written error message. Fixed with a _translate_lyra_errors decorator in server.py that re-raises any LyraError as a ToolError with the same text, at the MCP boundary only (core/errors.py stays independent of the SDK).

• Schema and actual data disagreed on field names. model_json_schema() defaults to by_alias=True; model_dump() defaults to by_alias=False. list_stages advertised SearchResultModel's aliased field as "id", but run_stage/run_pipeline were returning it as "document_id" — a real schema/data mismatch an agent trusting the advertised schema would hit immediately. Fixed by adding by_alias=True everywhere a StageOutput gets dumped for an external caller (pipelines/executor.py, server.py).

Also confirmed empirically (not assumed): a bare dict return-type annotation does not produce structured content from this SDK version — only a parameterized dict[str, Any] (or a real pydantic model) does; sync def tool functions are automatically offloaded to a worker thread (anyio.to_thread.run_sync), so no manual async/await was needed anywhere in server.py.

Related MCP server: AgentOps EvalBench MCP

Architecture: the subprocess boundary

WiQAS's internal code (from src.retrieval.retriever import WiQASRetriever, etc.) only imports cleanly inside WiQAS's own venv, because of its heavy ML dependencies (torch, chromadb, sentence-transformers). Rather than running Lyra itself inside WiQAS's venv — which would break down the moment a second, differently-built project (Orion, Phase 3) needs wrapping too — each Lyra stage invokes the target project's own interpreter as a subprocess, running a small bundled shim script that talks JSON over stdin/stdout.

Claude ──calls──> Lyra typed Stage ──subprocess──> target project's own venv
                   (stable interface,     runs a small runner shim script,
                    same every time)      talks JSON over stdin/stdout

Lyra itself stays a small, dependency-light project (pydantic only) — the framework (src/lyra/core/) has zero knowledge of WiQAS or Orion specifically, so the same run_subprocess_stage helper is what every adapter reuses unchanged.

A real gap this surfaced during implementation, worth knowing if you're wrapping a new project: don't trust a code investigation's reported function signatures for the exact wire shape — always confirm against the actual runtime output. Two mismatches turned up in WiQAS that weren't visible from reading signatures alone: SearchResult.to_dict() serializes its document_id attribute under the JSON key "id", and RerankerManager.rerank_search_results()'s output dicts omit search_type entirely. Both are handled in SearchResultModel (see its docstring in src/lyra/adapters/wiqas/common.py). Orion (Phase 3) turned out to have the identical mismatch, independently confirmed by reading its own source — not a coincidence worth assuming holds for a third project, but a real, recurring pattern worth checking for every time.

Also found live: WiQAS's own logging (a module-level rich.Console()) prints straight to stdout, which would have polluted the runner scripts' JSON-only stdout contract. Fixed on the Lyra side only (no WiQAS changes) with an OS-file-descriptor-level redirect — see src/lyra/adapters/_shared/shim_io.py. That helper was deliberately kept project-agnostic and outside any single adapter's folder specifically because this pattern was expected to recur — and it did: Orion has the exact same rich.Console()-to-stdout issue, fixed by reusing shim_io.py completely unchanged, zero new code needed.

Phase 3: validated against Orion, zero shared-code changes

Orion (D:\GitHub\Orion) is a FastAPI service with a Tauri desktop shell — a genuinely different shape from WiQAS's script/CLI style — wrapped the same way: adapters/orion/ mirrors adapters/wiqas/ file-for-file (common.py, retrieval.py, reranking.py, runners/), registered as two more STAGE_REGISTRY entries. No changes were needed to core/, adapters/_shared/shim_io.py, registry.py's shape, pipelines/executor.py, or pipelines/state.py — the entire cost of wrapping a second project was the new adapter files, exactly as the architecture was designed to allow. Live integration test (tests/integration/test_orion_live_pipeline.py) passed in ~21s against Orion's real ~232-chunk collection — dramatically faster than WiQAS's several minutes, since Orion's corpus is smaller and it does no query-decomposition/cross-lingual translation.

Eval stage: a stage type with a real live-LLM dependency

adapters/wiqas/eval.py wraps WiQAS's RAGAS answer_similarity metric (src/evaluation/[3] ragas/ragas_fixed.py) as wiqas.eval. Two things found by reading that code directly, not guessed:

• The file WiQAS's own docs point to as the RAGAS entrypoint (ragas_eval.py) is broken — it references json, argparse, evaluate, and the ragas metric objects without ever importing them, so it imports cleanly but raises NameError at call time. ragas_fixed.py is the working one.

• A real, hard live dependency, unlike retrieval/reranking. WiQAS's own setup_ollama_for_ragas() makes an unconditional chat-completion call to Ollama to test connectivity — even in answer_similarity-only mode, where that chat model is never actually used for scoring (answer_similarity is embeddings-only). Can't be skipped without modifying WiQAS's own code, which this adapter never does. A running Ollama server with the configured chat model (default mistral:latest) and nomic-embed-text both pulled is a hard precondition.

Scoped narrowly and deliberately: answer_similarity only (use_all_metrics=False, WiQAS's own default) — its own code comment calls this "the only one that works reliably," and warns the full 6-metric mode "may timeout." use_all_metrics=True is exposed as a config option but untested here. Live integration test (tests/integration/test_wiqas_eval_live.py) confirmed a real, discriminating score (0.697 similarity between a Filipino ground-truth answer and a differently-phrased English paraphrase) in ~12s wall time — far faster than WiQAS's own "this may take several minutes" comment, which describes the full-metric path this stage doesn't use. Not wired into wiqas_query_pipeline.json: eval needs a generated answer as input, which no existing stage produces (no generate stage exists yet) — registered in STAGE_REGISTRY alone, same precedent as retrieval/reranking being proven directly before Phase 2's MCP layer existed.

Setup

Requires uv and Python 3.12.

uv venv --python 3.12
uv sync

.env.example documents the environment variables Lyra reads if your WiQAS/Orion checkouts live somewhere other than their defaults. It's a template, not auto-loaded — nothing in this codebase reads .env — so set these as real environment variables in your shell/session:

WIQAS_REPO_ROOT=D:\GitHub\WiQAS
WIQAS_VENV_PYTHON=D:\GitHub\WiQAS\.venv\Scripts\python.exe
ORION_REPO_ROOT=D:\GitHub\Orion
ORION_VENV_PYTHON=D:\GitHub\Orion\.venv\Scripts\python.exe

Running on a different machine

Everything machine-specific is env vars plus one generated config file — nothing is hand-edited:

1. uv venv --python 3.12 && uv sync (see Setup above).

2. Set WIQAS_REPO_ROOT/WIQAS_VENV_PYTHON and/or ORION_REPO_ROOT/ORION_VENV_PYTHON as real environment variables for whichever projects you're wrapping on this machine (see .env.example).

3. Regenerate .mcp.json — it's gitignored on purpose (bakes in this machine's absolute uv.exe path and repo checkout paths, not portable as-is; see .mcp.json.example for the shape). Don't hand-copy the example — run:

   claude mcp add lyra -s project -e WIQAS_REPO_ROOT="<path>" -e WIQAS_VENV_PYTHON="<path>" -e ORION_REPO_ROOT="<path>" -e ORION_VENV_PYTHON="<path>" -- uv run --project <path to this checkout> lyra-mcp

   Restart/reconnect the Claude Code session afterward — new MCP servers load at session start.

If a path is wrong, the failure is a clear, actionable SubprocessLaunchError: Interpreter not found: <path> (see core/subprocess_adapter.py) — not a silent hang or a cryptic import error.

Running tests

# Unit tests (default) — fast, subprocess boundary mocked, no WiQAS/Orion needed
uv run pytest

# Integration tests (opt-in) — need a real WiQAS/Orion venv + an already-ingested
# Chroma collection at WIQAS_REPO_ROOT / ORION_REPO_ROOT respectively
uv run pytest -m integration -s

Layout

src/lyra/
  core/            # StageConfig/StageInput/StageOutput, Stage ABC, the shared subprocess-boundary helper
  registry.py      # stage_type -> concrete classes lookup, all wrapped projects register here
  server.py        # the MCP server (list_stages/get_stage_config/set_stage_config/run_stage/run_pipeline)
  mcp_models.py    # tool-output-only models (StageInfo, RunPipelineResult)
  adapters/
    _shared/       # project-agnostic runner-script helpers (e.g. the stdout-redirect fix), reused by every adapter
    wiqas/         # WiQAS-specific typed stages (retrieval, reranking, eval) + their runner shim scripts
    orion/         # Orion-specific typed stages, same shape as wiqas/ — proves the pattern generalizes
  pipelines/       # PipelineGraph (load/validate), executor.py (topological-order execution), state.py (mutable loaded-graph wrapper)
  graphs/          # Example pipeline-as-data graphs, one per wrapped project
tests/
  unit/           # subprocess boundary mocked, run by default
  integration/    # opt-in (`pytest -m integration`), needs real WiQAS/Orion