phenoforge

Semantic value set assembly for clinical cohort definitions, over MCP.

Status: early prototype. The vocabulary loader, hierarchy expansion, hybrid (BM25 + dense) search, curated OHDSI Phenotype Library matching, an eval harness scoring all of it against curated ground truth, a thin MCP server exposing them all, and a LangGraph agent that decomposes a population description, checks curated first, and pauses for human confirmation before including anything generated — all work end-to-end against a real Athena download. See Roadmap.

Try it now, no setup

git clone https://github.com/colbyw5/phenoforge && cd phenoforge
uv run python scripts/run_demo.py "diabetic nephropathy"   # or: make demo

Runs real hybrid search and hierarchy expansion against a small, bundled, redistributable ICD-10-CM-only demo vocabulary (see demo/README.md) — no Athena account, no API key, no model download. It only exercises the generated tier (BM25 + hierarchy expansion); the curated tier, dense retrieval, and the full agent need the real Setup below.

Related MCP server: OMOP MCP Server

Setup

Published to PyPI as py-phenoforge (the bare phenoforge name is taken by an unrelated package) — pip install py-phenoforge or uv add py-phenoforge gets you the library and the phenoforge-mcp CLI, importable as phenoforge. That alone isn't runnable yet, though: every tool needs the vocabulary database built below from your own Athena download, which can't be redistributed on PyPI.

Requires your own OHDSI Athena bulk download — vocabulary content carries its own license terms, so it can't be bundled with this repo.

1. Create a free account at athena.ohdsi.org.

2. Under Search, select at least ICD10CM and SNOMED as vocabularies, then click Download. SNOMED is used locally only — to resolve curated-set matching (find_curated_definition) — and is never shipped or exposed through any tool surface.

3. Unzip the download into data/athena/ (this directory is gitignored — nothing under data/ is ever committed or shipped). It contains OMOP CDM vocabulary tables (CONCEPT.csv, CONCEPT_RELATIONSHIP.csv, etc.) — Athena ships vocabulary content pre-shaped as OMOP, so no separate mapping step is needed.

uv sync
python scripts/load_vocab.py data/athena --output data/vocab.duckdb
phenoforge-mcp  # stdio MCP server; add to Claude Desktop's mcpServers config to try it

Or, once the Athena download is in place, make setup chains install + every optional data-build step below (vocab, phenotype library, dense index) in one call. make check runs lint, type checks, and the full test suite — the same checks CI runs on every push. See the Makefile for the full target list.

find_curated_definition needs a small local cache of OHDSI Phenotype Library cohort definitions. Like the Athena download, this content carries its own terms — the OHDSI PhenotypeLibrary GitHub repo has no confirmed LICENSE file (its R package DESCRIPTION claims Apache, but that's a manifest claim, not a verified grant for the cohort content itself) — so it's fetched into a gitignored local directory, never committed to this repo. Without this step, find_curated_definition still runs and reports why nothing matched.

python scripts/fetch_phenotype_library.py  # writes data/phenotype_library/

Bundles 23 hand-picked cohorts (not the full ~1,100-cohort library), spanning: diabetes (type 2/type 1/gestational, ketoacidosis, retinopathy), chronic kidney disease, cardiovascular (MI, atrial fibrillation, heart failure), respiratory (pneumonia, asthma), mental health (depression, anxiety), rheumatoid arthritis, neurological (epilepsy, migraine), sepsis, inflammatory bowel disease, cirrhosis, hyperlipidemia, and obesity. See scripts/fetch_phenotype_library.py for exact cohort ids and selection rationale.

search_concepts fuses lexical and semantic matching when a dense index has been built; without one, it falls back to lexical-only search automatically.

python scripts/build_index.py  # writes data/concept_index.lance; downloads BioLORD-2023 on first run

Scores each retrieval method (BM25, dense, hybrid, hierarchy expansion) against the curated demo cohorts as ground truth — hierarchical distance-weighted scoring, set-level coverage, and an over-inclusion penalty (partial credit for near-misses under the same hierarchy parent, not exact-match recall). Requires the phenotype library fetch step above; dense/hybrid scoring also needs the built index.

python scripts/run_eval.py                                 # bm25 + expand_descendants
python scripts/run_eval.py --index data/concept_index.lance # + dense + hybrid

Separately, scripts/eval_decomposition.py scores the agent's decompose step itself (needs ANTHROPIC_API_KEY and the agent extra — see Validation and limitations below):

python scripts/eval_decomposition.py --index data/concept_index.lance

The LangGraph agent decomposes a plain-English population description into seed terms, checks each against the curated phenotype library first, and pauses on the command line for you to accept or reject any generated (unverified) candidates before they're included. Requires an Anthropic API key (decomposition is a real Claude call) and the agent extra. scripts/run_agent.py loads a local .env automatically (gitignored — never commit real keys), or export the variable directly.

uv sync --extra agent
echo 'ANTHROPIC_API_KEY=...' > .env   # or: export ANTHROPIC_API_KEY=...
python scripts/run_agent.py "adults with type 2 diabetes and diabetic nephropathy"
python scripts/run_agent.py "..." --index data/concept_index.lance  # + dense retrieval for generated candidates

Hosted at colbyw5.github.io/phenoforge, rebuilt on every push to main. To build locally instead:

make docs  # writes docs/_build/index.html

Both notebooks are exploration only, never pushed to production — reusable logic stays in src/phenoforge/.

• notebooks/explore.ipynb calls the engine directly (no MCP transport) against your real built data/vocab.duckdb.

• notebooks/evaluate.ipynb runs the eval harness and walks through the metrics with explanatory text, a method-comparison chart, and a sortable per-cohort results table.

uv sync --extra dev
jupyter lab notebooks/explore.ipynb

What it does

Turns a plain-English patient population description into a defensible set of ICD-10-CM codes, where every code carries provenance — whether it came from a peer-reviewed phenotype definition, from hierarchy expansion, or from semantic retrieval that a human should check.

"adults with type 2 diabetes and diabetic nephropathy"
  → decomposes into seed clinical terms
  → checks OHDSI Phenotype Library for a validated definition for each
  → falls back to hybrid retrieval for terms with no curated match,
    pausing for human confirmation before including anything generated
  → returns a ConceptSet with per-code provenance and citations

Run it for real: python scripts/run_agent.py "adults with type 2 diabetes and diabetic nephropathy" (see Setup).

Why not an existing terminology server

Several good MCP terminology servers exist. They solve lookup — "what is code X", "map X to Y". This solves set assembly, which is the actual task in cohort definition. It also covers US ICD-10-CM, which the existing servers do not, and uses semantic retrieval rather than proxied keyword search, which fails when a population description and a code description share no vocabulary.

Architecture

Three layers — a retrieval engine, a thin MCP server, and a LangGraph agent. The MCP server and the agent are independent consumers of the same engine.

• v0.1 — vocabulary layer. Athena loader, DuckDB schema, hierarchy queries

• v0.2 — retrieval. BM25, BioLORD-2023 embeddings, LanceDB index, RRF hybrid scoring

• v0.3 — expansion + provenance. ConceptSet model, descendant expansion, and OHDSI PL curated matching (small hand-picked demo set — see Setup)

• v0.4 — MCP server. stdio transport, four tools (see below), Claude Desktop config

• v0.5 — eval harness. Distance-weighted scoring, coverage, over-inclusion penalty; curated demo cohorts as ground truth. Decomposition accuracy deferred to v0.7 (see phenoforge.eval) — nothing decomposes a population description yet

• v0.7 — LangGraph agent. Decompose → check curated first → generate only for unresolved terms → human confirmation gate on anything generated → assemble. Real interactive CLI (scripts/run_agent.py)

• v1.0 — packaging. One-command setup (make setup), CI (lint/typecheck/test on every push), a PyPI release pipeline (tag-triggered, trusted publishing) as py-phenoforge, a zero-setup demo (make demo, no account/key/download needed), hosted API docs (colbyw5.github.io/phenoforge, built from the same docstrings every function already carries), and a validation and limitations section (see below)

Skipped: v0.6 (encoder benchmark — BioLORD vs SapBERT vs MedCPT). The agent was the more demonstrable deliverable, so v0.7 was built first; the encoder benchmark may return later.

Deferred: literature-derived phenotype algorithms (published tier), RxNorm and LOINC domains

Watching: TypeSafe AI's Jev, a schema-constrained "System One Model" that could fit the decompose node's structured-output call. Early access/waitlist, proprietary, no independent benchmarks yet as of 2026-10 — not adopted; revisit if it leaves early access and the latency/cost tradeoff is verified independently.

find_curated_definition is the only source of curated provenance today, and only for the 23 bundled demo cohorts. Everything from expand_hierarchy and search_concepts is generated provenance — ungrounded, structural or lexical/semantic only, and meant to be confirmed by a human before use in a cohort definition.

The only quantitative evaluation that exists is scripts/run_eval.py scoring each retrieval method against the 23 bundled OHDSI Phenotype Library demo cohorts (spanning diabetes/kidney, cardiovascular, respiratory, mental health, and more — see Setup) as ground truth. Current numbers on that set:

(Earlier, measured against only the original 8 diabetes/kidney cohorts: bm25 0.535, expand_descendants 0.697, dense 0.586, hybrid 0.692 hierarchical. Every method's hierarchical score moved when the set broadened — hybrid down from 0.692 to 0.634 (since corrected to 0.654, see below), expand_descendants down from 0.697 to 0.667 — concrete evidence that the original 8-cohort numbers were narrower than they looked, not a general ICD-10-CM retrieval benchmark.)

hybrid's own numbers above reflect a later fix: hybrid_search's candidate pool (how many results each of BM25/dense considers before RRF fusion) used to default to k itself — the same number controlling how many fused results come back — so a concept ranked just outside k in one sub-retriever was invisible to fusion even when it would have won easily once pooled. Found via cohort 775 (below); fixed by decoupling the default (candidate_k now defaults to 5 * k, still overridable). Measured before/after on this same benchmark: coverage unchanged (0.873), over-inclusion down 0.480 → 0.436, hierarchical score up 0.634 → 0.654 — a real, measured improvement, not just a plausible-sounding default.

What this does and doesn't establish:

• 23 cohorts across several disease areas — still not a general benchmark. Broadening past diabetes/kidney measurably moved every method's score (above), which is itself the point: a benchmark this size is sensitive to exactly which conditions are in it. It says nothing about performance on conditions still absent from the set, rare diseases, or codes with sparse/ ambiguous natural-language descriptions.

• Over-inclusion is the dominant error mode, not coverage. Every method pulls in more codes than the curated ground truth (dense and hybrid worst: ~0.48-0.54 over-inclusion against ~0.83-0.87 coverage) — the system errs toward casting a wide net and relying on human review to reject false positives, not toward silently missing codes. Anyone using the generated tier should expect to reject a meaningful fraction of what it returns, not rubber-stamp it.

• Decomposition now has a test case for every bundled cohort (23/23), measured at the same search depth the real pipeline actually uses, and on average it helps rather than hurts — but several real, distinct failure modes showed up along the way. scripts/eval_decomposition.py (phenoforge.eval.decomposition) compares, per bundled cohort, "hybrid search using the cohort's own name directly" against "the real decompose step, then per-term resolution" — same ground truth, isolating what decomposition costs or saves (phrasing drawn from real ClinicalTrials.gov titles/inclusion criteria, not this project's own wording). An internal audit caught this eval defaulting to k=10 (previously k=25) — the earlier number measured a wider search than agent/nodes.py ever actually uses, silently inflating how good decomposition looked. At the corrected depth: mean gap across the original 8 diabetes/kidney cases was a near-wash (+0.019, +0.020 on rerun at the old depth); across all 23 at the corrected depth, it's -0.125 to -0.165 across reruns — decomposition still usually wins, because a single clean condition name (e.g. "atrial fibrillation") curated-matches better than a cohort's own longer, more clinical display name (e.g. "Diabetes Mellitus Type 2 or history of diabetes") did as a search query. (The mean gap itself isn't perfectly reproducible run-to-run: decompose_fn is a real, non-deterministic Claude call, so a handful of individual cases' seed terms — and therefore scores — shift between reruns even with no code change at all; the themes below hold across reruns, the exact per-case numbers don't.) Individual cases still range widely, and the depth correction itself surfaced a new one:

  • Near-duplicate cohorts cause find_curated_definition to deliberately refuse to guess. The library has three "type 2 diabetes" cohorts (40, 288, 503); a generic decomposed term matching more than one of them falls through to ungrounded generated retrieval instead of picking one (by design — see the function's docstring), losing the specificity a longer query carried. Worst case seen: +0.277 to +0.404 across reruns (cohort 288).

  • A cohort's display name can be misleading relative to what it actually contains. Cohort 602, bundled under the name "Anxiety," resolves — via its real SNOMED-to- ICD-10-CM mapping — exclusively to dementia-with-anxiety codes (F01.x4/F02.x4/ F03.x4), not primary anxiety disorders. The decomposed term "generalized anxiety disorder" correctly retrieves the real F41.1 code and its relatives — retrieval works correctly — but scores 0.000 against ground truth that doesn't actually contain it (and at the corrected depth, so does the baseline — both score 0.000 against this cohort now). A curated-library content-quality issue, not a decomposition or retrieval bug, and it likely already depressed 602's contribution to the retrieval-only numbers above before this was ever noticed.

  • Search depth itself changes which cases look solved — and widening the candidate pool narrows this but doesn't close it. Cohort 775 (inflammatory bowel disease) scored a perfect 1.000 at the old, wider k=25 depth but 0.000 at the corrected k=10. Investigated directly: the seed term "inflammatory bowel disease" has zero lexical overlap with any ICD-10-CM concept name at all (the vocabulary only names these codes "Crohn's disease" / "ulcerative colitis") — BM25 contributes nothing, and dense retrieval alone ranks the correct code (K50.9) only 5th of 50 candidates, short of cracking the fused top 10 even after hybrid_search's candidate_k default was widened from k to 5 * k (above). Widening the pool measurably helped the baseline query for this cohort (0.177 → 0.402, since the full cohort name has more lexical signal than the bare term) but the decomposed path stays at 0.000 — a genuine vocabulary/ terminology-coverage gap (umbrella clinical term vs. disease-specific vocabulary naming), not a bug, and not something a pooling default alone can fix. Left as a documented limitation rather than pursued further (e.g. query synonym expansion), since that's new scope with its own failure modes, not justified by one cohort. (Checked separately: none of the 23 bundled cohorts currently have a ground-truth code missing from the loaded vocabulary — unresolvable_ground_truth exists for exactly this case but had never actually been populated until now; see eval/harness.py.)

• Curated-tier coverage is still narrow by construction. 23 cohorts out of the library's ~1,100 are bundled; any population description outside them falls through entirely to the unverified generated tier. Separately, SNOMED-to-ICD-10-CM resolution deliberately drops concepts whose fan-out exceeds a configurable threshold (find_high_fanout_snomed_concepts) rather than including everything — a real cohort definition can lose codes this way.

• No scale or concurrency testing. Built and measured against one developer's own Athena download on one machine. No data on index build time, query latency, or memory at a larger vocabulary, or under concurrent MCP clients.

• English, US ICD-10-CM only. No other language, no RxNorm/LOINC/other domains (see Roadmap).

• The human-confirmation gate has no audit trail. confirm pauses for review, but nothing records who approved what or persists decisions across runs — each CLI invocation starts a fresh thread with no memory of past confirmations.

.github/workflows/publish.yml builds and publishes to PyPI on any v*.*.* tag push, via PyPI trusted publishing (OIDC — no stored API token). One-time setup, done outside this repo:

1. On PyPI: Account settings → Publishing → add a pending publisher for py-phenoforge, repository colbyw5/phenoforge, workflow publish.yml, environment pypi.

2. On GitHub: Settings → Environments → create an environment named pypi (optionally with required reviewers, for a manual approval gate before each publish).

To cut a release: bump version in pyproject.toml, commit, then git tag v0.1.0 && git push origin v0.1.0.

.github/workflows/docs.yml builds and deploys the API docs to GitHub Pages on every push to main. One-time setup: on GitHub, Settings → Pages → set "Source" to "GitHub Actions" (no separate environment to create — deploy-pages manages the github-pages environment itself).

License

Apache 2.0. Vocabulary content (ICD-10-CM, SNOMED) carries its own license terms from OHDSI/ Athena, separate from this repo's license — nothing from data/ is redistributed.

Not a clinical decision tool

This produces code sets for research and analytics. It does not make clinical determinations and has not been validated for patient care.