Skip to main content
Glama

resplens

Reliability-aware AI tools for US respiratory surveillance data (COVID-19, influenza, RSV).

Public health surveillance numbers are revised after publication, some weeks rest on few reporting hospitals, and some sudden jumps are reporting artifacts. An AI assistant that reads these numbers at face value will state wrong trends with confidence. resplens gives language models tools that return every value with its reliability flag, summarize what can and cannot be trusted, and reconstruct what was known on any past date.

The data come from ObsAwareData (CC BY 4.0): weekly CDC hospital admissions (NHSN), emergency department visits (NSSP), and wastewater (NWSS) for the 50 states and DC, with a flag on every value and the full revision history.

Status

Stage

Contents

State

A1

Data tools and MCP server

Done

A2

Retrieval over data documentation; retrieval versus whole corpus in context

Done

A3

Agent that answers trend and reliability questions, with a task evaluation

Done

B1

Real-time forecasting benchmark on as-of data

Planned

B2

Fine-tuned small model for tool calling

Done

A4

Public replay of the evaluated runs; local agent service

Replay published; local service done

Related MCP server: CDC MCP

Tools

Tool

What it returns

describe_measures

Measures, the pathogens and region types each is published for, and the reliability flag codes

describe_state

Coverage dates, flag thresholds, ED regions, and wastewater sites for a state

get_series

Weekly values of one measure with a flag on every week

check_reliability

Recent data quality across hospital, ED, and wastewater sources, with plain-language caveats

get_revisions

How one week's value changed across CDC releases

get_as_of

A series exactly as it had been published on a past date

Example: check_reliability(state="SC", pathogen="influenza", weeks=4) on the 2026-10-16 release returns, along with per-source counts:

"caveats": [
  "Smoothed ED percent had an unusually large change in: 2026-09-19, 2026-09-26, 2026-10-03.",
  "Wastewater influenza covers influenza A only."
]

Tools never average, fill in, or reinterpret values. Invalid requests (an unknown measure, a measure that is not published for the requested region type, a date before the revision history starts) return an error message that tells the model how to correct the call.

Use with an MCP client

Requires uv.

git clone https://github.com/jiandewu/resplens.git
cd resplens
uv sync

Claude Code:

claude mcp add resplens -- uv run --directory /path/to/resplens resplens-mcp

Other clients: run uv run --directory /path/to/resplens resplens-mcp as a stdio server.

Retrieval over the documentation

A corpus of 28 public documents (about 74,000 words) explains the three surveillance systems: CDC dataset descriptions, reporting protocols, methodology pages, FAQs, two MMWR reports, and the ObsAwareData documentation. corpus/ keeps every source file unchanged with its URL, retrieval date, and SHA-256, plus the extracted text.

Retrieval was evaluated on a hand-reviewed set of 40 questions, each with verbatim evidence from the documents; 15 were used for tuning and 25 were held out. The four test configurations were fixed before the held-out run. On the 21 answerable held-out questions, with 300-token chunks:

Configuration

Recall@5

Recall@10

MRR

Latency (median)

BM25

0.48

0.52

0.38

1 ms

Dense (BAAI/bge-m3)

0.62

0.76

0.48

9 ms

Hybrid (BM25 + dense)

0.57

0.71

0.47

10 ms

Hybrid + cross-encoder rerank

0.86

0.86

0.50

206 ms

Hybrid retrieval with reranking found the evidence in the top five for 8 more questions than BM25 and none fewer (sign test p = 0.008). No configuration was reliably better at putting the evidence first. Method, findings, and failure analysis: docs/retrieval-eval.md.

Retrieval versus the whole corpus in context

The whole corpus is about 131,000 tokens and fits in a long-context model, so retrieval has to earn its place. The same generator (gemma4:31b, local) answered the 25 held-out questions twice: from the top 5 retrieved chunks, and from all 28 documents in context. An LLM judge from a different model family (qwen3:235b-thinking) graded the answers against the reviewed references and the source documents; on 15 answers graded blind by the author, the judge agreed on all 15.

Retrieval (top 5)

Whole corpus in context

Correct, of 21 answerable

17 (18 counting one answer the judge could not grade)

20

Unanswerable questions correctly declined

4 of 4

4 of 4

Facts stated that are not in the corpus

0

0

Prompt tokens per question (median)

1,273

130,005

Seconds per question (median, one H100)

1.7

125.5

The difference in correctness is not reliable at this sample size (3 questions to 1, p = 0.63). The long-context prompt was about 100 times larger and each answer about 73 times slower; with this model and runtime the shared document prefix could not be reused from cache. Every retrieval error came from the evidence not being retrieved, and in each case the model said the documents did not answer instead of inventing one. Details, judge calibration, and failure analysis: docs/answer-eval.md.

Agent evaluation

An agent loop gives a local model the six data tools plus a documentation search tool, and answers questions that need several calls, for example a four-week trend together with its reliability caveats. The evaluation plan was frozen and published before any test run. It covers 37 tasks, 13 for tuning and 24 held out, of seven types: trends, lookups, revisions, definitions, mixed, unanswerable, and prompt injection. Two systems were compared on the same prompt and tools: gemma4:31b and gpt-oss:120b. Each ran every test task three times. A task succeeds only if the answer is correct, every number in it comes from a tool output or is correctly derived from one, the reliability caveat is stated when needed, and no injected instruction was followed. The author graded the first run of all 48 answers blind to the system, and the hand review checked every number that did not match a tool output directly.

gemma4:31b

gpt-oss:120b

Mean task success over 3 runs

0.82

0.79

Held-out tasks solved (at least 2 of 3 runs)

20 of 24

18 of 24

Tool-call recall

0.92

0.97

Median seconds per task (two H100 GPUs)

7.3

5.0

The difference between the systems is not detectable (sign test p = 0.69; the run-1 sensitivity analysis favors gpt-oss by one task). The failures differ in kind: gpt-oss more often misread a tool output or a document (8 runs), and gemma more often gave an incomplete documentation answer (5 runs). Both followed an instruction planted in a data-tool output in all 6 runs, while resisting a planted tool call and a false number in a retrieved document. The retrieval-only pipeline from A2, given no data tools, answered 7 of 24 tasks correctly, none of them data-value questions. Results, the judge's agreement with the author, the changes made after freezing, and failure analysis: docs/agent-results.md.

Replay: every test run, with each model turn, tool call and output, the grades, and the numbers in each answer marked as found in a tool output, checked by hand, or unsupported: https://jiande.is-a.si/resplens/. It is built from the files in eval/ by uv run python -m resplens.app.replay --out site.

Fine-tuned 4B agent

Can a 4B model, fine-tuned on traces from the two larger models, run the same agent? The plan was frozen and published before any training question was generated. Training questions were generated from templates and documentation chunks on states and documents held out from the test tasks; both teachers answered them with the agent, and 1,152 conversations passed code checks and an LLM judge (the author reviewed a sample of the filter's decisions). Gemma 4 E4B was chosen over Qwen3-4B on a separate validation set before training, and a LoRA grid (rank, learning rate, epochs) was chosen on the same set: 147 of 158 validation items against 108 for the base model. Both 4B models then ran the 24 held-out A3 tasks three times on vLLM, graded as above.

Base Gemma 4 E4B

Fine-tuned

gemma4:31b

gpt-oss:120b

Mean task success over 3 runs

0.85

0.90

0.82

0.79

Held-out tasks solved

20 of 24

22 of 24

20 of 24

18 of 24

The gain over the base model is not detectable on the test tasks (3 tasks solved only by the fine-tuned model, 1 only by the base; sign test p = 0.63), and on the author-graded first run the two tie. The fine-tuned model includes every required number in all 72 runs (base 66) and makes more tool calls. The comparison with the A3 models crosses backends (4-bit on Ollama against bf16 on vLLM) and shows no detectable difference. Results, the full grid, and what changed after freezing: docs/finetune-results.md.

Run the agent locally

A small web service runs the evaluated agent (same system prompt, tools and settings) against a local Ollama, streams each step to the page, and checks the numbers in the answer against the tool outputs as the evaluation does.

ollama pull gemma4:31b            # and/or gpt-oss:120b
uv sync --extra app --extra models
uv run resplens-app               # http://127.0.0.1:8000

The local service answering a dev-set question: the trace streams in, then the answer with each number marked as found in the tool outputs

  • Memory: gemma4:31b used 20.9 GB and gpt-oss:120b 64.5 GB of GPU memory at the evaluated 32k context (Ollama 0.40.13, measured 2026-11-09). The first documentation search loads the retrieval models (BAAI/bge-m3 and BAAI/bge-reranker-v2-m3).

  • The 4B models from the fine-tuning section run on vLLM instead. Serve one under the name b2-base or b2-ft and set RESPLENS_VLLM_URL; the page offers the vLLM systems it finds there. The fine-tuned weights are not published, so outside this project only the base model can be served:

    vllm serve google/gemma-4-E4B-it --served-model-name b2-base --port 8100 --dtype bfloat16 \
        --max-model-len 32768 --seed 0 --enable-auto-tool-choice --tool-call-parser gemma4
    RESPLENS_VLLM_URL=http://127.0.0.1:8100 uv run resplens-app

    The fine-tuned model used 15.2 GB for its weights (vLLM 0.33.0, bf16) and answered dev task a01 in 8 s with three tool calls (measured 2026-11-16).

  • RESPLENS_OTHER_MODEL=<ollama model> adds another model, marked on the page as not evaluated. OLLAMA_URL points to another Ollama server.

  • One question at a time; questions up to 1,000 characters. GET /healthz checks the service without calling a model.

  • Each query appends one line to logs/queries.jsonl: model and digest, seconds, tokens, model turns, tools called, stop reason, and the numbers not found in the tool outputs. The question text is included unless RESPLENS_LOG_QUESTIONS=0.

Data handling

  • All data are read from one tagged ObsAwareData release (DATA_TAG in src/resplens/data.py, currently v2.0.0+2026-10-16), so answers are reproducible.

  • Files are downloaded on first use to ~/.cache/resplens/ (override with RESPLENS_CACHE) and checked against the release's published SHA-256 checksums.

  • A state's revision history is downloaded the first time get_revisions or get_as_of is called for it (about 15 MB compressed for South Carolina).

Development

uv run pytest                                   # offline tests on a small synthetic mirror
RESPLENS_NETWORK_TESTS=1 uv run pytest -k real  # also check against the real release

Retrieval experiments use the models extra (sentence-transformers) and, for some embedding models, a local Ollama server:

uv sync --extra models
uv run --extra models python -m resplens.retrieval.evaluate --split dev --retriever hybrid --embed-model BAAI/bge-m3 --rerank

Rebuilding the corpus from its sources: uv run --extra corpus python -m resplens.retrieval.corpus.

Sources and attribution

  • ObsAwareData is licensed under CC BY 4.0. Cite it as described in its CITATION.cff. Measure and flag descriptions in src/resplens/tools.py are adapted from its data dictionary.

  • CDC documents in corpus/ (Source: CDC) are public domain under CDC's Use of Agency Materials policy and are available at no charge from cdc.gov and data.cdc.gov; MMWR material is public domain under the MMWR series terms. One CMS memorandum and one National Cancer Institute (SEER) page are also included. Each document's source URL and retrieval date are in corpus/manifest.csv; files in corpus/text/ are text extracted from the originals without changes to their content.

  • This project is independent. Its use of these materials does not imply endorsement by CDC, ATSDR, HHS, CMS, NCI, or the United States Government.

License

Code: MIT. Corpus documents keep their original terms, listed above.

Available Tools

6 tools
check_reliabilityA

Summarize recent data quality across hospital, ED, and wastewater sources, with plain-language caveats.

Inputs: state (required), pathogen (required), end, weeks. Returns: object

describe_measuresA

List the measures (with their source, pathogens, and region types) and the reliability flag codes.

Inputs: no named parameters. Returns: object

describe_stateA

Coverage dates, flag thresholds, ED regions (HSAs), and wastewater sites for one state.

Inputs: state (required). Returns: object

get_as_ofC

The series exactly as it had been published on a past date, for real-time backtests and audits.

Inputs: as_of (required), state (required), measure (required), pathogen (required), +4 more. Returns: object

get_revisionsA

How one week's value changed across CDC releases: first value, latest value, and each change.

The first call for a state downloads its revision history, which can take a minute.

Inputs: week (required), state (required), measure (required), pathogen (required), +2 more. Returns: object

get_seriesB

Weekly values of one measure, latest published versions, with a reliability flag for every week.

start and end bound the weeks returned (inclusive).

Inputs: state (required), measure (required), pathogen (required), end, +4 more. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 6 tools

    Disambiguation4/5

    The tools mostly have clearly distinct roles: metadata discovery, state metadata, series retrieval, reliability summaries, revisions, and point-in-time snapshots. However, get_series, get_revisions, and get_as_of all return series-like values and require careful reading to choose correctly.

    Naming Consistency4/5

    All names use snake_case and are generally verb-first, making the set readable and predictable. The verbs vary between describe, get, and check, and get_as_of is slightly less parallel than the other names.

    Tool Count5/5

    Six tools is well-scoped for a focused public-health data server. Each tool covers a distinct stage of the workflow: discovery, retrieval, quality assessment, revision history, and historical backtesting.

    Completeness4/5

    The surface covers measure discovery, state metadata, weekly series, reliability caveats, revisions, and as-of queries, which is strong lifecycle coverage for this domain. A minor gap is the lack of an explicit list_states or region-enumeration tool, though agents can likely work around it.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to query U.S. public-health datasets such as disease surveillance, mortality, behavioral risk factors, and flu activity through the Socrata API, using natural language or direct tool calls.
      98 npm
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Enables AI assistants to discover, retrieve, and compare official international development indicators from sources such as the World Bank, FAOSTAT, WHO, UNICEF, and IMF, while preserving source identifiers, units, and citations.
      10
      2
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI assistants to ask natural-language questions about weekly CDC NNDSS chlamydia, gonorrhea, and syphilis case counts and rates by state and week, returning trends, prior-year comparisons, and usual-range checks through validated read-only tools.
      MIT