Beyond Factual Accuracy: MCP-based LLM agents for heterogeneous public-sector data

Code, benchmark, frozen logs, scoring rubric and grounding annotations for the AIDA 2026 paper "Beyond Factual Accuracy: Evaluating MCP-Based LLM Agents for Heterogeneous Public-Sector Data Analytics" (Leonardo Castellón-Rodríguez and Pedro Palos-Sánchez; Procedia Computer Science, Elsevier).

Two LLM agents (GPT-4o and Gemini Flash) answer 30 frozen questions about public-health data for Costa Rica through a Model Context Protocol (MCP) server with three tools: World Bank (REST), OECD (SDMX) and a local corpus of 60,278 Google Play reviews of the EDUS digital-health app.

Two levels of reproducibility

Level 2 results will not match exactly: model endpoints change (Google served gemini-3.6-flash while gemini-3.5-flash was requested), and the live World Bank and OECD APIs may revise their data. The raw API responses received during the paper's runs are archived in raw_responses/ for that reason.

Related MCP server: datosgobdo-mcp

Level 1: re-compute the paper's figures

pip install -r requirements-analysis.txt
python verify_paper_figures.py      # accuracy, exact answers, external failures, wrong-country calls
python analisis_inferencial_v2.py   # paired difference, bootstrap CI, Wilcoxon -> salida_inferencial/
cd annotation && python acuerdo.py annotator2_labels_v20.xlsx grounding_annotations_audited_v1_1.json blinding_key.json

Where each result comes from

The September grounding audit uses run_openai_20260907T024524Z and run_gemini_20260907T025742Z.

Scoring rubric: versions

• scoring.py (v1) is the scorer used in the original submission.

• scoring_v2.py (v2) fixes negation handling ("the values do not match" was scored as a match) and adds a contextual country-validity check. All accuracy figures in the camera-ready paper use v2.

• The only October difference between v1 and v2 is chained_04 for GPT-4o (6/6 → 5/6 in four of five runs). Both outputs are kept: *_scored.json (v1) and *_scored_v2.json (v2).

• Expected values and check rules for each question are in questions_30.json (ground_truth).

• Factual checks and country validity are scored separately. Example: in chained_06 GPT-4o passes the factual direction checks in 4 of 5 October runs, but contextual_validity.country_valid_auto = 0 in all five, because it retrieved United States data. This is the gap between factual accuracy and grounding discussed in the paper.

• analisis_aida_integral.py is a descriptive companion analysis that reads the v1 score files; use it for latency, tool-call and failure tables, not for accuracy.

Level 2: re-run the experiment

pip install -r requirements.txt
cp .env.example .env            # add OPENAI_API_KEY and GOOGLE_API_KEY, then export them
python smoke_test.py            # checks the MCP server and the three tools
python run_experiment.py --questions questions_30.json --provider openai --model gpt-4o-2024-08-06
python run_experiment.py --questions questions_30.json --provider gemini --model gemini-3.5-flash
python scoring_v2.py --log experiment_logs/<new_run>.jsonl --questions questions_30.json --grounding annotation/grounding_annotations.json

Settings used in the paper (fixed in the code): temperature 0.0; at most 5 tool-calling rounds per question; up to 3 attempts for transient provider errors (backoff 2 s and 4 s, ±20% jitter); duplicate tool calls blocked; fallback synthesis if rounds are exhausted; no explicit reasoning configuration (provider defaults). The system prompt is the fixed instruction in run_experiment.py plus the data://schema resource published by mcp_server.py. GPT-4o uses Chat Completions; Gemini uses the google-genai SDK with automatic function calling disabled and thought_signature preserved in the history. Each log records the requested and the effective model.

Repository contents

Data, privacy and terms of use

• EDUS reviews. Public Google Play reviews of the EDUS app (Caja Costarricense de Seguro Social). Reviewer names and profile images were removed; only the four columns the tool needs are kept.

• Review identifiers. Google Play review IDs were replaced by sequential pseudonyms (EDUS-000001, ...), because an original ID can be used to open the review, and its author's profile, on Google Play. The mapping is kept by the authors and is not distributed.

• Personal data written inside review texts (names, national ID numbers, phone numbers and a birth year written by reviewers, names of individual health workers, and health-centre phone numbers) were replaced by [***] in 25 reviews (8 of them also in the BERTopic assignments file). reproducibility/redaction_log.csv lists the affected review_ids and the category of data removed, without the data itself. Because the assignments file changed (redactions and pseudonymous IDs), its expected SHA-256 in local_reviews_tool.py was updated (the hash used during the experiment is kept in a comment); topic assignments are unchanged. The tool returns identical results with the published files and with the original corpus on all 11 distinct EDUS queries made by the models. Review texts remain the property of their authors and are shared for research reproducibility only; they are not covered by the CC BY licence below. Please do not attempt to re-identify reviewers.

• World Bank data: CC BY 4.0 (World Bank Open Data terms). OECD data: OECD terms and conditions.

• No API keys or credentials are included. Keys are read from environment variables.

Licence

Code: MIT (LICENSE). Benchmark, scoring rubric, annotations and logs authored by us: CC BY 4.0. Third-party data keep their own terms (see above).

Citation

The version used in the paper is archived on Zenodo as release v1.0.0: https://doi.org/10.5281/zenodo.23816022 (all versions: https://doi.org/10.5281/zenodo.23816021). Please cite the paper and this archived release; see CITATION.cff.