Key4hep provenance and validation

Reusable tools around the existing FCC/Key4hep workflow. Generate events with k4run and the FCC-config cards as usual; use this package to archive committed inputs, record an output manifest, validate EDM4hep structure, and compare event content. It does not provide a generator launcher or a process-card catalogue. MCP is an optional, read-only interface to the same inspection tools.

Install

Python 3.11+ is required. The core has no pip dependencies:

uv sync --frozen
.venv/bin/python -I -m mcp_server_key4hep.cli --help
.venv/bin/python -I -m pytest -q

prepare and verify need only Python and Git (Git is needed for preparation). record, validate and compare need ROOT, podio and EDM4hep from your selected Key4hep environment. On LXPLUS, use a fresh shell and an explicit release:

source /cvmfs/sw.hsf.org/key4hep/setup.sh -r 2026-04-08
export PYTHONPATH="$PWD/src${PYTHONPATH:+:$PYTHONPATH}"
python3 -m mcp_server_key4hep.cli --help

Use stack Python for ROOT operations. -I deliberately ignores PYTHONPATH and is appropriate for isolated local tests, not this source-tree/stack invocation. Do not layer a second release over an already configured shell.

Related MCP server: rucio-mcp

Inspect files produced by your existing workflow

No server, job ID or provenance manifest is required:

python3 -m mcp_server_key4hep.cli validate events.e4h.root --events 10 --report validation.json
python3 -m mcp_server_key4hep.cli compare left.e4h.root right.e4h.root --report comparison.json

Reports are created exclusively; existing files are never overwritten. A failed operation still writes its {"valid": false, "error": ...} result to the report. CLI exit codes are 0 for success/match, 1 for an error or failed validation, and 2 for a completed comparison finding different content.

Validation reads every collection, checks event counts, and checks MCParticles finite momentum/mass/charge and parent/daughter references. It is structural validation, not a physics-quality assessment or comprehensive schema validation.

Comparison v2 covers ordered MCParticles, optional MCParticlesStable (cloned particles or subset membership), and optional EventHeader. Particle fields and relations are compared exactly; raw collection IDs are normalized to collection names and object indices. Finite floats use float.hex; signed zero and ordering are significant. Unknown collections fail rather than being silently excluded. ROOT storage metadata, file-level metadata and frame parameters are excluded. A match does not establish matching input configurations or physics correctness. The v2 hashes are intentionally distinct from historical v1 hashes.

Portable provenance for external runs

1. Write a JSON specification with release, explicit named seeds, the exact command argument list, and inputs containing repository and relative path. Inputs may come from multiple Git repositories.

2. Run prepare before executing the normal generator command. It checks committed input bytes, records commits and hashes, and archives copies.

3. Execute your usual command yourself; preserve stdout, stderr and environment information. The package does not execute the specification.

4. Run record to validate the output and write provenance.json beside it.

5. Run verify to check archived input and artifact hashes. Verification works after moving the whole run directory and without the original repositories.

python3 -m mcp_server_key4hep.cli prepare /absolute/run --spec spec.json
# Run your normal, explicitly seeded command in /absolute/run here.
python3 -m mcp_server_key4hep.cli record /absolute/run \
  --output events.e4h.root --events 10 --exit-code 0 \
  --artifact generation.stdout.log --artifact generation.stderr.log
python3 -m mcp_server_key4hep.cli verify /absolute/run/provenance.json

record rejects changed inputs or snapshots, failed declared exit codes, and outputs that change during validation. A failed external command leaves the pending manifest and logs for diagnosis. Use a fresh directory for another run.

The command, seeds, release and exit code are operator declarations. These utilities cannot prove that an external command actually used them. Hashes detect changes relative to the manifest; an unsigned manifest is not tamper-proof or an execution attestation. The snapshots cover only listed inputs, so list all files that can influence the run. Review actual logs and effective settings as well.

Demonstration using FCC-config

See the FCC tutorial demonstration. It uses pinned FCC-config and k4Gen commits, the existing dimuon card, and normal k4run. Only explicit Pythia and Gaudi seeds are added through a small options file. The upstream energy, decay settings, vertex smearing and collections are retained. No MCP server is involved.

Status: unit tests cover the portable tooling and canonicalization. This upstream-based demonstration still needs execution on LXPLUS with the selected release. The historical replay used the removed custom example and is not evidence that this demonstration passes.

Optional MCP interface

Install the extra only when an MCP client needs it:

uv sync --frozen --extra mcp

In a pinned stack environment, run the installed package using a Python that can import both the optional MCP dependency and the stack modules:

python3 -m mcp_server_key4hep.server --root /absolute/path/to/run-archive

The stdio server exposes only verify_provenance, validate_edm4hep_file and compare_event_content. Paths are confined to --root; ROOT inspection runs in a subprocess so native stdout cannot corrupt MCP framing. There are no tools for generation, shell execution, job scheduling, cancellation or writing manifests. Preparation and recording remain explicit CLI/library operations.

For a stack-derived venv on LXPLUS, install with uv sync --frozen --extra mcp --python "$(command -v python3)", then launch .venv/bin/python -m mcp_server_key4hep.server ... in that same pinned environment. Unlike isolated tests, stack inspection needs its configured module search paths.

Migration from 0.1

Version 0.2 removes Runner, the smoke launcher, custom generator examples, operator release/job configuration, and generation MCP tools. Existing ROOT files can be inspected directly. Old job manifests remain historical records; the new portable verifier accepts key4hep-external-run-v1, not the old runner schema. Use the old release to verify old manifests; do not rewrite them as new evidence. The package/repository name is retained for continuity; MCP is no longer required.