Hermes Enterprise Deployment Lab

./scripts/demo.sh boots a local enterprise API and walks one runbook action all the way through the failure that actually hurts: the API commits the write, then returns 500. Ten steps, no containers, no API keys, no model. What follows is what the code does, step by step, with the file and test that establishes each claim.

The arc the demo runs

Everything below is enterprise-mcp/enterprise_mcp/demo.py, which scripts/demo.sh executes and enterprise-mcp/tests/test_demo_arc.py runs as a test, so the demo cannot rot into a story the code no longer tells. Every tool call crosses a real MCP stdio boundary into a freshly spawned enterprise_mcp.server process. The caller is the script; no model has ever driven this arc.

1–2. The tool surface is decided by the server. With the read/plan allowlist, list_tools returns three tools and apply_incident_plan is not among them — build_server in enterprise-mcp/enterprise_mcp/server.py never registers a tool outside the allowlist, so an excluded tool is not merely refused, it does not exist on the wire. Calling it raises. Switch the allowlist to all and four tools appear. Proof: enterprise-mcp/tests/test_tool_filtering.py, and differentially against the real Hermes CLI in scripts/hermes-tool-filter-proof.sh (4 tools vs 1).

3–4. The first write stops itself. propose_incident_plan returns runbook steps with stable action_ids and approval_required flags. Calling apply_incident_plan without a capability returns status: pending_approval and an opaque approval_id — no capability, no idempotency key, and no HTTP write is issued at all. The demo counts the action store before and after to show the count did not move. Proof: workflow-runner/tests/test_executor.py::test_missing_approval_issues_only_an_id_and_writes_nothing, asserted against observed HTTP traffic, and enterprise-mcp/tests/test_approval_and_resume_over_mcp.py::test_unapproved_call_makes_no_side_effect over the real stdio boundary.

5. Approval comes from a different command. python -m workflow_runner.approval_operator approve <approval_id> --approver <identity> is a separate process the MCP server cannot invoke on its own. It records the approver, returns an expiring capability exactly once, and persists only its SHA-256 hash. Proof: workflow-runner/tests/test_executor.py::test_operator_identity_is_recorded_and_plaintext_capability_is_not; rationale in ADR 005. Forged, expired, wrongly bound, and already-applied capabilities are each refused before dispatch (test_forged_capability_writes_nothing, test_expired_capability_is_terminal_and_writes_nothing, test_capability_bound_to_a_different_incident_is_refused, test_pending_request_cannot_be_used_as_a_capability).

6. The ambiguous failure, on purpose. With ENTERPRISE_INJECT_FAILURE=error_after_commit, enterprise-api/app/main.py writes the record and then returns 500. The caller sees upstream_5xx plus resume instructions and cannot tell from the response whether the write landed. It did.

7–8. Resume replays instead of re-applying. The executor re-derives the incident/action pair's idempotency key at dispatch — it does not trust the key stored on the approval — so the resumed call returns the original record with replayed: true. The store holds exactly one record, and a third use of the same capability is refused with approval_already_applied without dispatching anything. Proof: enterprise-mcp/tests/test_approval_and_resume_over_mcp.py::test_forced_failure_then_resume_leaves_one_side_effect, plus workflow-runner/tests/test_executor.py::test_two_distinct_approvals_for_same_action_create_only_one_record and test_concurrent_distinct_approvals_converge_on_one_side_effect. The API's action store enforces the same one-record-per-pair invariant independently, returning HTTP 409 to a direct write-token caller who supplies a different key (enterprise-api/tests/test_actions.py).

9–10. Scope and trail. A write attempted with only the read credential is refused by the API with auth_failure and produces no record. The run's .jsonl audit log holds request, named operator grant, capability acceptance, failure, and replay events (workflow-runner/tests/test_audit.py). The log is not tamper-evident; see Limits.

The whole arc runs from a clean clone in ./scripts/fresh-clone-check.sh, and against the containerized API in the Docker-backed CI container-proof job — public CI run 31891411678 at 3da5938.

Provenance. This repository has real pre-publication history: the first 11 commits (git log --reverse --date=iso, 2026-07-27 through 2026-08-01) predate publication; nothing was squashed into a publication commit. Incident INC-2026-0042, its runbook, the operator identities, and the API data are fictional fixtures (enterprise-api/app/fixtures.py); no client data or credentials appear here, and private client history stays private.

Related MCP server: olivetin-mcp

What is in here

The LangGraph pipeline is deliberately the read-only half: it retrieves tenant-scoped runbook context with citations, fails closed without tenant scope or supporting evidence, and routes through explicit analysis and safety-review stages. Its retrieval is exact keyword-token overlap over an in-script fixture — not semantic relevance — and its evaluator is a regression check, not a mutation gate. The mutation gate is the approval machinery above.

Two repositories, one family

This lab is the execution half. The governance half — policy packs, approved configurations, independent checks, and human review gates — is the Hermes Enterprise Evaluation Kit, whose S3 Act mission runs against this lab's MCP tools. The split, and the exact code path that connects them, is in docs/hermes-enterprise-family.md.

Try the proof path

No provider API keys. Default proof stays on the host (tests + failure/resume + native telemetry/trace). Containers are optional / CI-attested.

./scripts/proof.sh

Treat a green Docker-backed container-proof job plus its uploaded receipt as the authority for the container path. Cloud IaC jobs here are validate only (no refresh/apply) and are never deployment evidence.

Hermes itself is an external client, not a Compose service: hermes mcp test lists the tools under an isolated HERMES_HOME and never touches ~/.hermes/config.yaml. It discovers; scripts/*.sh and pytest call. Model-driven invocation would need provider spend, which I declined on 2026-08-01 — this repository will not claim it.

What you can check

Each row is a rerunnable command. The command is the receipt.

Hermes is an external client, not a Compose service. The discovery scripts use an isolated HERMES_HOME and never touch ~/.hermes/config.yaml. Hermes lists the tools; scripts/*.sh and pytest call them.

Limits

• hermes mcp test only discovers tools. It makes no provider call. A true model-driven run would require hermes -z, a tool-call transcript, and provider spend. I declined that spend on 2026-08-01, so this repository does not claim model-driven invocation.

• The demo calls the operator command with demo-operator@example.com. The MCP server cannot grant its own approval, but the lab does not authenticate that identity or prove human judgment. A real deployment would put this command behind authenticated operator access.

• APPROVAL_STORE_PATH is an unauthenticated JSON file. It stores only a SHA-256 hash of the capability, but any local process that can edit the file can bypass the control. Same-host writers are serialized with a separate <path>.lock via Linux fcntl.flock; this is a single-host demonstration, not a distributed or production authorization service.

• The .jsonl audit log has no signature, chain hash, or WORM storage. run_started and run_finished have a null correlation ID. The API has no audit log of its own, so direct API writes do not appear here.

• The workflow derives one idempotency key per incident/action pair and rejects a later approved capability after that pair is applied. Concurrent approvals still converge on the same downstream key. The enterprise action store also enforces one record per pair for direct write-token callers: a different key receives HTTP 409 rather than a silent replay or second record. Dispatch re-derives the current pair key instead of trusting the persisted approval field, so pending approvals from the older random-key format also converge after an upgrade.

• Existing file-backed stores containing duplicate pairs fail closed on load. Preserve and reconcile or quarantine that fixture JSON before restart; the API cannot start merely to call its reset endpoint while the file is invalid.

• The server's allowlist is tested. Hermes's own tools.include behavior is not. On 2026-08-01, narrowing that list still made hermes mcp test print all three server-advertised tools.

• An independent AI-agent clean-checkout validation passed every executable native, demo, Hermes discovery, differential-filter, and adversarial path at 3da5938. It ran on the same VPS, not a different physical machine, and its container step was skipped because the validator could not access a Docker/Podman daemon. It is not human second-operator validation; the public Docker-capable CI result remains separate evidence.

• The approval guard lives in the workflow runner. A client with the write token can bypass it and call the fixture API directly, with no audit entry.

• This is not a production deployment: no OIDC, Kubernetes, real identity provider, cloud/hybrid scaling, or customer data. It has one deterministic incident (INC-2026-0042). A write adds a record to an in-memory store by default, or to an optional JSON file when ACTION_STORE_PATH is set (Compose uses a volume so restart proofs can survive process death).

• The workflow defines a container-proof job intended to check negative auth, post-commit failure, restart persistence, replay, and demo against the containerized API. A runtime pass applies only to an exact commit whose Docker-capable job is green and whose uploaded receipt reports a pass. Even then, it does not prove Kubernetes, OIDC, cloud deploy, or model-driven invocation. Missing Docker/Podman engine access fails closed (exit 2). proof.sh does not start containers unless you opt in; its default telemetry check still starts and cleans up temporary native processes.

• Prometheus metrics, SLO expressions, and alert-rule fixtures are implemented and proven with native localhost processes. Opt-in OpenTelemetry tracing is proven with loopback OTLP/HTTP capture. There is no Alertmanager, pager, collector backend, retention system, or production traffic. Native proofs are not evidence that the Compose telemetry path ran.

• The hardened Stage-1 LangGraph proof is public CI-attested synthetic evidence at 3da5938 / run 31891411678. Retrieval is exact keyword-token overlap over an in-script two-document fixture: any shared token, including a common word, produces a nonzero score and ready_for_review. This is not semantic relevance. There is no vector store, model call, action execution, authorization, or external validation. The evaluator is a regression check, not a mutation gate.

Component map

Hermes / script ──stdio──► enterprise-mcp ──Bearer+Idempotency-Key──► enterprise-api
        │                     ▲       │
        │ approval_id         │       └──────────────► audit log
        ▼                     │
 approval store ◄── operator command --approver <identity>
        │                     │
        └─ one-time capability (plaintext never persisted)

Prometheus ──scrape /metrics──► enterprise-api
OTLP capture (opt-in, loopback) ◄── workflow-runner CLIENT + enterprise-api SERVER

The step-by-step behaviour behind this diagram is the walkthrough at the top of this file; the security model is in docs/architecture.md.

Fresh-clone setup

Prerequisites: Python 3.11, 3.12, or 3.13. Not 3.14 — pydantic-core has no wheel for it and its vendored PyO3 tops out at 3.13, so a source build fails. The native telemetry proof also needs curl, sha256sum, tar, and network access to the pinned Prometheus release. The trace proof needs curl only and stays on loopback. Docker or Podman and the Hermes CLI are optional and only needed for the container and Hermes proofs.

git clone https://github.com/dbett4/hermes-enterprise-deployment-lab
cd hermes-enterprise-deployment-lab
cp .env.example .env

python3 -m venv .venv
.venv/bin/pip install -r requirements-dev.txt \
                      -r workflow-runner/requirements.txt \
                      -r enterprise-mcp/requirements.txt
.venv/bin/python -m pytest -q

./scripts/demo.sh          # the whole arc; boots its own API, no containers needed

Optional focused and self-contained container checks:

./scripts/telemetry-proof.sh          # native API + Prometheus, no containers
./scripts/trace-proof.sh              # native API + loopback OTLP capture, no containers
bash ./scripts/container-proof.sh     # dynamic free host ports; restart + replay + demo; performs teardown

For a manual fixed-port Compose stack that stays up for Hermes/MCP checks, start and stop it separately from container-proof.sh:

docker compose up -d --build enterprise-api prometheus
ENTERPRISE_API_URL=http://127.0.0.1:8080 ./scripts/mcp-smoke.sh
ENTERPRISE_API_URL=http://127.0.0.1:8080 ./scripts/hermes-tool-filter-proof.sh
docker compose down -v --remove-orphans

workflow-runner is a run-to-completion container that exits 0 by design; some compose providers report that as a failure under --wait.

Protocol-only smoke (CI mode)

MCP_SMOKE_PROTOCOL_ONLY=1 ./scripts/mcp-smoke.sh

Runs the FastMCP inspect/list/call checks without the Hermes CLI. Full local smoke requires Hermes and fails closed when it is absent.

Configuration

MCP stdio does not inherit your environment. The SDK forwards only HOME, LOGNAME, PATH, SHELL, USER. Anything else must be passed explicitly via env= on the stdio transport or the Hermes env: block. This repository got that wrong once and the failure was silent — see ADR 004.

Hermes MCP config (isolated)

Never merge this into ~/.hermes/config.yaml.

export ENTERPRISE_API_TOKEN=lab-read-token
export HERMES_HOME=/tmp/hermes-mcp-lab
mkdir -p "$HERMES_HOME"
{
  echo "_config_version: 9"
  ./scripts/emit-hermes-mcp-config.sh "$PWD" all
} > "$HERMES_HOME/config.yaml"
hermes mcp test enterprise_ops

The second argument is the server-side allowlist. Example shape: config/hermes-mcp-example.yaml.

Fixture tokens

Local lab only — non-secret test data, documented in .env.example:

ENTERPRISE_API_TOKEN=lab-read-token
ENTERPRISE_API_WRITE_TOKEN=lab-write-token

MCP tool surface

More detail

Build status

Work not done

These boundaries remain open or deliberately out of scope.

What production approval would require

The local implementation returns an opaque request ID, grants through a different command, records the supplied identity, stores a capability hash, enforces expiry and binding, and safely resumes after an ambiguous commit. It does not authenticate that identity or protect the JSON file from a local writer. A production version would replace the command and file with an IdP-backed approval service and transactional audit store. Details are in docs/architecture.md.

License

MIT — see LICENSE. Security notes: SECURITY.md.