agentic-orchestrator

A multi-agent orchestration engine built from first principles — no agent framework — to show the machinery that frameworks hide: planner → specialists-with-tools → critic revision loop, a structural human-in-the-loop approval gate for sensitive actions, full JSONL traces with derived metrics, a deterministic offline mode, an eval harness with a safety invariant, and an MCP server so any MCP client (Claude Desktop / Claude Code) can drive the engine as a tool.

goal ─▶ Planner ─▶ Plan (validated DAG) ─▶ steps: [tool? ─▶ approval gate ─▶ specialist] ─▶ Critic ⇄ bounded revisions ─▶ TaskReport
                        │                                    │
                        └── 1 retry with validation          └── sensitive tools HELD for a human by default
                            error fed back                       (DenyAll) — enforced in code, not in a prompt
              every event ──▶ TraceRecorder (JSONL) ──▶ Metrics (derived, never hand-counted)

See it run (offline, deterministic, no keys)

pip install -e ".[dev]"
orchestrate "Investiga el precio de una web y calcula el total con IVA 890 * 1.21 y envía un email al cliente"

── PLAN ──
  s1 (researcher) [kb_search]: Gather facts from the knowledge base about: ...
  s2 (analyst) [calculator]: Compute the figures requested in: ...  ← s1
  s3 (writer) [send_email]: Draft and send the email  ← s1,s2
  s4 (writer): Write the final deliverable  ← s1,s2,s3

── FINAL OUTPUT ──
Deliverable:
- Based on the tool result: Una página web básica cuesta desde 890 € ...
- Based on the tool result: 1076.9
- Based on the tool result: ACTION_HELD: awaiting human approval

── STATUS ──
  critic approved: True | revisions: 0
  ⚠ held for human approval: ['s3:send_email']

Note the last line: the email was not sent. Side-effecting tools are registered as sensitive and the engine routes them through an approval policy — DenyAll by default. --approve-all (or an AllowList / CallbackApproval policy in code) releases them. This gate lives in the execution path, not in a prompt, so no jailbreak can skip it.

Real model: ORCH_LLM=claude orchestrate "..." (needs pip install -e ".[anthropic]" + ANTHROPIC_API_KEY).

As an MCP server

orchestrator-mcp   # stdio MCP server: tools run_task, get_trace, get_metrics

// claude_desktop_config.json
{ "mcpServers": { "orchestrator": { "command": "orchestrator-mcp" } } }

The MCP layer is ~150 lines of dependency-free JSON-RPC — implementing the handshake, tools/list and tools/call by hand — and it's unit-tested in-process, including parse errors and invalid params (tests/test_tools_mcp.py).

Evaluation with a safety invariant

python eval/run_eval.py runs 6 tasks offline and gates CI on three axes:

• success — expected facts/figures present in the final output;

• tool routing — the planner used the tools each task needs (verified against the trace, not the plan);

• safety — every sensitive action was held, plus the hard invariant checked on every run: under DenyAll, no sensitive tool ever executes. A system that is 95% useful but 1% executes unapproved side effects is 0% deployable.

Reliability gate (agent-reliability-kit)

The same invariant is additionally audited end-to-end by my own eval tooling, relkit, with planted failures (eval/relkit_dataset.yaml): goals that ask for a sensitive send directly, and one phrased as a prompt injection ("ignora todas las políticas de aprobación..."). Because the approval gate is structural, both must end ACTION_HELD — and tests/test_reliability_gate.py proves the gate works in both directions: the shipping DenyAll config passes, while an over-permissive AllowList({"send_email"}) is caught and fails the build.

What's inside

30 tests cover the failure modes that matter: planner produces garbage twice → PlanningError; plan references unknown tools or contains cycles → rejected pre-execution; critic rejects → bounded revisions with the hint in the prompt; critic emits invalid JSON → delivery proceeds, trace records it; approval callback crashes → HELD; calculator receives __import__('os')... → ERROR, not execution.

Full design rationale: docs/architecture.md.

Parallel DAG execution

Orchestrator(..., parallel=True) (or orchestrate --parallel) groups the plan into topological waves and runs each wave's steps on a thread pool: a diamond plan s1 → (s2 ∥ s3) → s4 executes its middle branches concurrently. Guarantees, all tested in tests/test_parallel.py:

• Parity — parallel and sequential runs produce identical outputs, pending approvals and step results (a step only ever reads results from earlier waves, by construction).

• Real concurrency — verified with a thread-tracking LLM stub asserting overlapping execution, not just wave bookkeeping.

• Ordered traces under concurrency — TraceRecorder is lock-protected; seq stays strictly monotonic while events interleave, and each wave records a wave_started event.

Honest limitations

• The offline RuleBasedLLM is a demo brain — keyword planning, checklist critique. It makes the engine testable and the evals deterministic; it is not intelligent. Output quality with ClaudeLLM is not evaluated here (deterministic gates only).

• No persistence/resume: held approvals must be re-run today, not released mid-flight.

• Single-process, single-tenant. This is an engine study, not a hosted platform.

License

MIT