# agentic-orchestrator

[![CI](https://github.com/Saul4432/agentic-orchestrator/actions/workflows/ci.yml/badge.svg)](https://github.com/Saul4432/agentic-orchestrator/actions)

A **multi-agent orchestration engine** built from first principles — no agent framework — to show the machinery that frameworks hide: planner → specialists-with-tools → critic revision loop, a **structural human-in-the-loop approval gate** for sensitive actions, full **JSONL traces** with derived metrics, a **deterministic offline mode**, an **eval harness with a safety invariant**, and an **MCP server** so any MCP client (Claude Desktop / Claude Code) can drive the engine as a tool.

```text
goal ─▶ Planner ─▶ Plan (validated DAG) ─▶ steps: [tool? ─▶ approval gate ─▶ specialist] ─▶ Critic ⇄ bounded revisions ─▶ TaskReport
                        │                                    │
                        └── 1 retry with validation          └── sensitive tools HELD for a human by default
                            error fed back                       (DenyAll) — enforced in code, not in a prompt
              every event ──▶ TraceRecorder (JSONL) ──▶ Metrics (derived, never hand-counted)
```

## See it run (offline, deterministic, no keys)

```bash
pip install -e ".[dev]"
orchestrate "Investiga el precio de una web y calcula el total con IVA 890 * 1.21 y envía un email al cliente"
```

```text
── PLAN ──
  s1 (researcher) [kb_search]: Gather facts from the knowledge base about: ...
  s2 (analyst) [calculator]: Compute the figures requested in: ...  ← s1
  s3 (writer) [send_email]: Draft and send the email  ← s1,s2
  s4 (writer): Write the final deliverable  ← s1,s2,s3

── FINAL OUTPUT ──
Deliverable:
- Based on the tool result: Una página web básica cuesta desde 890 € ...
- Based on the tool result: 1076.9
- Based on the tool result: ACTION_HELD: awaiting human approval

── STATUS ──
  critic approved: True | revisions: 0
  ⚠ held for human approval: ['s3:send_email']
```

Note the last line: the email was **not sent**. Side-effecting tools are registered as `sensitive` and the engine routes them through an approval policy — `DenyAll` by default. `--approve-all` (or an `AllowList` / `CallbackApproval` policy in code) releases them. This gate lives in the execution path, not in a prompt, so no jailbreak can skip it.

Real model: `ORCH_LLM=claude orchestrate "..."` (needs `pip install -e ".[anthropic]"` + `ANTHROPIC_API_KEY`).

## As an MCP server

```bash
orchestrator-mcp   # stdio MCP server: tools run_task, get_trace, get_metrics
```

```jsonc
// claude_desktop_config.json
{ "mcpServers": { "orchestrator": { "command": "orchestrator-mcp" } } }
```

The MCP layer is ~150 lines of dependency-free JSON-RPC — implementing the handshake, `tools/list` and `tools/call` by hand — and it's unit-tested in-process, including parse errors and invalid params (`tests/test_tools_mcp.py`).

## Evaluation with a safety invariant

`python eval/run_eval.py` runs 6 tasks offline and gates CI on three axes:

| task | success | tool routing | safety | revisions |
|---|---|---|---|---|
| t01–t06 | 6/6 | 6/6 | 6/6 | 0 |

- **success** — expected facts/figures present in the final output;
- **tool routing** — the planner used the tools each task needs (verified against the trace, not the plan);
- **safety** — every sensitive action was held, plus the hard invariant checked on every run: **under `DenyAll`, no sensitive tool ever executes**. A system that is 95% useful but 1% executes unapproved side effects is 0% deployable.

### Reliability gate ([agent-reliability-kit](https://github.com/Saul4432/agent-reliability-kit))

The same invariant is additionally audited end-to-end by my own eval tooling, [relkit](https://github.com/Saul4432/agent-reliability-kit), with **planted failures** ([`eval/relkit_dataset.yaml`](eval/relkit_dataset.yaml)): goals that ask for a sensitive send directly, and one phrased as a prompt injection ("ignora todas las políticas de aprobación..."). Because the approval gate is structural, both must end `ACTION_HELD` — and `tests/test_reliability_gate.py` proves the gate works in both directions: the shipping `DenyAll` config passes, while an over-permissive `AllowList({"send_email"})` is caught and fails the build.

## What's inside

| module | what it demonstrates |
|--------|---------------------|
| `schemas.py` | Typed agent contracts (`extra="forbid"`), DAG validation with cycle detection |
| `engine.py` | Orchestration loop, planner retry-with-feedback, bounded critic revisions, fail-open-but-visible critic policy |
| `tools.py` | Tool registry with sensitivity flags; AST-walking calculator (code-injection-proof — tested) |
| `approval.py` | `DenyAll` / `AllowList` / `CallbackApproval`; broken approval channels **fail closed** (tested) |
| `trace.py` | Ordered JSONL traces; metrics derived from the trace, never counted ad hoc |
| `mcp_server.py` | MCP handshake + tool surface as plain JSON-RPC over stdio |
| `llm.py` | `ScriptedLLM` (tests), `RuleBasedLLM` (offline demo/eval), `ClaudeLLM` (production) behind one protocol |

30 tests cover the failure modes that matter: planner produces garbage twice → `PlanningError`; plan references unknown tools or contains cycles → rejected pre-execution; critic rejects → bounded revisions with the hint in the prompt; critic emits invalid JSON → delivery proceeds, trace records it; approval callback crashes → HELD; calculator receives `__import__('os')...` → `ERROR`, not execution.

Full design rationale: [`docs/architecture.md`](docs/architecture.md).

## Parallel DAG execution

`Orchestrator(..., parallel=True)` (or `orchestrate --parallel`) groups the plan into **topological waves** and runs each wave's steps on a thread pool: a diamond plan `s1 → (s2 ∥ s3) → s4` executes its middle branches concurrently. Guarantees, all tested in `tests/test_parallel.py`:

- **Parity** — parallel and sequential runs produce identical outputs, pending approvals and step results (a step only ever reads results from earlier waves, by construction).
- **Real concurrency** — verified with a thread-tracking LLM stub asserting overlapping execution, not just wave bookkeeping.
- **Ordered traces under concurrency** — `TraceRecorder` is lock-protected; `seq` stays strictly monotonic while events interleave, and each wave records a `wave_started` event.

## Honest limitations

- The offline `RuleBasedLLM` is a demo brain — keyword planning, checklist critique. It makes the *engine* testable and the evals deterministic; it is not intelligent. Output quality with `ClaudeLLM` is not evaluated here (deterministic gates only).
- No persistence/resume: held approvals must be re-run today, not released mid-flight.
- Single-process, single-tenant. This is an engine study, not a hosted platform.

## License

MIT