# agent-eval-gate

Trace Inspector + Eval Gate for Codex, OpenAI-compatible agents, and MCP clients.
This is an independent MIT-licensed implementation; it does not copy restricted source code or claim an official OpenAI/Codex plugin.

## Install

Requires Python 3.11+.

```bash
python -m pip install -e .
python -m pip install -e '.[dev]'
```

## CLI

```bash
agent-eval-gate inspect fixtures/normal.jsonl
agent-eval-gate eval cases/safe_search.json
agent-eval-gate gate cases/budget_fail.json  # exits 1 when a check fails
agent-eval-gate schema fixtures/normal.jsonl
```

`eval` prints JSON and is informational; `gate` is CI-friendly and returns non-zero on failure.

## Trace schema

One JSON object per line. `type` is one of `run`, `model_call`, `tool_call`, `tool_result`, or `error`.
Optional fields include `id`, `parent_id`, `timestamp`, `status`, `tool`, `arguments`, `latency_ms`, `attempt`, `tokens: {input, output}`, and `error_type`.

Metrics count tool calls, unique tools, repeated `(tool, arguments)` signatures, retries (`attempt > 1`), latency, token totals, error types, normalized failure classes, final status, and context-budget usage. A context budget is compared against input plus output tokens; tokenization is delegated to the producer, so this tool does not estimate hidden tokens.

Failure classes include `schema_error`, `permission_denied`, `timeout`, `tool_error`, `duplicate_operation`, `unknown_side_effect`, `model_error`, and `unknown`.

## Golden cases

Case JSON names a trace and optional constraints: `expected_tools`, `forbidden_tools`, `max_tokens`, `max_steps`, `expected_final_status`, and `context_budget`. Fixtures in this repository are synthetic and public.

## MCP

Install the optional SDK with `pip install -e '.[mcp]'`, then run `python -m agent_eval_gate.mcp_server`. It exposes read-only `inspect_trace` and `evaluate_case_file` tools returning structured JSON. Configure your MCP client to launch this command; no official plugin relationship is implied.

## Codex integration

The CLI is directly callable from a Codex skill or plugin. A minimal skill can invoke `agent-eval-gate inspect <trace>` and `agent-eval-gate gate <case>`; the MCP server can be registered as a local stdio server. Keep traces and cases in the calling workspace.

## Architecture and development

```text
JSONL -> parser -> typed events -> deterministic analysis -> eval checks -> CLI / MCP JSON
```

Run `pytest`, `ruff check .`, and `mypy agent_eval_gate`. GitHub Actions runs these checks on pushes and pull requests. Known limits: no vendor-specific trace adapters, no probabilistic scoring, and no token re-counting from raw prompts.

## 中文说明

这是一个面向 Codex、OpenAI-compatible Agent 和 MCP 客户端的 Trace Inspector + Eval Gate 独立实现，采用 MIT 许可证，不是 OpenAI 或 Codex 官方插件，也不复制受限仓库代码。

安装需要 Python 3.11+：`python -m pip install -e .`。`inspect` 输出 JSON 指标，`eval` 执行并展示 golden case 检查，`gate` 在约束失败时返回非零退出码，适合接入 CI。事件 JSONL 的 `type` 支持 `run`、`model_call`、`tool_call`、`tool_result`、`error`；可选字段包括工具名、参数、延迟、token、重试次数和错误类型。指标中的 token 使用生产者提供的 `tokens.input` 与 `tokens.output`，本工具不会猜测隐藏 token。

仓库 fixtures 和 case 都是自造公开数据。MCP SDK 是可选依赖，服务只读分析本地 trace 和 case。当前限制包括没有厂商专用适配器、没有概率评分，也不会从原始 prompt 重新分词计数。