agent-eval-gate

Trace Inspector + Eval Gate for Codex, OpenAI-compatible agents, and MCP clients. This is an independent MIT-licensed implementation; it does not copy restricted source code or claim an official OpenAI/Codex plugin.

Install

Requires Python 3.11+.

python -m pip install -e .
python -m pip install -e '.[dev]'

Related MCP server: agent-eval-mcp

CLI

agent-eval-gate inspect fixtures/normal.jsonl
agent-eval-gate eval cases/safe_search.json
agent-eval-gate gate cases/budget_fail.json  # exits 1 when a check fails
agent-eval-gate schema fixtures/normal.jsonl

eval prints JSON and is informational; gate is CI-friendly and returns non-zero on failure.

Trace schema

One JSON object per line. type is one of run, model_call, tool_call, tool_result, or error. Optional fields include id, parent_id, timestamp, status, tool, arguments, latency_ms, attempt, tokens: {input, output}, and error_type.

Metrics count tool calls, unique tools, repeated (tool, arguments) signatures, retries (attempt > 1), latency, token totals, error types, normalized failure classes, final status, and context-budget usage. A context budget is compared against input plus output tokens; tokenization is delegated to the producer, so this tool does not estimate hidden tokens.

Failure classes include schema_error, permission_denied, timeout, tool_error, duplicate_operation, unknown_side_effect, model_error, and unknown.

Golden cases

Case JSON names a trace and optional constraints: expected_tools, forbidden_tools, max_tokens, max_steps, expected_final_status, and context_budget. Fixtures in this repository are synthetic and public.

MCP

Install the optional SDK with pip install -e '.[mcp]', then run python -m agent_eval_gate.mcp_server. It exposes read-only inspect_trace and evaluate_case_file tools returning structured JSON. Configure your MCP client to launch this command; no official plugin relationship is implied.

Codex integration

The CLI is directly callable from a Codex skill or plugin. A minimal skill can invoke agent-eval-gate inspect <trace> and agent-eval-gate gate <case>; the MCP server can be registered as a local stdio server. Keep traces and cases in the calling workspace.

Architecture and development

JSONL -> parser -> typed events -> deterministic analysis -> eval checks -> CLI / MCP JSON

Run pytest, ruff check ., and mypy agent_eval_gate. GitHub Actions runs these checks on pushes and pull requests. Known limits: no vendor-specific trace adapters, no probabilistic scoring, and no token re-counting from raw prompts.

中文说明

这是一个面向 Codex、OpenAI-compatible Agent 和 MCP 客户端的 Trace Inspector + Eval Gate 独立实现，采用 MIT 许可证，不是 OpenAI 或 Codex 官方插件，也不复制受限仓库代码。

安装需要 Python 3.11+：python -m pip install -e .。inspect 输出 JSON 指标，eval 执行并展示 golden case 检查，gate 在约束失败时返回非零退出码，适合接入 CI。事件 JSONL 的 type 支持 run、model_call、tool_call、tool_result、error；可选字段包括工具名、参数、延迟、token、重试次数和错误类型。指标中的 token 使用生产者提供的 tokens.input 与 tokens.output，本工具不会猜测隐藏 token。

仓库 fixtures 和 case 都是自造公开数据。MCP SDK 是可选依赖，服务只读分析本地 trace 和 case。当前限制包括没有厂商专用适配器、没有概率评分，也不会从原始 prompt 重新分词计数。