Co-Designing the Tool Surface with the Agent Loop: Efficiency and Feasibility in Agentic Design Automation

IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD)

Tongyang Dai, Dimitrios Stamoulis, Hongyu Yuan, Jing Jin, and Jie Liu

This repository contains the artifact for the above paper. It includes the Ptolemy MCP server, the full evaluation harness, pre-collected per-run trace files for all four evaluation environments, and a script that reproduces every paper table from those traces without requiring an LLM API key.

Reproducing the paper tables (no API key required)

The pre-collected traces under eval/results/per_run/ (and the corresponding directories for Verilog/RTL, SystemC, and Simulink) are all that is needed to regenerate the LaTeX tables. No Ptolemy II installation and no Java are needed.

pip install -e .
python reproduce_tables.py

Output .tex and .json files are written to reproduced_tables/. The numbers should match the published paper exactly. An alternative output directory can be specified with --out-dir.

Related MCP server: COMSOL MCP Server

Installation

Python 3.10 or later is required.

pip install -e .

For Ptolemy II integration (needed to run new evaluations, not for table reproduction), set two environment variables before starting the server:

export PTII=/path/to/ptII          # Ptolemy II installation root
export JAVA_HOME=/path/to/java     # JRE 11+

On Windows use set or PowerShell $env: equivalents.

Running the smoke demo

The smoke demo starts the MCP server over stdio, builds a small SDF model, and verifies a signal property end-to-end. It requires PTII and JAVA_HOME to be set.

python demos/mcp_stdio_smoke.py

Re-running the full evaluation

Re-running requires a live LLM API key and a working Ptolemy II installation. See eval/README.md for sweep scripts and configuration details.

The Claude-based agent is configured through eval/ptolemy_mcp_config.json. The GPT-5 cross-model probe reads the endpoint from the PTOLEMY_GPT5_BASE environment variable (defaults to the standard OpenAI-compatible endpoint).

Repository layout

ptolemy_mcp/          MCP server and Ptolemy II bridge (Python package)
eval/
  benchmarks/         Task specifications for Benchmark A and C
  results/per_run/    Pre-collected per-run trace JSON files (1 260 runs)
  aggregate_*.py      Aggregation scripts that produce the paper's tables
  runner.py           Evaluation harness entry point
eval_eda/             Verilog/RTL cross-environment evaluation (54 runs)
eval_systemc/         SystemC cross-environment evaluation (27 runs)
eval_simulink/        Simulink cross-environment evaluation (18 runs)
agent/                System prompt and case-study notes used during evaluation
demos/                End-to-end smoke demo and reference MoML models
examples/             Example MCP client configuration
scripts/              Helper scripts for running sweeps
reproduce_tables.py   Code Ocean entry point — regenerates all tables

Trace file format

Each file in eval/results/per_run/ is a JSON object with the following top-level fields:

License

See LICENSE.