sim-agent

Natural language → sandboxed simulation code, with a self-correction loop, an MCP server and an honest evaluation harness. Ask a question about photon transport in plain language ("sweep the lead thickness from 1 to 10 cm and tell me where transmission drops below 1 %"). The agent writes a Python script against a small Monte Carlo simulator, checks it statically, runs it in a locked-down Docker sandbox, and when the run fails, feeds the error back to the model and repairs the script. It runs on small local models (Ollama, 4 GB of VRAM) or hosted ones, and it is measured: single-shot vs repair loop, across models, with accuracy, latency, tokens and cost.

A recorded benchmark run (scripts/render_demo.py draws it from the run's saved artifacts): the local 3B model forgets an import, the sandbox returns the traceback, the repair passes.

Quickstart

Python 3.11+, Docker. No API key or GPU needed for the first step.

pip install -e ".[dev,ollama]"
docker build -f sandbox/Dockerfile -t simagent-sandbox .     # or: make sandbox-image
simagent doctor                                               # Docker, image, Ollama, GPU, keys

# 1. Offline: a scripted model replays a buggy script, then the fix (shows a full repair cycle)
simagent solve "transmission of 1 MeV photons through 5 cm of water" --provider fake

# 2. A small local model
ollama pull qwen2.5-coder:3b
simagent solve "What fraction of 0.662 MeV photons gets through 10 cm of concrete?" \
    --provider ollama --model qwen2.5-coder:3b

# 3. As MCP tools for Claude Desktop / Claude Code / any MCP client
simagent mcp

Other commands: simagent run-script FILE (policy + sandbox, no model), --json for machine output, --max-attempts 1 for single-shot. Exit codes: 0 solved, 3 not solved or refused, 2 invalid input or missing sandbox (one line, no stack trace), 1 unexpected error. MCP setup: docs/mcp.md; a recorded client session: docs/mcp_demo.md.

Related MCP server: genpark-agent-sandboxed-virtual-code-runner-skill

How it works

flowchart LR
    U[request] --> G[generate]
    G --> P{check_policy}
    P -- passes --> X[execute in sandbox]
    P -- violation --> E[evaluate_run]
    X --> E
    E -- failed, attempts left --> R[repair]
    R --> P
    E -- solved / refused / out of attempts --> F[finalize]

• photonsim: a vectorised NumPy Monte Carlo of photons crossing a stack of slabs (transmitted / reflected / absorbed). Attenuation data for six materials comes from NIST SRD 126, downloaded and parsed by scripts/build_materials.py. Validated against Beer–Lambert within 4 standard errors. The physics is deliberately simple (isotropic, energy-preserving scattering); this is a test bed for the agent, not a shielding code.

• Agent (LangGraph StateGraph, docs/graph.mmd): the model sees a compact generated API reference (~850 tokens, docs/api.md), the result contract (simagent_io.submit(answer, **data)) and the request wrapped as data. The repair prompt gets the previous script plus a compact error report (policy violation, exit code and the last 40 stderr lines, timeout, or "no result submitted"). Every run saves each attempt's reply, script, output and a trace.json with timings and token counts under runs/.

• Providers: Ollama (local), Groq, OpenAI, and a scripted fake used by every test. Each model call records input/output tokens, time to first token, latency and tokens/s; anything a provider does not report is null, never estimated.

Security model

Model output is untrusted code. It only ever runs through a Runner, and the model never chooses the runner, the timeout or the limits.

The CI workflow's sandbox job builds the image and runs these Docker tests, and fails rather than skips if Docker is missing.

Results

36 tasks per run. Full table, per-category rates, prompt v1 vs v2 and per-task tokens: docs/eval_results.md.

What the numbers say (full analysis): with the original prompt the repair loop lifted the 3B coder from 7 to 20 of 36, mostly by fixing one missing import; a one-line prompt change (v2) got the same model to 23 single-shot, after which repair adds little. Repair fixes crashes, not wrong logic ("solved but wrong" rises in repair mode), and for the weakest model it lowered the adversarial score by pushing it to produce code. The hosted 120B model passes 36/36 with repair; the best local model on 4 GB of VRAM passes 24/36.

How it is measured. 36 tasks in seven categories: single runs, sweeps, comparisons, threshold searches, multi-step analyses, ambiguous or misleading requests (unknown material, energy out of range, impossible target, mm and keV units), and adversarial ones (file, network and shell access, prompt injection). Expected values come only from running reference scripts in the sandbox (scripts/make_expected.py); checkers are deterministic (Monte Carlo tolerance from the reference standard error, threshold within one grid step, must-refuse...), with no LLM judge. Baseline = the same graph with one attempt. Prompt v1 is the original; v2 adds one general instruction (import photonsim and simagent_io), logged with its before/after numbers in docs/NOTES.md. Run it yourself: python scripts/evaluate.py --provider ollama --model qwen2.5-coder:3b --mode repair --resume, or make bench for the whole plan in data/bench_plan.json.

Design decisions

• Static policy and sandbox, both. The policy gives the model fast, specific feedback ("import of os is not allowed; allowed modules are ..."); the sandbox is what actually holds. Each is tested on its own.

• Docker through the CLI, not the SDK. Every flag is visible in one function (DockerRunner.command) and unit-tested without Docker. Results leave the container through a nonce-marked envelope on stdout, because /out is a tmpfs that disappears with --rm.

• Deterministic checkers from reference runs. No hand-typed expected numbers and no LLM judge, so scores are reproducible and cannot be argued with.

• Small context, small models. The API reference is generated and kept under ~1 500 tokens so the whole prompt fits a 4–6k context on a 4 GB GPU.

• Honest numbers. The README and report are generated from committed result files; values a provider does not report stay n/a; no prices are typed from memory.

Limitations

• Toy physics: mono-energetic photons, 1D slabs, isotropic energy-preserving scattering, and mu_en used as the absorption probability. Good enough to check that code is right, not to design shielding.

• status = solved only means "the script ran and submitted a result"; correctness comes from the checkers. The report counts "solved but wrong" separately.

• 36 tasks, one run per configuration: differences of a few tasks are within noise.

• The reasoning model's TTFT and tokens/s include hidden reasoning tokens and are not comparable with the local models.

Roadmap

• Retrieval over the docs and a small physics-notes corpus; compare with and without it.

• An adapter for a real Monte Carlo code (e.g. OpenMC) behind the same Runner and API-reference pattern.

• Use the checkers as a reward signal for a small preference-tuning experiment.

Make targets

On Windows without make, run the commands from the Makefile directly.

Project layout

src/photonsim (simulator), src/simagent (policy, runners, LLM layer, agent graph, CLI, MCP server, evaluation), sandbox/Dockerfile, data/ (tasks, reference scripts, generated expected values, bench plan), results/ (committed benchmark results), scripts/ (generators, evaluation, report, demo), docs/. Spec: SPEC.md; plan: PLAN.md; progress and notes: docs/PROGRESS.md, docs/NOTES.md. MIT licence.