Skip to main content
Glama

Game Agent Harness

Game Agent Harness is a deterministic test runner for Godot games. It launches the demo in a separate process, calls a narrow domain API, validates JSON responses, simulates input, captures screenshots, and writes diagnostic artifacts.

Licensed under the Apache License 2.0.

The repository contains an original 8x6 tactics micro-game. It uses primitive Godot drawing only; no reference-project code or artwork is included.

Requirements

  • Godot 4.6.3

  • Python 3.11 or newer

  • macOS, Linux, or Windows with a display available for screenshot and UI tests

The Python client and vendored Godot addon are both pinned to godot-e2e==1.2.0. The stdio MCP facade uses the official mcp==2.0.0 Python SDK.

Related MCP server: godot-mcp-rts

Install

python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'

Open examples/tactics_demo/project.godot in Godot to play the demo manually. The automation server remains dormant in a normal game session.

Run scenarios

.venv/bin/python -m game_agent_harness run scenarios/tactics_smoke.yaml
.venv/bin/python -m game_agent_harness run scenarios/tactics_full.yaml
.venv/bin/python -m game_agent_harness run scenarios/tactics_ui.yaml
.venv/bin/python -m game_agent_harness soak scenarios/tactics_smoke.yaml --runs 100

Exit code 0 means pass, 1 means a game assertion failed, and 2 means a configuration, process, transport, or protocol error.

Every run creates artifacts/<run-id>/ with the resolved scenario, commands, events, states, screenshots, captured Godot logs, and reports in JSON and Markdown. Give report.md and its adjacent JSONL files to Codex when diagnosing a failed run.

MCP server

Start the local stdio server with an absolute workspace path:

.venv/bin/game-agent-harness-mcp \
  --workspace-root /absolute/path/to/game-agent-codex

An MCP host can launch it with this configuration:

{
  "mcpServers": {
    "game-agent-harness": {
      "command": "/absolute/path/to/game-agent-codex/.venv/bin/game-agent-harness-mcp",
      "args": [
        "--workspace-root",
        "/absolute/path/to/game-agent-codex"
      ]
    }
  }
}

The server publishes four tools:

  • list_scenarios lists YAML files under scenarios/.

  • run_scenario runs one listed scenario.

  • soak_scenario repeats a listed scenario, with a hard limit of 100 runs.

  • read_report reads report.json from one direct child of artifacts/.

The MCP layer uses the same ScenarioRunner as the CLI. It does not expose arbitrary filesystem paths, Godot node calls, property writes, or a code-editing tool. Scenario projects and generated artifacts must stay inside the configured workspace.

Game Agent API

/root/GameAgent exposes six methods during an --e2e session:

agent_get_manifest()
agent_reset(options)
agent_get_state()
agent_list_actions()
agent_perform_action(request)
agent_get_events(after_sequence)

The 0.1.0 contract is defined by the schemas under protocol/. An action request carries a unique request_id, the caller's expected_revision, an action name, and its arguments. A stale revision or invalid action is rejected without changing game state.

To add an action:

  1. Implement validation and behavior in TacticsGameController.perform_action.

  2. Add its argument schema to GameAgent.agent_get_manifest.

  3. Return a stable error code without incrementing revision on failure.

  4. Add a domain test and a scenario step.

The CLI deliberately does not expose arbitrary Godot node calls.

Verify

scripts/verify.sh

The script runs headless Godot domain tests, Python unit tests, Ruff, mypy, and the smoke scenario. It uses .venv/bin/python when available, otherwise python3. Set GAME_AGENT_PYTHON=/path/to/python to select another installed environment. The package is loaded directly from orchestrator/, but its third-party dependencies must be installed first.

VisualProbe

VisualProbe V1 is an optional, versioned contract for deterministic 3D inspection. It is isolated in examples/visual_probe_fixture, a Forward+ fixture that does not affect the Compatibility renderer used by tactics_demo. gameplay is deliberately outside V1: supported views are only front, back, left, right, and top; supported passes are beauty, silhouette, depth, normal, object_id, and wireframe.

The manifest is the only public capability registry. capabilities, when present, remains a unique list of strings; it must not contain a VisualProbe object. A V1 game publishes this shape (targets and variants are game-owned):

{
  "protocol_version": "0.1.0",
  "capabilities": ["domain_action"],
  "visual_probe": {
    "version": 1,
    "targets": {
      "asset_name": {
        "views": ["front", "back", "left", "right", "top"],
        "render_passes": ["beauty", "silhouette"],
        "max_resolution": {"width": 1280, "height": 720},
        "variants": {
          "height_scale": {"type": "number", "default": 1.0, "minimum": 0.5, "maximum": 2.0}
        }
      }
    }
  }
}

/root/GameAgent exposes these V1 methods. The JSON Schemas in protocol/visual-probe.schema.json are normative.

  • agent_get_manifest() returns the normal manifest above.

  • agent_prepare_visual_capture(request) accepts {target, views, render_passes, resolution: {width, height}, projection: "orthographic"}. Views and passes are unique, non-empty manifest values, at most 5×6 pairs; dimensions are positive and at most 1280×720. It returns {ok: true, count}.

  • agent_read_visual_capture() returns exactly one pending image per call: {ok: true, images: [{key, view, render_pass, width, height, png_base64}], renderer}. renderer contains projection, source, godot_version, and platform. The harness waits frames between reads. This two-phase API is the V1 capture API; agent_capture_visual is not part of V1 and is rejected by the VisualProbe transport.

  • agent_inspect_geometry({target}) returns {ok: true, geometry_snapshot}.

  • agent_apply_visual_variant({target, values}) applies one registered, type- and range-checked variant and returns {ok: true, token}. Only one variant can be active.

  • agent_restore_visual_variant({token}) restores it and returns {ok: true}.

Stable VisualProbe errors are visual_probe_unavailable, unknown_visual_target, unsupported_view, unsupported_render_pass, invalid_visual_variant, invalid_visual_request, visual_capture_failed, and visual_capture_not_ready. Errors have shape {ok: false, error: {code, message}}; no method accepts Godot node paths, method names, or host save paths.

Install the optional image stack and use the dedicated verifier on a GPU runner:

.venv/bin/pip install -e '.[dev,visual]'
scripts/verify-visual.sh

Visual runs store PNGs beneath visual/targets/<target>/; variant captures use visual/targets/<target>/variants/<id>/. A baseline package is exactly scenarios/baselines/<id>/metadata.json plus images/<target>/<view>-<pass>.png. metadata.json has visual_probe_version, godot_version, renderer, platform, width, height, and an images map keyed by <target>:<view>:<pass>. Renderer and dimensions must match before comparison; the V1 harness compares silhouette IoU, occupancy, centre/size deltas, and contour distance. PNGs and baseline files are bounded regular files read without following symlinks.

Each visual scenario has a fresh Godot process. A capture failure closes that process; on client disconnect or wait timeout the loopback server clears pending capture state and restores the active variant. The fixture transport listens only on 127.0.0.1, requires the non-empty random token generated by godot-e2e, fails closed after a 10-second handshake timeout, accepts frames up to 64 MiB, and permits only the methods listed above on /root/GameAgent. It rejects generic set_property, call_method, change_scene, batch, and all screenshots (including client-supplied save paths). A missing Godot 4.6 installation, GPU environment, or visual dependency is a configuration error (exit code 2), never a passing substitute.

Reference projects

The design was informed by:

No code or art was copied from these projects. See THIRD_PARTY_NOTICES.md for the vendored dependency.

Available Tools

9 tools
compare_visual_runsC

Return the VisualProbe reports for two confined artifact directories.

Inputs: left_artifact_dir (required), right_artifact_dir (required). Returns: object

list_scenariosA

List scenario files allowed within this workspace.

Inputs: no named parameters. Returns: object

list_visual_artifactsB

List safe PNG files directly inside an artifact directory's visual folder.

Inputs: artifact_dir (required). Returns: object

read_reportA

Read report.json from one direct child of the workspace artifacts directory.

Inputs: artifact_dir (required). Returns: object

read_visual_artifactB

Read one safe PNG as an MCP image content block.

Inputs: path (required), artifact_dir (required). Returns: An image provided to or from an LLM.

read_visual_reportB

Read VisualProbe JSON reports from one direct artifact directory.

Inputs: artifact_dir (required). Returns: object

run_scenarioA

Run one allowlisted scenario path relative to the scenarios directory.

Inputs: scenario (required). Returns: object

run_visual_scenarioB

Run one allowlisted scenario containing VisualProbe steps.

Inputs: scenario (required). Returns: object

soak_scenarioB

Repeat an allowlisted scenario within the server's hard run limit.

Inputs: runs (required), scenario (required), minimum_pass_rate. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 9 tools

    Disambiguation4/5

    Most tools are clearly distinct, but run_scenario/run_visual_scenario and read_report/read_visual_report could be confused without knowing scenario types or report nesting. Descriptions clarify the visual/VisualProbe specializations enough for an agent to choose.

    Naming Consistency5/5

    All tools follow a consistent verb_noun snake_case pattern with an optional visual modifier, such as list_scenarios, run_visual_scenario, read_visual_artifact, and compare_visual_runs. There are no mixed conventions or inconsistent verb styles.

    Tool Count5/5

    9 tools is well within the ideal range for a focused harness: scenario discovery/execution, report access, and visual artifact handling are each represented without obvious bloat. The set is appropriately scoped for its purpose.

    Completeness4/5

    Core workflows are covered: list/run/soak scenarios, read standard reports, and run/read/list/compare visual scenarios and artifacts. Minor gaps include no direct way to list artifact/report directories or inspect a scenario for VisualProbe steps before running, but these can often be worked around by convention.

    Maintenance

    ActivityInactive
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers