Skip to main content
Glama

ARC-AGI-3 World Model Lab

中文说明 | English

A research workbench for reconstructing ARC-AGI-3 games as explicit, executable world models. The project combines three layers:

  1. a deterministic C11 actor/rule engine exposed through Python and MCP;

  2. a lightweight Grid-SLAM and replay workbench for stable maps, masks, and trajectories;

  3. a visual discovery harness for human gold masks, traditional tracking, and multimodal-model experiments over complete action animations.

This is an independent research prototype, not an official ARC Prize project or submission. It deliberately focuses on discovering and verifying game dynamics, not planning or leaderboard optimization.

Grid-SLAM workbench concept

Why this project exists

ARC-AGI-3 interactions are short, discrete, and visually exact. A useful agent should not have to retain a long stream of screenshots in hidden neural state. Instead, it should be able to construct an external model:

action animation
  -> stable coordinates and candidate masks
  -> action-conditioned object events
  -> actor-owned event-condition-effect rules
  -> deterministic simulation and observation comparison

The engine is both a simulator and a constrained visual-description language. An agent can propose actors and rules, replay observations, inspect residuals, and revise the model without writing arbitrary native code.

Related MCP server: Meaning Model MCP Server

Current status

Area

Status

What works now

C world engine

Usable prototype

Actors, static map, ARC colors 0-15, collision, topology transforms, state/model hashes, snapshots, traces, rendering, diffs

ECE rule system

Usable prototype

Action/click/tick/collision/all-event triggers, validated condition expressions, multi-frame effect programs

MCP interface

Usable prototype

Typed actor, topology, rule, snapshot, simulation, render, and observation-comparison operations

Grid-SLAM

Experimental

Integer camera registration, typed grid-corner features, static/dynamic separation, status-bar filtering, object tracks

Web workbench

Integrated

Real/demo backend, full animation playback, replay, map/object views, actor/rule editor, native debug trace

Visual harness

Operational experiment

Full-sequence annotator, unified ROI, traditional bidirectional tracker, UniPixel/K3/Seed adapters and benchmarks

Automatic ontology discovery

Open research

Candidate masks exist, but deciding actor boundaries and grouping still needs model/human hypotheses

Full game reconstruction verifier

Partial

Engine can compare predictions with observations; automatic synthesis of complete game rules is not finished

Planner

Out of scope

No gameplay planner is included yet

World engine

world_engine/ contains the deterministic C11 kernel and Python bindings. The static map is separate from dynamic actors. Every actor can own event-condition-effect (ECE) rules.

Condition code is parsed with a restricted AST and compiled to a native postfix program; Python eval is not used:

actor.visible == true and
(actor.color == 8 or (action.id == 6 and action.x >= actor.x))

The validator checks actor references, property fields, booleans, ARC color ranges, viewport coordinates, and action fields before changing the model.

Effect code uses one validated operation per line and can expose intermediate frames:

frame 0: actor.recolor(8)
frame 1: actor.move(1, 0, block)
frame 2: actor.rotate_to(actor_12, 0, -1)

rotate_to lets an agent use another actor as an explicit post-transform pixel topology. This supports ARC-style rotations and reflections without making the LLM manually rewrite the full frame.

MCP

Start the stdio MCP server:

arc-world-engine-mcp

Set ARC_WORLD_MCP_TRANSPORT=streamable-http for an HTTP transport. The MCP surface intentionally exposes typed cognitive operations rather than arbitrary pixel writes.

Grid-SLAM workbench

The workbench processes every intermediate frame returned by one ARC action. Its current pipeline is:

  1. detect typed cell-corner features and estimate an integer camera offset;

  2. align frames before computing added/removed masks;

  3. update a confidence-weighted static map while excluding dynamic cells;

  4. match masks under translation to maintain object trajectories;

  5. detect one-cell-wide, two-color monotonic status bars and remove them from ordinary object candidates and map landmarks.

When a level finishes, map and object tracking reset. Exact previous-level object templates remain available as cross-level priors. A full game reset also clears replay, action history, and Grid-SLAM state.

Run the demo

Requirements: Python 3.12+, a C compiler, Node.js 22+, and uv.

make -C world_engine test build
uv venv
uv pip install -e ".[dev]"

cd frontend
npm ci
npm run build
cd ..

python -m grid_slam.server --demo --game demo-grid

Open http://127.0.0.1:8765. To use a real public ARC environment, copy .env.example to .env, set ARC_API_KEY, and run:

python -m grid_slam.server --game ft09 --operation-mode normal

The frontend is desktop-only by design.

Docker

docker build -t arc-world-model-lab .
docker run --rm -p 8765:8765 arc-world-model-lab

Visual discovery and summarization harness

vision_harness/ is a separate Python package. It imports action animations from the workbench, builds one shared ROI across all frames, and stores one human mask per target per frame. The browser annotator supports target splitting, multi-label change types, traditional forward/backward propagation, and side-by-side model overlays.

python -m pip install -e vision_harness
python -m arc_unipixel.import_workbench \
  --url http://127.0.0.1:8765 --output vision_harness/data/real --limit 12
python -m arc_unipixel.annotator \
  --data-root vision_harness/data --port 8788

Open http://127.0.0.1:8788. Real replay frames, human annotations, model weights, and provider outputs are intentionally excluded from this repository.

Experiment conclusions so far

  • The complete action animation matters. Reducing an interaction to two endpoint frames loses transient motion, blinking, occlusion, and causal grouping evidence.

  • A shared ROI is useful, but it is only a search window. It should not be treated as an object mask or supplied as semantic ground truth.

  • Traditional tracking is strong for exact rigid motion. Integer translation and color-map matching outperform general vision models on easy pixel-exact cases, but fail on deformation, split/merge, and ambiguous appearance changes.

  • UniPixel-7B can often describe the animation but is unreliable at exact ARC masks without domain adaptation. Natural-video object priors do not map cleanly to small, disconnected, discrete-color game actors.

  • Tool-assisted matrix reasoning is promising. Kimi K3 and Seed 2.1 Pro can use a native grayscale matrix tool to recover precise target topology, but generated origins and masks still require deterministic matching and review.

  • Semantic quality and mask quality must be measured separately. A model may correctly describe a target but emit no mask, or split one gold actor into several masks whose union is exact.

  • Incomplete gold can reverse model rankings. In the 12-sample endpoint benchmark K3 scored 0.8206 mean endpoint IoU and Seed scored 0.7986, but the difference is driven by one Seed mask-generation failure and by unlabelled status-bar changes that Seed detected. Excluding that failure, Seed's endpoint score is higher. These numbers must not be interpreted as a general model ranking.

The next evaluation revision should report semantic correctness, endpoint union, full temporal coverage, target grouping, grounding reliability, and unlabelled-change coverage as separate dimensions.

See vision_harness/README.md for protocols and vision_harness/docs/annotation-guide.md for the annotation format.

Tests

make -C world_engine test build
python -m pytest -q tests/unit

cd frontend
npm ci
npm run typecheck
npm run build

cd ../vision_harness
python -m unittest discover -s tests -p "test_*.py"

Repository layout

world_engine/    C11 kernel, Python bindings, ECE DSL, MCP server
grid_slam/       map registration, masks, tracking, status bars, backend
frontend/        Preact/Vite desktop workbench
vision_harness/  annotation and multimodal visual-discovery experiments
tests/           engine and Grid-SLAM tests
docs/            architecture images

License and attribution

Original project code is licensed under Apache License 2.0. Portions derived from arcprize/arc-agi-3-benchmarking remain available under its MIT license; the original notice is preserved in LICENSES/ARC-PRIZE-BENCHMARKING-MIT.txt. See NOTICE and THIRD_PARTY_NOTICES.md.

Available Tools

18 tools
add_click_effectC

Apply an effect when ACTION6 clicks the source actor.

Inputs: effect (required), source_actor_id (required), args, target_actor_id. Returns: object

add_collision_effectB

Apply an effect when one actor collides with another actor or static map.

Inputs: effect (required), moving_actor_id (required), collider_actor_id (required), args, +1 more. Returns: object

add_directional_movementC

Bind ACTION1-4 to up/down/left/right movement for an actor.

Inputs: actor_id (required), collision_policy. Returns: object

add_ruleB

Attach actor ECE code. Use actor.value predicates and frame N effects.

condition_code supports and/or, for example: actor.visible == true and (actor.color == 8 or action.id == 6). ACTION6 coordinates are action.x and action.y. Effects use one operation per line, for example frame 0: actor.recolor(8) followed by frame 1: actor.rotate_to(actor_2, 1, -1). Invalid code is rejected with an agent-facing line/column error. Legacy structured fields remain accepted.

Inputs: name (required), effect (required), trigger (required), args, +17 more. Returns: object

add_timer_effectC

Apply an actor effect on discrete action ticks, not wall-clock time.

Inputs: every (required), effect (required), source_actor_id (required), args, +2 more. Returns: object

compare_observationB

Compare a real ARC frame against the model's current rendered viewport.

Inputs: observed (required). Returns: object

create_actorC

Create a generic actor; cells are [local_x, local_y, optional_color].

Inputs: x (required), y (required), name (required), color (required), +6 more. Returns: object

create_snapshotB

Clone the complete native world for later rollback.

Inputs: name. Returns: object

create_worldB

Create an empty deterministic world using raw ARC colors 0-15.

Inputs: width (required), height (required), debug, background. Returns: object

get_world_stateA

Return rendered layers, actors, rules, hashes, trace, and debug history.

Inputs: no named parameters. Returns: object

load_static_mapA

Replace the static color and collision layers without creating actors.

Inputs: colors (required), blocked. Returns: object

merge_actorsC

Merge actors into the first ID using their rendered world topology.

Inputs: actor_ids (required). Returns: object

remove_actorA

Delete an actor and automatically remove ECE rules that reference it.

Inputs: actor_id (required). Returns: object

rollbackB

Restore a previously named native snapshot.

Inputs: name (required). Returns: object

set_viewportC

Set the integer camera origin and viewport, up to 64 by 64 cells.

Inputs: x (required), y (required), width (required), height (required). Returns: object

simulate_actionB

Advance one deterministic turn; ACTION6 uses viewport click coordinates.

Inputs: action (required), x, y. Returns: object

split_actorC

Split one actor by partitioning all of its local topology cells.

Inputs: groups (required), actor_id (required). Returns: object

update_actorC

Update only the provided generic actor state and transform fields.

Inputs: actor_id (required), x, y, color, +7 more. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.2/5.0

    Scored across 18 tools

    Disambiguation4/5

    Most tools have clear, distinct purposes, such as creating/updating/removing actors, adding effects by trigger type, and managing snapshots. The only minor overlap is between the general add_rule and the specific add_*_effect tools, but descriptions clarify their scopes.

    Naming Consistency4/5

    Tool names mostly follow a consistent verb_noun snake_case pattern (create_actor, update_actor, remove_actor, add_rule). Minor deviations include merge_actors using plural while other actor operations are singular, and rollback lacking a noun object.

    Tool Count4/5

    18 tools is on the higher side but each addresses a distinct aspect of the world engine: world setup, actor management, simulation, effects, and state inspection. No tool seems redundant, and the count is appropriate for the complex domain.

    Completeness5/5

    The tool surface covers the full lifecycle of world management: creating worlds, loading static maps, actor CRUD, simulation, rule/effect binding, observation comparison, and snapshot rollback. There are no obvious gaps that would prevent an agent from completing common tasks.

    Maintenance

    ActivityInactive
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables LLMs to interact with the Blocksworld Simulation to perform block manipulation actions, query real-time state, and verify multi-step plans. It bridges the Model Context Protocol with the simulation's REST API for AI planning research and agent development.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables an AI assistant to build and revise an explicit shared world model, then reason, create, narrate, remember, and give writing feedback through inspectable processes, concepts, and cuts. Supports general-purpose modeling, storytelling, agent/user memory, and ideation over one engine and graph while keeping estimates, interpretations, and observations distinct.
      MIT