# ARC-AGI-3 World Model Lab

[中文说明](README.zh-CN.md) | English

A research workbench for reconstructing ARC-AGI-3 games as explicit,
executable world models. The project combines three layers:

1. a deterministic C11 actor/rule engine exposed through Python and MCP;
2. a lightweight Grid-SLAM and replay workbench for stable maps, masks, and
   trajectories;
3. a visual discovery harness for human gold masks, traditional tracking, and
   multimodal-model experiments over complete action animations.

This is an independent research prototype, not an official ARC Prize project
or submission. It deliberately focuses on **discovering and verifying game
dynamics**, not planning or leaderboard optimization.

![Grid-SLAM workbench concept](docs/grid-slam-workbench-concept.png)

## Why this project exists

ARC-AGI-3 interactions are short, discrete, and visually exact. A useful agent
should not have to retain a long stream of screenshots in hidden neural state.
Instead, it should be able to construct an external model:

```text
action animation
  -> stable coordinates and candidate masks
  -> action-conditioned object events
  -> actor-owned event-condition-effect rules
  -> deterministic simulation and observation comparison
```

The engine is both a simulator and a constrained visual-description language.
An agent can propose actors and rules, replay observations, inspect residuals,
and revise the model without writing arbitrary native code.

## Current status

| Area | Status | What works now |
|---|---|---|
| C world engine | Usable prototype | Actors, static map, ARC colors 0-15, collision, topology transforms, state/model hashes, snapshots, traces, rendering, diffs |
| ECE rule system | Usable prototype | Action/click/tick/collision/all-event triggers, validated condition expressions, multi-frame effect programs |
| MCP interface | Usable prototype | Typed actor, topology, rule, snapshot, simulation, render, and observation-comparison operations |
| Grid-SLAM | Experimental | Integer camera registration, typed grid-corner features, static/dynamic separation, status-bar filtering, object tracks |
| Web workbench | Integrated | Real/demo backend, full animation playback, replay, map/object views, actor/rule editor, native debug trace |
| Visual harness | Operational experiment | Full-sequence annotator, unified ROI, traditional bidirectional tracker, UniPixel/K3/Seed adapters and benchmarks |
| Automatic ontology discovery | Open research | Candidate masks exist, but deciding actor boundaries and grouping still needs model/human hypotheses |
| Full game reconstruction verifier | Partial | Engine can compare predictions with observations; automatic synthesis of complete game rules is not finished |
| Planner | Out of scope | No gameplay planner is included yet |

## World engine

`world_engine/` contains the deterministic C11 kernel and Python bindings.
The static map is separate from dynamic actors. Every actor can own
event-condition-effect (ECE) rules.

Condition code is parsed with a restricted AST and compiled to a native postfix
program; Python `eval` is not used:

```text
actor.visible == true and
(actor.color == 8 or (action.id == 6 and action.x >= actor.x))
```

The validator checks actor references, property fields, booleans, ARC color
ranges, viewport coordinates, and action fields before changing the model.

Effect code uses one validated operation per line and can expose intermediate
frames:

```text
frame 0: actor.recolor(8)
frame 1: actor.move(1, 0, block)
frame 2: actor.rotate_to(actor_12, 0, -1)
```

`rotate_to` lets an agent use another actor as an explicit post-transform pixel
topology. This supports ARC-style rotations and reflections without making the
LLM manually rewrite the full frame.

### MCP

Start the stdio MCP server:

```bash
arc-world-engine-mcp
```

Set `ARC_WORLD_MCP_TRANSPORT=streamable-http` for an HTTP transport. The MCP
surface intentionally exposes typed cognitive operations rather than arbitrary
pixel writes.

## Grid-SLAM workbench

The workbench processes every intermediate frame returned by one ARC action.
Its current pipeline is:

1. detect typed cell-corner features and estimate an integer camera offset;
2. align frames before computing added/removed masks;
3. update a confidence-weighted static map while excluding dynamic cells;
4. match masks under translation to maintain object trajectories;
5. detect one-cell-wide, two-color monotonic status bars and remove them from
   ordinary object candidates and map landmarks.

When a level finishes, map and object tracking reset. Exact previous-level
object templates remain available as cross-level priors. A full game reset
also clears replay, action history, and Grid-SLAM state.

### Run the demo

Requirements: Python 3.12+, a C compiler, Node.js 22+, and `uv`.

```bash
make -C world_engine test build
uv venv
uv pip install -e ".[dev]"

cd frontend
npm ci
npm run build
cd ..

python -m grid_slam.server --demo --game demo-grid
```

Open `http://127.0.0.1:8765`. To use a real public ARC environment, copy
`.env.example` to `.env`, set `ARC_API_KEY`, and run:

```bash
python -m grid_slam.server --game ft09 --operation-mode normal
```

The frontend is desktop-only by design.

### Docker

```bash
docker build -t arc-world-model-lab .
docker run --rm -p 8765:8765 arc-world-model-lab
```

## Visual discovery and summarization harness

`vision_harness/` is a separate Python package. It imports action animations
from the workbench, builds one shared ROI across all frames, and stores one
human mask per target per frame. The browser annotator supports target
splitting, multi-label change types, traditional forward/backward propagation,
and side-by-side model overlays.

```bash
python -m pip install -e vision_harness
python -m arc_unipixel.import_workbench \
  --url http://127.0.0.1:8765 --output vision_harness/data/real --limit 12
python -m arc_unipixel.annotator \
  --data-root vision_harness/data --port 8788
```

Open `http://127.0.0.1:8788`. Real replay frames, human annotations, model
weights, and provider outputs are intentionally excluded from this repository.

### Experiment conclusions so far

- **The complete action animation matters.** Reducing an interaction to two
  endpoint frames loses transient motion, blinking, occlusion, and causal
  grouping evidence.
- **A shared ROI is useful, but it is only a search window.** It should not be
  treated as an object mask or supplied as semantic ground truth.
- **Traditional tracking is strong for exact rigid motion.** Integer
  translation and color-map matching outperform general vision models on easy
  pixel-exact cases, but fail on deformation, split/merge, and ambiguous
  appearance changes.
- **UniPixel-7B can often describe the animation but is unreliable at exact ARC
  masks without domain adaptation.** Natural-video object priors do not map
  cleanly to small, disconnected, discrete-color game actors.
- **Tool-assisted matrix reasoning is promising.** Kimi K3 and Seed 2.1 Pro can
  use a native grayscale matrix tool to recover precise target topology, but
  generated origins and masks still require deterministic matching and review.
- **Semantic quality and mask quality must be measured separately.** A model
  may correctly describe a target but emit no mask, or split one gold actor into
  several masks whose union is exact.
- **Incomplete gold can reverse model rankings.** In the 12-sample endpoint
  benchmark K3 scored `0.8206` mean endpoint IoU and Seed scored `0.7986`, but
  the difference is driven by one Seed mask-generation failure and by unlabelled
  status-bar changes that Seed detected. Excluding that failure, Seed's endpoint
  score is higher. These numbers must not be interpreted as a general model
  ranking.

The next evaluation revision should report semantic correctness, endpoint
union, full temporal coverage, target grouping, grounding reliability, and
unlabelled-change coverage as separate dimensions.

See [vision_harness/README.md](vision_harness/README.md) for protocols and
[vision_harness/docs/annotation-guide.md](vision_harness/docs/annotation-guide.md)
for the annotation format.

## Tests

```bash
make -C world_engine test build
python -m pytest -q tests/unit

cd frontend
npm ci
npm run typecheck
npm run build

cd ../vision_harness
python -m unittest discover -s tests -p "test_*.py"
```

## Repository layout

```text
world_engine/    C11 kernel, Python bindings, ECE DSL, MCP server
grid_slam/       map registration, masks, tracking, status bars, backend
frontend/        Preact/Vite desktop workbench
vision_harness/  annotation and multimodal visual-discovery experiments
tests/           engine and Grid-SLAM tests
docs/            architecture images
```

## License and attribution

Original project code is licensed under Apache License 2.0. Portions derived
from `arcprize/arc-agi-3-benchmarking` remain available under its MIT license;
the original notice is preserved in
[`LICENSES/ARC-PRIZE-BENCHMARKING-MIT.txt`](LICENSES/ARC-PRIZE-BENCHMARKING-MIT.txt).
See [NOTICE](NOTICE) and [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).