omp-laya-judge

Local System-1 judge for oh-my-pi, powered by laya. Typed decisions (choice/bool/score) in ~1s on CPU, 0 LLM tokens burned, nothing leaves the machine.

Measured head-to-head on the same classification questions (real model via omp -p vs this plugin on CPU, no GPU):

Checkpoint A/B/C + random (benchmark/compare_models.py):

Default stays english: tied-best on EN with the highest usable confidence (above the 0.6 auto-accept line), and the server already routes non-Latin scripts to multilingual per request. TR sample is small (4); the honest read is laya >> chance (2–3 vs 0), not a checkpoint coronation.

Grounded quiz (python demo/quiz.py): 6/8, mean 282ms, 0 tokens — both misses came in under 0.2 confidence, exactly the cases the escalate rule covers.

Snake (python demo/snake.py): score 12, 300 moves, alive, 12 shield interventions, ~0.9s/move on CPU, 0 tokens. Watch every decision in the browser: open web/snake.html (canvas replay with live probability bars, play/pause/speed/scrub); quiz at web-quiz.html. Recipe (from laya-mlx): a deterministic planner describes each direction over a tiny state (Safe route: yes. Food reachable: a safety shield executes the best SAFE move. Short parallel criteria are the whole trick — greedy end-to-end choice without the planner scores 0 (see demo/snake.py header for the measured failure modes). Honest calibration: per-move confidences on game states run at noise level (0.005-0.05; the quiz gets 0.2-1.0 on text) — navigation is planner + shield with laya ranking, the same division of labor as laya-mlx. The maze variant (demo/maze.py`) is a documented negative: corridors trap the noise walk (1 crumb/200 steps).

Tetris (python demo/tetris.py): score 5720, 43 lines, 120 pieces, alive, ~1s/placement on CPU, 0 tokens. Per piece the planner enumerates every legal landing, shortlists 6 by classic features (lines, holes, height, bumpiness), laya picks one in a batched call, and a shield keeps the stack out of the top-4 danger zone. Replay in browser: web/tetris.html (colored board, candidate cards with live probabilities, scrub).

Languages

The server lets laya route per request: English goes to the english checkpoint, other scripts to multilingual (downloaded once, ~0.7GB extra). Turkish works today (e.g. fatura/departman routing correct) but with lower confidence than English — treat Turkish answers under 0.6 as escalate, same rule as everything else. Set LAYA_MODEL=english in .mcp.json to pin.

Related MCP server: laya-mcp

Install (step by step)

Requirements: Python 3.10+, pip, ~3GB disk (checkpoints), oh-my-pi.

pip install -r server/requirements.txt   # laya, mcp<2, torch, transformers>=4.48,<5
omp plugin marketplace add F0Rextasy/omp-marketplace
omp plugin install laya-judge@forextasy  # or: omp plugin link ./omp-laya-judge
omp plugin list                          # laya-judge should show ● enabled

Verify inside omp:

Call the tool mcp__laya-judge__judge_info and reply with its exact JSON output.

Expected: {"model": "laya/english", "device": "cpu", ...}. First judgment takes ~30s (checkpoint load), then ~0.3s each. If the server times out on connect, raise timeout in .mcp.json (default 30s is shorter than the load).

Troubleshooting (all hit during development, all fixed in this repo):

• transformers < 4.48 cannot load ModernBERT → pin transformers>=4.48,<5.

• mcp>=2 renamed FastMCP → pin mcp<2.

• On Windows, torch must init in the main thread → the server preloads eagerly.

• python.EXE/py.EXE uppercase spawn failures → use the shipped .cmd wrapper.

Use

The plugin exposes mcp__laya-judge__judge(state, questions) plus the laya-judge skill (triage → pre-filter → escalate). Question shapes mirror oh-my-pi's eval judge():

• choice: criteria: {label: rubric} → {choice, probabilities, confidence}

• bool: yes/no statement → {bool: P(true)} (laya noul head is [false, true])

• score: criteria: [lowest … highest] → {score, legend, probabilities, confidence}

Every reply carries model: "laya/english", per-call latency_ms, and laya's own usage/routing block so provenance survives in the transcript.

Status: tool + skill, not yet the agent loop

Honest scope: this plugin makes laya callable from any oh-my-pi session today. It is not yet wired into the eval judge() chain — the agent won't call it on its own; you (or your prompt) invoke it explicitly. Auto-routing cheap judgments to laya inside resolveJudge is the planned next step, as a PR to oh-my-pi core.

Layout

• .mcp.json — stdio server declaration (relative paths, lengthened timeout)

• server/server.py — FastMCP server, eager checkpoint load

• server/laya-judge.cmd — Windows launcher

• skills/laya-judge/SKILL.md — when to use / when to escalate

• benchmark/ — reproducible accuracy + latency proof

• assets/ — before-after.gif, benchmark.svg, quiz.gif