Skip to main content
Glama
Oneiros MCP Server

Oneiros MCP Server

by mal0ware

Oneiros

CI

Status: complete research artifact with verified results. v0.2 adds image-based perception, backed by the four pixel gates below; v0.1 had listed it as the next step. The V-JEPA 2 scale-up is still future work because it needs a GPU and pretrained weights.

A JEPA-style latent world model that an agent calls over MCP to plan.

Oneiros is a small, CPU-only research artifact that combines agentic systems with world models. It trains a Joint Embedding Predictive Architecture (JEPA) world model on a 2D point-mass environment, then exposes that model as a set of Model Context Protocol tools. Agent tools are usually hand-written functions that an LLM calls. Here the tool is the learned predictive world model: the agent encodes observations into latents, rolls the latent dynamics forward, and runs model-predictive control entirely in latent space.

Why predict in latent space

A JEPA predicts the future in a learned latent space, not in pixels or tokens. Given an observation o_t and an action a_t, it learns an encoder f and a predictor g such that g(f(o_t), a_t) matches f(o_{t+1}). There is no decoder and no pixel-reconstruction loss. A reconstruction loss spends capacity on detail that is perceptually salient but irrelevant to control (texture, lighting, background), while a latent predictor can discard anything that does not help it anticipate the future. The price is a known failure mode: latent prediction can collapse to a constant, with every observation mapped to the same point so that prediction is trivially perfect. Oneiros prevents collapse with an EMA target encoder, stop-gradients, and a VICReg-style variance and covariance penalty, then plans with the resulting latent dynamics.

Related MCP server: MuJoCo MCP Server

Architecture

flowchart LR
    subgraph Env["Point-mass environment (numpy)"]
        O["obs o_t"]
        ON["obs o_t+1"]
    end

    subgraph WM["JEPA world model (torch, CPU)"]
        F["encoder f"]
        FT["EMA target f_target<br/>(stop-grad)"]
        G["predictor g"]
        O --> F --> Z["latent z_t"]
        Z --> G
        A["action a_t"] --> G
        G --> ZH["z_hat_t+1"]
        ON --> FT --> ZT["z_t+1 (target)"]
        ZH -. "MSE + VICReg<br/>variance/covariance" .-> ZT
    end

    subgraph MCP["MCP server (agentic interface)"]
        T1["encode_observation"]
        T2["predict_rollout"]
        T3["plan_to_goal"]
        T4["reset_env / step_env"]
    end

    subgraph Agent["Agent loop"]
        P["latent MPC planner<br/>(CEM over g)"]
    end

    WM --> MCP
    MCP <--> Agent
    P -->|"first action"| Env

The agent never sees the environment's dynamics. It calls plan_to_goal, which encodes the current and goal observations, searches action sequences by rolling the predictor g forward H steps in latent space (cross-entropy method), scores each candidate by predicted-latent distance to the goal, and returns the first action. The planner replans every step (receding-horizon MPC).

Verified results

The numbers below come from one run on my machine (CPU only, seed 0). Train with python -m oneiros.train and reproduce the diagnostics with python -m oneiros.demo_agent.

Honesty gate

Metric

Result

(a) Predictor beats no-op baseline

next-latent MSE vs identity baseline

0.0185 vs 0.1076 (ratio 0.17, ~5.8x better)

(b) Latent not collapsed

per-dim latent std (mean / min)

1.04 / 1.00 (threshold 0.1)

(c) MPC beats random

goal-reaching success over 20 seeds

MPC 95% vs random 15-20%

Training takes about 21 seconds for 4000 steps. The checkpoint (oneiros/checkpoint.pt, ~240 KB) is committed so the demo, MCP server, and planning tests run without retraining.

Pixel gates (v0.2): image-based perception

In v0.1 the image encoder could not beat the identity baseline. The diagnosis had two parts, and v0.2 fixes each one:

  1. Consecutive frames were nearly identical. The blob moves about a pixel per step, so "predict no change" was already an excellent predictor. Fix: the swift environment preset (PointMassConfig.swift()), with a larger dt and acceleration so the agent moves several pixels per frame, plus speed-proportional drag so the dynamics are non-linear.

  2. A single frame hides velocity. The dynamics are second-order, so no single-frame predictor can recover the next state, and the convolutional encoder exploited this by temporal smoothing: it mapped consecutive frames to nearly identical latents. The dataset-wide variance penalty does not forbid that, and more training made it worse (MSE ratio from 0.89 to 0.94). Fixes: two-frame stacking, which makes velocity observable from pixels (the same reason pixel world models from DQN to V-JEPA take clips instead of single frames), and a delta-variance penalty, a VICReg-style hinge on the std of z_{t+1} - z_t that rules out the temporal collapse.

Results for the committed checkpoint_image.pt (~735 KB) on heldout data, enforced as tests in tests/test_gates_image.py:

Pixel gate

Metric

Result

(d) Image predictor beats no-op

heldout next-latent MSE ratio vs identity

0.11 (vector model: 0.17)

(e) Latent not collapsed, incl. temporally

per-dim std / one-step delta MSE

1.10 / 1.08 (pre-fix delta was 0.012)

(f) Pixel MPC beats random

goal-reach rate + median steps over 20 seeds

20/20, median 9.5 steps vs 27.5 random

(g) Imagination useful at horizon

compounded H-step rollout MSE ratio

0.45 at H=4, 0.86 at H=12

One negative result is measured but deliberately left out of the gates: a privileged linear-dynamics MPC that reads the true 4D state still reaches the goal about twice as fast as the pixel planner (median ~5 vs ~9.5 steps). Planning from pixels has not caught up with planning from privileged state, and gate (g) shows open-loop imagination degrading by horizon 12, which is why the planner replans every step. The baselines are in oneiros/baselines.py; to reproduce, use the image-training command under Run.

Latent-MPC plan to goal

The agent drives the point-mass to the goal (green star) using only the world-model planning interface.

Latent prediction error

Planning success

latent prediction

planning vs random

Scope and limits

Oneiros is an end-to-end demonstration that (1) a non-trivial latent dynamics model can be learned without collapse and without reconstruction, and (2) planning in that latent space solves a control task far better than chance, all behind an agentic tool interface.

It does not show any of this at scale. The environments are toy 2D point-masses and the models are tiny (a few hundred KB). The default committed model uses a vector-state observation on the original environment. The committed image model (v0.2, checkpoint_image.pt) sees stacked 32x32 frames on the swift environment and clears its own four gates above. The scale-up is still future work: a frozen pretrained perception encoder such as V-JEPA 2 with a learned latent dynamics head on top, which is the recipe this toy mirrors. That step needs a GPU and pretrained weights, and this artifact is CPU-only on purpose.

Install

Requires Python 3.12. A project-local virtual environment is recommended.

python -m venv .venv
# Windows: .venv\Scripts\activate    |    Unix: source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -e ".[dev]"

torch is installed from the CPU wheel index, so no GPU is needed or used.

Run

# Train the JEPA world model (writes oneiros/checkpoint.pt). ~21s on CPU.
python -m oneiros.train --obs-mode vector --steps 4000

# Train the image world model on the swift environment (~3 min on CPU).
# This is the exact recipe of the committed checkpoint: two-frame stacking,
# the delta-variance penalty, and an explicit --checkpoint so the vector
# model's default path is not clobbered.
python -m oneiros.train --obs-mode image --env swift --steps 6000 \
    --frame-stack 2 --delta-var-weight 25 --checkpoint oneiros/checkpoint_image.pt

# Run the scripted agent: drive to the goal via latent MPC, write the GIF +
# diagnostic plots to assets/.
python -m oneiros.demo_agent --seed 0 --k 20

# Tests, including the seven honesty gates (uses the committed checkpoints).
pytest -q

# Lint.
ruff check oneiros tests

MCP server

The world model is exposed over MCP as a stdio server:

# vector model (default)
python -m oneiros.mcp_server

# the committed image model: stacked-frame observations, swift environment
python -m oneiros.mcp_server --obs-mode image

# any other checkpoint
python -m oneiros.mcp_server --checkpoint /path/to/checkpoint.pt

--obs-mode selects which committed world model the server exposes. The environment tools serve observations in the loaded model's format (6D vectors, or the stacked rendered frames the image model trains on), and reset_env returns a ready-made goal_observation for the planning tools. The served environment always uses the configuration the checkpoint was trained on (the swift preset for the image model).

Tools:

Tool

Purpose

encode_observation

observation -> latent z (the trained JEPA encoder)

predict_rollout

roll latent dynamics g forward over an action sequence

plan_to_goal

latent-space MPC; returns the next action toward a goal

plan_trajectory

full best action sequence + the imagined latent path

model_info

loaded model's obs mode, dims, and honesty-gate metrics

reset_env / step_env

drive the point-mass environment; a session id keeps concurrent agent sessions independent

To register the server with Claude Desktop, add this to claude_desktop_config.json (use absolute paths for your checkout):

{
  "mcpServers": {
    "oneiros": {
      "command": "C:/path/to/Oneiros/.venv/Scripts/python.exe",
      "args": ["-m", "oneiros.mcp_server"],
      "cwd": "C:/path/to/Oneiros"
    }
  }
}

On Unix the command is .venv/bin/python. Append "--obs-mode", "image" to args to serve the committed image world model instead of the vector one.

Repository layout

oneiros/
  env.py          # deterministic 2D point-mass environment (numpy)
  model.py        # JEPA encoder + predictor + VICReg regularizers
  data.py         # random-policy rollout replay buffer
  train.py        # JEPA training loop, evaluation, checkpoint I/O
  planner.py      # latent-space MPC (CEM / random shooting)
  baselines.py    # pixel episode runners + privileged linear-MPC baseline
  mcp_server.py   # MCP tools exposing the world model
  demo_agent.py   # scripted agent + diagnostics (GIF, plots)
  checkpoint.pt        # committed vector model (~240 KB)
  checkpoint_image.pt  # committed image model, swift env (~735 KB)
tests/            # determinism, shapes, and the seven honesty gates
assets/           # generated GIF and diagnostic figures

See SYNERGY.md for how the same latent-dynamics idea connects to regime-aware modeling in time series.

License

MIT. See LICENSE.

Available Tools

5 tools
encode_observationA

Encode a 6D observation [x, y, vx, vy, goal_x, goal_y] into a latent vector.

Returns the latent embedding produced by the trained JEPA encoder. The latent is the space the world model predicts in; downstream tools operate on these latents rather than on raw observations.

Inputs: observation (required).

plan_to_goalA

Plan the next action that drives the agent toward a goal, via latent MPC.

Encodes the current and goal observations, searches action sequences by rolling the world model forward in latent space (cross-entropy method), and returns the first action of the best plan. Call repeatedly (replanning each step) to follow a receding-horizon trajectory to the goal.

Inputs: current_obs (required), goal_obs (required).

predict_rolloutA

Roll the learned latent dynamics forward from a latent over an action sequence.

Given a starting latent and a list of 2D acceleration actions, applies the predictor g step by step and returns the latent trajectory. This is the model imagining a future without touching the real environment.

Inputs: latent (required), actions (required).

reset_envA

Reset the point-mass environment to a deterministic start for the given seed.

Returns the initial observation, the agent state, and the goal position.

Inputs: seed.

step_envA

Apply a 2D acceleration action to the environment and advance one timestep.

Returns the new observation, reward (negative distance to goal), a done flag (agent reached the goal), and the distance to the goal.

Inputs: action (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.2/5.0

    Scored across 5 tools

    Disambiguation5/5

    Each tool has a clear, distinct purpose: encoding observations, planning to a goal, predicting rollouts, resetting the environment, and stepping the environment. No two tools overlap in function.

    Naming Consistency5/5

    All tool names follow a consistent verb_noun pattern (encode_observation, plan_to_goal, predict_rollout, reset_env, step_env) and use snake_case uniformly.

    Tool Count5/5

    With 5 tools, the server covers the core operations for a point-mass control domain with latent dynamics perfectly. The count is well-scoped and each tool earns its place.

    Completeness4/5

    The tool set covers the main workflow: observation encoding, planning, prediction, environment reset and stepping. A minor gap is the lack of a tool to directly set or inspect the goal, but it is implicitly covered via encode_observation and plan_to_goal.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      A domain-agnostic MCP server for autonomous experimentation, generalizing Karpathy's autoresearch pattern into a reusable server that any AI agent can drive, pointed at any domain defined by a JSON configuration.
      3
      Apache 2.0
    • A
      license
      Not graded
      quality
      D
      maintenance
      Exposes MuJoCo physics simulation to AI assistants via 65 MCP tools, enabling natural language control of robotics simulation, trajectory optimization, contact analysis, and video export.
      12
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server that enables LLM agents to control a two-joint MuJoCo robot arm through tools for moving, reading state, and resetting. It deliberately omits inverse kinematics, so the agent must infer joint angles from observations.
      MIT