Oneiros MCP Server
This server enables agents to interact with a 2D point-mass environment by utilizing a JEPA-style latent world model for planning and prediction.
Encode high-dimensional observations (like images or 6D vectors) into a compact latent vector.
Simulate the future state of the environment by projecting actions forward in the latent space (rollouts).
Determine the optimal next action toward a goal using latent-space Model Predictive Control (MPC).
Control the actual point-mass environment by executing actions and resetting the simulation.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Oneiros MCP Server plan a path to the goal at (5,3)"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Oneiros
Status: complete research artifact with verified results. v0.2 adds image-based perception, backed by the four pixel gates below; v0.1 had listed it as the next step. The V-JEPA 2 scale-up is still future work because it needs a GPU and pretrained weights.
A JEPA-style latent world model that an agent calls over MCP to plan.
Oneiros is a small, CPU-only research artifact that combines agentic systems with world models. It trains a Joint Embedding Predictive Architecture (JEPA) world model on a 2D point-mass environment, then exposes that model as a set of Model Context Protocol tools. Agent tools are usually hand-written functions that an LLM calls. Here the tool is the learned predictive world model: the agent encodes observations into latents, rolls the latent dynamics forward, and runs model-predictive control entirely in latent space.
Why predict in latent space
A JEPA predicts the future in a learned latent space, not in pixels or tokens.
Given an observation o_t and an action a_t, it learns an encoder f and a
predictor g such that g(f(o_t), a_t) matches f(o_{t+1}). There is no
decoder and no pixel-reconstruction loss. A reconstruction loss spends capacity
on detail that is perceptually salient but irrelevant to control (texture,
lighting, background), while a latent predictor can discard anything that does
not help it anticipate the future. The price is a known failure mode: latent
prediction can collapse to a constant, with every observation mapped to the
same point so that prediction is trivially perfect. Oneiros prevents collapse
with an EMA target encoder, stop-gradients, and a VICReg-style variance and
covariance penalty, then plans with the resulting latent dynamics.
Related MCP server: MuJoCo MCP Server
Architecture
flowchart LR
subgraph Env["Point-mass environment (numpy)"]
O["obs o_t"]
ON["obs o_t+1"]
end
subgraph WM["JEPA world model (torch, CPU)"]
F["encoder f"]
FT["EMA target f_target<br/>(stop-grad)"]
G["predictor g"]
O --> F --> Z["latent z_t"]
Z --> G
A["action a_t"] --> G
G --> ZH["z_hat_t+1"]
ON --> FT --> ZT["z_t+1 (target)"]
ZH -. "MSE + VICReg<br/>variance/covariance" .-> ZT
end
subgraph MCP["MCP server (agentic interface)"]
T1["encode_observation"]
T2["predict_rollout"]
T3["plan_to_goal"]
T4["reset_env / step_env"]
end
subgraph Agent["Agent loop"]
P["latent MPC planner<br/>(CEM over g)"]
end
WM --> MCP
MCP <--> Agent
P -->|"first action"| EnvThe agent never sees the environment's dynamics. It calls plan_to_goal, which
encodes the current and goal observations, searches action sequences by rolling
the predictor g forward H steps in latent space (cross-entropy method),
scores each candidate by predicted-latent distance to the goal, and returns the
first action. The planner replans every step (receding-horizon MPC).
Verified results
The numbers below come from one run on my machine (CPU only, seed 0). Train
with python -m oneiros.train and reproduce the diagnostics with
python -m oneiros.demo_agent.
Honesty gate | Metric | Result |
(a) Predictor beats no-op baseline | next-latent MSE vs identity baseline | 0.0185 vs 0.1076 (ratio 0.17, ~5.8x better) |
(b) Latent not collapsed | per-dim latent std (mean / min) | 1.04 / 1.00 (threshold 0.1) |
(c) MPC beats random | goal-reaching success over 20 seeds | MPC 95% vs random 15-20% |
Training takes about 21 seconds for 4000 steps. The checkpoint
(oneiros/checkpoint.pt, ~240 KB) is committed so the demo, MCP server, and
planning tests run without retraining.
Pixel gates (v0.2): image-based perception
In v0.1 the image encoder could not beat the identity baseline. The diagnosis had two parts, and v0.2 fixes each one:
Consecutive frames were nearly identical. The blob moves about a pixel per step, so "predict no change" was already an excellent predictor. Fix: the swift environment preset (
PointMassConfig.swift()), with a largerdtand acceleration so the agent moves several pixels per frame, plus speed-proportional drag so the dynamics are non-linear.A single frame hides velocity. The dynamics are second-order, so no single-frame predictor can recover the next state, and the convolutional encoder exploited this by temporal smoothing: it mapped consecutive frames to nearly identical latents. The dataset-wide variance penalty does not forbid that, and more training made it worse (MSE ratio from 0.89 to 0.94). Fixes: two-frame stacking, which makes velocity observable from pixels (the same reason pixel world models from DQN to V-JEPA take clips instead of single frames), and a delta-variance penalty, a VICReg-style hinge on the std of
z_{t+1} - z_tthat rules out the temporal collapse.
Results for the committed checkpoint_image.pt (~735 KB) on heldout data,
enforced as tests in tests/test_gates_image.py:
Pixel gate | Metric | Result |
(d) Image predictor beats no-op | heldout next-latent MSE ratio vs identity | 0.11 (vector model: 0.17) |
(e) Latent not collapsed, incl. temporally | per-dim std / one-step delta MSE | 1.10 / 1.08 (pre-fix delta was 0.012) |
(f) Pixel MPC beats random | goal-reach rate + median steps over 20 seeds | 20/20, median 9.5 steps vs 27.5 random |
(g) Imagination useful at horizon | compounded H-step rollout MSE ratio | 0.45 at H=4, 0.86 at H=12 |
One negative result is measured but deliberately left out of the gates: a
privileged linear-dynamics MPC that reads the true 4D state still reaches the
goal about twice as fast as the pixel planner (median ~5 vs ~9.5 steps).
Planning from pixels has not caught up with planning from privileged state, and
gate (g) shows open-loop imagination degrading by horizon 12, which is why the
planner replans every step. The baselines are in oneiros/baselines.py; to
reproduce, use the image-training command under Run.

The agent drives the point-mass to the goal (green star) using only the world-model planning interface.
Latent prediction error | Planning success |
|
|
Scope and limits
Oneiros is an end-to-end demonstration that (1) a non-trivial latent dynamics model can be learned without collapse and without reconstruction, and (2) planning in that latent space solves a control task far better than chance, all behind an agentic tool interface.
It does not show any of this at scale. The environments are toy 2D point-masses
and the models are tiny (a few hundred KB). The default committed model uses a
vector-state observation on the original environment. The committed image
model (v0.2, checkpoint_image.pt) sees stacked 32x32 frames on the swift
environment and clears its own four gates above. The scale-up is still future
work: a frozen pretrained perception encoder such as
V-JEPA 2 with a learned latent dynamics head on
top, which is the recipe this toy mirrors. That step needs a GPU and pretrained
weights, and this artifact is CPU-only on purpose.
Install
Requires Python 3.12. A project-local virtual environment is recommended.
python -m venv .venv
# Windows: .venv\Scripts\activate | Unix: source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -e ".[dev]"torch is installed from the CPU wheel index, so no GPU is needed or used.
Run
# Train the JEPA world model (writes oneiros/checkpoint.pt). ~21s on CPU.
python -m oneiros.train --obs-mode vector --steps 4000
# Train the image world model on the swift environment (~3 min on CPU).
# This is the exact recipe of the committed checkpoint: two-frame stacking,
# the delta-variance penalty, and an explicit --checkpoint so the vector
# model's default path is not clobbered.
python -m oneiros.train --obs-mode image --env swift --steps 6000 \
--frame-stack 2 --delta-var-weight 25 --checkpoint oneiros/checkpoint_image.pt
# Run the scripted agent: drive to the goal via latent MPC, write the GIF +
# diagnostic plots to assets/.
python -m oneiros.demo_agent --seed 0 --k 20
# Tests, including the seven honesty gates (uses the committed checkpoints).
pytest -q
# Lint.
ruff check oneiros testsMCP server
The world model is exposed over MCP as a stdio server:
# vector model (default)
python -m oneiros.mcp_server
# the committed image model: stacked-frame observations, swift environment
python -m oneiros.mcp_server --obs-mode image
# any other checkpoint
python -m oneiros.mcp_server --checkpoint /path/to/checkpoint.pt--obs-mode selects which committed world model the server exposes. The
environment tools serve observations in the loaded model's format (6D vectors,
or the stacked rendered frames the image model trains on), and reset_env
returns a ready-made goal_observation for the planning tools. The served
environment always uses the configuration the checkpoint was trained on (the
swift preset for the image model).
Tools:
Tool | Purpose |
| observation -> latent |
| roll latent dynamics |
| latent-space MPC; returns the next action toward a goal |
| full best action sequence + the imagined latent path |
| loaded model's obs mode, dims, and honesty-gate metrics |
| drive the point-mass environment; a |
To register the server with Claude Desktop, add this to
claude_desktop_config.json (use absolute paths for your checkout):
{
"mcpServers": {
"oneiros": {
"command": "C:/path/to/Oneiros/.venv/Scripts/python.exe",
"args": ["-m", "oneiros.mcp_server"],
"cwd": "C:/path/to/Oneiros"
}
}
}On Unix the command is .venv/bin/python. Append "--obs-mode", "image"
to args to serve the committed image world model instead of the vector one.
Repository layout
oneiros/
env.py # deterministic 2D point-mass environment (numpy)
model.py # JEPA encoder + predictor + VICReg regularizers
data.py # random-policy rollout replay buffer
train.py # JEPA training loop, evaluation, checkpoint I/O
planner.py # latent-space MPC (CEM / random shooting)
baselines.py # pixel episode runners + privileged linear-MPC baseline
mcp_server.py # MCP tools exposing the world model
demo_agent.py # scripted agent + diagnostics (GIF, plots)
checkpoint.pt # committed vector model (~240 KB)
checkpoint_image.pt # committed image model, swift env (~735 KB)
tests/ # determinism, shapes, and the seven honesty gates
assets/ # generated GIF and diagnostic figuresSee SYNERGY.md for how the same latent-dynamics idea connects to regime-aware modeling in time series.
License
MIT. See LICENSE.
Available Tools
5 toolsencode_observationA
Encode a 6D observation [x, y, vx, vy, goal_x, goal_y] into a latent vector.
Returns the latent embedding produced by the trained JEPA encoder. The latent is the space the world model predicts in; downstream tools operate on these latents rather than on raw observations.
Inputs: observation (required).
plan_to_goalA
Plan the next action that drives the agent toward a goal, via latent MPC.
Encodes the current and goal observations, searches action sequences by rolling the world model forward in latent space (cross-entropy method), and returns the first action of the best plan. Call repeatedly (replanning each step) to follow a receding-horizon trajectory to the goal.
Inputs: current_obs (required), goal_obs (required).
predict_rolloutA
Roll the learned latent dynamics forward from a latent over an action sequence.
Given a starting latent and a list of 2D acceleration actions, applies the predictor g step by step and returns the latent trajectory. This is the model imagining a future without touching the real environment.
Inputs: latent (required), actions (required).
reset_envA
Reset the point-mass environment to a deterministic start for the given seed.
Returns the initial observation, the agent state, and the goal position.
Inputs: seed.
step_envA
Apply a 2D acceleration action to the environment and advance one timestep.
Returns the new observation, reward (negative distance to goal), a done flag (agent reached the goal), and the distance to the goal.
Inputs: action (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 5 tools
Each tool has a clear, distinct purpose: encoding observations, planning to a goal, predicting rollouts, resetting the environment, and stepping the environment. No two tools overlap in function.
All tool names follow a consistent verb_noun pattern (encode_observation, plan_to_goal, predict_rollout, reset_env, step_env) and use snake_case uniformly.
With 5 tools, the server covers the core operations for a point-mass control domain with latent dynamics perfectly. The count is well-scoped and each tool earns its place.
The tool set covers the main workflow: observation encoding, planning, prediction, environment reset and stepping. A minor gap is the lack of a tool to directly set or inspect the goal, but it is implicitly covered via encode_observation and plan_to_goal.
Maintenance
Related MCP Connectors
Your personal world model for any agent.
Deterministic MiniMindsLab utilities for AI agents over MCP.
Goal and task planning MCP for Codex and AI agents, with evidence-backed completion.
Daily puzzle arena for AI agents over MCP, ranked by step-efficiency on a public leaderboard.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceA domain-agnostic MCP server for autonomous experimentation, generalizing Karpathy's autoresearch pattern into a reusable server that any AI agent can drive, pointed at any domain defined by a JSON configuration.3Apache 2.0
- AlicenseNot gradedqualityDmaintenanceExposes MuJoCo physics simulation to AI assistants via 65 MCP tools, enabling natural language control of robotics simulation, trajectory optimization, contact analysis, and video export.12MIT
- AlicenseAqualityDmaintenanceEnables AI agents to interactively explore PDDL planning problems by exposing a PDDL engine as MCP tools for initialization, action execution, state inspection, and goal checking.104Apache 2.0
- AlicenseNot gradedqualityDmaintenanceAn MCP server that enables LLM agents to control a two-joint MuJoCo robot arm through tools for moving, reading state, and resetting. It deliberately omits inverse kinematics, so the agent must infer joint angles from observations.MIT

