This server enables agents to interact with a 2D point-mass environment by utilizing a JEPA-style latent world model for planning and prediction.
Encode high-dimensional observations (like images or 6D vectors) into a compact latent vector.
Simulate the future state of the environment by projecting actions forward in the latent space (rollouts).
Determine the optimal next action toward a goal using latent-space Model Predictive Control (MPC).
Control the actual point-mass environment by executing actions and resetting the simulation.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Oneiros MCP Server plan a path to the goal at (5,3)"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Status: complete research artifact with verified results. v0.2 adds image-based perception, backed by the four pixel gates below; v0.1 had listed it as the next step. The V-JEPA 2 scale-up is still future work because it needs a GPU and pretrained weights.
A JEPA-style latent world model that an agent calls over MCP to plan.
Oneiros is a small, CPU-only research artifact that combines agentic systems with world models. It trains a Joint Embedding Predictive Architecture (JEPA) world model on a 2D point-mass environment, then exposes that model as a set of Model Context Protocol tools. Agent tools are usually hand-written functions that an LLM calls. Here the tool is the learned predictive world model: the agent encodes observations into latents, rolls the latent dynamics forward, and runs model-predictive control entirely in latent space.
A JEPA predicts the future in a learned latent space, not in pixels or tokens.
Given an observation o_t and an action a_t, it learns an encoder f and a
predictor g such that g(f(o_t), a_t) matches f(o_{t+1}). There is no
decoder and no pixel-reconstruction loss. A reconstruction loss spends capacity
on detail that is perceptually salient but irrelevant to control (texture,
lighting, background), while a latent predictor can discard anything that does
not help it anticipate the future. The price is a known failure mode: latent
prediction can collapse to a constant, with every observation mapped to the
same point so that prediction is trivially perfect. Oneiros prevents collapse
with an EMA target encoder, stop-gradients, and a VICReg-style variance and
covariance penalty, then plans with the resulting latent dynamics.
Related MCP server: MuJoCo MCP Server
flowchart LR
subgraph Env["Point-mass environment (numpy)"]
O["obs o_t"]
ON["obs o_t+1"]
end
subgraph WM["JEPA world model (torch, CPU)"]
F["encoder f"]
FT["EMA target f_target<br/>(stop-grad)"]
G["predictor g"]
O --> F --> Z["latent z_t"]
Z --> G
A["action a_t"] --> G
G --> ZH["z_hat_t+1"]
ON --> FT --> ZT["z_t+1 (target)"]
ZH -. "MSE + VICReg<br/>variance/covariance" .-> ZT
end
subgraph MCP["MCP server (agentic interface)"]
T1["encode_observation"]
T2["predict_rollout"]
T3["plan_to_goal"]
T4["reset_env / step_env"]
end
subgraph Agent["Agent loop"]
P["latent MPC planner<br/>(CEM over g)"]
end
WM --> MCP
MCP <--> Agent
P -->|"first action"| EnvThe agent never sees the environment's dynamics. It calls plan_to_goal, which
encodes the current and goal observations, searches action sequences by rolling
the predictor g forward H steps in latent space (cross-entropy method),
scores each candidate by predicted-latent distance to the goal, and returns the
first action. The planner replans every step (receding-horizon MPC).
The numbers below come from one run on my machine (CPU only, seed 0). Train
with python -m oneiros.train and reproduce the diagnostics with
python -m oneiros.demo_agent.
Honesty gate | Metric | Result |
(a) Predictor beats no-op baseline | next-latent MSE vs identity baseline | 0.0185 vs 0.1076 (ratio 0.17, ~5.8x better) |
(b) Latent not collapsed | per-dim latent std (mean / min) | 1.04 / 1.00 (threshold 0.1) |
(c) MPC beats random | goal-reaching success over 20 seeds | MPC 95% vs random 15-20% |
Training takes about 21 seconds for 4000 steps. The checkpoint
(oneiros/checkpoint.pt, ~240 KB) is committed so the demo, MCP server, and
planning tests run without retraining.
In v0.1 the image encoder could not beat the identity baseline. The diagnosis had two parts, and v0.2 fixes each one:
Consecutive frames were nearly identical. The blob moves about a pixel per
step, so "predict no change" was already an excellent predictor. Fix: the
swift environment preset (PointMassConfig.swift()), with a larger dt and
acceleration so the agent moves several pixels per frame, plus
speed-proportional drag so the dynamics are non-linear.
A single frame hides velocity. The dynamics are second-order, so no
single-frame predictor can recover the next state, and the convolutional
encoder exploited this by temporal smoothing: it mapped consecutive frames
to nearly identical latents. The dataset-wide variance penalty does not
forbid that, and more training made it worse (MSE ratio from 0.89 to 0.94).
Fixes: two-frame stacking, which makes velocity observable from pixels (the
same reason pixel world models from DQN to V-JEPA take clips instead of
single frames), and a delta-variance penalty, a VICReg-style hinge on the
std of z_{t+1} - z_t that rules out the temporal collapse.
Results for the committed checkpoint_image.pt (~735 KB) on heldout data,
enforced as tests in tests/test_gates_image.py:
Pixel gate | Metric | Result |
(d) Image predictor beats no-op | heldout next-latent MSE ratio vs identity | 0.11 (vector model: 0.17) |
(e) Latent not collapsed, incl. temporally | per-dim std / one-step delta MSE | 1.10 / 1.08 (pre-fix delta was 0.012) |
(f) Pixel MPC beats random | goal-reach rate + median steps over 20 seeds | 20/20, median 9.5 steps vs 27.5 random |
(g) Imagination useful at horizon | compounded H-step rollout MSE ratio | 0.45 at H=4, 0.86 at H=12 |
One negative result is measured but deliberately left out of the gates: a
privileged linear-dynamics MPC that reads the true 4D state still reaches the
goal about twice as fast as the pixel planner (median ~5 vs ~9.5 steps).
Planning from pixels has not caught up with planning from privileged state, and
gate (g) shows open-loop imagination degrading by horizon 12, which is why the
planner replans every step. The baselines are in oneiros/baselines.py; to
reproduce, use the image-training command under Run.
The agent drives the point-mass to the goal (green star) using only the world-model planning interface.
Latent prediction error | Planning success |
|
|
Oneiros is an end-to-end demonstration that (1) a non-trivial latent dynamics model can be learned without collapse and without reconstruction, and (2) planning in that latent space solves a control task far better than chance, all behind an agentic tool interface.
It does not show any of this at scale. The environments are toy 2D point-masses
and the models are tiny (a few hundred KB). The default committed model uses a
vector-state observation on the original environment. The committed image
model (v0.2, checkpoint_image.pt) sees stacked 32x32 frames on the swift
environment and clears its own four gates above. The scale-up is still future
work: a frozen pretrained perception encoder such as
V-JEPA 2 with a learned latent dynamics head on
top, which is the recipe this toy mirrors. That step needs a GPU and pretrained
weights, and this artifact is CPU-only on purpose.
Requires Python 3.12. A project-local virtual environment is recommended.
python -m venv .venv
# Windows: .venv\Scripts\activate | Unix: source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -e ".[dev]"torch is installed from the CPU wheel index, so no GPU is needed or used.
# Train the JEPA world model (writes oneiros/checkpoint.pt). ~21s on CPU.
python -m oneiros.train --obs-mode vector --steps 4000
# Train the image world model on the swift environment (~3 min on CPU).
# This is the exact recipe of the committed checkpoint: two-frame stacking,
# the delta-variance penalty, and an explicit --checkpoint so the vector
# model's default path is not clobbered.
python -m oneiros.train --obs-mode image --env swift --steps 6000 \
--frame-stack 2 --delta-var-weight 25 --checkpoint oneiros/checkpoint_image.pt
# Run the scripted agent: drive to the goal via latent MPC, write the GIF +
# diagnostic plots to assets/.
python -m oneiros.demo_agent --seed 0 --k 20
# Tests, including the seven honesty gates (uses the committed checkpoints).
pytest -q
# Lint.
ruff check oneiros testsThe world model is exposed over MCP as a stdio server:
# vector model (default)
python -m oneiros.mcp_server
# the committed image model: stacked-frame observations, swift environment
python -m oneiros.mcp_server --obs-mode image
# any other checkpoint
python -m oneiros.mcp_server --checkpoint /path/to/checkpoint.pt--obs-mode selects which committed world model the server exposes. The
environment tools serve observations in the loaded model's format (6D vectors,
or the stacked rendered frames the image model trains on), and reset_env
returns a ready-made goal_observation for the planning tools. The served
environment always uses the configuration the checkpoint was trained on (the
swift preset for the image model).
Tools:
Tool | Purpose |
| observation -> latent |
| roll latent dynamics |
| latent-space MPC; returns the next action toward a goal |
| full best action sequence + the imagined latent path |
| loaded model's obs mode, dims, and honesty-gate metrics |
| drive the point-mass environment; a |
To register the server with Claude Desktop, add this to
claude_desktop_config.json (use absolute paths for your checkout):
{
"mcpServers": {
"oneiros": {
"command": "C:/path/to/Oneiros/.venv/Scripts/python.exe",
"args": ["-m", "oneiros.mcp_server"],
"cwd": "C:/path/to/Oneiros"
}
}
}On Unix the command is .venv/bin/python. Append "--obs-mode", "image"
to args to serve the committed image world model instead of the vector one.
oneiros/
env.py # deterministic 2D point-mass environment (numpy)
model.py # JEPA encoder + predictor + VICReg regularizers
data.py # random-policy rollout replay buffer
train.py # JEPA training loop, evaluation, checkpoint I/O
planner.py # latent-space MPC (CEM / random shooting)
baselines.py # pixel episode runners + privileged linear-MPC baseline
mcp_server.py # MCP tools exposing the world model
demo_agent.py # scripted agent + diagnostics (GIF, plots)
checkpoint.pt # committed vector model (~240 KB)
checkpoint_image.pt # committed image model, swift env (~735 KB)
tests/ # determinism, shapes, and the seven honesty gates
assets/ # generated GIF and diagnostic figuresSee SYNERGY.md for how the same latent-dynamics idea connects to regime-aware modeling in time series.
MIT. See LICENSE.
Encode a 6D observation [x, y, vx, vy, goal_x, goal_y] into a latent vector.
Returns the latent embedding produced by the trained JEPA encoder. The latent is the space the world model predicts in; downstream tools operate on these latents rather than on raw observations.
| Name | Required | Description | Default |
|---|---|---|---|
| observation | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses the use of a trained JEPA encoder and explains the latent space role, but omits side effects, idempotency, or authorization needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two efficient sentences: first for purpose, second for context. No redundancy, well front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers core functionality and output purpose well, but lacks detail on latent vector dimensions or type, which would be helpful given no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema only specifies an array of numbers with 0% description coverage. The description compensates fully by defining the 6D structure and field meanings, adding significant value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it encodes a 6D observation into a latent vector, specifying the exact fields [x, y, vx, vy, goal_x, goal_y]. It differentiates itself from siblings by explaining that the latent is used by the world model and downstream tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context (encoding before world model prediction) but does not explicitly state when to use this tool vs alternatives like plan_to_goal or predict_rollout. No when-not or exclusion criteria provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Plan the next action that drives the agent toward a goal, via latent MPC.
Encodes the current and goal observations, searches action sequences by rolling the world model forward in latent space (cross-entropy method), and returns the first action of the best plan. Call repeatedly (replanning each step) to follow a receding-horizon trajectory to the goal.
| Name | Required | Description | Default |
|---|---|---|---|
| current_obs | Yes | ||
| goal_obs | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses the algorithmic approach (latent MPC, cross-entropy method) and that it returns the first action of the best plan. No annotations are provided, but the description adequately covers the tool's behavior and expected output.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise: three sentences covering purpose, method, and usage. Every sentence adds value with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of annotations and output schema, the description sufficiently explains what the tool does and how to use it. It could be slightly improved by mentioning the return format (e.g., action shape), but it is largely complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% description coverage, but the description clarifies that current_obs is the current observation and goal_obs is the goal observation. This adds necessary context beyond the raw array type.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: planning the next action toward a goal via latent MPC. It explains the method (encoding observations, searching action sequences) and distinguishes it from sibling tools like encode_observation and predict_rollout.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises calling the tool repeatedly for receding-horizon control. While it doesn't list when not to use it or compare to alternatives, the replanning guidance is clear and practical.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Roll the learned latent dynamics forward from a latent over an action sequence.
Given a starting latent and a list of 2D acceleration actions, applies the predictor g step by step and returns the latent trajectory. This is the model imagining a future without touching the real environment.
| Name | Required | Description | Default |
|---|---|---|---|
| latent | Yes | ||
| actions | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries burden. It discloses that it is a simulation using predictor g, no real environment interaction. Lacks details on side effects or error handling.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with main purpose, efficient and no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema or annotations, description is fairly complete: explains input, process, and output. Could specify output structure more.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but description explains latent is starting state and actions are list of 2D acceleration actions, adding meaning beyond schema types.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it rolls latent dynamics forward from a latent over an action sequence, using specific verbs and resources. It distinguishes from sibling tools (e.g., step_env interacts with environment).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for simulation without environment interaction, but does not explicitly state when not to use or compare to alternatives like plan_to_goal.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Reset the point-mass environment to a deterministic start for the given seed.
Returns the initial observation, the agent state, and the goal position.
| Name | Required | Description | Default |
|---|---|---|---|
| seed | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears full burden. It discloses return values (initial observation, agent state, goal position) and mentions deterministic start, but does not explicitly state that the tool destroys the current environment state or any side effects. Some behavioral context is given, but more would improve transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with no unnecessary words. It front-loads the main action ('Reset the point-mass environment') and concisely details return values. Every sentence is purposeful.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with one parameter and no output schema or annotations, the description is nearly complete: it explains the purpose, the role of the seed, and the return values. It could mention that it resets the environment to its initial state, but overall it's adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% coverage, meaning no parameter descriptions. The description adds meaning by stating that the seed controls the deterministic start, clarifying the parameter's role beyond the default value and type in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Reset' and the resource 'point-mass environment', specifying it resets to a deterministic start for a given seed. This distinguishes it from sibling tools like step_env (which advances the environment) and encode_observation (which encodes observations).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use this tool (to reset to a deterministic start with a seed) but provides no explicit guidance on when not to use it or alternatives among siblings. The context is clear but lacks exclusions or comparative context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Apply a 2D acceleration action to the environment and advance one timestep.
Returns the new observation, reward (negative distance to goal), a done flag (agent reached the goal), and the distance to the goal.
| Name | Required | Description | Default |
|---|---|---|---|
| action | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must convey behavioral traits. It discloses return values but fails to mention side effects (e.g., environment state mutation), safety concerns, or requirements (e.g., prior reset). For a step function, critical behavioral context is missing.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no filler. The first sentence states the core action, the second lists the return values. Information is front-loaded and every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the return values, but lacks details about the environment's state, action effects, and termination conditions beyond 'agent reached the goal'. Given no output schema, more context (e.g., observation structure) would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It adds meaning by specifying '2D acceleration action', suggesting the action array has two numeric components. However, it does not specify the exact dimensions, allowed ranges, or semantics of the acceleration, leaving some ambiguity.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool applies a 2D acceleration action and advances one timestep, and lists return values. This distinguishes it from siblings (reset_env, encode_observation, etc.) by specifying the verb 'apply' and the resource 'environment', making the purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for stepping the environment, but does not provide explicit guidance on when to use this tool versus alternatives like predict_rollout or when not to use it. No exclusions or prerequisites are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0encode_observationplan_to_goalpredict_rolloutreset_envstep_envScored across 5 tools
Each tool has a clear, distinct purpose: encoding observations, planning to a goal, predicting rollouts, resetting the environment, and stepping the environment. No two tools overlap in function.
All tool names follow a consistent verb_noun pattern (encode_observation, plan_to_goal, predict_rollout, reset_env, step_env) and use snake_case uniformly.
With 5 tools, the server covers the core operations for a point-mass control domain with latent dynamics perfectly. The count is well-scoped and each tool earns its place.
The tool set covers the main workflow: observation encoding, planning, prediction, environment reset and stepping. A minor gap is the lack of a tool to directly set or inspect the goal, but it is implicitly covered via encode_observation and plan_to_goal.
Your personal world model for any agent.
Deterministic MiniMindsLab utilities for AI agents over MCP.
Goal and task planning MCP for Codex and AI agents, with evidence-backed completion.
Read-only MCP server for the OPERANT AI operating-agent calibration benchmark.
