The Oneiros MCP Server exposes a JEPA-style latent world model as agentic tools, enabling planning and navigation in a 2D point-mass environment via learned latent space.
encode_observation: Convert a 6D observation vector [x, y, vx, vy, goal_x, goal_y] into a compact latent embedding using the trained JEPA encoder.
predict_rollout: Given a starting latent and a sequence of 2D acceleration actions, roll the learned latent dynamics forward step-by-step to imagine future trajectories — without touching the real environment.
plan_to_goal: Use latent-space Model Predictive Control (MPC via the Cross-Entropy Method) to search over action sequences, score them by predicted latent distance to the goal, and return the best next action. Call repeatedly each step for receding-horizon planning.
reset_env: Initialize the 2D point-mass environment to a deterministic starting state (with an optional seed), returning the initial observation and goal position.
step_env: Apply a 2D acceleration action to advance the environment one timestep, returning the new observation, reward (negative distance to goal), a done flag, and current distance to goal.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Oneiros MCP Server plan a path to the goal at (5,3)"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Status: complete research artifact — results verified. v0.2 adds image-based perception as a demonstrated result (the four pixel gates below), closing v0.1's documented next step; the V-JEPA 2 scale-up remains future work (needs GPU + pretrained weights).
A JEPA-style latent world model that an agent calls as a tool — over MCP — to plan.
Oneiros is a small, CPU-only research artifact at the intersection of agentic systems and world models. It trains a Joint Embedding Predictive Architecture (JEPA) world model on a 2D point-mass environment, then exposes that model as a set of Model Context Protocol tools. An agent plans by calling the learned predictive world model as a tool — encoding observations into latents, rolling latent dynamics forward, and running model-predictive control entirely in latent space — rather than the usual pattern of an LLM calling hand-written functions.
A JEPA predicts the future in a learned latent space, not in pixels or
tokens. Given an observation o_t and an action a_t, it learns an encoder
f and a predictor g such that g(f(o_t), a_t) matches f(o_{t+1}) — there
is no decoder and no pixel-reconstruction loss. This matters because
reconstruction wastes capacity modeling perceptually salient but
control-irrelevant detail (texture, lighting, background), while a latent
predictor is free to discard everything that does not help it anticipate the
future. The cost is a well-known failure mode: latent prediction can collapse to
a constant (every observation maps to the same point, making prediction
trivially perfect). Oneiros defeats collapse with an EMA target encoder,
stop-gradients, and a VICReg-style variance + covariance penalty, and then uses
the resulting latent dynamics for planning.
Related MCP server: MuJoCo MCP Server
flowchart LR
subgraph Env["Point-mass environment (numpy)"]
O["obs o_t"]
ON["obs o_t+1"]
end
subgraph WM["JEPA world model (torch, CPU)"]
F["encoder f"]
FT["EMA target f_target<br/>(stop-grad)"]
G["predictor g"]
O --> F --> Z["latent z_t"]
Z --> G
A["action a_t"] --> G
G --> ZH["z_hat_t+1"]
ON --> FT --> ZT["z_t+1 (target)"]
ZH -. "MSE + VICReg<br/>variance/covariance" .-> ZT
end
subgraph MCP["MCP server (agentic interface)"]
T1["encode_observation"]
T2["predict_rollout"]
T3["plan_to_goal"]
T4["reset_env / step_env"]
end
subgraph Agent["Agent loop"]
P["latent MPC planner<br/>(CEM over g)"]
end
WM --> MCP
MCP <--> Agent
P -->|"first action"| EnvThe agent never sees the environment's dynamics. It calls plan_to_goal, which
encodes the current and goal observations, searches action sequences by rolling
the predictor g forward H steps in latent space (cross-entropy method),
scores each candidate by predicted-latent distance to the goal, and returns the
first action. The planner replans every step (receding-horizon MPC).
Numbers below are from an actual run on this machine (CPU only, seed 0). Train
with python -m oneiros.train and reproduce the diagnostics with
python -m oneiros.demo_agent.
Honesty gate | Metric | Result |
(a) Predictor beats no-op baseline | next-latent MSE vs identity baseline | 0.0185 vs 0.1076 (ratio 0.17 — ~5.8x better) |
(b) Latent not collapsed | per-dim latent std (mean / min) | 1.04 / 1.00 (threshold 0.1) |
(c) MPC beats random | goal-reaching success over 20 seeds | MPC 95% vs random 15-20% |
Training takes about 21 seconds for 4000 steps. The checkpoint
(oneiros/checkpoint.pt, ~240 KB) is committed so the demo, MCP server, and
planning tests run without retraining.
v0.1 documented why the image encoder could not beat the identity baseline. The diagnosis had two parts, and each got a principled fix rather than a knob-twiddle:
Consecutive frames were nearly identical (the blob moves ~a pixel per
step), so "predict no change" was already an excellent predictor. Fix: the
swift environment preset (PointMassConfig.swift()) — larger dt and
acceleration so the agent moves several pixels per frame, plus
speed-proportional drag so the dynamics are genuinely non-linear.
A single frame hides velocity — the dynamics are second-order, so no
single-frame predictor can recover the next state, and the convolutional
encoder exploited this by temporal smoothing (mapping consecutive frames
to nearly identical latents; the dataset-wide variance penalty does not
forbid it — more training made it worse, 0.89 → 0.94 MSE ratio). Fixes:
two-frame stacking (velocity becomes observable from pixels, the same
reason pixel world models from DQN to V-JEPA consume clips, not stills)
and a delta-variance penalty (VICReg-style hinge on the std of
z_{t+1} - z_t) that forbids the temporal collapse outright.
Results from the committed checkpoint_image.pt (~735 KB), heldout data,
enforced as tests in tests/test_gates_image.py:
Pixel gate | Metric | Result |
(d) Image predictor beats no-op | heldout next-latent MSE ratio vs identity | 0.11 (vector model: 0.17) |
(e) Latent not collapsed, incl. temporally | per-dim std / one-step delta MSE | 1.10 / 1.08 (pre-fix delta was 0.012) |
(f) Pixel MPC beats random | goal-reach rate + median steps over 20 seeds | 20/20, median 9.5 steps vs 27.5 random |
(g) Imagination useful at horizon | compounded H-step rollout MSE ratio | 0.45 at H=4, 0.86 at H=12 |
One honest negative, measured and deliberately not gated: a privileged
linear-dynamics MPC reading the true 4D state still reaches the goal about
twice as fast as the pixel planner (median ~5 vs ~9.5 steps). Planning from
pixels has not caught planning from privileged state, and gate (g) shows
open-loop imagination degrading by horizon 12 — which is exactly why the
planner replans every step. The baselines live in oneiros/baselines.py;
reproduce with the image-training command under Run.
The agent drives the point-mass to the goal (green star) using only the world-model planning interface.
Latent prediction error | Planning success |
|
|
This is a genuine, end-to-end demonstration that (1) a non-trivial latent dynamics model can be learned without collapse and without reconstruction, and (2) planning in that latent space solves a control task far better than chance, all behind an agentic tool interface.
It is not at scale. The environments are toy 2D point-masses, the models
are tiny (a few hundred KB). The default committed model uses a
vector-state observation on the original environment; the committed
image model (v0.2, checkpoint_image.pt) perceives stacked 32x32 frames
on the swift environment and clears its own four gates above — v0.1's
"image-based perception is the documented next step" is now a demonstrated
result, with the diagnosis (frame similarity + hidden velocity + temporal
smoothing) and fixes documented rather than hand-waved. What remains future
work is the scale-up: a frozen pretrained perception encoder such as
V-JEPA 2 with a learned latent dynamics head on
top, exactly the recipe this toy mirrors — that step needs a GPU and
pretrained weights, which this CPU-only artifact deliberately does not
assume.
Requires Python 3.12. A project-local virtual environment is recommended.
python -m venv .venv
# Windows: .venv\Scripts\activate | Unix: source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -e ".[dev]"torch is installed from the CPU wheel index — no GPU is needed or used.
# Train the JEPA world model (writes oneiros/checkpoint.pt). ~21s on CPU.
python -m oneiros.train --obs-mode vector --steps 4000
# Train the image world model on the swift environment (~3 min on CPU).
# This is the exact recipe of the committed checkpoint: two-frame stacking,
# the delta-variance penalty, and an explicit --checkpoint so the vector
# model's default path is not clobbered.
python -m oneiros.train --obs-mode image --env swift --steps 6000 \
--frame-stack 2 --delta-var-weight 25 --checkpoint oneiros/checkpoint_image.pt
# Run the scripted agent: drive to the goal via latent MPC, write the GIF +
# diagnostic plots to assets/.
python -m oneiros.demo_agent --seed 0 --k 20
# Tests, including the three honesty gates (uses the committed checkpoint).
pytest -q
# Lint.
ruff check oneiros testsThe world model is exposed over MCP as a stdio server:
# vector model (default)
python -m oneiros.mcp_server
# the committed image model: stacked-frame observations, swift environment
python -m oneiros.mcp_server --obs-mode image
# any other checkpoint
python -m oneiros.mcp_server --checkpoint /path/to/checkpoint.pt--obs-mode selects which committed world model the server exposes. The
environment tools serve observations in the loaded model's format — 6D
vectors, or the stacked rendered frames the image model trains on — and
reset_env returns a ready-made goal_observation for the planning tools.
The served environment always uses the configuration the checkpoint was
trained on (the swift preset for the image model).
Tools:
Tool | Purpose |
| observation -> latent |
| roll latent dynamics |
| latent-space MPC; returns the next action toward a goal |
| full best action sequence + the imagined latent path |
| loaded model's obs mode, dims, and honesty-gate metrics |
| drive the point-mass environment; a |
To register the server with Claude Desktop, add this to
claude_desktop_config.json (use absolute paths for your checkout):
{
"mcpServers": {
"oneiros": {
"command": "C:/path/to/Oneiros/.venv/Scripts/python.exe",
"args": ["-m", "oneiros.mcp_server"],
"cwd": "C:/path/to/Oneiros"
}
}
}On Unix the command is .venv/bin/python. Append "--obs-mode", "image"
to args to serve the committed image world model instead of the vector one.
oneiros/
env.py # deterministic 2D point-mass environment (numpy)
model.py # JEPA encoder + predictor + VICReg regularizers
data.py # random-policy rollout replay buffer
train.py # JEPA training loop, evaluation, checkpoint I/O
planner.py # latent-space MPC (CEM / random shooting)
baselines.py # pixel episode runners + privileged linear-MPC baseline
mcp_server.py # MCP tools exposing the world model
demo_agent.py # scripted agent + diagnostics (GIF, plots)
checkpoint.pt # committed vector model (~240 KB)
checkpoint_image.pt # committed image model, swift env (~735 KB)
tests/ # determinism, shapes, and the three honesty gates
assets/ # generated GIF and diagnostic figuresSee SYNERGY.md for how the same latent-dynamics idea connects to regime-aware modeling in time series.
MIT — see LICENSE.
Encode a 6D observation [x, y, vx, vy, goal_x, goal_y] into a latent vector.
Returns the latent embedding produced by the trained JEPA encoder. The latent is the space the world model predicts in; downstream tools operate on these latents rather than on raw observations.
| Name | Required | Description | Default |
|---|---|---|---|
| observation | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses the use of a trained JEPA encoder and explains the latent space role, but omits side effects, idempotency, or authorization needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two efficient sentences: first for purpose, second for context. No redundancy, well front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers core functionality and output purpose well, but lacks detail on latent vector dimensions or type, which would be helpful given no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema only specifies an array of numbers with 0% description coverage. The description compensates fully by defining the 6D structure and field meanings, adding significant value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it encodes a 6D observation into a latent vector, specifying the exact fields [x, y, vx, vy, goal_x, goal_y]. It differentiates itself from siblings by explaining that the latent is used by the world model and downstream tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context (encoding before world model prediction) but does not explicitly state when to use this tool vs alternatives like plan_to_goal or predict_rollout. No when-not or exclusion criteria provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Plan the next action that drives the agent toward a goal, via latent MPC.
Encodes the current and goal observations, searches action sequences by rolling the world model forward in latent space (cross-entropy method), and returns the first action of the best plan. Call repeatedly (replanning each step) to follow a receding-horizon trajectory to the goal.
| Name | Required | Description | Default |
|---|---|---|---|
| current_obs | Yes | ||
| goal_obs | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses the algorithmic approach (latent MPC, cross-entropy method) and that it returns the first action of the best plan. No annotations are provided, but the description adequately covers the tool's behavior and expected output.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise: three sentences covering purpose, method, and usage. Every sentence adds value with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of annotations and output schema, the description sufficiently explains what the tool does and how to use it. It could be slightly improved by mentioning the return format (e.g., action shape), but it is largely complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% description coverage, but the description clarifies that current_obs is the current observation and goal_obs is the goal observation. This adds necessary context beyond the raw array type.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: planning the next action toward a goal via latent MPC. It explains the method (encoding observations, searching action sequences) and distinguishes it from sibling tools like encode_observation and predict_rollout.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises calling the tool repeatedly for receding-horizon control. While it doesn't list when not to use it or compare to alternatives, the replanning guidance is clear and practical.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Roll the learned latent dynamics forward from a latent over an action sequence.
Given a starting latent and a list of 2D acceleration actions, applies the predictor g step by step and returns the latent trajectory. This is the model imagining a future without touching the real environment.
| Name | Required | Description | Default |
|---|---|---|---|
| latent | Yes | ||
| actions | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries burden. It discloses that it is a simulation using predictor g, no real environment interaction. Lacks details on side effects or error handling.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with main purpose, efficient and no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema or annotations, description is fairly complete: explains input, process, and output. Could specify output structure more.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but description explains latent is starting state and actions are list of 2D acceleration actions, adding meaning beyond schema types.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it rolls latent dynamics forward from a latent over an action sequence, using specific verbs and resources. It distinguishes from sibling tools (e.g., step_env interacts with environment).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for simulation without environment interaction, but does not explicitly state when not to use or compare to alternatives like plan_to_goal.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Reset the point-mass environment to a deterministic start for the given seed.
Returns the initial observation, the agent state, and the goal position.
| Name | Required | Description | Default |
|---|---|---|---|
| seed | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears full burden. It discloses return values (initial observation, agent state, goal position) and mentions deterministic start, but does not explicitly state that the tool destroys the current environment state or any side effects. Some behavioral context is given, but more would improve transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with no unnecessary words. It front-loads the main action ('Reset the point-mass environment') and concisely details return values. Every sentence is purposeful.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with one parameter and no output schema or annotations, the description is nearly complete: it explains the purpose, the role of the seed, and the return values. It could mention that it resets the environment to its initial state, but overall it's adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% coverage, meaning no parameter descriptions. The description adds meaning by stating that the seed controls the deterministic start, clarifying the parameter's role beyond the default value and type in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Reset' and the resource 'point-mass environment', specifying it resets to a deterministic start for a given seed. This distinguishes it from sibling tools like step_env (which advances the environment) and encode_observation (which encodes observations).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use this tool (to reset to a deterministic start with a seed) but provides no explicit guidance on when not to use it or alternatives among siblings. The context is clear but lacks exclusions or comparative context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Apply a 2D acceleration action to the environment and advance one timestep.
Returns the new observation, reward (negative distance to goal), a done flag (agent reached the goal), and the distance to the goal.
| Name | Required | Description | Default |
|---|---|---|---|
| action | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must convey behavioral traits. It discloses return values but fails to mention side effects (e.g., environment state mutation), safety concerns, or requirements (e.g., prior reset). For a step function, critical behavioral context is missing.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no filler. The first sentence states the core action, the second lists the return values. Information is front-loaded and every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the return values, but lacks details about the environment's state, action effects, and termination conditions beyond 'agent reached the goal'. Given no output schema, more context (e.g., observation structure) would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It adds meaning by specifying '2D acceleration action', suggesting the action array has two numeric components. However, it does not specify the exact dimensions, allowed ranges, or semantics of the acceleration, leaving some ambiguity.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool applies a 2D acceleration action and advances one timestep, and lists return values. This distinguishes it from siblings (reset_env, encode_observation, etc.) by specifying the verb 'apply' and the resource 'environment', making the purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for stepping the environment, but does not provide explicit guidance on when to use this tool versus alternatives like predict_rollout or when not to use it. No exclusions or prerequisites are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v0.1.0encode_observationplan_to_goalpredict_rolloutreset_envstep_envEach tool has a clear, distinct purpose: encoding observations, planning to a goal, predicting rollouts, resetting the environment, and stepping the environment. No two tools overlap in function.
All tool names follow a consistent verb_noun pattern (encode_observation, plan_to_goal, predict_rollout, reset_env, step_env) and use snake_case uniformly.
With 5 tools, the server covers the core operations for a point-mass control domain with latent dynamics perfectly. The count is well-scoped and each tool earns its place.
The tool set covers the main workflow: observation encoding, planning, prediction, environment reset and stepping. A minor gap is the lack of a tool to directly set or inspect the goal, but it is implicitly covered via encode_observation and plan_to_goal.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
Deterministic MiniMindsLab utilities for AI agents over MCP.
Goal and task planning MCP for Codex and AI agents, with evidence-backed completion.
Read-only MCP server for the OPERANT AI operating-agent calibration benchmark.
Agent Replay Debugger MCP — record every agent step + deterministic replay. Step-debugger for
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/mal0ware/Oneiros'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
