Skip to main content
Glama
embodify-mcp

embodify-mcp

by YidaYang

First release (0.1.0a1). Supports LIBERO and RoboDojo today; support for RoboTwin and the LeRobot SO-101 arm is coming soon.

Why Embodify

Most embodied agents are built from scratch: a dedicated harness wraps a model, hands it a fixed set of robot actions, and nothing else. The agent you use every day already has what those harnesses lack:

  • Context and memory. It manages long sessions and remembers across them.

  • It knows you. Your preferences, your projects, your lab setup.

  • It talks with you in the terminal, IDE, desktop app or chat you already use. You can ask how it is going, step in, correct it, or teach it mid-task.

  • It has a computer. It writes and runs code, uses its tools, searches the web and reads papers.

Embodify keeps all of that and adds a body. Install the plugin and your agent gets robot observation and control tools over MCP — a protocol it already uses for everything else — plus skills that teach it how to operate robots and how to achieve recursive self-improvement (RSI) from its own experience.

Why "best"

A general agent with a body is stronger than a harness that can only move a robot:

  1. It can think with tools, not only act. When a task needs geometry it can write a script; when it needs a fact it can search; when it needs perception it can run a model.

  2. It keeps track. Long-horizon manipulation fails when the agent forgets what it already tried. Mature agents manage context and memory well.

  3. It works with you. It asks you when a task is ambiguous, takes your guidance when it gets stuck, and remembers your corrections next time.

  4. It controls robots in its native language. Robot control arrives as ordinary MCP tool calls, the same shape as every other tool the agent uses.

  5. It builds on frontier embodied models. Your agent runs on models with frontier embodied manipulation skills, such as GPT-6 Astra and Claude Opus 5.5, and every model upgrade makes your robot better, with no retraining.

  6. It improves itself recursively. It turns every episode into lessons, lessons into rules and rules into new skills, and gets better the more it works.

  7. Zero-shot, few-shot and in-context learning come easily. A general agent takes on new tasks without training: describe a task in plain language (zero-shot), show it a few examples (few-shot), or put instructions, demonstrations and past experience in its context (in-context learning, ICL).

Related MCP server: VELMA

What's inside

Embodify is an agent plugin with two parts, and installing the plugin gives your agent both:

Part

What it gives your agent

MCP server (embodify-mcp, registered as embodify)

Tools to list tasks, start an episode, observe cameras and robot state, move end effectors, open and close grippers, and coordinate two arms.

Skills

Know-how: running a careful observe–act loop, keeping a profile of the robot body and cameras, and learning from past episodes. More perception skills are coming soon.

flowchart LR
  A["Your agent<br/>Claude Code · Codex · …"] -- "MCP (stdio)" --> B["embodify-mcp<br/>episodes · budgets · logs"]
  K["Embodify skills"] -. "loaded by" .-> A
  B --> I["Backend interface"]
  I --> L["LIBERO"]
  I --> R["RoboDojo"]
  I --> T["RoboTwin (WIP)"]
  I --> H["SO-101 and other real robots (WIP)"]
  I -. "SSH / TCP" .-> G["Remote server"]

The MCP server is the front end. Each simulator, benchmark or robot is a backend behind one small interface, so adding a new one does not change what the agent sees. Highlights:

  • Works with any MCP host. No LLM calls inside; your agent keeps its own model, memory, skills and tools.

  • One call, one motion. The control loop runs next to the simulator. Each action returns the actual displacement, remaining error, a stop reason and fresh camera images.

  • Remote simulation. Run the simulator on your lab's GPU server over SSH while the agent stays on your laptop. Heartbeats keep high-latency links alive; if the link drops, the episode aborts cleanly and reset_task reconnects.

  • Fair evaluation. When you use Embodify to evaluate an agent's manipulation skills, task success is recorded for humans and never exposed to the agent.

  • Live monitor and replay. Watch running episodes live in a browser, or replay past ones frame by frame.

Backends

Backend

Robot

Status

fake, fake-two-arm

Kinematic diagnostic, one or two arms

✅ Included, no simulator needed

libero

Franka Panda, 130 tasks in 5 suites

✅ Included

robodojo

Dual ARX X5, all 54 simulation tasks

✅ Included

RoboTwin

Dual-arm manipulation benchmark

🚧 Coming soon

LeRobot SO-101

Real robot: 5-DoF arm with gripper

🚧 Coming soon

remote

Any backend above, on another machine

✅ SSH or trusted TCP

Coming soon: robot arms such as xArm, mobile manipulators, humanoids and dexterous hands. If there is a simulator or robot you would like us to support, open a GitHub issue with details about it. We welcome contributions of new backends.

For research

Embodify doubles as a testbed for the embodied manipulation abilities of language models and agent frameworks. Run GPT, Claude, Gemini, Qwen and other models, through Claude Code, Codex or your own agent framework, on the same simulated benchmarks with the same tools, and compare them directly.

  • Standard benchmarks: LIBERO (130 tasks in 5 suites) and RoboDojo (54 tasks), with RoboTwin coming soon.

  • Fair by design: task success is recorded for evaluators and never shown to the agent, and --lock-task fixes the scene.

  • Controlled budgets: episode budgets and per-call step limits are configurable and logged.

  • Complete records: tool calls, robot states and camera frames are saved for replay and analysis.

Typical studies include zero-shot robot manipulation by frontier models, comparisons of agent frameworks on the same model, the effect of skills and memory, and recursive self-improvement across episodes. See citation for how to cite Embodify and what to report.

Quick start

One-line setup

Paste this into Claude Code, Codex or any other agent you use:

Install Embodify (https://github.com/YidaYang/embodify) for yourself, then set up a LIBERO environment or connect to my existing simulator.

That's it. Your agent installs the Embodify plugin, sets up the simulator and tells you when to restart the session. (Agents: follow the agent setup guide.)

Manual install

Embodify starts its MCP server with uv, straight from PyPI. If you do not have uv yet, install it with python -m pip install uv or the official installer.

Claude Code

claude plugin marketplace add YidaYang/embodify
claude plugin install embodify@embodify

Codex

codex plugin marketplace add YidaYang/embodify
codex plugin add embodify@embodify

The plugin brings both the MCP server, registered as embodify, and the skills. The server starts on the Fake backend, so you can try the tools right away; connect a real simulator next. It reads its settings from ~/.embodify/config.json (see backend setup). To check the package on its own, without an agent:

uvx --from embodify-mcp embodify-mcp-smoke   # end-to-end check: no simulator, GPU or model needed

Other agents

  1. Register the server in your agent's MCP configuration using examples/mcp.json.

  2. If your agent supports Agent Skills (SKILL.md folders), copy or link the folders in plugin/skills/ into its skills directory.

Restart the session, then ask your agent something like "Reset the task, describe what the cameras show, then raise the gripper 5 cm."

Connect a real simulator

Let your agent do this step. Paste:

Connect Embodify to a real simulator for me: set up LIBERO on this machine, or connect to my existing simulator. Follow https://github.com/YidaYang/embodify/blob/main/docs/agent-setup.md.

Your agent installs the simulator or connects to yours over SSH, writes the server settings to ~/.embodify/config.json and asks you to restart the session. To do it by hand, see backend setup.

Watch robots live and replay episodes

Add "monitor-port": 8765 to ~/.embodify/config.json and open http://127.0.0.1:8765 to watch the robot work live, with every camera view, the robot state and each tool call your agent makes, or to replay any past episode frame by frame. To browse saved runs without a running server:

uvx --from embodify-mcp embodify-mcp-monitor   # reads ~/.embodify/runs

The monitor listens on localhost only.

MCP tools

Tool

Purpose

get_session_info

Current run, task, step budget and robot state; no images

list_tasks

Browse the backend's task catalogue

reset_task

Start an episode and return the first camera images

observe

Camera images and robot state, without moving

move_relative

Move one end effector by a translation and optional rotation

set_gripper

Open or close a gripper

control_arms

Move several arms together with common progress (two-arm backends)

stop_episode

End the episode and write its record

Translations are in meters, rotations in radians, quaternions in xyzw order. Frames and stop reasons are defined in the action contract.

Skills

The plugin contains these skills:

Skill

Status

What it does

embodied-control

v0

The observe–act loop: small moves, reading stop reasons, verifying grasps, releasing in a separate call

robot-profile

v0

Keep a profile of the robot: arms, cameras, frames, calibration and measured behavior

task-experience

v0

Write a lesson after every episode, read relevant lessons before the next, promote repeated lessons to rules

object-segmentation

Coming soon

Open-vocabulary segmentation of camera images

depth-ranging

Coming soon

Pixel to 3D position from depth and calibration

Skills are grouped into three families that will keep growing:

  1. Embodiment knowledge: how this particular robot is built, where its cameras are and how it actually moves.

  2. Manipulation tools: perception, measurement and planning tools, including ideas from work such as Code as Policies.

  3. Recursive self-improvement: turning experience into lessons, lessons into rules and eventually into new skills, improving recursively.

Roadmap

  • Backends: RoboTwin; LeRobot SO-101 as the first real robot, with workspace limits, human-judged success and an emergency stop; more real robots and simulators.

  • Embodiments: mobile manipulators, humanoids, dexterous hands.

  • Observations: depth images, camera calibration and more through the tools.

  • Skills: segmentation, depth ranging and other perception tools; code-as-policy style tools; a stronger recursive self-improvement loop, and more.

Development

python -m pip install -e ".[dev]"
python -m pytest -q
python tools/check_release.py

See contributing, writing a backend, backend setup, action contract and security.

License and citation

Original code is licensed under Apache-2.0. Bundled third-party code keeps its original notices. See citation for how to cite Embodify.

Available Tools

7 tools
get_session_infoA

Read the state of the current run: task context, instruction, step count and robot state. Renders no images and uses no step budget; callable at any time, including before the first reset_task. This only says where you are now. To see which tasks are available, use list_tasks. Returns run (short run ID), status (idle/running/stopped), step (simulation steps used), max_steps (the episode limit), steps_left (simulation steps left), task (suite, task_index, name, instruction, init_state_index), task_locked (true means the server fixed this run's scene, and reset_task accepts no scene arguments), and state (eef_pos_m is the XYZ position in meters, eef_rpy_rad the RPY orientation in radians, gripper holds command/opening_m). While status is idle there is no task or state; instead it returns default_scene (the scene a reset_task without arguments starts).

Inputs: no named parameters.

list_tasksA

List the tasks FakeSim offers. Independent of the current run: callable at any time, renders no images, uses no step budget and does not change a running episode. Without arguments: returns suites, the name and task count of every suite, plus the total task count. Start here, pick a suite, then look inside it. With suite: returns task_index, instruction and n_init_states (the number of initial layouts of the same task) for every task in that suite. Pass the chosen suite and task_index to reset_task to start.

Inputs: suite.

move_relativeA

Move the end effector by delta_xyz meters in the FakeSim world frame (z up, table surface at z=0.80 m), optionally rotating by delta_rpy about the robot base X/Y/Z axes (radians). The server approaches the target step by step with P control and returns only when a definite stop condition triggers, not after a fixed number of steps. action.stop_reason says why the call stopped, one of four:

  • reached: Within the target tolerance (8 mm in position, 0.02 rad in orientation); the motion is complete.

  • stalled: Barely moved for 3 consecutive steps (less than 1 mm in total) without reaching the target, which usually means something is blocking it: the table, an object, or a joint limit. Repeating the same command has no effect; change direction, or lift or back off first. This is the main signal of contact.

  • step_cap: Hit the per-call internal step limit while the end effector was still moving toward the target. Call again with the remaining displacement in the same direction to continue.

  • budget: The episode's total step budget ran out during the motion; only stop_episode can be called afterwards. Motion is a best-effort approach and may fall short of the requested delta: judge progress by action.achieved_delta_xyz (actual displacement of this call, meters) and action.remaining_distance_m (meters still missing to the requested target), and do not assume the request was achieved. Returns images (two separate PNGs), state (eef_pos_m is the end-effector XYZ position in meters, eef_rpy_rad the RPY orientation in radians, gripper holds command/opening_m), action (the requested delta_xyz/delta_rpy, executed_steps, stop_reason, achieved_delta_xyz, remaining_distance_m), step (simulation steps used after the call), steps_left (simulation steps left) and status (running). When the budget is already 0, returns a step_limit error instead of silently doing nothing.

Inputs: delta_xyz (required), delta_rpy.

observeA

Read the current RGB images from two cameras and the end-effector/gripper state, without advancing the simulation or using the step budget. Returns images (agentview and robot0_eye_in_hand: two separate PNGs), state (eef_pos_m is the end-effector XYZ position in meters, eef_rpy_rad the orientation about the base XYZ axes in radians, gripper holds the target command and opening_m in meters), step (simulation steps used), steps_left (simulation steps left) and status (running).

Inputs: no named parameters.

reset_taskA

Start a new FakeSim episode. All three arguments are optional: omit them all to use the server's default scene (usually task 0 of the suite with initial layout 0), which is the most common use. To switch tasks, use list_tasks first: without arguments it lists the suites, with suite it gives the task_index and instruction of every task in that suite; then pass the chosen suite / task_index. init_state_index picks another object layout of the same task. If the server was started with --lock-task (task_locked is true in get_session_info), passing any of these arguments returns a task_locked error. If an episode is already running, returns already_running; call stop_episode first. Returns run (short run ID), task (suite, task_index, name, instruction, init_state_index), images (agentview and robot0_eye_in_hand: two separate PNGs), state (eef_pos_m is the end-effector XYZ position in meters; eef_rpy_rad.roll_x, pitch_y and yaw_z are the orientation about the robot base X/Y/Z axes in radians; gripper.command is the open/close target and gripper.opening_m the gripper opening in meters), step (simulation steps used) and steps_left (simulation steps left in this episode).

Inputs: suite, task_index, init_state_index.

set_gripperA

Open or close the gripper. The server keeps sending the command and watches the opening width, and returns only when a definite stop condition triggers, not after a fixed number of steps. action.stop_reason says why the call stopped, one of four:

  • settled: After at least 10 internal steps of the same command, the opening varied by less than 0.5 mm over the last 2 steps. This only means the opening is momentarily stable, possibly because something blocks it; it does not guarantee a fully open gripper, a firm grasp or a released object. Check state.gripper.opening_m and the images.

  • step_cap: Hit the per-call internal step limit before the opening was confirmed stable; the gripper may still be starting or moving. Repeat the same command to continue.

  • budget: The episode's total step budget ran out during the action; only stop_episode can be called afterwards.

  • no_op: The request matches the current gripper target and no gripper call is pending; nothing was executed and no budget was used. It does not mean an object was released. Returns images (two separate PNGs), state (eef_pos_m is the end-effector XYZ position in meters, eef_rpy_rad the RPY orientation in radians, gripper.command the gripper target and gripper.opening_m the opening in meters), action (the requested gripper target, executed_steps, stop_reason, opening_m before and after), step (simulation steps used after the call), steps_left (simulation steps left) and status (running). When the budget is already 0, returns a step_limit error and the gripper target stays unchanged.

Inputs: state (required).

stop_episodeA

End the current simulation and save its frames, event log and summary. Returns run (short run ID), status (stopped) and artifacts (events: path of events.jsonl, frames: PNG frame directory, summary: path of summary.json).

Inputs: no named parameters.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.3/5.0

    Scored across 7 tools

    Disambiguation4/5

    The tools map cleanly to distinct phases of the simulation loop: start (reset_task), observe (observe), act (move_relative, set_gripper), end (stop_episode), and introspect (get_session_info, list_tasks). There is minor overlap in that observe and get_session_info both return robot state, but their descriptions clearly separate their roles (images during a run vs. run metadata callable anytime).

    Naming Consistency4/5

    Six of seven tools follow a verb_noun pattern (reset_task, move_relative, set_gripper, stop_episode, get_session_info, list_tasks). observe is a bare verb without a noun, a minor deviation, but overall the naming is predictable and readable.

    Tool Count5/5

    Seven tools exactly cover the full episode lifecycle without redundancy or bloat. Every tool earns its place: reset, observe, two action primitives, stop, session info, and task listing are all necessary for the stated purpose.

    Completeness4/5

    The surface covers task discovery, episode reset, observation, action, episode termination, and session introspection, which is the core loop. A minor gap is the lack of an explicit reward/success query, though stop_episode artifacts (summary, events) likely provide results for agents able to read them.

    Maintenance

    ActivitySlowing
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      An MCP server that lets AI agents control iOS and Android devices (tap, scroll, type, take screenshots, read UI trees, and run code). Works with multiple devices at the same time.
      91 npm
      53
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server that enables LLM agents to control a two-joint MuJoCo robot arm through tools for moving, reading state, and resetting. It deliberately omits inverse kinematics, so the agent must infer joint angles from observations.
      MIT
    • F
      license
      Not graded
      quality
      D
      maintenance
      Enables agents to safely drive a simulated Franka Panda pick-and-place cell through MCP tools, with gated motion, emergency stop, structured errors, and audit logging.
      -