embodied-mcp
This server gives a language model a live robot's eyes and wheels: look around, ground what it names, drive there, count what it sees, and park.
get_view— grab the 360° camera as four perspective faces (front, right, back, left) plus current pose.ground— turn a referring expression into a pixel box, lift it to lidar metres, and return a converter-snapped waypoint (publish_xy) for driving.drive_to— publish one map-frame(x, y)waypoint and wait until arrival or settle (arrival from/state_estimation).stop— park at the current pose so the planner stops chasing the last waypoint.parse_numerical— split a counting question into target, attribute, relation, and anchor.count_in_view— count matching instances from the current vantage only, lidar-lifting each one for de-duplication.commit_count— cluster every look by position and publish a single integer on/numerical_response(geometry decides the number, not the model).publish_object— publish a 3D cube marker on/selected_object_markerfor the last grounded object, sized by its lidar cluster.
Together these support the CMU VLN Challenge task types — numerical counting, object reference, and instruction-following navigation.
Integrates directly with a ROS 2 (Jazzy) autonomy stack on a live robot or simulator: reads camera, registered scan, terrain map, and state estimation topics, and publishes map-frame waypoints, 3D object markers, and numerical counting answers to the challenge topics.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@embodied-mcp How many plush toys are on the bed?"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
embodied-nav-mcp
A language model that can look around a room, go to what it names, count what it sees, and stop when it is done.
The model decides what to do. The sensors decide where things are.
Tools
Tool | What it does |
| Four perspective faces from the 360° camera, plus pose |
| Referring expression → pixel box → lidar metres → converter-snapped waypoint |
| Publish one map-frame |
| Park so the planner stops chasing the last waypoint |
| Split a counting question into target / anchor |
| Count this vantage only; each instance is lidar-lifted |
| Cluster looks by position; publish the integer |
| Lidar AABB of the last |
commit_count does not take a number from the model. Two views of four pillows become 4, not 8.
Geometry is the Type 3 stack from Team Xiao Hei’s feat/xiao-hei-instruction-merge: size gate, blind-cone handling, converter settle pose, park-on-exit.
Related MCP server: Webots MCP
Install
uv sync --extra dev
export ANTHROPIC_API_KEY=...
uv run pytestPython 3.11+. Live robot/sim needs ROS Jazzy and the autonomy stack. Tests use a fake robot and need neither.
Try it
# Dry run, no ROS
uv run embodied-mcp-agent --fake "How many blue chairs are between the table and the wall?"
uv run embodied-mcp-agent --fake "Find the potted plant on the kitchen island closest to the fridge."
uv run embodied-mcp-agent --fake "Take the path near the window to the fridge."On a robot (or in the challenge container) with the stack up:
uv run embodied-mcp-challenge
# evaluators still run: ros2 launch dummy_vlm dummy_vlm.launchembodied-mcp --attach-ros is the same tools over stdio MCP. The challenge node calls them in-process: a robot container has no public URL for a remote MCP connector.
CMU VLN Challenge
Drop-in adapter: examples/cmu_vln/. Package name dummy_vlm is unchanged.
Type | Tools | ROS out |
Numerical |
|
|
Object reference |
|
|
Instruction-following |
|
|
Allowed sensors: /camera/image, /registered_scan, /terrain_map, /terrain_map_ext, /state_estimation. Arrival is /state_estimation, not /way_point_reached. One question per process.
Configuration
Variable | Default | |
| — | required |
|
| host and pixel-grounding / counting model |
|
| seconds from process start |
|
| camera topic |
License
MIT. Vendored geometry and ROS I/O from Team Xiao Hei; see NOTICE.
Available Tools
8 toolscommit_countA
Cluster every count_in_view look by lidar position and publish one integer on /numerical_response. Two views of the same four pillows become 4, not 8. You do not pass the number — geometry does.
Inputs: no named parameters. Returns: result
count_in_viewA
Count matching instances from the current vantage point only. The call is never told the running total — do not add previous looks yourself. Each instance is boxed and lifted with lidar so later looks can be de-duplicated. If sufficient is false, drive as next_view suggests (or ground the anchor) and count_in_view again. When you are done looking, call commit_count — that publishes the clustered integer.
Inputs: target (required), question (required), anchor. Returns: result
drive_toA
Publish one waypoint (map-frame x, y) and wait until the vehicle arrives or settles. Arrival is decided from /state_estimation, not from a 'waypoint reached' topic. Prefer publish_xy from ground() over coordinates you invent.
Inputs: x (required), y (required), timeout_s. Returns: result
get_viewA
Capture the robot's current 360° camera, split into four perspective faces (front, right, back, left), and return them with the current pose. Call this when you need to see the room. Do not guess what is around you.
Inputs: no named parameters. Returns: result
groundA
Ground a referring expression in the live camera + lidar. Pass the object or place to go to, in the instruction's own words (e.g. 'the lantern closest to the fan decoration'). The tool asks a pixel-grounding model for a box, lifts metres from the registered scan, size-checks the range, and snaps the waypoint to where the platform converter will actually settle. Use publish_xy as the argument to drive_to. If committed is false, this is a step to a better viewpoint — drive there and ground again. If visible is false, drive to suggested_xy and look again.
Inputs: query (required). Returns: result
parse_numericalA
Split a counting question into target, attribute, relation, and anchor. Use this first on numerical questions so you know what to ground and what to pass to count_in_view.
Inputs: question (required). Returns: result
publish_objectA
Publish a 3D cube marker on /selected_object_marker for the object you last grounded. Call ground() on the unique referred object first. The box centre and size come from the lidar cluster, not from a guess.
Inputs: label (required). Returns: result
stopA
Park at the current pose so the local planner stops chasing the last waypoint. Call this when the instruction is finished, or after you have published a numerical or object-reference answer.
Inputs: no named parameters. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 8 tools
Each tool serves a distinct function: perception (get_view), grounding (ground), navigation (drive_to, stop), numerical parsing (parse_numerical), counting (count_in_view), finalizing counts (commit_count), and object publication (publish_object). There is no overlap or ambiguous boundaries between them.
All tool names follow a consistent verb_phrase pattern with snake_case (e.g., get_view, drive_to, commit_count). The verbs are imperative and descriptive, making the intent of each tool predictable from its name alone.
With 8 tools, the server is well-scoped for embodied navigation and counting tasks. Each tool covers a necessary step in the workflow without redundancy or excessive granularity, fitting within the ideal 3-15 tool range.
The tool surface fully covers the lifecycle of the domain: perceiving the environment, grounding references, moving, stopping, parsing questions, counting from multiple views, committing aggregated results, and publishing object markers. There are no obvious gaps or dead ends for the intended tasks.
Related MCP Connectors
Persistent workspace and visual memory for humans and autonomous agents.
Video scene understanding for AI agents via the Primate Vision API.
E2LLM gives your AI eyes and hands in a real browser: structured perception (SiFR) plus action.
Task and planning workspace for humans collaborating with AI agents
Related MCP Servers
- AlicenseAqualityDmaintenanceEnables controlling robots in ROS environments through natural language, supporting topics, services, actions, and GUI tools.2436MIT
- AlicenseNot gradedqualityDmaintenanceBridges AI agents with Webots robotics simulation, enabling natural language control of Nao robots and visual perception.MIT
- FlicenseAqualityDmaintenanceEnables agents to identify appliances and retrieve grounded, robot-executable operation packages, reducing hallucinations from 24% to nearly 0%.4-
- FlicenseNot gradedqualityDmaintenanceEnables LLM agents to control robot simulations (grasping, pouring) in the Genesis physics simulator through persistent sessions, allowing multi-turn interaction with the same scene.-