embodied-nav-mcp

A language model that can look around a room, go to what it names, count what it sees, and stop when it is done.

The model decides what to do. The sensors decide where things are.

Tools

commit_count does not take a number from the model. Two views of four pillows become 4, not 8.

Geometry is the Type 3 stack from Team Xiao Hei’s feat/xiao-hei-instruction-merge: size gate, blind-cone handling, converter settle pose, park-on-exit.

Install

uv sync --extra dev
export ANTHROPIC_API_KEY=...
uv run pytest

Python 3.11+. Live robot/sim needs ROS Jazzy and the autonomy stack. Tests use a fake robot and need neither.

Try it

# Dry run, no ROS
uv run embodied-mcp-agent --fake "How many blue chairs are between the table and the wall?"
uv run embodied-mcp-agent --fake "Find the potted plant on the kitchen island closest to the fridge."
uv run embodied-mcp-agent --fake "Take the path near the window to the fridge."

On a robot (or in the challenge container) with the stack up:

uv run embodied-mcp-challenge
# evaluators still run: ros2 launch dummy_vlm dummy_vlm.launch

embodied-mcp --attach-ros is the same tools over stdio MCP. The challenge node calls them in-process: a robot container has no public URL for a remote MCP connector.

CMU VLN Challenge

Drop-in adapter: examples/cmu_vln/. Package name dummy_vlm is unchanged.

Allowed sensors: /camera/image, /registered_scan, /terrain_map, /terrain_map_ext, /state_estimation. Arrival is /state_estimation, not /way_point_reached. One question per process.

Configuration

License

MIT. Vendored geometry and ROS I/O from Team Xiao Hei; see NOTICE.