[![M8ven Score](https://m8ven.ai/badge/mcp/aryanmangal769/embodied-nav-mcp)](https://m8ven.ai/mcp/aryanmangal769/embodied-nav-mcp)
# embodied-nav-mcp

A language model that can look around a room, go to what it names, count what it sees, and stop when it is done.

The model decides *what* to do. The sensors decide *where things are*.

<p align="center">
  <img src="docs/figures/fig2-after.png" alt="The model calls tools on a live robot: look, ground, count, drive. Lidar decides metres." width="100%">
</p>


## Tools

| Tool | What it does |
|---|---|
| `get_view` | Four perspective faces from the 360° camera, plus pose |
| `ground` | Referring expression → pixel box → lidar metres → converter-snapped waypoint |
| `drive_to` | Publish one map-frame `(x, y)` and wait until arrived or settled |
| `stop` | Park so the planner stops chasing the last waypoint |
| `parse_numerical` | Split a counting question into target / anchor |
| `count_in_view` | Count this vantage only; each instance is lidar-lifted |
| `commit_count` | Cluster looks by position; publish the integer |
| `publish_object` | Lidar AABB of the last `ground`; publish a 3D marker |

`commit_count` does not take a number from the model. Two views of four pillows become 4, not 8.

Geometry is the Type 3 stack from Team Xiao Hei’s [`feat/xiao-hei-instruction-merge`](https://github.com/ginlov/CMU-VLN-Challenge-2026/tree/feat/xiao-hei-instruction-merge): size gate, blind-cone handling, converter settle pose, park-on-exit.

## Install

```bash
uv sync --extra dev
export ANTHROPIC_API_KEY=...
uv run pytest
```

Python 3.11+. Live robot/sim needs ROS Jazzy and the autonomy stack. Tests use a fake robot and need neither.

## Try it

```bash
# Dry run, no ROS
uv run embodied-mcp-agent --fake "How many blue chairs are between the table and the wall?"
uv run embodied-mcp-agent --fake "Find the potted plant on the kitchen island closest to the fridge."
uv run embodied-mcp-agent --fake "Take the path near the window to the fridge."
```

On a robot (or in the challenge container) with the stack up:

```bash
uv run embodied-mcp-challenge
# evaluators still run: ros2 launch dummy_vlm dummy_vlm.launch
```

`embodied-mcp --attach-ros` is the same tools over stdio MCP. The challenge node calls them in-process: a robot container has no public URL for a remote MCP connector.

## CMU VLN Challenge

Drop-in adapter: [`examples/cmu_vln/`](examples/cmu_vln/). Package name `dummy_vlm` is unchanged.

| Type | Tools | ROS out |
|---|---|---|
| Numerical | `parse_numerical`, `count_in_view`, `commit_count` | `/numerical_response` |
| Object reference | `ground`, `publish_object` | `/selected_object_marker` |
| Instruction-following | `get_view`, `ground`, `drive_to`, `stop` | `/way_point_with_heading` |

Allowed sensors: `/camera/image`, `/registered_scan`, `/terrain_map`, `/terrain_map_ext`, `/state_estimation`. Arrival is `/state_estimation`, not `/way_point_reached`. One question per process.

## Configuration

| Variable | Default | |
|---|---|---|
| `ANTHROPIC_API_KEY` | — | required |
| `EMBODIED_MCP_MODEL` | `claude-opus-5` | host and pixel-grounding / counting model |
| `EMBODIED_MCP_BUDGET_S` | `540` | seconds from process start |
| `XIAO_HEI_IMAGE_TOPIC` | `/camera/image` | camera topic |

## License

MIT. Vendored geometry and ROS I/O from Team Xiao Hei; see [NOTICE](NOTICE).