# VELMA

Vision-Enabled LLM Manipulation Arm. VELMA is a tabletop SO-101 arm in MuJoCo that runs a full
pick-and-place from vision. A fixed side camera locates the object to within a centimetre or so,
and the wrist camera orbits that seed over five views and triangulates it to about 1 mm. The arm
grasps and places from those coordinates, and can be driven by an LLM over MCP, or by hand from a
terminal control panel.

## What's here

```
so101-Models/
  main.py            entry point: loads config.yaml, starts the MCP server (or hardware backend)
  config.yaml        runtime config (backend, viewer on/off, port, LLM provider)
  requirements.txt   robot venv deps
  server/            the MCP server — 15 tools (see below)
  perception/        find_object (coarse) + wrist triangulation + the sidecar client
    requirements.txt   perception sidecar venv deps (torch CPU + ultralytics + transformers)
  percept_venv/      torch/SAM/Depth-Anything sidecar interpreter (CPU)
  wrist_refine.py    wrist-camera hover/state helpers used by perception/wrist_triangulate.py
  robot/, kinematics/, models/   backend, IK, MuJoCo scenes
  control_panel.py   terminal UI to drive the arm by hand through the MCP server
  agent.py           thin MCP client that connects any LLM (Ollama/OpenAI-compatible) to the arm
  SKILL.md           system-prompt guide for an LLM driving the arm (fed in by agent.py)
  .mcp.json          MCP server entry for CLI tools (Claude Code, Codex CLI, ...)
  venv/              robot interpreter (MuJoCo, no torch)
```

## Install

The two virtual environments are the only heavy pieces. Recreate them from the pinned deps
(run from inside `so101-Models/`):

```bash
# robot venv (renders MuJoCo, runs the MCP server — no torch)
python3 -m venv venv && ./venv/bin/pip install -r requirements.txt

# perception sidecar venv (torch CPU + ultralytics + transformers; needs Python 3.11/3.12)
python3.11 -m venv percept_venv && ./percept_venv/bin/pip install -r perception/requirements.txt
```

Then, optionally, run the setup wizard to write `config.yaml` (backend, LLM connection, server
port) — it's a config helper, not part of the install:

```bash
./venv/bin/python setup.py
```

A committed `config.yaml` with sane defaults already ships in this repo, so `setup.py` is only
needed to change something (e.g. switch to the hardware backend, or add an API key).

## Run the server (headless — required for perception)

Perception needs offscreen rendering, so run the server **without** the viewer:

```bash
SO101_VIEWER=0 MUJOCO_GL=egl ./venv/bin/python main.py
```

`SO101_SCENE=models/so101/<scene>.xml` overrides the scene (default `pick_and_place_scene.xml`).
Set `SO101_VIEWER=1` to watch the arm instead — but perception is then disabled by design (the
offscreen and on-screen renderers cannot coexist on this box).

## Drive it by hand

In a second terminal:

```bash
./venv/bin/python control_panel.py
```

It connects to the running server, lists the 15 tools with their live schemas, and lets you call
any of them with guided input. **Any MCP client must set a long read timeout** (the panel uses
`read_timeout_seconds=300`) — `grasp`/`place`/`stop_recording` take several seconds, and a client
that times out mid-call takes the server down with it.

## Connect an LLM

With the headless server running (above), pick one of two paths:

**A CLI tool that speaks MCP** (Claude Code, Codex CLI, Gemini CLI, Cursor, ...): this repo's
`.mcp.json` already points at `http://localhost:3001/mcp` — just run the CLI tool from inside
`so101-Models/` (e.g. `claude`). If you changed `config.yaml`'s `server.port`, re-run
`./venv/bin/python setup.py` (Server settings) to regenerate `.mcp.json` to match.

**Any Ollama / OpenAI-compatible model**, via the bundled thin agent:

```bash
./venv/bin/python agent.py --model gemma3:4b --task "pick up the red cube and place it in the container"
```

`agent.py` feeds the model `SKILL.md` as a system prompt plus the *live* tool schemas from the
server (never hard-coded, so it can't drift from what the server actually exposes), and relays
tool calls both ways — it doesn't inject a procedure or second-guess the model. See `agent.py
--help` for endpoint/model flags (`LLM_BASE_URL`, `LLM_API_KEY`, `--mode auto|native|text`).

## The pick-and-place path (what an LLM calls)

The object's position is **measured**, never given:

1. `find_object("the red cube")` — coarse `(x, y)` from the fixed side camera (~cm, enough to aim).
2. `triangulate("the red cube", x, y)` — refine to a grasp-ready 3D point with the wrist camera
   (~1 mm). If `confidence < 0.6` the views disagree — re-run `find_object` or decline.
3. `grasp(x, y, z, trust_coords=true, grip_width_mm=width_mm+10)` — gentle, never-bat top grasp.
4. `find_object("the container")` — the container's position is measured too, not assumed.
5. `place(container_x, container_y, 0.05)` — put it down; check `cube_in_container`.

## The 15 tools

Perception: `find_object`, `triangulate`. State: `get_robot_state`, `capture_cameras`,
`get_initial_instructions`. Manipulation: `grasp`, `place`. Motion: `move_to_position`,
`set_joint_angles`. Gripper: `set_gripper`, `open_gripper`, `close_gripper`. Harness:
`reset_scene`, `start_recording`, `stop_recording`.

## Honesty note

The perception path (`find_object` → `triangulate`) is measurement-only — nothing in it reads
ground truth. `get_robot_state` and every tool's returned state deliberately **omit** the cube's
and container's true positions, so an LLM cannot shortcut perception by reading the answer out of
state (a benchmark harness that needs ground truth for scoring can opt back in by starting the
server with `SO101_EXPOSE_TRUTH=1` — never set this when an LLM is connected).

Two internal reads of ground truth remain inside `grasp` itself (object shape for the never-bat
decline, and a position drift-abort guard) and are pending removal. Full audit, exact locations,
and severity: [`research/HONESTY_REVIEW.md`](../research/HONESTY_REVIEW.md).