# A control runtime for remote robot policies

A simulated Franka Panda does pick-and-place, driven by a policy that lives
behind an HTTP boundary — and keeps working when that boundary misbehaves.

![the runtime holding through a server outage and resuming](docs/outage_timeline.png)

The interesting part is not the arm. It is everything between the arm and the
model: action-chunk scheduling, staleness rejection, retries with backoff, a
circuit breaker, a protective hold that clears itself, and an MCP tool surface
so an agent can drive the cell without being able to hurt it.

Runs entirely on a laptop. No GPU, no ROS install, no hardware.

---

## Why the network is the hard part

Manipulation policies want a GPU. Robots want a real-time control loop. They are
rarely the same machine, so in practice the model sits behind a network hop —
which is why policies emit *action chunks* rather than single steps. You cannot
round-trip to an inference server at 50 Hz, but you can ask for 400 ms of
actions at a time and keep executing while the next chunk is in flight.

Every hard problem in this repo follows from that one hop:

- A chunk describes a world that existed when the observation was taken. By the
  time it lands it is already out of date. **How out of date is too much?**
- The control loop must command the arm every 20 ms whether or not the server
  has answered. **What does it do when there is nothing valid to run?**
- Requests fail, retry, and arrive out of order. **What stops an older answer
  from overwriting a newer one?**
- A model can emit NaNs; a mis-versioned server can send targets for a different
  robot. **What refuses to pass that to the actuators?**

## Results

25 seeds per condition, real HTTP, faults injected from a seeded RNG.
`python experiments/latency_sweep.py --seeds 25 --ablations`

| condition | task success | ended safely | median time | held | recoveries | p50 latency | stale rejects | retries |
|---|---|---|---|---|---|---|---|---|
| `clean` | **100%** | 100% | 6.4 s | 0.0 s | 0 | 20 ms | 0 | 0 |
| `lan` — 20 ms ± 5 | **100%** | 100% | 6.7 s | 0.0 s | 0 | 40 ms | 0 | 0 |
| `wan` — 150 ms ± 40 | **100%** | 100% | 11.1 s | 0.0 s | 0 | 160 ms | 0 | 0 |
| `congested` — 250 ms ± 150, 5% loss | **100%** | 100% | 12.6 s | 0.36 s | 68 | 280 ms | 222 | 64 |
| `lossy` — 20% loss | **100%** | 100% | 9.8 s | 0.26 s | 50 | 60 ms | 197 | 222 |
| `flaky_server` — 20% 5xx | **100%** | 100% | 8.1 s | 0.0 s | 0 | 60 ms | 0 | 248 |
| `outage` — server gone 3 s | **100%** | 100% | 13.9 s | 6.2 s | 25 | 60 ms | 0 | 24 |

**Task success** is the cube on the target. **Ended safely** is a separate
column on purpose: a run can fail the task and still be correct, because
stopping is sometimes the right answer. Collapsing the two would hide the
difference between *the network was bad* and *the robot did something it
shouldn't have*.

The pattern across the table is the design goal: as the link degrades the robot
gets **slower, not wrong**. A 250 ms congested link doubles cycle time and
rejects 222 stale chunks; it does not drop the cube or reach somewhere it
shouldn't.

### Ablations — every mitigation, removed

A safety check you have never watched fail is a safety check you cannot claim
works.

| removed | task success | median time | note |
|---|---|---|---|
| *(nothing — baseline `congested`)* | 100% | 12.6 s | |
| **recovery from hold** (`outage`) | **0%** | 0.5 s | latching stop, never resumes |
| **staleness check** (`congested`) | **88%** | 23.7 s | executes plans for a world that moved |
| **retries** (`congested`) | **96%** | 17.5 s | |
| **adaptive lead** (`congested`) | 100% | 12.4 s | but 1.10 s held vs 0.36 s |
| **retries** (`lossy`) | 100% | **7.9 s** | *faster without them — see below* |

## What the fault sweep actually found

Both of these were real defects. Neither was visible against a healthy
localhost server; both showed up the first time the sweep ran.

**1. The protective stop had no way back.** On the `outage` profile the runtime
correctly detected the dead server, held position, and latched an e-stop — then
sat there while the server came back three seconds later. Correct, and useless.
A robot that needs a human to walk over and re-arm it after every network blip
gets unplugged in week two.

The fix splits one concept into two: a **protective hold** that clears itself
the instant a valid chunk arrives, and a **latched e-stop** eight seconds later
if it never does. `outage` went 0% → 100%, and the same change fixed
`congested`. The figure at the top is that fix working.

**2. The request lead time was shorter than the latency.** The runtime asked for
the next chunk when 120 ms of actions remained. On the congested link the round
trip was 280 ms. Every request was issued 140 ms too late to be useful, so the
arm starved at nearly every chunk boundary. No amount of retrying fixes a
request that was sent too late — you have to ask sooner.

The runtime now measures its own p95 latency and scales the lead to it. Held
time on `congested` dropped from 1.10 s to 0.36 s.

**3. A mitigation that doesn't pay for itself.** On the `lossy` profile,
turning retries *off* made things **faster** (7.9 s vs 9.8 s) with no loss of
success rate, and eliminated 197 stale rejections. On a low-latency link,
chunking already provides the redundancy: by the time a retry lands, a fresh
request would have been more useful. Retries earn their place on `congested`
(96% → 100%) and not on `lossy`. It is in the table because reporting only the
mitigations that worked is how you end up shipping the ones that don't.

## How it works

```
        robot side                          │            policy side
                                            │
  ┌──────────────────────────────┐          │       ┌────────────────────┐
  │ runtime.py  50 Hz loop       │          │       │ server.py          │
  │   1 collect ── poll ─────────┼── HTTP ──┼──────▶│  POST /predict     │
  │   2 request ── submit        │          │       │  obs → 20 actions  │
  │   3 act                      │◀─────────┼───────│                    │
  │   4 check                    │          │       └────────────────────┘
  └──┬────────┬────────┬─────────┘          │        stateless; knows
     │        │        │                    │        nothing about episodes
     ▼        ▼        ▼                    │        or scheduling
  client   scheduler  safety                │
  retries  staleness  NaN/limits/workspace  │
  backoff  ordering   rate limit            │
  breaker  discards   e-stop                │
```

| module | one job |
|---|---|
| [contracts.py](robot_runtime/contracts.py) | every type that crosses the wire, defined once |
| [clock.py](robot_runtime/clock.py) | time, injectable — real or virtual |
| [sim.py](robot_runtime/sim.py) | MuJoCo behind six methods; swap for hardware here |
| [policies/scripted.py](robot_runtime/policies/scripted.py) | stands in for a VLA: stateless, chunked, reactive |
| [server.py](robot_runtime/server.py) | the policy, behind HTTP |
| [client.py](robot_runtime/client.py) | submit/poll, deadlines, retries, backoff, circuit breaker |
| [scheduler.py](robot_runtime/scheduler.py) | which chunks to trust, which actions to execute |
| [safety.py](robot_runtime/safety.py) | assumes the policy is wrong |
| [runtime.py](robot_runtime/runtime.py) | the 50 Hz loop |
| [recording.py](robot_runtime/recording.py) | MCAP logging |
| [mcp_server.py](robot_runtime/mcp_server.py) | the cell as MCP tools |

Three decisions worth calling out:

**The loop never blocks on the network.** Step 2 submits, step 1 polls, nothing
waits. A control loop that a slow server can stall is not a control loop.

**Staleness is measured from `observed_at`, not arrival.** A chunk that took
300 ms to come back is 300 ms out of date the moment it lands.

**Safety checks run at two different rates.** Chunk validation is expensive
(forward kinematics on every action) and runs once per chunk at the trust
boundary. Rate limiting is cheap and runs every tick. Rejecting a bad plan
wholesale beats clamping it into something subtly wrong.

### The clock trick

Virtual time runs ~100× faster than real time, so 400 ms of robot time elapses
in 4 ms of wall clock — quicker than an HTTP round trip to localhost. Without
care, every response looks late and the experiment measures the harness instead
of the runtime.

So `SimClock.settle()` blocks in *real* seconds without advancing *virtual*
ones. The only delay the runtime ever observes is the delay the fault profile
asked for. Same client code, same retry paths, same staleness logic — under
`WallClock` on hardware, `settle()` is a no-op. That is what makes every number
above reproducible to the bit.

## Driving it from an agent (MCP)

```bash
python -m robot_runtime.mcp_server
```

Twelve tools. Six read-only (state, cameras, fault profiles, recordings, audit
log), six that move the robot. The gating is server-side state, not a request in
the prompt:

```
run_pick_and_place  → {"ok": false, "error": "cell is not armed",
                       "hint": "call arm_cell with a reason before commanding motion"}
arm_cell("  ")      → {"ok": false, "error": "a reason is required"}
arm_cell("demo")    → {"ok": true, "armed": true, "expires_in_s": 120.0}
emergency_stop()    → {"ok": true, "estopped": true}
run_pick_and_place  → {"ok": false, "error": "cell is e-stopped"}
clear_estop()       → {"ok": false, "error": "confirmation required"}
```

- **Motion is gated; reading is not; the stop button never is.** A safety
  control you have to authenticate to reach is not a safety control.
- **Arming takes a reason and expires**, and the reason is logged.
- **Errors are structured results with a `hint`, never exceptions.** An agent
  that reads `hint: start the policy server` can fix the problem. A stack trace
  makes it guess.
- **Every call is appended to an audit log** readable through the same
  interface, so "what exactly did it call?" always has an answer.

## Observability and replay

Every episode records to **MCAP** — the container ROS 2 logs into — on four
topics: `/observation`, `/action_chunk`, `/command`, `/event`. The event topic
is the one that matters, because it records *decisions*, not just data:

```
3.28s  request_failed:     unreachable
3.52s  breaker_rejected:   circuit open, request not sent
3.80s  protective_hold:    no valid action for 0.50s
6.66s  hold_released:      resumed on chunk 136
```

Nine lines, not the 128 the first version wrote — repeated events collapse. The
breaker rejects a request on every one of the 50 ticks a second it is open, and
writing all fifty says nothing the first one didn't.

Replaying a recording re-runs the exact commands into a freshly seeded simulator:

```
$ python experiments/replay.py recordings/outage-seed0.mcap
  commands_replayed: 533
  placement_error_m: 0.007850735794278705   # live run: 0.007850735794278705
  time drift: 0.000 ms
```

Bit-exact. It was not, at first: `/command` was logged rounded to six decimals,
which put a micron of drift between a run and its own replay. Small — and it
made "reproduces exactly" false, which is the entire point of recording.

This is also how a field failure gets fixed: ship the MCAP back from the site,
replay it, watch the arm do the wrong thing again on your laptop.

## Running it

```bash
python3 -m venv ~/.venvs/robotarm && ~/.venvs/robotarm/bin/pip install -r requirements.txt
```

The venv goes on the internal disk deliberately — this repo sits on an exFAT
volume, where macOS scatters AppleDouble `._` files that MuJoCo's plugin loader
tries to `dlopen` and dies on, and that `git` cannot maintain a pack index
through.

```bash
python experiments/latency_sweep.py --seeds 25 --ablations   # the results table
mjpython demos/run_with_viewer.py --profile outage           # watch it hold and recover
python experiments/replay.py recordings/outage-seed0.mcap    # replay a recording
python -m robot_runtime.mcp_server                           # agent-facing tools
mjpython demos/pick_and_place.py                             # the original scripted demo
pytest -q                                                    # 38 tests, ~5 s
```

`mjpython` rather than `python` for anything with a viewer — on macOS the window
must own the main thread.

## Tests

38 tests, about five seconds, no mocks of the thing under test. The client tests
use a fake transport but the real fault injector; the runtime tests go over real
HTTP to a real server in a thread.

Three of them are the defects above, kept as regressions:
`test_server_outage_holds_then_recovers`,
`test_adaptive_lead_reduces_time_spent_holding`, and
`test_the_stage_machine_does_not_oscillate` — an earlier policy flipped
`lift→carry→lift→carry` because it checked height before position.

## What this is not

Stated plainly, because overclaiming to robotics engineers fails the interview
rather than the screen.

- **Simulation only.** No hardware, no sim-to-real transfer, no contact-model
  calibration against a real Panda.
- **The policy is hand-written, not learned.** It is deliberately shaped like a
  VLA — stateless, chunked, language-conditioned, reactive — so the runtime is
  exercised as it would be by a real model. But nothing here is trained, and no
  claim about model quality is being made.
- **Object poses come from the simulator, not perception.** The observation
  carries a camera image and the contract supports it; the scripted policy
  ignores the pixels. A real deployment needs a perception stack, and that gap
  is the largest one here.
- **Single arm, single rigid object, one task.**
- **Not ROS.** MCAP is used because it is the right container and the ecosystem
  reads it, but there are no nodes, no TF tree, no launch files.
- **Built in about a day**, as a focused demonstration of the runtime layer.

## Where it goes next

In rough order of value:

1. **Perception in the loop** — pose from the camera image instead of from the
   simulator. Closes the biggest gap on this list.
2. **Train the policy.** Generate demonstrations with the scripted controller,
   fit an action-chunking behaviour-cloning model, serve it through the *same*
   `/predict` contract. The runtime should not need one line changed — that is
   the claim the `Policy` interface makes, and it is currently untested.
3. **Two arms**, which turns chunk scheduling into a genuine coordination
   problem rather than a bookkeeping one.
4. **A real Panda**, where `settle()` becomes a no-op and every latency number
   in this README gets measured again for real.

## Credits

Panda model from [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie)
(Apache 2.0). Physics: [MuJoCo](https://mujoco.org). Logging:
[MCAP](https://mcap.dev).