A control runtime for remote robot policies

A simulated Franka Panda does pick-and-place, driven by a policy that lives behind an HTTP boundary — and keeps working when that boundary misbehaves.

The interesting part is not the arm. It is everything between the arm and the model: action-chunk scheduling, staleness rejection, retries with backoff, a circuit breaker, a protective hold that clears itself, and an MCP tool surface so an agent can drive the cell without being able to hurt it.

Runs entirely on a laptop. No GPU, no ROS install, no hardware.

Why the network is the hard part

Manipulation policies want a GPU. Robots want a real-time control loop. They are rarely the same machine, so in practice the model sits behind a network hop — which is why policies emit action chunks rather than single steps. You cannot round-trip to an inference server at 50 Hz, but you can ask for 400 ms of actions at a time and keep executing while the next chunk is in flight.

Every hard problem in this repo follows from that one hop:

• A chunk describes a world that existed when the observation was taken. By the time it lands it is already out of date. How out of date is too much?

• The control loop must command the arm every 20 ms whether or not the server has answered. What does it do when there is nothing valid to run?

• Requests fail, retry, and arrive out of order. What stops an older answer from overwriting a newer one?

• A model can emit NaNs; a mis-versioned server can send targets for a different robot. What refuses to pass that to the actuators?

Related MCP server: omni-kit-mcp

Results

25 seeds per condition, real HTTP, faults injected from a seeded RNG. python experiments/latency_sweep.py --seeds 25 --ablations

Task success is the cube on the target. Ended safely is a separate column on purpose: a run can fail the task and still be correct, because stopping is sometimes the right answer. Collapsing the two would hide the difference between the network was bad and the robot did something it shouldn't have.

The pattern across the table is the design goal: as the link degrades the robot gets slower, not wrong. A 250 ms congested link doubles cycle time and rejects 222 stale chunks; it does not drop the cube or reach somewhere it shouldn't.

Ablations — every mitigation, removed

A safety check you have never watched fail is a safety check you cannot claim works.

What the fault sweep actually found

Both of these were real defects. Neither was visible against a healthy localhost server; both showed up the first time the sweep ran.

1. The protective stop had no way back. On the outage profile the runtime correctly detected the dead server, held position, and latched an e-stop — then sat there while the server came back three seconds later. Correct, and useless. A robot that needs a human to walk over and re-arm it after every network blip gets unplugged in week two.

The fix splits one concept into two: a protective hold that clears itself the instant a valid chunk arrives, and a latched e-stop eight seconds later if it never does. outage went 0% → 100%, and the same change fixed congested. The figure at the top is that fix working.

2. The request lead time was shorter than the latency. The runtime asked for the next chunk when 120 ms of actions remained. On the congested link the round trip was 280 ms. Every request was issued 140 ms too late to be useful, so the arm starved at nearly every chunk boundary. No amount of retrying fixes a request that was sent too late — you have to ask sooner.

The runtime now measures its own p95 latency and scales the lead to it. Held time on congested dropped from 1.10 s to 0.36 s.

3. A mitigation that doesn't pay for itself. On the lossy profile, turning retries off made things faster (7.9 s vs 9.8 s) with no loss of success rate, and eliminated 197 stale rejections. On a low-latency link, chunking already provides the redundancy: by the time a retry lands, a fresh request would have been more useful. Retries earn their place on congested (96% → 100%) and not on lossy. It is in the table because reporting only the mitigations that worked is how you end up shipping the ones that don't.

How it works

        robot side                          │            policy side
                                            │
  ┌──────────────────────────────┐          │       ┌────────────────────┐
  │ runtime.py  50 Hz loop       │          │       │ server.py          │
  │   1 collect ── poll ─────────┼── HTTP ──┼──────▶│  POST /predict     │
  │   2 request ── submit        │          │       │  obs → 20 actions  │
  │   3 act                      │◀─────────┼───────│                    │
  │   4 check                    │          │       └────────────────────┘
  └──┬────────┬────────┬─────────┘          │        stateless; knows
     │        │        │                    │        nothing about episodes
     ▼        ▼        ▼                    │        or scheduling
  client   scheduler  safety                │
  retries  staleness  NaN/limits/workspace  │
  backoff  ordering   rate limit            │
  breaker  discards   e-stop                │

Three decisions worth calling out:

The loop never blocks on the network. Step 2 submits, step 1 polls, nothing waits. A control loop that a slow server can stall is not a control loop.

Staleness is measured from observed_at, not arrival. A chunk that took 300 ms to come back is 300 ms out of date the moment it lands.

Safety checks run at two different rates. Chunk validation is expensive (forward kinematics on every action) and runs once per chunk at the trust boundary. Rate limiting is cheap and runs every tick. Rejecting a bad plan wholesale beats clamping it into something subtly wrong.

The clock trick

Virtual time runs ~100× faster than real time, so 400 ms of robot time elapses in 4 ms of wall clock — quicker than an HTTP round trip to localhost. Without care, every response looks late and the experiment measures the harness instead of the runtime.

So SimClock.settle() blocks in real seconds without advancing virtual ones. The only delay the runtime ever observes is the delay the fault profile asked for. Same client code, same retry paths, same staleness logic — under WallClock on hardware, settle() is a no-op. That is what makes every number above reproducible to the bit.

Driving it from an agent (MCP)

python -m robot_runtime.mcp_server

Twelve tools. Six read-only (state, cameras, fault profiles, recordings, audit log), six that move the robot. The gating is server-side state, not a request in the prompt:

run_pick_and_place  → {"ok": false, "error": "cell is not armed",
                       "hint": "call arm_cell with a reason before commanding motion"}
arm_cell("  ")      → {"ok": false, "error": "a reason is required"}
arm_cell("demo")    → {"ok": true, "armed": true, "expires_in_s": 120.0}
emergency_stop()    → {"ok": true, "estopped": true}
run_pick_and_place  → {"ok": false, "error": "cell is e-stopped"}
clear_estop()       → {"ok": false, "error": "confirmation required"}

• Motion is gated; reading is not; the stop button never is. A safety control you have to authenticate to reach is not a safety control.

• Arming takes a reason and expires, and the reason is logged.

• Errors are structured results with a hint, never exceptions. An agent that reads hint: start the policy server can fix the problem. A stack trace makes it guess.

• Every call is appended to an audit log readable through the same interface, so "what exactly did it call?" always has an answer.

Observability and replay

Every episode records to MCAP — the container ROS 2 logs into — on four topics: /observation, /action_chunk, /command, /event. The event topic is the one that matters, because it records decisions, not just data:

3.28s  request_failed:     unreachable
3.52s  breaker_rejected:   circuit open, request not sent
3.80s  protective_hold:    no valid action for 0.50s
6.66s  hold_released:      resumed on chunk 136

Nine lines, not the 128 the first version wrote — repeated events collapse. The breaker rejects a request on every one of the 50 ticks a second it is open, and writing all fifty says nothing the first one didn't.

Replaying a recording re-runs the exact commands into a freshly seeded simulator:

$ python experiments/replay.py recordings/outage-seed0.mcap
  commands_replayed: 533
  placement_error_m: 0.007850735794278705   # live run: 0.007850735794278705
  time drift: 0.000 ms

Bit-exact. It was not, at first: /command was logged rounded to six decimals, which put a micron of drift between a run and its own replay. Small — and it made "reproduces exactly" false, which is the entire point of recording.

This is also how a field failure gets fixed: ship the MCAP back from the site, replay it, watch the arm do the wrong thing again on your laptop.

Running it

python3 -m venv ~/.venvs/robotarm && ~/.venvs/robotarm/bin/pip install -r requirements.txt

The venv goes on the internal disk deliberately — this repo sits on an exFAT volume, where macOS scatters AppleDouble ._ files that MuJoCo's plugin loader tries to dlopen and dies on, and that git cannot maintain a pack index through.

python experiments/latency_sweep.py --seeds 25 --ablations   # the results table
mjpython demos/run_with_viewer.py --profile outage           # watch it hold and recover
python experiments/replay.py recordings/outage-seed0.mcap    # replay a recording
python -m robot_runtime.mcp_server                           # agent-facing tools
mjpython demos/pick_and_place.py                             # the original scripted demo
pytest -q                                                    # 38 tests, ~5 s

mjpython rather than python for anything with a viewer — on macOS the window must own the main thread.

Tests

38 tests, about five seconds, no mocks of the thing under test. The client tests use a fake transport but the real fault injector; the runtime tests go over real HTTP to a real server in a thread.

Three of them are the defects above, kept as regressions: test_server_outage_holds_then_recovers, test_adaptive_lead_reduces_time_spent_holding, and test_the_stage_machine_does_not_oscillate — an earlier policy flipped lift→carry→lift→carry because it checked height before position.

What this is not

Stated plainly, because overclaiming to robotics engineers fails the interview rather than the screen.

• Simulation only. No hardware, no sim-to-real transfer, no contact-model calibration against a real Panda.

• The policy is hand-written, not learned. It is deliberately shaped like a VLA — stateless, chunked, language-conditioned, reactive — so the runtime is exercised as it would be by a real model. But nothing here is trained, and no claim about model quality is being made.

• Object poses come from the simulator, not perception. The observation carries a camera image and the contract supports it; the scripted policy ignores the pixels. A real deployment needs a perception stack, and that gap is the largest one here.

• Single arm, single rigid object, one task.

• Not ROS. MCAP is used because it is the right container and the ecosystem reads it, but there are no nodes, no TF tree, no launch files.

• Built in about a day, as a focused demonstration of the runtime layer.

Where it goes next

In rough order of value:

1. Perception in the loop — pose from the camera image instead of from the simulator. Closes the biggest gap on this list.

2. Train the policy. Generate demonstrations with the scripted controller, fit an action-chunking behaviour-cloning model, serve it through the same /predict contract. The runtime should not need one line changed — that is the claim the Policy interface makes, and it is currently untested.

3. Two arms, which turns chunk scheduling into a genuine coordination problem rather than a bookkeeping one.

4. A real Panda, where settle() becomes a no-op and every latency number in this README gets measured again for real.

Credits

Panda model from MuJoCo Menagerie (Apache 2.0). Physics: MuJoCo. Logging: MCAP.