# cs20-lidar-mcp

An MCP server that lets an LLM take its own depth measurements with a Synexens
CS20 solid-state time-of-flight lidar — plus the Linux toolkit underneath it:
ctypes SDK binding, live viewer, CLI and Python API.

![Depth capture: an armchair in the foreground, a bag hanging on a door, a skateboard leaning in an alcove](docs/depth-capture.jpg)

*One capture, false-coloured by distance — blue near, red far. Five clean
frames median-combined; 99.5% of pixels returned a reading.*

## Problem

The CS20 is a capable 640×480 ToF sensor that is close to unusable out of the
box on Linux, for three separate reasons.

- **It lies about what it is.** It enumerates as a UVC camera and `uvcvideo`
  binds it at `/dev/video0`, so every normal camera tool will happily open it
  and show you garbage. The stream is raw time-of-flight phase data, not an
  image. Depth reconstruction happens on the host inside a vendor binary. The
  USB ID `2222:6666` is an unassigned vendor ID that `usb.ids` maps to
  "MacAlly", which sends you looking in the wrong place entirely.
- **Roughly half its frames are silently corrupt.** The dual-frequency phase
  unwrap intermittently picks the wrong wrap index, and the affected pixels
  land a whole range ambiguity away from the truth — scattered flyers plus
  contiguous blobs up to a quarter of the image. Nothing in the API tells you
  this happened. The sensor reports a frame; the frame is wrong.
- **Polling for frames tears them.** The obvious read path returns the SDK's
  rolling buffer even while it is half-written, producing captures with clean
  rectangular bands where two frames meet.

None of this is documented, and the reconstruction library is closed — so none
of it can be fixed upstream.

## Solution

Everything here is diagnosed against the hardware and measured, not assumed.

- **A ctypes binding, no build step.** Every SDK entry point is `extern "C"`
  with flat arguments, so it binds directly. Enums, packed structs and ~60
  functions, with the library-loading problem solved transparently.
- **Push-callback frame delivery.** Rather than polling, the toolkit registers
  a real `ISYFrameObserver` — synthesising the C++ vtable through ctypes —
  which only fires on a complete frame. Measured: 15 consecutive frames, zero
  torn, zero duplicates.
- **Corrupt frames detected and rejected.** The quantisation that causes the
  unwrap bug also identifies it: corrupted pixels sit a whole ambiguity
  (~1585 mm) from their neighbours, which real geometry does not do. Scoring
  correlates 0.94 with ground truth, cleanly separating good frames (1.2–1.4%)
  from bad (4.7–13.8%).
- **Better-than-native captures by merging.** Median-combining several clean
  frames cut the residual aliasing score from 1.49 to 0.41 while keeping 99.5%
  of pixels valid — a better frame than the sensor can produce on its own.
- **A viewer for the settings that matter.** All 11 depth filters with live
  parameter editing, because the vendor defaults are not obviously right and
  the effect has to be seen to be judged.
- **An MCP server that reports its own confidence.** Every capture returns a
  quality verdict, and problems come back as instructions a model can act on
  ("37% of pixels saturated; lower integral_time") rather than error codes.

## Install

Udev rules, so the device is reachable without root:

```bash
sudo tee /etc/udev/rules.d/99-synexens.rules >/dev/null <<'EOF'
SUBSYSTEM=="usb", ATTRS{idVendor}=="2222", ATTRS{idProduct}=="6666", MODE="0666", GROUP="plugdev"
KERNEL=="video[0-9]*", ATTRS{idVendor}=="2222", ATTRS{idProduct}=="6666", MODE="0666"
EOF
sudo udevadm control --reload-rules && sudo udevadm trigger
```

The vendor SDK, which is **not redistributed here** (see *Third-party
components*). It is a free download, no registration:

```bash
mkdir -p sdk && cd sdk
curl -L -o synexens.tar.xz \
  "https://support.tofsensors.com/sdk/4.2.5.0/SynexensSDK_4.2.5.0_ubuntu(18.04+)_202602060238.tar.xz"
tar -xjf synexens.tar.xz      # despite the name, the file is bzip2
```

It is auto-discovered at `sdk/SynexensSDK_*`; set `CS20_SDK_PATH` to override.

Then the Python dependencies:

```bash
pip install -r requirements.txt
```

`numpy` and `pillow` are required. `scipy` is strongly recommended — it makes
frame-quality scoring about 10× faster, though there is a numpy fallback.
`PyQt6` is only needed for the viewer, `mcp` only for the MCP server, and
`matplotlib` is optional and just adds extra colormaps.

## Usage

### Live viewer

```bash
python3 cs20_viewer.py
```

![The live viewer: depth image on the left, camera and filter controls on the right](docs/viewer.jpg)

Depth/IR toggle, colormap, manual or auto distance window, exposure,
mirror/flip, resolution, and a panel for all 11 depth filters with live
parameter editing. **A/B compare** grabs one filtered and one unfiltered frame
side by side with valid-pixel and speckle counts — the quickest way to settle
on a filter chain. Hovering the image reads out the distance under the cursor.
Snapshot writes a false-colour PNG, a 16-bit depth PNG, and a coloured PLY.

The window takes about 15 seconds to appear; see *Limitations*.

### Command line

```bash
python3 cs20_cli.py list                      # enumerate devices
python3 cs20_cli.py info                      # identity, intrinsics, settings
python3 cs20_cli.py capture -o scan --merge 5 # PNGs + PLY
python3 cs20_cli.py capture -o scan --colormap viridis --vmin 400 --vmax 4000
python3 cs20_cli.py set --integral-time 1200 --filter on
python3 cs20_cli.py get
```

`capture` writes `<prefix>_depth_color.png`, `<prefix>_depth_mm.png` (16-bit,
lossless millimetres), `<prefix>_ir.png` and `<prefix>.ply`.

### Python API

```python
from cs20 import Device, StreamType, Resolution, colorize_depth, write_ply, valid_points

with Device() as cam:
    cam.start(StreamType.DEPTHIR, Resolution.R640_480)
    cam.integral_time = 1200

    frame = cam.capture()                 # retries until a clean frame
    print(frame.quality)                  # score, clean, problems
    print(frame.stats())                  # valid %, min/max/mean/median, centre

    best = cam.capture_merged(merge=5)    # median-combine 5 clean frames
    rgb, window = colorize_depth(best.depth)
    cloud = cam.point_cloud(best.depth)   # (H, W, 3) float32 XYZ, millimetres

    points, colors = valid_points(cloud, best.depth, rgb, scale=0.001)  # metres
    write_ply("scan.ply", points, colors)
```

`frame.depth` is uint16 millimetres; `frame.ir` is uint16 amplitude. Depth `0`
means no return, `1` means the pixel saturated.

### MCP server

```bash
python3 cs20_mcp.py          # stdio transport
```

```json
{
  "mcpServers": {
    "cs20-lidar": {
      "command": "python3",
      "args": ["/path/to/cs20-lidar-mcp/cs20_mcp.py"]
    }
  }
}
```

| Tool | Purpose |
|---|---|
| `describe_camera` | identity, optics, capabilities, current settings |
| `capture_image` | false-colour depth or IR image, returned inline to the model |
| `capture_point_cloud` | writes PLY/XYZ, returns bounding box and a preview |
| `measure_distance` | distance at a pixel, averaged over a patch |
| `get_settings` / `set_settings` | read and change settings |
| `release_camera` | free the USB device for other programs |

The server key is `cs20-lidar` rather than `cs20-lidar-mcp`; "mcp" is redundant
inside an `mcpServers` block, and the key becomes the tool prefix the model
sees. Rename it if you prefer — nothing depends on it.

Captures land in `$CS20_OUTPUT_DIR` (default `~/cs20-captures`). The camera is
held open between calls and released after `$CS20_IDLE_TIMEOUT` seconds
(default 180), because enumeration is slow.

Images returned to the model are JPEG, downscaled to 400px by default
(`detail`: low 256 / normal 400 / high 640). This matters more than it sounds:
a false-colour depth image is thousands of distinct noisy colours that PNG
cannot compress, so a full-resolution PNG costs ~166k tokens against ~6k for
JPEG at 400px — a 25× difference, for no gain in what the picture shows. The
lossless full-resolution PNG is still written to disk.

## Architecture

```
cs20/sdk.py        ctypes binding: enums, packed structs, ~60 entry points
cs20/device.py     Device class, push-callback frame delivery, settings
cs20/quality.py    phase-unwrap detection, frame scoring, temporal merge
cs20/colorize.py   false colour (turbo/jet/gray + any matplotlib colormap)
cs20/export.py     PLY (binary/ascii), 16-bit depth PNG, XYZ, NPZ
cs20_viewer.py     PyQt6 live viewer
cs20_cli.py        command line
cs20_mcp.py        MCP server
```

Three decisions are worth explaining, because each one looks wrong until you
know what it is working around.

**Frames arrive by push callback, not polling.** `GetLastFrameData` hands back
the SDK's rolling buffer mid-write. `GetLastFrameDataSafety`, the obvious
alternative, returns `NOFRAME` on this firmware — which is presumably why the
vendor's own demos have it commented out. So the toolkit registers a real
`ISYFrameObserver`. That interface is an abstract C++ class with a single
virtual method, so a conforming object is synthesised in ctypes: a one-word
object whose vptr points into a fake vtable, per the Itanium ABI.

**The library loader re-execs the interpreter.** `libSynexensSDK.so` lists two
private dependencies as `DT_NEEDED` but carries no `DT_RUNPATH`, and neither
dependency declares a `SONAME` — so preloading them by absolute path does not
satisfy the loader. glibc caches the search path at process start, so
`LD_LIBRARY_PATH` cannot be set after the fact. `ensure_library_path()` sets it
and re-execs. Long-running hosts should call it at startup, before opening any
transport, so the re-exec cannot happen mid-session. `CS20_NO_REEXEC=1` opts
out.

**Frame quality is scored from a single frame, with no reference.** The unwrap
error is quantised — a wrong wrap index displaces a pixel by a whole multiple
of the ~1585 mm ambiguity (a 94.6 MHz modulation), never by an arbitrary
amount. Counting pixels that sit a whole ambiguity from their local median
separates corrupt frames from clean ones without anything to compare against,
because real scene geometry does not place depth steps at exactly 1.585 m.

## Tests

```bash
python3 test_mcp_client.py
```

Drives `cs20_mcp.py` as a real MCP client over stdio and exercises all seven
tools end to end against the hardware, including decoding the returned images.
Requires the camera to be connected and not held by another process.

## Limitations

**Only one process can own the camera.** The viewer, CLI and MCP server each
take exclusive control. Close one before starting another.

**Enumeration takes about 15 seconds.** `FindDevice` probes for network
cameras on every local network interface before returning, whether or not any
exist. Nothing can be done about it in the SDK; the MCP server works around it
by keeping the device open between calls.

**Effective frame rate is about half the nominal rate.** The sensor delivers
~7.5 fps at 640×480, but with corrupt frames discarded the usable rate is
~3.7 fps. That is the real throughput of this device.

**The SDK changes your working directory.** It writes `log/` and `parameters/`
relative to the CWD, so the toolkit `chdir`s to `$CS20_STATE_DIR` (default
`~/.cache/cs20`) when it initialises. Resolve output paths to absolute before
opening a device, or relative writes will land somewhere surprising.

**Identity and exposure need opposite stream states.** `GetDeviceSN` and
`GetDeviceHWVersion` work only while streaming is stopped; `GetIntegralTime`
only while it is running.

**Several controls are unimplemented on this firmware.** Frame rate is not
adjustable, and temperature and trigger mode return errors. `Device.settings()`
reports these rather than raising, so it is safe to call on any unit.

**`EDGE` filtering is inverted from what you would guess: lower is more
aggressive.** On one scene `EDGE=80` kept 84.4% of pixels and `EDGE=20` kept
73.3%, both with zero speckle. To recover pixels, raise it or drop it from the
chain.

**Intrinsics report `height` as 640 at 640×480.** An SDK quirk; the real height
is 480. `fx`, `fy`, `cx`, `cy` and the distortion coefficients are correct.

**Tested against one unit.** A CS20_DUAL on firmware `XY-V1-2108200201`, SDK
4.2.5.0, x86-64 Ubuntu. Other variants (CS20_P, CS30, CS40) are in the SDK's
enums and should mostly work, but are untested here.

## Third-party components

The **Synexens SDK** (`libSynexensSDK.so`, `libcsreconstruction2.0.so`,
`libSonixCamera.so`) is proprietary vendor software and is **not redistributed
in this repository**. Download it from
[support.tofsensors.com](https://support.tofsensors.com/resource/sdk/sdk.html)
under Synexens' own terms; the *Install* section has the exact URL used here.
This project only binds to its public C API.

Product documentation: [CS20 product
page](https://support.tofsensors.com/product/CS20.html).

## License

MIT — see [LICENSE](LICENSE).