cs20-lidar-mcp

An MCP server that lets an LLM take its own depth measurements with a Synexens CS20 solid-state time-of-flight lidar — plus the Linux toolkit underneath it: ctypes SDK binding, live viewer, CLI and Python API.

One capture, false-coloured by distance — blue near, red far. Five clean frames median-combined; 99.5% of pixels returned a reading.

Problem

The CS20 is a capable 640×480 ToF sensor that is close to unusable out of the box on Linux, for three separate reasons.

• It lies about what it is. It enumerates as a UVC camera and uvcvideo binds it at /dev/video0, so every normal camera tool will happily open it and show you garbage. The stream is raw time-of-flight phase data, not an image. Depth reconstruction happens on the host inside a vendor binary. The USB ID 2222:6666 is an unassigned vendor ID that usb.ids maps to "MacAlly", which sends you looking in the wrong place entirely.

• Roughly half its frames are silently corrupt. The dual-frequency phase unwrap intermittently picks the wrong wrap index, and the affected pixels land a whole range ambiguity away from the truth — scattered flyers plus contiguous blobs up to a quarter of the image. Nothing in the API tells you this happened. The sensor reports a frame; the frame is wrong.

• Polling for frames tears them. The obvious read path returns the SDK's rolling buffer even while it is half-written, producing captures with clean rectangular bands where two frames meet.

None of this is documented, and the reconstruction library is closed — so none of it can be fixed upstream.

Related MCP server: OBSBOT Camera MCP Server

Solution

Everything here is diagnosed against the hardware and measured, not assumed.

• A ctypes binding, no build step. Every SDK entry point is extern "C" with flat arguments, so it binds directly. Enums, packed structs and ~60 functions, with the library-loading problem solved transparently.

• Push-callback frame delivery. Rather than polling, the toolkit registers a real ISYFrameObserver — synthesising the C++ vtable through ctypes — which only fires on a complete frame. Measured: 15 consecutive frames, zero torn, zero duplicates.

• Corrupt frames detected and rejected. The quantisation that causes the unwrap bug also identifies it: corrupted pixels sit a whole ambiguity (~1585 mm) from their neighbours, which real geometry does not do. Scoring correlates 0.94 with ground truth, cleanly separating good frames (1.2–1.4%) from bad (4.7–13.8%).

• Better-than-native captures by merging. Median-combining several clean frames cut the residual aliasing score from 1.49 to 0.41 while keeping 99.5% of pixels valid — a better frame than the sensor can produce on its own.

• A viewer for the settings that matter. All 11 depth filters with live parameter editing, because the vendor defaults are not obviously right and the effect has to be seen to be judged.

• An MCP server that reports its own confidence. Every capture returns a quality verdict, and problems come back as instructions a model can act on ("37% of pixels saturated; lower integral_time") rather than error codes.

Install

Udev rules, so the device is reachable without root:

sudo tee /etc/udev/rules.d/99-synexens.rules >/dev/null <<'EOF'
SUBSYSTEM=="usb", ATTRS{idVendor}=="2222", ATTRS{idProduct}=="6666", MODE="0666", GROUP="plugdev"
KERNEL=="video[0-9]*", ATTRS{idVendor}=="2222", ATTRS{idProduct}=="6666", MODE="0666"
EOF
sudo udevadm control --reload-rules && sudo udevadm trigger

The vendor SDK, which is not redistributed here (see Third-party components). It is a free download, no registration:

mkdir -p sdk && cd sdk
curl -L -o synexens.tar.xz \
  "https://support.tofsensors.com/sdk/4.2.5.0/SynexensSDK_4.2.5.0_ubuntu(18.04+)_202602060238.tar.xz"
tar -xjf synexens.tar.xz      # despite the name, the file is bzip2

It is auto-discovered at sdk/SynexensSDK_*; set CS20_SDK_PATH to override.

Then the Python dependencies:

pip install -r requirements.txt

numpy and pillow are required. scipy is strongly recommended — it makes frame-quality scoring about 10× faster, though there is a numpy fallback. PyQt6 is only needed for the viewer, mcp only for the MCP server, and matplotlib is optional and just adds extra colormaps.

Usage

Live viewer

python3 cs20_viewer.py

Depth/IR toggle, colormap, manual or auto distance window, exposure, mirror/flip, resolution, and a panel for all 11 depth filters with live parameter editing. A/B compare grabs one filtered and one unfiltered frame side by side with valid-pixel and speckle counts — the quickest way to settle on a filter chain. Hovering the image reads out the distance under the cursor. Snapshot writes a false-colour PNG, a 16-bit depth PNG, and a coloured PLY.

The window takes about 15 seconds to appear; see Limitations.

Command line

python3 cs20_cli.py list                      # enumerate devices
python3 cs20_cli.py info                      # identity, intrinsics, settings
python3 cs20_cli.py capture -o scan --merge 5 # PNGs + PLY
python3 cs20_cli.py capture -o scan --colormap viridis --vmin 400 --vmax 4000
python3 cs20_cli.py set --integral-time 1200 --filter on
python3 cs20_cli.py get

capture writes <prefix>_depth_color.png, <prefix>_depth_mm.png (16-bit, lossless millimetres), <prefix>_ir.png and <prefix>.ply.

Python API

from cs20 import Device, StreamType, Resolution, colorize_depth, write_ply, valid_points

with Device() as cam:
    cam.start(StreamType.DEPTHIR, Resolution.R640_480)
    cam.integral_time = 1200

    frame = cam.capture()                 # retries until a clean frame
    print(frame.quality)                  # score, clean, problems
    print(frame.stats())                  # valid %, min/max/mean/median, centre

    best = cam.capture_merged(merge=5)    # median-combine 5 clean frames
    rgb, window = colorize_depth(best.depth)
    cloud = cam.point_cloud(best.depth)   # (H, W, 3) float32 XYZ, millimetres

    points, colors = valid_points(cloud, best.depth, rgb, scale=0.001)  # metres
    write_ply("scan.ply", points, colors)

frame.depth is uint16 millimetres; frame.ir is uint16 amplitude. Depth 0 means no return, 1 means the pixel saturated.

MCP server

python3 cs20_mcp.py          # stdio transport

{
  "mcpServers": {
    "cs20-lidar": {
      "command": "python3",
      "args": ["/path/to/cs20-lidar-mcp/cs20_mcp.py"]
    }
  }
}

The server key is cs20-lidar rather than cs20-lidar-mcp; "mcp" is redundant inside an mcpServers block, and the key becomes the tool prefix the model sees. Rename it if you prefer — nothing depends on it.

Captures land in $CS20_OUTPUT_DIR (default ~/cs20-captures). The camera is held open between calls and released after $CS20_IDLE_TIMEOUT seconds (default 180), because enumeration is slow.

Images returned to the model are JPEG, downscaled to 400px by default (detail: low 256 / normal 400 / high 640). This matters more than it sounds: a false-colour depth image is thousands of distinct noisy colours that PNG cannot compress, so a full-resolution PNG costs ~166k tokens against ~6k for JPEG at 400px — a 25× difference, for no gain in what the picture shows. The lossless full-resolution PNG is still written to disk.

Architecture

cs20/sdk.py        ctypes binding: enums, packed structs, ~60 entry points
cs20/device.py     Device class, push-callback frame delivery, settings
cs20/quality.py    phase-unwrap detection, frame scoring, temporal merge
cs20/colorize.py   false colour (turbo/jet/gray + any matplotlib colormap)
cs20/export.py     PLY (binary/ascii), 16-bit depth PNG, XYZ, NPZ
cs20_viewer.py     PyQt6 live viewer
cs20_cli.py        command line
cs20_mcp.py        MCP server

Three decisions are worth explaining, because each one looks wrong until you know what it is working around.

Frames arrive by push callback, not polling. GetLastFrameData hands back the SDK's rolling buffer mid-write. GetLastFrameDataSafety, the obvious alternative, returns NOFRAME on this firmware — which is presumably why the vendor's own demos have it commented out. So the toolkit registers a real ISYFrameObserver. That interface is an abstract C++ class with a single virtual method, so a conforming object is synthesised in ctypes: a one-word object whose vptr points into a fake vtable, per the Itanium ABI.

The library loader re-execs the interpreter. libSynexensSDK.so lists two private dependencies as DT_NEEDED but carries no DT_RUNPATH, and neither dependency declares a SONAME — so preloading them by absolute path does not satisfy the loader. glibc caches the search path at process start, so LD_LIBRARY_PATH cannot be set after the fact. ensure_library_path() sets it and re-execs. Long-running hosts should call it at startup, before opening any transport, so the re-exec cannot happen mid-session. CS20_NO_REEXEC=1 opts out.

Frame quality is scored from a single frame, with no reference. The unwrap error is quantised — a wrong wrap index displaces a pixel by a whole multiple of the ~1585 mm ambiguity (a 94.6 MHz modulation), never by an arbitrary amount. Counting pixels that sit a whole ambiguity from their local median separates corrupt frames from clean ones without anything to compare against, because real scene geometry does not place depth steps at exactly 1.585 m.

Tests

python3 test_mcp_client.py

Drives cs20_mcp.py as a real MCP client over stdio and exercises all seven tools end to end against the hardware, including decoding the returned images. Requires the camera to be connected and not held by another process.

Limitations

Only one process can own the camera. The viewer, CLI and MCP server each take exclusive control. Close one before starting another.

Enumeration takes about 15 seconds. FindDevice probes for network cameras on every local network interface before returning, whether or not any exist. Nothing can be done about it in the SDK; the MCP server works around it by keeping the device open between calls.

Effective frame rate is about half the nominal rate. The sensor delivers ~7.5 fps at 640×480, but with corrupt frames discarded the usable rate is ~3.7 fps. That is the real throughput of this device.

The SDK changes your working directory. It writes log/ and parameters/ relative to the CWD, so the toolkit chdirs to $CS20_STATE_DIR (default ~/.cache/cs20) when it initialises. Resolve output paths to absolute before opening a device, or relative writes will land somewhere surprising.

Identity and exposure need opposite stream states. GetDeviceSN and GetDeviceHWVersion work only while streaming is stopped; GetIntegralTime only while it is running.

Several controls are unimplemented on this firmware. Frame rate is not adjustable, and temperature and trigger mode return errors. Device.settings() reports these rather than raising, so it is safe to call on any unit.

EDGE filtering is inverted from what you would guess: lower is more aggressive. On one scene EDGE=80 kept 84.4% of pixels and EDGE=20 kept 73.3%, both with zero speckle. To recover pixels, raise it or drop it from the chain.

Intrinsics report height as 640 at 640×480. An SDK quirk; the real height is 480. fx, fy, cx, cy and the distortion coefficients are correct.

Tested against one unit. A CS20_DUAL on firmware XY-V1-2108200201, SDK 4.2.5.0, x86-64 Ubuntu. Other variants (CS20_P, CS30, CS40) are in the SDK's enums and should mostly work, but are untested here.

Third-party components

The Synexens SDK (libSynexensSDK.so, libcsreconstruction2.0.so, libSonixCamera.so) is proprietary vendor software and is not redistributed in this repository. Download it from support.tofsensors.com under Synexens' own terms; the Install section has the exact URL used here. This project only binds to its public C API.

Product documentation: CS20 product page.

License

MIT — see LICENSE.