Skip to main content
Glama

OpenEye

Open-source eyes for the agent era.

PyPI Downloads Python License: MIT GitHub stars

Website · Docs · Models · Blog · PyPI

OpenEye is a CLI-first perception engine that turns raw video into structured world models for robots and autonomous agents. Pull, run, and serve computer-vision models from your terminal — like Ollama, but for vision AI.

Features

  • Object Detection — YOLOv8, YOLO26 (currently ships YOLO11n weights), Grounding DINO, RF-DETR, and custom adapters

  • Depth Estimation — Monocular depth via Depth Anything V2

  • Segmentation — SAM2 segment anything

  • Scene Understanding — Spatial relationships, scene graphs, natural language queries

  • VLM Reasoning — Vision-language model integration (Nebius, OpenRouter)

  • Agentic Loop — Continuous perception → reasoning → planning pipeline

  • Safety Monitoring — Zone-based awareness with real-time human detection and robot halt

  • CLI-Native — Every capability accessible from the terminal

  • REST + WebSocket API — Serve models over HTTP with a single command

  • Fleet Management — Register, deploy, and monitor edge devices at scale

  • MLOps — Model registry, versioning, A/B testing, batch inference

  • Governance — Policy enforcement, audit logging, safety rule management

  • Hosted API — https://openeye.sh/v1 for object detection without running anything locally (pay per call in credits)

  • Web Dashboard — openeye.sh/dashboard: API keys, credits, history, fleet management, and — connected to your own openeye serve — inference, live streams, and more

Related MCP server: vision-eyes

Quick Install

pipx install openeye-sh

With YOLO support:

pipx install "openeye-sh[yolo]"

Quick Start

# Pull a model
openeye pull yolov8

# Run inference on an image
openeye run yolov8 photo.jpg

# Start the server with REST API, WebSocket, and browser dashboard
openeye serve yolov8

# Live camera feed with real-time detections
openeye watch --models yolov8

# Live camera feed with safety monitoring
openeye watch --models yolov8 --safety

Give Your AI Agent Eyes (MCP)

OpenEye ships an MCP server, so Claude, Cursor, or any MCP client can pull models and run detection, depth, and segmentation on your machine:

pipx install "openeye-sh[mcp]"
claude mcp add openeye -- openeye-mcp

Then ask: "Run yolov8 on ~/Pictures/desk.jpg and tell me what's on my desk." Setup for Claude Desktop and other clients: docs/claude-skill.

Hosted API Quick Start

No GPU or local install needed for detection:

  1. Sign in at openeye.sh (email/password or 1Profile).

  2. Create a key in Dashboard → Settings → API Keys (keys start with oe_ and are shown once).

  3. Buy credits in Dashboard → Settings → Credits.

  4. Call the API:

curl -X POST https://openeye.sh/v1/detect \
  -H "Authorization: Bearer oe_your_key_here" \
  -F "file=@photo.jpg"

# or with the CLI
export OPENEYE_API_KEY=oe_your_key_here
openeye api detect photo.jpg
openeye api usage

Endpoint

Status

Credits

POST /v1/detect

Live (YOLOv8n, JPEG/PNG under 4 MB)

1

POST /v1/describe

Returns 503 until the VLM is enabled on the hosted API

3

POST /v1/depth

Returns 501 — self-hosted only (openeye serve depth-anything)

2

GET /v1/models, GET /v1/usage

Live

free

The hosted API has no WebSocket streaming; for streaming, run openeye serve. See the docs for details.

Use the Dashboard with a Local Server

openeye serve allows requests from https://openeye.sh by default, so the hosted dashboard can drive a server on your machine:

openeye serve yolov8          # http://localhost:8000

Open openeye.sh/dashboard, click Server on the Overview page (the "OpenEye Server URL" dialog), and enter http://localhost:8000. Inference, Live, Scene Graph, Metrics, Benchmark, Config, Governance, MLOps, and the live agent loop then run against your server. If your browser blocks the connection from an HTTPS page to localhost, serve OpenEye over HTTPS instead.

Interactive CLI Demo

An interactive walkthrough that runs real OpenEye commands against sample images. Great for evaluating the tool, onboarding, or live presentations.

Prerequisites

  • Python 3.12+

  • jq (optional, for JSON pipeline steps — brew install jq)

  • curl (for the API server step)

Setup

# Install the CLI with YOLO and camera support
pipx install -e "./cli[yolo,camera]"

# Pull the YOLOv8 model (6 MB)
openeye pull yolov8

Run the Demo

./demo.sh

demo.sh is a bash script: run it on macOS, Linux, WSL, or Git Bash on Windows. It uses openeye from your PATH (pipx), falling back to ./.venv or cli/.venv. In plain PowerShell or cmd, run the same steps yourself:

openeye list
openeye pull yolov8
openeye run yolov8 src/assets/demo/scene-warehouse.jpg --pretty
openeye run yolov8 src/assets/demo/scene-warehouse.jpg --visualize
openeye bench yolov8 --warmup 2 --runs 5
openeye serve yolov8 --port 8111

The demo is interactive — press Enter to advance between steps. Each step shows the command being run and its live output.

What the Demo Covers

Step

Command

What it demonstrates

1

openeye list

Browse the model registry (7 vision models)

2

openeye pull yolov8

Download a model — like ollama pull

3

openeye run yolov8 scene-warehouse.jpg --pretty

Object detection with structured JSON output

4

openeye run yolov8 scene-kitchen.jpg --pretty

Detection on a different scene (apple, cup, donut)

5

openeye run ... --visualize

Save an annotated image with bounding boxes

6

openeye run ... | jq

Unix pipeline composability — pipe JSON to jq

7

openeye bench yolov8

Benchmark inference speed (latency, FPS, P95)

8

openeye serve yolov8

Launch a REST API server and query it with curl

9

Batch loop

Process all 4 demo images and summarize results

Demo Images

Four sample images are included in src/assets/demo/:

Image

Scene

Typical detections

scene-warehouse.jpg

Warehouse with people and vehicles

person, truck, tv

scene-kitchen.jpg

Kitchen countertop

apple, cup, donut, bottle

scene-workshop.jpg

Workshop environment

person, dining table, remote, knife

safety-workspace.jpg

Safety-critical workspace

(used for safety guardian demos)

Output

Annotated images are saved to demo_output/. The API server step temporarily runs on port 8111 and shuts down automatically.

Beyond the Demo

Commands not covered in the scripted demo that you can try interactively:

# Live webcam detection with terminal overlay
openeye watch --models yolov8

# Safety Guardian — zone-based human proximity alerts
openeye watch --models yolov8 --safety --danger-m 0.5 --caution-m 1.5

# Unitree G1 robot safety demo (dry-run, no robot needed)
openeye g1-demo --control-mode dry_run --demo

# Agentic perception loop — perceive, reason, act
openeye agent run

# Edge device fleet management
openeye fleet ls

# Open-vocabulary detection with text prompts (requires grounding-dino)
openeye pull grounding-dino
openeye run grounding-dino photo.jpg --prompt "person. hard hat. forklift."

Project Structure

openeye.sh/
├── cli/              # openeye-sh CLI & inference server (Python)
├── backend/          # Perception engine runtime (Python)
├── src/              # React web frontend (TypeScript)
├── api/              # Vercel functions (auth, data, credits, /v1 API)
├── db/               # Postgres schema
└── docs/             # MkDocs documentation site

Package

Description

Language

cli/

CLI tool & FastAPI inference server (port 8000), MCP server (openeye-mcp)

Python

backend/

Perception pipeline, self-hosted fleet control plane (port 8001), governance engine

Python

src/

Website and dashboard at openeye.sh

TypeScript/React

api/

Vercel functions: Better Auth (/api/auth), dashboard data (/api/data), credits proxy (/api/cred), hosted inference API (/v1)

TypeScript

db/

Postgres schema (schema.sql) shared by the website and the fleet control plane

SQL

Tech Stack

CLI & Backend: Python, Typer, Rich, FastAPI, Uvicorn, Pydantic

Frontend: React, TypeScript, Vite, Tailwind CSS, shadcn/ui, TanStack Query

Website & API: Vercel (static SPA + functions in api/), Neon Postgres, Vercel Blob (avatars)

Auth: Better Auth — email/password and "Continue with 1Profile"; Google/Apple sign-in when their OAuth credentials are configured

Credits: Cred, proxied through /api/cred

AI/ML: Ultralytics (YOLO), Transformers, ONNX Runtime, TensorRT, LangChain

Development

Frontend

npm install
npm run dev          # Vite only, http://localhost:8080 (no api/ functions)

To run the api/ functions too (sign-in, dashboard data, credits, /v1), use the Vercel CLI:

vercel env pull .env.local   # DATABASE_URL, BETTER_AUTH_SECRET, ... (see .env.example)
npm run db:migrate           # apply db/schema.sql to that database
vercel dev

Without the server functions you can still use Continue as Demo User on the login page.

CLI

cd cli
pip install -e ".[all]" --group dev

Backend

cd backend
pip install -e . --group dev

The fleet control plane in backend/src/fleet is self-hosted: point DATABASE_URL at a Postgres database (Neon works) with db/schema.sql applied. See backend/.env.example.

Documentation

  • openeye.sh/docs — the main docs (source: src/data/docs/)

  • docs/ — MkDocs Material site with longer guides and tutorials

License

See LICENSE for details.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      D
      maintenance
      Provides vision understanding capabilities such as image analysis, OCR, object localization, and video frame analysis, plus optional image generation and editing, to coding agents via OpenAI-compatible multimodal models. Runs as a local MCP server with HTTP and stdio transports, configurable for clients like Codex, Claude Code, Kimi, and Cursor.
      3
      17 npm
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to perform detailed local image analysis, including pixel-level queries, cropping, grid coordinates, comparisons, and zero-shot classification, all without cloud APIs, keys, or payments.
      1
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI coding models that lack native vision to analyze screenshots, error messages, UI layouts, documents, and OCR-heavy images by routing each request to one of several major vision LLM providers through a single unified tool. Handles local files, remote URLs, and Data URIs with automatic compression, multi-cropping for large images, retries, and optional HTTP/Docker deployment for shared access.
      512 npm
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI coding assistants without native vision to analyze screenshots, errors, UI designs, and documents through a unified image understanding tool backed by multiple vision model providers. Supports local files, remote URLs, Data URIs, OCR, multi-crop processing, and stdio or HTTP deployment for shared access.
      512 npm
      MIT