Skip to main content
Glama
README.md
# OpenEye

**Open-source eyes for the agent era.**

[![PyPI](https://img.shields.io/pypi/v/openeye-sh)](https://pypi.org/project/openeye-sh/)
[![Downloads](https://img.shields.io/pypi/dm/openeye-sh)](https://pypistats.org/packages/openeye-sh)
[![Python](https://img.shields.io/pypi/pyversions/openeye-sh)](https://pypi.org/project/openeye-sh/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
[![GitHub stars](https://img.shields.io/github/stars/aw3-technology/openeye.sh?style=social)](https://github.com/aw3-technology/openeye.sh/stargazers)

[Website](https://openeye.sh) · [Docs](https://openeye.sh/docs) · [Models](https://openeye.sh/models) · [Blog](https://openeye.sh/blog) · [PyPI](https://pypi.org/project/openeye-sh/)

OpenEye is a CLI-first perception engine that turns raw video into structured world models for robots and autonomous agents. Pull, run, and serve computer-vision models from your terminal — like Ollama, but for vision AI.

## Features

- **Object Detection** — YOLOv8, YOLO26 (currently ships YOLO11n weights), Grounding DINO, RF-DETR, and custom adapters
- **Depth Estimation** — Monocular depth via Depth Anything V2
- **Segmentation** — SAM2 segment anything
- **Scene Understanding** — Spatial relationships, scene graphs, natural language queries
- **VLM Reasoning** — Vision-language model integration (Nebius, OpenRouter)
- **Agentic Loop** — Continuous perception → reasoning → planning pipeline
- **Safety Monitoring** — Zone-based awareness with real-time human detection and robot halt
- **CLI-Native** — Every capability accessible from the terminal
- **REST + WebSocket API** — Serve models over HTTP with a single command
- **Fleet Management** — Register, deploy, and monitor edge devices at scale
- **MLOps** — Model registry, versioning, A/B testing, batch inference
- **Governance** — Policy enforcement, audit logging, safety rule management
- **Hosted API** — `https://openeye.sh/v1` for object detection without running anything locally (pay per call in credits)
- **Web Dashboard** — [openeye.sh/dashboard](https://openeye.sh/dashboard): API keys, credits, history, fleet management, and — connected to your own `openeye serve` — inference, live streams, and more

## Quick Install

```bash
pipx install openeye-sh
```

With YOLO support:

```bash
pipx install "openeye-sh[yolo]"
```

## Quick Start

```bash
# Pull a model
openeye pull yolov8

# Run inference on an image
openeye run yolov8 photo.jpg

# Start the server with REST API, WebSocket, and browser dashboard
openeye serve yolov8

# Live camera feed with real-time detections
openeye watch --models yolov8

# Live camera feed with safety monitoring
openeye watch --models yolov8 --safety
```

## Give Your AI Agent Eyes (MCP)

OpenEye ships an MCP server, so Claude, Cursor, or any MCP client can pull models and run detection, depth, and segmentation on your machine:

```bash
pipx install "openeye-sh[mcp]"
claude mcp add openeye -- openeye-mcp
```

Then ask: *"Run yolov8 on ~/Pictures/desk.jpg and tell me what's on my desk."* Setup for Claude Desktop and other clients: [docs/claude-skill](https://openeye.sh/docs/claude-skill).

## Hosted API Quick Start

No GPU or local install needed for detection:

1. Sign in at [openeye.sh](https://openeye.sh/login) (email/password or 1Profile).
2. Create a key in **Dashboard → Settings → API Keys** (keys start with `oe_` and are shown once).
3. Buy credits in **Dashboard → Settings → Credits**.
4. Call the API:

```bash
curl -X POST https://openeye.sh/v1/detect \
  -H "Authorization: Bearer oe_your_key_here" \
  -F "file=@photo.jpg"

# or with the CLI
export OPENEYE_API_KEY=oe_your_key_here
openeye api detect photo.jpg
openeye api usage
```

| Endpoint | Status | Credits |
|----------|--------|---------|
| `POST /v1/detect` | Live (YOLOv8n, JPEG/PNG under 4 MB) | 1 |
| `POST /v1/describe` | Returns `503` until the VLM is enabled on the hosted API | 3 |
| `POST /v1/depth` | Returns `501` — self-hosted only (`openeye serve depth-anything`) | 2 |
| `GET /v1/models`, `GET /v1/usage` | Live | free |

The hosted API has no WebSocket streaming; for streaming, run `openeye serve`. See the [docs](https://openeye.sh/docs) for details.

## Use the Dashboard with a Local Server

`openeye serve` allows requests from `https://openeye.sh` by default, so the hosted dashboard can drive a server on your machine:

```bash
openeye serve yolov8          # http://localhost:8000
```

Open [openeye.sh/dashboard](https://openeye.sh/dashboard), click **Server** on the Overview page (the "OpenEye Server URL" dialog), and enter `http://localhost:8000`. Inference, Live, Scene Graph, Metrics, Benchmark, Config, Governance, MLOps, and the live agent loop then run against your server. If your browser blocks the connection from an HTTPS page to `localhost`, serve OpenEye over HTTPS instead.

## Interactive CLI Demo

An interactive walkthrough that runs real OpenEye commands against sample images. Great for evaluating the tool, onboarding, or live presentations.

### Prerequisites

- **Python 3.12+**
- **jq** (optional, for JSON pipeline steps — `brew install jq`)
- **curl** (for the API server step)

### Setup

```bash
# Install the CLI with YOLO and camera support
pipx install -e "./cli[yolo,camera]"

# Pull the YOLOv8 model (6 MB)
openeye pull yolov8
```

### Run the Demo

```bash
./demo.sh
```

`demo.sh` is a bash script: run it on macOS, Linux, WSL, or Git Bash on Windows. It uses `openeye` from your `PATH` (pipx), falling back to `./.venv` or `cli/.venv`. In plain PowerShell or cmd, run the same steps yourself:

```bash
openeye list
openeye pull yolov8
openeye run yolov8 src/assets/demo/scene-warehouse.jpg --pretty
openeye run yolov8 src/assets/demo/scene-warehouse.jpg --visualize
openeye bench yolov8 --warmup 2 --runs 5
openeye serve yolov8 --port 8111
```

The demo is interactive — press **Enter** to advance between steps. Each step shows the command being run and its live output.

### What the Demo Covers

| Step | Command | What it demonstrates |
|------|---------|----------------------|
| 1 | `openeye list` | Browse the model registry (7 vision models) |
| 2 | `openeye pull yolov8` | Download a model — like `ollama pull` |
| 3 | `openeye run yolov8 scene-warehouse.jpg --pretty` | Object detection with structured JSON output |
| 4 | `openeye run yolov8 scene-kitchen.jpg --pretty` | Detection on a different scene (apple, cup, donut) |
| 5 | `openeye run ... --visualize` | Save an annotated image with bounding boxes |
| 6 | `openeye run ... \| jq` | Unix pipeline composability — pipe JSON to jq |
| 7 | `openeye bench yolov8` | Benchmark inference speed (latency, FPS, P95) |
| 8 | `openeye serve yolov8` | Launch a REST API server and query it with curl |
| 9 | Batch loop | Process all 4 demo images and summarize results |

### Demo Images

Four sample images are included in `src/assets/demo/`:

| Image | Scene | Typical detections |
|-------|-------|--------------------|
| `scene-warehouse.jpg` | Warehouse with people and vehicles | person, truck, tv |
| `scene-kitchen.jpg` | Kitchen countertop | apple, cup, donut, bottle |
| `scene-workshop.jpg` | Workshop environment | person, dining table, remote, knife |
| `safety-workspace.jpg` | Safety-critical workspace | (used for safety guardian demos) |

### Output

Annotated images are saved to `demo_output/`. The API server step temporarily runs on port 8111 and shuts down automatically.

### Beyond the Demo

Commands not covered in the scripted demo that you can try interactively:

```bash
# Live webcam detection with terminal overlay
openeye watch --models yolov8

# Safety Guardian — zone-based human proximity alerts
openeye watch --models yolov8 --safety --danger-m 0.5 --caution-m 1.5

# Unitree G1 robot safety demo (dry-run, no robot needed)
openeye g1-demo --control-mode dry_run --demo

# Agentic perception loop — perceive, reason, act
openeye agent run

# Edge device fleet management
openeye fleet ls

# Open-vocabulary detection with text prompts (requires grounding-dino)
openeye pull grounding-dino
openeye run grounding-dino photo.jpg --prompt "person. hard hat. forklift."
```

## Project Structure

```
openeye.sh/
├── cli/              # openeye-sh CLI & inference server (Python)
├── backend/          # Perception engine runtime (Python)
├── src/              # React web frontend (TypeScript)
├── api/              # Vercel functions (auth, data, credits, /v1 API)
├── db/               # Postgres schema
└── docs/             # MkDocs documentation site
```

| Package | Description | Language |
|---------|-------------|----------|
| `cli/` | CLI tool & FastAPI inference server (port 8000), MCP server (`openeye-mcp`) | Python |
| `backend/` | Perception pipeline, self-hosted fleet control plane (port 8001), governance engine | Python |
| `src/` | Website and dashboard at openeye.sh | TypeScript/React |
| `api/` | Vercel functions: Better Auth (`/api/auth`), dashboard data (`/api/data`), credits proxy (`/api/cred`), hosted inference API (`/v1`) | TypeScript |
| `db/` | Postgres schema (`schema.sql`) shared by the website and the fleet control plane | SQL |

## Tech Stack

**CLI & Backend**: Python, Typer, Rich, FastAPI, Uvicorn, Pydantic

**Frontend**: React, TypeScript, Vite, Tailwind CSS, shadcn/ui, TanStack Query

**Website & API**: Vercel (static SPA + functions in `api/`), Neon Postgres, Vercel Blob (avatars)

**Auth**: Better Auth — email/password and "Continue with 1Profile"; Google/Apple sign-in when their OAuth credentials are configured

**Credits**: [Cred](https://cred.diy), proxied through `/api/cred`

**AI/ML**: Ultralytics (YOLO), Transformers, ONNX Runtime, TensorRT, LangChain

## Development

### Frontend

```bash
npm install
npm run dev          # Vite only, http://localhost:8080 (no api/ functions)
```

To run the `api/` functions too (sign-in, dashboard data, credits, `/v1`), use the Vercel CLI:

```bash
vercel env pull .env.local   # DATABASE_URL, BETTER_AUTH_SECRET, ... (see .env.example)
npm run db:migrate           # apply db/schema.sql to that database
vercel dev
```

Without the server functions you can still use **Continue as Demo User** on the login page.

### CLI

```bash
cd cli
pip install -e ".[all]" --group dev
```

### Backend

```bash
cd backend
pip install -e . --group dev
```

The fleet control plane in `backend/src/fleet` is self-hosted: point `DATABASE_URL` at a Postgres database (Neon works) with `db/schema.sql` applied. See `backend/.env.example`.

## Documentation

- [openeye.sh/docs](https://openeye.sh/docs) — the main docs (source: `src/data/docs/`)
- `docs/` — MkDocs Material site with longer guides and tutorials

## License

See [LICENSE](LICENSE) for details.