OpenEye
README.md
# OpenEye
**Open-source eyes for the agent era.**
[](https://pypi.org/project/openeye-sh/)
[](https://pypistats.org/packages/openeye-sh)
[](https://pypi.org/project/openeye-sh/)
[](LICENSE)
[](https://github.com/aw3-technology/openeye.sh/stargazers)
[Website](https://openeye.sh) · [Docs](https://openeye.sh/docs) · [Models](https://openeye.sh/models) · [Blog](https://openeye.sh/blog) · [PyPI](https://pypi.org/project/openeye-sh/)
OpenEye is a CLI-first perception engine that turns raw video into structured world models for robots and autonomous agents. Pull, run, and serve computer-vision models from your terminal — like Ollama, but for vision AI.
## Features
- **Object Detection** — YOLOv8, YOLO26 (currently ships YOLO11n weights), Grounding DINO, RF-DETR, and custom adapters
- **Depth Estimation** — Monocular depth via Depth Anything V2
- **Segmentation** — SAM2 segment anything
- **Scene Understanding** — Spatial relationships, scene graphs, natural language queries
- **VLM Reasoning** — Vision-language model integration (Nebius, OpenRouter)
- **Agentic Loop** — Continuous perception → reasoning → planning pipeline
- **Safety Monitoring** — Zone-based awareness with real-time human detection and robot halt
- **CLI-Native** — Every capability accessible from the terminal
- **REST + WebSocket API** — Serve models over HTTP with a single command
- **Fleet Management** — Register, deploy, and monitor edge devices at scale
- **MLOps** — Model registry, versioning, A/B testing, batch inference
- **Governance** — Policy enforcement, audit logging, safety rule management
- **Hosted API** — `https://openeye.sh/v1` for object detection without running anything locally (pay per call in credits)
- **Web Dashboard** — [openeye.sh/dashboard](https://openeye.sh/dashboard): API keys, credits, history, fleet management, and — connected to your own `openeye serve` — inference, live streams, and more
## Quick Install
```bash
pipx install openeye-sh
```
With YOLO support:
```bash
pipx install "openeye-sh[yolo]"
```
## Quick Start
```bash
# Pull a model
openeye pull yolov8
# Run inference on an image
openeye run yolov8 photo.jpg
# Start the server with REST API, WebSocket, and browser dashboard
openeye serve yolov8
# Live camera feed with real-time detections
openeye watch --models yolov8
# Live camera feed with safety monitoring
openeye watch --models yolov8 --safety
```
## Give Your AI Agent Eyes (MCP)
OpenEye ships an MCP server, so Claude, Cursor, or any MCP client can pull models and run detection, depth, and segmentation on your machine:
```bash
pipx install "openeye-sh[mcp]"
claude mcp add openeye -- openeye-mcp
```
Then ask: *"Run yolov8 on ~/Pictures/desk.jpg and tell me what's on my desk."* Setup for Claude Desktop and other clients: [docs/claude-skill](https://openeye.sh/docs/claude-skill).
## Hosted API Quick Start
No GPU or local install needed for detection:
1. Sign in at [openeye.sh](https://openeye.sh/login) (email/password or 1Profile).
2. Create a key in **Dashboard → Settings → API Keys** (keys start with `oe_` and are shown once).
3. Buy credits in **Dashboard → Settings → Credits**.
4. Call the API:
```bash
curl -X POST https://openeye.sh/v1/detect \
-H "Authorization: Bearer oe_your_key_here" \
-F "file=@photo.jpg"
# or with the CLI
export OPENEYE_API_KEY=oe_your_key_here
openeye api detect photo.jpg
openeye api usage
```
| Endpoint | Status | Credits |
|----------|--------|---------|
| `POST /v1/detect` | Live (YOLOv8n, JPEG/PNG under 4 MB) | 1 |
| `POST /v1/describe` | Returns `503` until the VLM is enabled on the hosted API | 3 |
| `POST /v1/depth` | Returns `501` — self-hosted only (`openeye serve depth-anything`) | 2 |
| `GET /v1/models`, `GET /v1/usage` | Live | free |
The hosted API has no WebSocket streaming; for streaming, run `openeye serve`. See the [docs](https://openeye.sh/docs) for details.
## Use the Dashboard with a Local Server
`openeye serve` allows requests from `https://openeye.sh` by default, so the hosted dashboard can drive a server on your machine:
```bash
openeye serve yolov8 # http://localhost:8000
```
Open [openeye.sh/dashboard](https://openeye.sh/dashboard), click **Server** on the Overview page (the "OpenEye Server URL" dialog), and enter `http://localhost:8000`. Inference, Live, Scene Graph, Metrics, Benchmark, Config, Governance, MLOps, and the live agent loop then run against your server. If your browser blocks the connection from an HTTPS page to `localhost`, serve OpenEye over HTTPS instead.
## Interactive CLI Demo
An interactive walkthrough that runs real OpenEye commands against sample images. Great for evaluating the tool, onboarding, or live presentations.
### Prerequisites
- **Python 3.12+**
- **jq** (optional, for JSON pipeline steps — `brew install jq`)
- **curl** (for the API server step)
### Setup
```bash
# Install the CLI with YOLO and camera support
pipx install -e "./cli[yolo,camera]"
# Pull the YOLOv8 model (6 MB)
openeye pull yolov8
```
### Run the Demo
```bash
./demo.sh
```
`demo.sh` is a bash script: run it on macOS, Linux, WSL, or Git Bash on Windows. It uses `openeye` from your `PATH` (pipx), falling back to `./.venv` or `cli/.venv`. In plain PowerShell or cmd, run the same steps yourself:
```bash
openeye list
openeye pull yolov8
openeye run yolov8 src/assets/demo/scene-warehouse.jpg --pretty
openeye run yolov8 src/assets/demo/scene-warehouse.jpg --visualize
openeye bench yolov8 --warmup 2 --runs 5
openeye serve yolov8 --port 8111
```
The demo is interactive — press **Enter** to advance between steps. Each step shows the command being run and its live output.
### What the Demo Covers
| Step | Command | What it demonstrates |
|------|---------|----------------------|
| 1 | `openeye list` | Browse the model registry (7 vision models) |
| 2 | `openeye pull yolov8` | Download a model — like `ollama pull` |
| 3 | `openeye run yolov8 scene-warehouse.jpg --pretty` | Object detection with structured JSON output |
| 4 | `openeye run yolov8 scene-kitchen.jpg --pretty` | Detection on a different scene (apple, cup, donut) |
| 5 | `openeye run ... --visualize` | Save an annotated image with bounding boxes |
| 6 | `openeye run ... \| jq` | Unix pipeline composability — pipe JSON to jq |
| 7 | `openeye bench yolov8` | Benchmark inference speed (latency, FPS, P95) |
| 8 | `openeye serve yolov8` | Launch a REST API server and query it with curl |
| 9 | Batch loop | Process all 4 demo images and summarize results |
### Demo Images
Four sample images are included in `src/assets/demo/`:
| Image | Scene | Typical detections |
|-------|-------|--------------------|
| `scene-warehouse.jpg` | Warehouse with people and vehicles | person, truck, tv |
| `scene-kitchen.jpg` | Kitchen countertop | apple, cup, donut, bottle |
| `scene-workshop.jpg` | Workshop environment | person, dining table, remote, knife |
| `safety-workspace.jpg` | Safety-critical workspace | (used for safety guardian demos) |
### Output
Annotated images are saved to `demo_output/`. The API server step temporarily runs on port 8111 and shuts down automatically.
### Beyond the Demo
Commands not covered in the scripted demo that you can try interactively:
```bash
# Live webcam detection with terminal overlay
openeye watch --models yolov8
# Safety Guardian — zone-based human proximity alerts
openeye watch --models yolov8 --safety --danger-m 0.5 --caution-m 1.5
# Unitree G1 robot safety demo (dry-run, no robot needed)
openeye g1-demo --control-mode dry_run --demo
# Agentic perception loop — perceive, reason, act
openeye agent run
# Edge device fleet management
openeye fleet ls
# Open-vocabulary detection with text prompts (requires grounding-dino)
openeye pull grounding-dino
openeye run grounding-dino photo.jpg --prompt "person. hard hat. forklift."
```
## Project Structure
```
openeye.sh/
├── cli/ # openeye-sh CLI & inference server (Python)
├── backend/ # Perception engine runtime (Python)
├── src/ # React web frontend (TypeScript)
├── api/ # Vercel functions (auth, data, credits, /v1 API)
├── db/ # Postgres schema
└── docs/ # MkDocs documentation site
```
| Package | Description | Language |
|---------|-------------|----------|
| `cli/` | CLI tool & FastAPI inference server (port 8000), MCP server (`openeye-mcp`) | Python |
| `backend/` | Perception pipeline, self-hosted fleet control plane (port 8001), governance engine | Python |
| `src/` | Website and dashboard at openeye.sh | TypeScript/React |
| `api/` | Vercel functions: Better Auth (`/api/auth`), dashboard data (`/api/data`), credits proxy (`/api/cred`), hosted inference API (`/v1`) | TypeScript |
| `db/` | Postgres schema (`schema.sql`) shared by the website and the fleet control plane | SQL |
## Tech Stack
**CLI & Backend**: Python, Typer, Rich, FastAPI, Uvicorn, Pydantic
**Frontend**: React, TypeScript, Vite, Tailwind CSS, shadcn/ui, TanStack Query
**Website & API**: Vercel (static SPA + functions in `api/`), Neon Postgres, Vercel Blob (avatars)
**Auth**: Better Auth — email/password and "Continue with 1Profile"; Google/Apple sign-in when their OAuth credentials are configured
**Credits**: [Cred](https://cred.diy), proxied through `/api/cred`
**AI/ML**: Ultralytics (YOLO), Transformers, ONNX Runtime, TensorRT, LangChain
## Development
### Frontend
```bash
npm install
npm run dev # Vite only, http://localhost:8080 (no api/ functions)
```
To run the `api/` functions too (sign-in, dashboard data, credits, `/v1`), use the Vercel CLI:
```bash
vercel env pull .env.local # DATABASE_URL, BETTER_AUTH_SECRET, ... (see .env.example)
npm run db:migrate # apply db/schema.sql to that database
vercel dev
```
Without the server functions you can still use **Continue as Demo User** on the login page.
### CLI
```bash
cd cli
pip install -e ".[all]" --group dev
```
### Backend
```bash
cd backend
pip install -e . --group dev
```
The fleet control plane in `backend/src/fleet` is self-hosted: point `DATABASE_URL` at a Postgres database (Neon works) with `db/schema.sql` applied. See `backend/.env.example`.
## Documentation
- [openeye.sh/docs](https://openeye.sh/docs) — the main docs (source: `src/data/docs/`)
- `docs/` — MkDocs Material site with longer guides and tutorials
## License
See [LICENSE](LICENSE) for details.
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues