# jev-eyes

[![ci](https://github.com/LeddoEngano/jev-eyes/actions/workflows/ci.yml/badge.svg)](https://github.com/LeddoEngano/jev-eyes/actions/workflows/ci.yml)

**Give Jev eyes — honestly.**

[Jev](https://typesafe.ai), TypeSafe's System One model, is text-only. It has no world knowledge and does not see images. Any demo where Jev "looks at" a screenshot is doing perception somewhere else — usually hidden.

jev-eyes does it in the open. It turns an image into the text `state` Jev can actually decide over, **locally**, and hands it back as a plain dict. The state *is* the return value: you always see exactly what Jev saw.

```bash
pip install jev-eyes               # until the PyPI release lands: pip install git+https://github.com/LeddoEngano/jev-eyes
jev-eyes see screenshot.png --show # no API key needed
```

![jev-eyes see on a banking screen: OCR text and a spatial layout map, in about two seconds](docs/demo.gif)

<sub>Rendered from real output of the command above — nothing typed in by hand. `docs/demo.tape` reproduces it as a true recording with [vhs](https://github.com/charmbracelet/vhs).</sub>

## In your code: one new line

```python
from jev_eyes import see
from typesafe_sdk import TypeSafeClient, Noul, Choice

result = TypeSafeClient().system_one(
    state=see("screenshot.png"),          # ← that's the whole integration
    questions={
        "decorative": Noul(instructions="Is this image purely decorative, carrying no information?"),
        "kind": Choice(instructions="What is this image?",
                       criteria={"screenshot": None, "chart": None, "photo": None, "document": None}),
        "needs_vision": Noul(instructions="Is the extracted text insufficient to describe what the image shows?"),
    },
)
result.answers["decorative"].noul      # float 0..1 — Jev's calibrated probability
result.answers["kind"].choice          # str; .probabilities has every option
result.answers["needs_vision"].noul    # your gate: escalate to a VLM only when this is high
```

`see()` never needs a key. `ask(image, questions)` does both steps when `TYPESAFE_API_KEY` is set.

## What Jev receives

Real output on the 1440×900 banking screen committed at `examples/sample_screen.png`. Nothing was edited — reproduce it with `jev-eyes see examples/sample_screen.png --no-blocks --cols 80`.

**`text`** — OCR in reading order; same-row blocks joined by two spaces:

```
Conta corrente
Início  Saldo disponível  Fatura do cartão
Extrato  R$ 4.820,15  Fecha em 3 dias · R$ 1.284,77
Cartões  Limite disponível R$ 1.200,00
Investimentos  atualizado às 14:32· agência 0341 · conta 88192-4
Ajustes
Últimas transações
18 set  Mercado Pão de Açúcar  -R$ 214,90
17 set  Transferência PIX recebida  +R$ 1.500,00
17 set  Assinatura Spotify Premium  -R$ 34,90
16 set  Posto Ipiranga Centro  -R$ 180,00
15 set  Farmácia Droga Raia  -R$ 62,45
14 set  Salário  +R$ 7.200,00
Transferir  Extrato
```

**`layout`** — the same words at their true position on a character grid. This is what lets Jev answer *where* things are ("which region is the main content?", "is the error next to the form?") without a single pixel reaching it:

```
            Conta corrente

 Início       Saldo disponível              Fatura do cartão
 Extrato      R$ 4.820,15                   Fecha em 3 dias · R$ 1.284,77
 Cartões                                    Limite disponível R$ 1.200,00
 Investimentos atualizado às 14:32· agência 0341 · conta 88192-4
 Ajustes
              Últimas transações
              18 set Mercado Pão de Açúcar               -R$ 214,90

              17 set Transferência PIX recebida          +R$ 1.500,00
              17 set Assinatura Spotify Premium          -R$ 34,90

              16 set Posto Ipiranga Centro               -R$ 180,00
              15 set Farmácia Droga Raia                 -R$ 62,45
              14 set Salário                             +R$ 7.200,00

              Transferir Extrato
```

Also in the state: `image` (size, source), `blocks` (`[x, y, w, h]` boxes with OCR confidence) and, if installed, `labels`. `see(img, compact=True)` keeps only `image`, `text` and top label names when tokens matter more than positions.

## Why this and not…

- **…ASCII art?** It only becomes legible at ~200 columns, where it costs **15× the tokens of OCR** for a blurry copy of what OCR returns exactly — and a text model does not read ASCII the way your eyes do. Measured in [`benchmarks/`](benchmarks/README.md).
- **…a vision-language model?** Slower, and it generates prose that Jev then has to re-read. Keep one for what OCR and layout cannot answer; the `needs_vision` question above is the gate.
- **…training a classifier?** Jev *decides*; it does not recognise. Recognition belongs in the perception layer, where pretrained encoders already do it for free.

## Optional: zero-shot labels

```bash
pip install 'jev-eyes[labels]'      # SigLIP via torch + transformers, ~1 GB
```

```python
see("photo.jpg", labels=["a flower", "a dog", "a car"])
```

Scores are a softmax over *the candidates you pass* — a relative ranking, not a calibrated probability, and it says nothing about labels you did not offer. On the banking screen above, the default vocabulary ranks it as **"a text document"** (0.70), then "a blank or empty screen" (0.12) and "a table of data" (0.08) — a fair description of a white synthetic UI, and a reminder that labels are hints for Jev, not truth.

## For agents

The repo ships a skill in the same format TypeSafe uses for Jev itself:

```bash
npx skills add LeddoEngano/jev-eyes --skill jev-eyes     # or copy skills/jev-eyes into your agent's skills dir
```

And an MCP server:

```bash
pip install 'jev-eyes[mcp]'
claude mcp add jev-eyes -- jev-eyes mcp
```

Tools: `see(image_path, labels?, compact?, layout?, cols?)` and `ask(image_path, questions, ...)`. Both return the state alongside any notes about missing backends, so an agent can never silently pretend.

## Honest limits

- **OCR on CPU takes seconds, not milliseconds** — about 1.5–2.5 s per full-resolution screenshot on an 8-core laptop, measured. Crop to the region you care about, cache by image hash, or downscale to ~1280 px. An Apple Vision backend is on the roadmap.
- **Labels cost a model load** — the first call in a process loads ~400 MB (around a minute); after that ~200 ms per image on Apple Silicon, ~2.8 s on plain CPU.
- The default OCR is PP-OCR (PP-OCRv6, keeps accents). `pip install 'jev-eyes[tesseract]'` and `ocr="tesseract"` if you prefer Tesseract.
- **Jev decides; it does not describe.** If your task is "describe this image", you need a vision-language model, not Jev.
- jev-eyes is an independent project, **not affiliated with or endorsed by TypeSafe AI.**

## Roadmap

JavaScript package for `@typesafe-ai/sdk` users · Apple Vision OCR backend · accessibility-tree input for browsers · perceptual-hash cache · PDF pages · a labelled evaluation set so `benchmarks/jev_accuracy.py` has real numbers.

## Development

```bash
python -m venv .venv && .venv/bin/pip install -e '.[dev]'
.venv/bin/pytest -q -m "not slow"
```

MIT.