VAAS

VAAS is a compact prototype for treating vision as a queryable data source, not a stream of disposable screenshots. It gives an agent a CLI and an MCP server for image portfolios, recorded video, sampled camera feeds, attention timelines, coarse face signals, and persistent visual entities.

This is portfolio-grade infrastructure: real, testable, and deliberately modular. It is not a production biometric, surveillance, or safety-critical system.

See VAAS in action

Related MCP server: Librarian

What it demonstrates

60-second demo

Requires Python 3.10+.

python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

# Creates a tiny synthetic portfolio, indexes it, and runs visual search.
python examples/quickstart.py

Or point the CLI at real media:

vaas index ~/Pictures/portfolio --tag portfolio
vaas search portfolio --limit 5
vaas search --image ~/Pictures/query.jpg --limit 5
vaas inspect asset_abc123

Video and coarse facial signals are opt-in:

pip install -e '.[video]'
vaas --face-signals index demo.mp4 --sample-every 1.5 --tag interview
vaas timeline demo.mp4
vaas export asset_abc123 exports/interesting-frame.jpg

The default descriptor is intentionally tiny and supports example-image similarity. For semantic text-to-image search, select OpenCLIP:

pip install -e '.[semantic]'
vaas --embedder openclip index ~/Pictures/portfolio
vaas --embedder openclip search "a red bicycle beside a brick wall"

The first OpenCLIP run downloads the selected model weights. A catalog only compares vectors produced by the same backend, so index and search with the same --embedder value.

MCP server

Install the official SDK adapter and run over stdio:

pip install -e '.[mcp,video]'
vaas --db ~/.local/share/vaas/catalog.db serve

Example MCP client configuration:

{
  "mcpServers": {
    "vaas": {
      "command": "/absolute/path/to/.venv/bin/vaas",
      "args": ["--db", "/absolute/path/to/vaas.db", "serve"]
    }
  }
}

The agent receives seven focused tools:

A useful agent loop looks like this:

index_visual_path("meeting.mp4", sample_every_seconds=1)
→ search_visual_memory("whiteboard", limit=3)
→ inspect_visual_asset(best_asset_id)
→ export_visual_frame(best_asset_id, "exports/whiteboard.jpg")
→ agent inspects the exported image with its native vision model

Python API

from vaas import VAAS

vision = VAAS("vaas.db", embedder="visual", face_signals=False)
frames = vision.index_video("meeting.mp4", sample_every=2.0, max_frames=300)

matches = vision.search(image="whiteboard-photo.jpg", limit=5)
best = matches[0]
print(best.record.source_uri, best.record.timestamp, best.record.attention)

entity = vision.resolve_entity(best.record.id, kind="scene", label="planning board")
vision.export_frame(best.record.id, "exports/planning-board.jpg")

See examples/agent_workflow.py for a complete media-to-evidence flow.

How it fits together

flowchart LR
  A["Images · video · camera snapshots"] --> B["Sampler + provenance"]
  B --> C["Pluggable classifiers"]
  C --> D["Visual embeddings"]
  C --> E["Attention + face signals"]
  D --> F[("SQLite visual catalog")]
  E --> F
  F --> G["Similarity + metadata search"]
  F --> H["Entity prototypes"]
  G --> I["CLI"]
  H --> I
  G --> J["MCP tools"]
  H --> J
  J --> K["Any multimodal agent"]
  K --> L["Export exact frame"]

The package has one orchestration API, VAAS, used by both interfaces. Adapters conform to a small Embedder protocol. SQLite holds metadata and float32 vectors; NumPy performs exact cosine search. That makes the demo transparent and portable. At portfolio scale, swap the vector scan for Faiss, Qdrant, Milvus, or pgvector without changing the agent tools.

Data model

source ──< asset/frame ──> embedding
                 │
                 ├── attention {score, focus_x, focus_y, entropy, 8×8 grid}
                 ├── signals   {faces, smiles, adapter-specific outputs}
                 └── observation >── entity {kind, label, running centroid}

Every result preserves its original source, video timestamp, frame number, content hash, dimensions, embedding model, and analysis metadata. VAAS stores indexes—not copied media—unless export is explicitly called.

Extending the sensing library

VAAS does not vendor model code. Add an adapter and keep weights/licenses with their upstream project:

class MyEmbedder:
    name = "my-model:v1"

    def embed_image(self, image):
        return normalized_numpy_vector

    def embed_text(self, text):
        return normalized_numpy_vector


vision = VAAS("catalog.db", embedder=MyEmbedder())

Good next adapters include:

• DINOv2 for general-purpose visual similarity and entity features.

• SAM 2 for promptable object masks and temporal object tracking.

• MediaPipe Face Landmarker for blendshapes, head pose, and richer interaction signals.

• PySceneDetect for production-grade shot boundaries.

• Faiss/HNSW for million-scale approximate nearest-neighbor search.

Boundaries and responsible use

• The built-in face adapter detects coarse face/smile patterns. It does not identify people, infer emotion, or establish intent.

• Smile detection is a noisy visual signal, not an emotional truth. Treat all facial outputs as uncertain observations.

• Obtain consent before processing cameras, calls, faces, or private media. Follow retention and access-control requirements.

• Keep a human in the loop for consequential uses. Benchmark every selected model on the actual domain and demographic mix.

• Validate paths and add authentication/authorization before exposing the MCP server beyond a trusted local process.

Model & library credits

VAAS uses or is designed to interoperate with these excellent projects. Their code is referenced through normal dependencies; none is copied into this repository.

• Pillow and NumPy — image I/O and local numeric features.

• SQLite — portable catalog and entity registry.

• OpenCV — optional video decoding, shot/motion statistics, Haar face/smile signals.

• OpenCLIP — optional text/image embeddings; based on Radford et al., Learning Transferable Visual Models From Natural Language Supervision.

• MCP Python SDK — the stdio agent interface.

• Oquab et al., DINOv2: Learning Robust Visual Features without Supervision — recommended general visual features adapter.

• Ravi et al., SAM 2: Segment Anything in Images and Videos — recommended mask and object-tracking adapter.

• Lugaresi et al., MediaPipe: A Framework for Building Perception Pipelines — recommended real-time landmark/blendshape adapter.

• Johnson, Douze, and Jégou, Billion-scale similarity search with GPUs and Faiss — recommended large-scale vector index.

Development

pip install -e '.[dev]'
pytest -q
ruff check .

# Regenerate the README MP4 and GIF (requires ffmpeg).
python scripts/render_readme_demo.py

The MIT license covers VAAS itself. Optional models, weights, and dependencies retain their own licenses and usage terms.