Skip to main content
Glama
vision-gesture-control

vision-gesture-control

by 1nc0gn30

Vision Gesture Control

Touchless 21-Joint 3D Hand Tracking, Spatial Air Drawing, Dwell Navigation & Model Context Protocol (MCP) Server (v2.5.0)

Overview

Vision Gesture Control combines a client-side WebAssembly/WebGL computer vision workstation (public/) with a zero-dependency Python standard-library HTTP server, CLI, and JSON-RPC 2.0 Model Context Protocol (MCP) server (src/vision_gesture_control/).

Key capabilities:

  • Sub-15ms Latency Hand & Face Kinematics: Tracks 21 3D hand landmarks per hand with adaptive OneEuroFilter smoothing, curl-angle heuristics, pinch ratios, dual-hand span/ROI lens framing, and 468-point facial EAR/MAR metrics.

  • Interactive Browser Workspaces: Touchless presentation slide controller with dwell rings, velocity-sensitive spline air-drawing canvas, 3D Three.js spatial geometry portal, 13 optical shader filters, and a 41-preset WebAudio spatial loop/Solfeggio studio (including a built-in Virtual Hand Simulator when no webcam is attached).

  • Zero Cloud Egress: Video frames remain in volatile local memory for coordinate extraction and are never uploaded or written to disk.

Related MCP server: APIVerve MCP Server

Architecture

vision-gesture-control/
├── server.py                             # Standalone workstation entrypoint on 127.0.0.1:8089
├── bin/vision-gesture.js                 # Unified Node/ZothOS CLI & Electron desktop wrapper
├── public/
│   ├── index.html                        # Multi-tab Vision Studio UI & Virtual Hand Simulator
│   └── js/
│       ├── tool-air-drawing.js           # Quadratic spline air-drawing & palm-hold clear engine
│       ├── tool-presentation-spatial.js  # Slide laser/dwell controller & Three.js 3D portal
│       ├── tool-playground-lens.js       # Dual-hand EMA/deadzone ROI shader lens
│       ├── tool-gesturefx.js             # Gesture particle emitters, macropad & Biquad audio filter
│       └── tool-spatial-audio.js         # 41-preset WebAudio loop, synth, vocal & Solfeggio engine
└── src/vision_gesture_control/
    ├── cli.py                            # `vision-control` CLI (serve, classify, dwell, mcp, map, platform, test, status)
    ├── ui_server.py                      # ThreadingHTTPServer & REST API on 127.0.0.1:8089
    ├── mcp_server.py                     # Stdio JSON-RPC 2.0 MCP server (8 tools)
    ├── gestures.py                       # 3D joint-angle, pinch, dwell & facial EAR/MAR classifiers
    ├── filter_pipeline.py                # OneEuroFilter, 3D landmark smoother & filter pipeline
    ├── trajectory.py                     # Swipe velocity & pinch-zoom delta tracker
    └── compat.py                         # Multi-OS display server (X11/Wayland/macOS/Windows) & camera discovery

Quick Start

git clone https://github.com/1nc0gn30/vision-gesture-control.git
cd vision-gesture-control
pip install -e .
python3 server.py --port 8089

Open http://127.0.0.1:8089 in any WebGL-capable browser. No external Python runtime packages are required to run server.py or PYTHONPATH=src python3 -m vision_gesture_control.

CLI & MCP Usage

Python & ZothOS v3.4.0 CLI

On ZothOS v3.4.0 (or after pip install -e .), invoke vision-control or node bin/vision-gesture.js:

# Inspect CLI flags and subcommands
vision-control --help
node bin/vision-gesture.js --status --json

# Start the web studio on 127.0.0.1:8089
vision-control serve --host 127.0.0.1 --port 8089

# Classify 21 hand landmarks from a JSON file, inline JSON, or built-in sample
vision-control classify
vision-control classify --landmarks-json landmarks.json --handedness Right --confidence-threshold 0.6

# Evaluate pointer dwell progress inside an interactive bounding box (x,y,width,height)
vision-control dwell --x 0.45 --y 0.50 --zone 0.2,0.2,0.6,0.6 --duration 1000 --elapsed 650

# Export gesture-to-action key maps (presentation, media, drawing, gaming, accessibility)
vision-control map --preset presentation --format table

# Run OS, display server, and camera telemetry diagnostics
vision-control platform --extended

# Dump MCP tool schemas or client configuration snippets
vision-control mcp --tools
vision-control mcp --config claude_desktop

REST API Endpoints (127.0.0.1:8089)

Method

Endpoint

Description

GET

/health, /api/health

Service health status, version (2.5.0), and uptime in seconds

GET

/api/presets

Gesture action maps (?preset=presentation|media|drawing|gaming|accessibility|all)

GET

/api/mcp/config

Client config JSON (?client=claude_desktop|cursor|cline|zed|all)

GET

/api/diagnostics

System, Python, display server, CPU, and camera telemetry (?extended=true)

POST

/api/classify

Classify { "landmarks": [...21 pts...], "handedness": "Right" }

POST

/api/dwell

Compute dwell progress for { "x", "y", "zone", "duration_ms", "elapsed_ms" }

POST

/api/face

Classify facial blink/smile/mouth states from { "face_landmarks": ... }

GET / POST

/api/export-bundle

Download standalone offline vision-gesture-studio-standalone.zip bundle

MCP Server Tools (stdio)

Configure your MCP host with python3 -m vision_gesture_control.mcp_server (see docs/MCP_GUIDE.md):

  • vision_classify_gesture — Classify 21 3D/2D hand landmarks with confidence, pinch state, and finger extension flags.

  • vision_analyze_dual_hands — Compute inter-hand wrist/index span, Z-depth delta, dual-pinch state, and ROI lens bounds.

  • vision_calculate_dwell — Evaluate normalized dwell progress and trigger state inside a bounding box.

  • vision_classify_face — Evaluate facial landmarks or EAR/MAR ratios for blink, smile, and mouth-open states.

  • vision_generate_action_map — Generate preset or custom-overridden gesture key bindings.

  • vision_get_shader_filters — List the 13 optical shader filters and their gesture triggers.

  • vision_get_solfeggio_frequencies — List the 9 Solfeggio frequencies (174 Hz–963 Hz) and Theremin axis mappings.

  • vision_get_diagnostics — Query OS, display server, CPU, and camera diagnostics.

Testing & Verification

# Run full unit & integration test suite (ephemeral port=0)
PYTHONPATH=src python3 -m pytest -q

# Run built-in stdlib self-tests
PYTHONPATH=src python3 -m vision_gesture_control --test

# Verify CLI status JSON
python3 server.py --status --json

Support

Vision Gesture Control is 100% free and open-source with zero paywalls or feature gates.

  • Optional support: Buy Us a Coffee (Stripe)

  • Solana Tip Address: GG2eQbVh8ut1BK6j1A2tz7pSySdLpD38BgyEhGcxdXoK (solana:GG2eQbVh8ut1BK6j1A2tz7pSySdLpD38BgyEhGcxdXoK)

License & Attribution

Released under the MIT License. Engineered by Neal Frazier Tech (@1nc0gn30, github.com/1nc0gn30/vision-gesture-control) • NullAI Platform • Cataloged in Zoth Studio.

Available Tools

8 tools
vision_analyze_dual_handsC

Analyzes dual-hand spatial geometry, wrist-to-wrist span, index span, dual pinch state, and dynamic ROI filter lens bounding box coordinates.

Inputs: hand0_landmarks (required), hand1_landmarks (required), confidence_threshold.

vision_calculate_dwellB

Evaluates dwell progress for a pointer inside a target interactive bounding box zone. Returns normalized progress (0.0 to 1.0), trigger state, remaining time, and dwell status.

Inputs: x (required), y (required), zone (required), elapsed_ms, +1 more.

vision_classify_faceB

Evaluates facial landmarks or features for mouth opening, smile expression, eye blinks, and eye aspect ratios (EAR).

Inputs: face_landmarks.

vision_classify_gestureB

Classifies 21 3D/2D hand landmarks (MediaPipe format) into gesture types (e.g. open_palm, fist, pointing_up, peace_sign, pinch, thumbs_up, rock_on, ok_sign) with confidence score, pinch metrics, and individual finger extension states.

Inputs: landmarks (required), handedness, confidence_threshold.

vision_generate_action_mapB

Generates complete gesture-to-action key bindings for presentations, media players, drawing studios, air gaming, or accessibility navigation.

Inputs: preset, overrides.

vision_get_diagnosticsC

Returns system, camera, display server, and compute telemetry diagnostics.

Inputs: extended.

vision_get_shader_filtersA

Returns all available real-time optical shader filters (Matrix Rain, Thermal IR, Sobel Neon, Cyber VHS, Hologram, Solar Amber) and their hand gesture triggers.

Inputs: no named parameters.

vision_get_solfeggio_frequenciesB

Returns the 9 sacred Solfeggio acoustic frequencies (174Hz to 963Hz) and spatial audio Theremin mapping equations.

Inputs: no named parameters.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.4/5.0

    Scored across 8 tools

    Disambiguation4/5

    Most tools have clearly distinct purposes defined by verb+resource (classify vs calculate vs generate). The only plausible confusion is between vision_analyze_dual_hands and vision_classify_gesture, since both consume hand landmarks, though one is dual-hand spatial geometry and the other is single-hand gesture typing. The two static config getters (shader filters, solfeggio frequencies) are distinct but feel like a different category of tool.

    Naming Consistency5/5

    All eight tools use the same vision_ prefix plus a snake_case verb_noun pattern (analyze_, get_, classify_, calculate_, generate_). Verbs are chosen meaningfully (get for static data, classify for inference, calculate for computation), with no mixed conventions.

    Tool Count4/5

    Eight tools is a well-scoped size for a gesture-control server, covering perception, classification, interaction, and diagnostics. A couple of the static-data getters (shader filters, solfeggio frequencies) are tangential flavor rather than core capability, slightly diluting focus.

    Completeness3/5

    Core pipeline is covered: hand/face analysis, gesture classification, dwell triggering, action-map generation, and diagnostics. However, there is no temporal/sequence tracking (gestures over time), no pointer/smoothing or coordinate-mapping tool despite dwell-by-position interaction, and no calibration step, leaving gaps for continuous control workflows.

    Maintenance

    ActivityActive
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables access to over 249 APIs through the Model Context Protocol, simplifying integration and usage without extensive technical knowledge.
      2 npm
      1
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables Windows GUI automation through the Model Context Protocol, providing tools, workflows, and generative capabilities for natural language-driven interaction with desktop applications.
      MIT