vision-gesture-control
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@vision-gesture-control What gesture am I making right now?"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Vision Gesture Control
Touchless 21-Joint 3D Hand Tracking, Spatial Air Drawing, Dwell Navigation & Model Context Protocol (MCP) Server (v2.5.0)
Zoth Studio Dossier: https://zoth.nullai.tech/tools/vision-gesture-control
Local Workstation URL:
http://127.0.0.1:8089Repository: https://github.com/1nc0gn30/vision-gesture-control
Overview
Vision Gesture Control combines a client-side WebAssembly/WebGL computer vision workstation (public/) with a zero-dependency Python standard-library HTTP server, CLI, and JSON-RPC 2.0 Model Context Protocol (MCP) server (src/vision_gesture_control/).
Key capabilities:
Sub-15ms Latency Hand & Face Kinematics: Tracks 21 3D hand landmarks per hand with adaptive
OneEuroFiltersmoothing, curl-angle heuristics, pinch ratios, dual-hand span/ROI lens framing, and 468-point facial EAR/MAR metrics.Interactive Browser Workspaces: Touchless presentation slide controller with dwell rings, velocity-sensitive spline air-drawing canvas, 3D Three.js spatial geometry portal, 13 optical shader filters, and a 41-preset WebAudio spatial loop/Solfeggio studio (including a built-in Virtual Hand Simulator when no webcam is attached).
Zero Cloud Egress: Video frames remain in volatile local memory for coordinate extraction and are never uploaded or written to disk.
Related MCP server: APIVerve MCP Server
Architecture
vision-gesture-control/
├── server.py # Standalone workstation entrypoint on 127.0.0.1:8089
├── bin/vision-gesture.js # Unified Node/ZothOS CLI & Electron desktop wrapper
├── public/
│ ├── index.html # Multi-tab Vision Studio UI & Virtual Hand Simulator
│ └── js/
│ ├── tool-air-drawing.js # Quadratic spline air-drawing & palm-hold clear engine
│ ├── tool-presentation-spatial.js # Slide laser/dwell controller & Three.js 3D portal
│ ├── tool-playground-lens.js # Dual-hand EMA/deadzone ROI shader lens
│ ├── tool-gesturefx.js # Gesture particle emitters, macropad & Biquad audio filter
│ └── tool-spatial-audio.js # 41-preset WebAudio loop, synth, vocal & Solfeggio engine
└── src/vision_gesture_control/
├── cli.py # `vision-control` CLI (serve, classify, dwell, mcp, map, platform, test, status)
├── ui_server.py # ThreadingHTTPServer & REST API on 127.0.0.1:8089
├── mcp_server.py # Stdio JSON-RPC 2.0 MCP server (8 tools)
├── gestures.py # 3D joint-angle, pinch, dwell & facial EAR/MAR classifiers
├── filter_pipeline.py # OneEuroFilter, 3D landmark smoother & filter pipeline
├── trajectory.py # Swipe velocity & pinch-zoom delta tracker
└── compat.py # Multi-OS display server (X11/Wayland/macOS/Windows) & camera discoveryQuick Start
git clone https://github.com/1nc0gn30/vision-gesture-control.git
cd vision-gesture-control
pip install -e .
python3 server.py --port 8089Open http://127.0.0.1:8089 in any WebGL-capable browser. No external Python runtime packages are required to run server.py or PYTHONPATH=src python3 -m vision_gesture_control.
CLI & MCP Usage
Python & ZothOS v3.4.0 CLI
On ZothOS v3.4.0 (or after pip install -e .), invoke vision-control or node bin/vision-gesture.js:
# Inspect CLI flags and subcommands
vision-control --help
node bin/vision-gesture.js --status --json
# Start the web studio on 127.0.0.1:8089
vision-control serve --host 127.0.0.1 --port 8089
# Classify 21 hand landmarks from a JSON file, inline JSON, or built-in sample
vision-control classify
vision-control classify --landmarks-json landmarks.json --handedness Right --confidence-threshold 0.6
# Evaluate pointer dwell progress inside an interactive bounding box (x,y,width,height)
vision-control dwell --x 0.45 --y 0.50 --zone 0.2,0.2,0.6,0.6 --duration 1000 --elapsed 650
# Export gesture-to-action key maps (presentation, media, drawing, gaming, accessibility)
vision-control map --preset presentation --format table
# Run OS, display server, and camera telemetry diagnostics
vision-control platform --extended
# Dump MCP tool schemas or client configuration snippets
vision-control mcp --tools
vision-control mcp --config claude_desktopREST API Endpoints (127.0.0.1:8089)
Method | Endpoint | Description |
|
| Service health status, version ( |
|
| Gesture action maps ( |
|
| Client config JSON ( |
|
| System, Python, display server, CPU, and camera telemetry ( |
|
| Classify |
|
| Compute dwell progress for |
|
| Classify facial blink/smile/mouth states from |
|
| Download standalone offline |
MCP Server Tools (stdio)
Configure your MCP host with python3 -m vision_gesture_control.mcp_server (see docs/MCP_GUIDE.md):
vision_classify_gesture— Classify 21 3D/2D hand landmarks with confidence, pinch state, and finger extension flags.vision_analyze_dual_hands— Compute inter-hand wrist/index span, Z-depth delta, dual-pinch state, and ROI lens bounds.vision_calculate_dwell— Evaluate normalized dwell progress and trigger state inside a bounding box.vision_classify_face— Evaluate facial landmarks or EAR/MAR ratios for blink, smile, and mouth-open states.vision_generate_action_map— Generate preset or custom-overridden gesture key bindings.vision_get_shader_filters— List the 13 optical shader filters and their gesture triggers.vision_get_solfeggio_frequencies— List the 9 Solfeggio frequencies (174 Hz–963 Hz) and Theremin axis mappings.vision_get_diagnostics— Query OS, display server, CPU, and camera diagnostics.
Testing & Verification
# Run full unit & integration test suite (ephemeral port=0)
PYTHONPATH=src python3 -m pytest -q
# Run built-in stdlib self-tests
PYTHONPATH=src python3 -m vision_gesture_control --test
# Verify CLI status JSON
python3 server.py --status --jsonSupport
Vision Gesture Control is 100% free and open-source with zero paywalls or feature gates.
Optional support: Buy Us a Coffee (Stripe)
Solana Tip Address:
GG2eQbVh8ut1BK6j1A2tz7pSySdLpD38BgyEhGcxdXoK(solana:GG2eQbVh8ut1BK6j1A2tz7pSySdLpD38BgyEhGcxdXoK)
License & Attribution
Released under the MIT License. Engineered by Neal Frazier Tech (@1nc0gn30, github.com/1nc0gn30/vision-gesture-control) • NullAI Platform • Cataloged in Zoth Studio.
Available Tools
8 toolsvision_analyze_dual_handsC
Analyzes dual-hand spatial geometry, wrist-to-wrist span, index span, dual pinch state, and dynamic ROI filter lens bounding box coordinates.
Inputs: hand0_landmarks (required), hand1_landmarks (required), confidence_threshold.
vision_calculate_dwellB
Evaluates dwell progress for a pointer inside a target interactive bounding box zone. Returns normalized progress (0.0 to 1.0), trigger state, remaining time, and dwell status.
Inputs: x (required), y (required), zone (required), elapsed_ms, +1 more.
vision_classify_faceB
Evaluates facial landmarks or features for mouth opening, smile expression, eye blinks, and eye aspect ratios (EAR).
Inputs: face_landmarks.
vision_classify_gestureB
Classifies 21 3D/2D hand landmarks (MediaPipe format) into gesture types (e.g. open_palm, fist, pointing_up, peace_sign, pinch, thumbs_up, rock_on, ok_sign) with confidence score, pinch metrics, and individual finger extension states.
Inputs: landmarks (required), handedness, confidence_threshold.
vision_generate_action_mapB
Generates complete gesture-to-action key bindings for presentations, media players, drawing studios, air gaming, or accessibility navigation.
Inputs: preset, overrides.
vision_get_diagnosticsC
Returns system, camera, display server, and compute telemetry diagnostics.
Inputs: extended.
vision_get_shader_filtersA
Returns all available real-time optical shader filters (Matrix Rain, Thermal IR, Sobel Neon, Cyber VHS, Hologram, Solar Amber) and their hand gesture triggers.
Inputs: no named parameters.
vision_get_solfeggio_frequenciesB
Returns the 9 sacred Solfeggio acoustic frequencies (174Hz to 963Hz) and spatial audio Theremin mapping equations.
Inputs: no named parameters.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 8 tools
Most tools have clearly distinct purposes defined by verb+resource (classify vs calculate vs generate). The only plausible confusion is between vision_analyze_dual_hands and vision_classify_gesture, since both consume hand landmarks, though one is dual-hand spatial geometry and the other is single-hand gesture typing. The two static config getters (shader filters, solfeggio frequencies) are distinct but feel like a different category of tool.
All eight tools use the same vision_ prefix plus a snake_case verb_noun pattern (analyze_, get_, classify_, calculate_, generate_). Verbs are chosen meaningfully (get for static data, classify for inference, calculate for computation), with no mixed conventions.
Eight tools is a well-scoped size for a gesture-control server, covering perception, classification, interaction, and diagnostics. A couple of the static-data getters (shader filters, solfeggio frequencies) are tangential flavor rather than core capability, slightly diluting focus.
Core pipeline is covered: hand/face analysis, gesture classification, dwell triggering, action-map generation, and diagnostics. However, there is no temporal/sequence tracking (gestures over time), no pointer/smoothing or coordinate-mapping tool despite dwell-by-position interaction, and no calibration step, leaving gaps for continuous control workflows.
Maintenance
Related MCP Connectors
- DNS-DATA RemoteDeprecatedcom.dns-data
Connect AI assistants to authorized Windows and macOS devices for files, commands, and management.
Connect AI assistants to your approved computer for files, commands, browser and desktop tasks.
Real-time planetary signal engine and Model Context Protocol (MCP) server for autonomous AI agents.
A comprehensive Model Context Protocol (MCP) server that enables AI assistants to interact with yo…
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceA Model Context Protocol implementation with a modular architecture that exposes capabilities through specialized agents, enabling seamless integration with Claude Desktop and web applications.2-
- AlicenseNot gradedqualityCmaintenanceEnables access to over 249 APIs through the Model Context Protocol, simplifying integration and usage without extensive technical knowledge.2 npm1MIT
- FlicenseNot gradedqualityDmaintenanceGive any AI full control of your desktop. Mouse, keyboard, screenshots — all through the Model Context Protocol.-
- AlicenseNot gradedqualityCmaintenanceEnables Windows GUI automation through the Model Context Protocol, providing tools, workflows, and generative capabilities for natural language-driven interaction with desktop applications.MIT