Skip to main content
Glama
pongsakornp

vision-mcp

by pongsakornp
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Local

vision-mcp

MCP server that gives text-only AI coding agents the ability to understand images and videos. Works with OpenCode, Claude Desktop, Claude Code, Cursor, VS Code, and any MCP-compatible client — including agents running on models without native vision.

agent (text-only) → vision-mcp → Gemini / Grok / OpenRouter → text description

Why

Most coding agents run on strong text models (DeepSeek, Claude, etc.) that can't see images. When you paste a screenshot, a UI mockup, or an error screenshot, they say "I can't view images". vision-mcp fixes that: it takes the file, sends it to a vision-capable model, and returns a plain-text description the agent can reason over.

Related MCP server: vision-mcp

Features

  • 2 tools: analyze_image and analyze_video

  • 3 backends, automatic fallback: Gemini → Grok (xAI) → OpenRouter — tried in priority order, first success wins

  • Zero config file: keys come from environment variables only

  • Image preprocessing: auto-rotate (EXIF), downscale to ≤1024px, JPEG re-encode — small payloads, fast responses

  • Video support: ffmpeg keyframe extraction (up to 8 frames) with timestamps

  • Typed & tested: TypeScript, 21 unit/integration tests, MCP protocol verified over real stdio

  • No Python: pure Node.js (sharp for images, ffmpeg binary for video)

Requirements

  • Node.js ≥ 18

  • ffmpeg (for video analysis only; images work without it)

  • At least one of these API keys:

Backend

Env var

Get it

Gemini (default)

GEMINI_API_KEY

https://aistudio.google.com/app/apikey

Grok (xAI)

XAI_API_KEY

https://console.x.ai

OpenRouter

OPENROUTER_API_KEY

https://openrouter.ai/keys

Install

npx from GitHub (recommended — no npm publish needed)

# Run directly (no install)
npx -y github:pongsakornp/vision-mcp

npx clones the repo, installs deps, auto-builds via the prepare script, and runs the server over stdio.

From source

git clone https://github.com/pongsakornp/vision-mcp.git
cd vision-mcp
pnpm install
pnpm run build

Usage

MCP client config

OpenCode (opencode.jsonc):

{
  "mcp": {
    "vision-mcp": {
      "type": "local",
      "command": ["npx", "-y", "github:pongsakornp/vision-mcp"],
      "environment": {
        "GEMINI_API_KEY": "AIza...",
        "XAI_API_KEY": "xai-...",
        "OPENROUTER_API_KEY": "sk-or-..."
      },
      "enabled": true
    }
  }
}

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "vision-mcp": {
      "command": "npx",
      "args": ["-y", "github:pongsakornp/vision-mcp"],
      "env": {
        "GEMINI_API_KEY": "AIza..."
      }
    }
  }
}

VS Code / Cursor (.vscode/mcp.json or workspace):

{
  "servers": {
    "vision-mcp": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "github:pongsakornp/vision-mcp"],
      "environment": {
        "GEMINI_API_KEY": "AIza..."
      }
    }
  }
}

Keys live in the MCP config — no shell profile edits needed. Keep the config out of git if it contains keys.

CLI (for scripting / debugging)

# Spawn the server and talk JSON-RPC over stdio
GEMINI_API_KEY=AIza... node dist/index.js

Tools

analyze_image(path, prompt?)

Analyze an image file (png, jpg, jpeg, webp, bmp, gif, avif, tiff). Path can be absolute, relative, or ~/....

analyze_video(path, prompt?)

Analyze a video (mp4, webm, mov, avi, mkv, flv, wmv, m4v). Extracts up to 8 evenly-spaced keyframes with ffmpeg, then describes the sequence.

Configuration

Env var

Default

Description

GEMINI_API_KEY

—

Gemini API key (tried first)

XAI_API_KEY

—

Grok/xAI API key (tried second)

OPENROUTER_API_KEY

—

OpenRouter key (tried last)

VISION_GEMINI_MODEL

gemini-flash-latest

Gemini model override

VISION_GROK_MODEL

grok-4.5

Grok model override

VISION_OPENROUTER_MODEL

google/gemini-flash-latest

OpenRouter model override

VISION_TIMEOUT_MS

20000

Per-backend request timeout

Backend behavior

  • Only backends with keys configured are tried.

  • Backends are tried sequentially in priority order (Gemini → Grok → OpenRouter); the first success wins and later backends are never called.

  • If all fail, the error reports every backend's failure for diagnosis.

Development

pnpm run build        # compile TypeScript → dist/
pnpm test             # run the 21 tests (vitest)
pnpm run typecheck    # tsc --noEmit

Test coverage: config parsing, backend fallback logic, path resolution, image preprocessing (sharp), video keyframe extraction (real ffmpeg), and the full MCP stdio protocol.

License

MIT — see LICENSE.

Available Tools

2 tools
analyze_imageAnalyze imageA

Analyze an image file (png, jpg, webp, bmp, gif, avif) and return a text description. Use for screenshots, diagrams, UI mockups, photos. Provide an absolute or relative path.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYesPath to the image file
promptNoOptional custom prompt for the analysis

TDQS

A4.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must carry the behavioral burden; it does disclose output type ('text description'), accepted path convention ('absolute or relative path'), and supported formats. However, it does not explicitly state that the operation is read-only, whether local file access is required, or how errors like an invalid path are handled, leaving some inference to the agent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences carry all the essential information with no filler; the core action and output are front-loaded, followed by use cases and a path note. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter, no-output-schema tool, the description provides the necessary invocation details: path form, supported formats, and expected return type. The main gap is lack of explicit read-only assurance and error handling, but an agent can select and call this tool confidently.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds value beyond the schema: it enumerates accepted file extensions and clarifies that the path may be absolute or relative. The prompt parameter remains covered only by the schema's description, but the total parameter knowledge is strong.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with the specific verb 'Analyze' and the resource 'image file', lists six supported formats, and states the output is a text description. This clearly separates it from the sibling analyze_video by restricting the media type to images.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit use cases ('screenshots, diagrams, UI mockups, photos'), which tells an agent when to reach for it. It doesn't name the alternative analyze_video or provide exclusions, so it falls short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

analyze_videoAnalyze videoA

Analyze a video file (mp4, webm, mov, avi, mkv) by extracting keyframes and describing the content. Requires ffmpeg installed.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYesPath to the video file
promptNoOptional custom prompt for the analysis

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the behavioral burden. It usefully discloses the ffmpeg dependency and the keyframe-extraction approach, but it does not mention output format, side effects, or limitations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single dense sentence with no filler. It front-loads the action and resource, then adds formats and dependencies efficiently.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Adequate for a straightforward tool, but because there is no output schema and no annotations, the agent is left to infer what the tool returns. A brief note on the output would make it complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents the path and prompt parameters. The description adds format and dependency context but no additional parameter-specific meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly specifies the action (analyze), the resource (video files), the accepted formats, and the method (extracting keyframes and describing content). This cleanly distinguishes it from the sibling analyze_image by scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied through the supported video formats and the ffmpeg prerequisite, but there is no explicit guidance on when to choose this tool over analyze_image or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updatesv0.2.0
    • First observedanalyze_image
    • First observedanalyze_video
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.1/5.0

Scored across 2 tools

Disambiguation5/5

The two tools are clearly separated by media type: images versus videos. An agent can easily select the right tool based on the file extension or input format, with no meaningful overlap.

Naming Consistency5/5

Both tools follow the exact same verb_noun snake_case pattern: analyze_image and analyze_video. The naming is clean, predictable, and immediately signals what each tool does.

Tool Count4/5

Two tools is minimal but reasonable for a vision server focused on content description of static and moving media. Each tool earns its place, though the set feels slightly thin compared to broader vision-capable servers.

Completeness4/5

The core workflow of analyzing visual media is covered: images and videos both have a dedicated tool. Minor gaps exist, such as no URL-based input and no specialized vision capabilities like OCR or object detection, but these are not clearly implied by the server's stated scope.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers