agent-vision
This server exposes two MCP tools for vision analysis: image understanding and video frame analysis.
Analyze an image from a local path, HTTP(S) URL, or
data:image/...URI, with an optional specific question prompt.Analyze a local or remote video by extracting 1–16 evenly spaced keyframes (default 8) and sending them with timestamps to a configured vision model.
Use it for screenshots, UI inspection, OCR, diagrams, photos, visual debugging, screen recordings, demos, and event summaries.
analyze_imageis intended only when the agent cannot view the image natively.The provided schema does not expose
generate_imageor any other generation tool; both tools have async task support forbidden.
Enables analyzing images and videos via OpenAI-compatible vision models, sending images or extracted video keyframes to the configured API and returning textual analysis.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@agent-visionAnalyze this image and tell me what it shows."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
agent-vision
一个小而直接的视觉 MCP Server + Agent Skill:让纯文本 Agent 通过一个 OpenAI-compatible 视觉模型理解图片和视频,也能通过生图模型从文字生成图片。
纯 Node.js / TypeScript,
npx自动下载运行MCP 工具:
analyze_image、analyze_video、generate_imageAgent Skills 标准目录:
skills/agent-vision/,能力自适应:能看图的 Agent 直接看,纯文本 Agent 直接跑 CLI图片:本地路径、HTTP(S) URL、Data URI
视频:本地路径或 URL,自动下载 npm 内的 ffmpeg 并均匀抽帧
识图多 provider 多模型并发竞速:所有 target 同时请求,谁先成功用谁,其余请求 abort
生图:支持 OpenAI-compatible 和火山方舟 Seedream,独立配置 provider / model,按顺序失败回退
为什么做这个项目
参考项目各有优点,但组合起来偏重:
luma-mcp:Node.js、npx 和图片预处理体验好,但主要面向单图。
deepseek-vision-mcp:MCP 执行层与 Skill 决策层分离得很清楚,但依赖 Python,且本地视频受 provider 限制。
vision-tool:支持视频抽帧,但包含大量 provider 探测、并行 fallback 和安装逻辑,并采用 GPL-3.0。
agent-vision-toolkit:Skill 和任务工作流很强,但核心是多组 Python/Shell CLI,不是 MCP 视频服务。
agent-vision 保持两条简单链路:
理解:MCP/CLI → 图片或视频帧 → 你配置的视觉 API → 文本
生成:MCP/CLI → 文字描述 → 你配置的生图 API → 图片 本仓库是独立实现,没有复制
vision-tool的 GPL 源码。
Related MCP server: VisionPower
要求
Node.js 20+
图片/视频理解:支持
/chat/completions和image_url的 OpenAI-compatible 视觉模型生图:OpenAI-compatible 生图模型,或火山方舟 Seedream(API Key 鉴权);理解和生图可以单独配置
ffmpeg 由 @ffmpeg-installer/ffmpeg 按平台自动安装,无需另行安装。
配置
推荐用配置文件(一次配置,所有宿主共用):
mkdir -p ~/.config/agent-vision
cat > ~/.config/agent-vision/settings.json <<'EOF'
{
"providers": [
{
"name": "dashscope",
"baseUrl": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"apiKey": "sk-...",
"models": ["qwen-vl-max", "qwen-vl-plus"]
},
{
"name": "openai",
"baseUrl": "https://api.openai.com/v1",
"apiKey": "sk-...",
"models": ["gpt-4o"]
}
]
}
EOF
chmod 600 ~/.config/agent-vision/settings.json位置:
$AGENT_VISION_CONFIG显式路径,或$XDG_CONFIG_HOME/agent-vision/settings.json(默认~/.config/agent-vision/settings.json)providers用于图片/视频理解,每个 provider × model 是一个 target;多个 target 时并发请求,第一个成功者胜出,其余请求 abort。注意成本:N 个 target 意味着每次分析最多 N 份上传与推理费用provider 字段:
baseUrl(必填)、models(必填非空)、name/apiKey/headers(可选;apiKey缺省回退环境变量)顶层可选:
maxTokens、timeoutMs、headers、maxImageMb、maxVideoMb、allowPrivateUrls、ffmpegPath文件定义的字段优先于环境变量;含 key 的文件建议
chmod 600
环境变量作为回退仍然完整支持(无配置文件时):
设置 | 兼容回退 |
|
|
|
|
|
|
生图配置
在同一个 settings.json 中添加 generationProviders,字段与识图的 providers
类似,额外通过 type 选择生图协议;也可以只配置生图,不配置识图:
{
"generationProviders": [
{
"name": "image-api",
"type": "openai-images",
"baseUrl": "https://your-image-api.example/v1",
"apiKey": "sk-...",
"models": ["your-image-generation-model"]
}
]
}识图与生图模型完全分开,不会把视觉聊天模型自动当作生图模型。
type支持openai-images(默认,兼容旧配置)和volcengine-ark。按显式协议适配请求,不根据域名或模型名称猜测;未知类型会报配置错误。generationProviders内按 provider、model 的配置顺序依次尝试,拿到第一张有效图片就停止,不并发竞速。失败回退仍可能产生多次费用(例如上游超时但实际已完成生成);只想调用一个模型时配置一个 target。每次只生成一张图片;目前支持文生图,不包含参考图编辑或异步任务轮询。
两种协议均支持
data[0].b64_json或data[0].url返回。openai-images省略response_format,避免不支持该参数的模型报错;volcengine-ark请求 base64 返回。参考 OpenAI 官方 Images API 文档。与识图共用顶层
timeoutMs、maxImageMb、headers、allowPrivateUrls,生图较慢时可调高timeoutMs。doctor会分别列出识图和生图 targets,并显示生图协议,只配置生图时不要求 ffmpeg。
火山引擎方舟(Seedream)
在已有配置中加入下面的生图部分即可,保留原来的 providers:
{
"generationProviders": [
{
"name": "volcengine",
"type": "volcengine-ark",
"baseUrl": "https://ark.cn-beijing.volces.com/api/v3",
"apiKey": "你的方舟 API Key",
"models": ["doubao-seedream-5-0-lite-260128"]
}
]
}models 填写你已开通的 Seedream Model ID,也支持配置了生图模型的 ep-... 接入点 ID。
apiKey 可省略并从 ARK_API_KEY 读取;若设置了 AGENT_VISION_GENERATION_API_KEY,则后者优先。
方舟不会自动继承识图的 AGENT_VISION_API_KEY / OPENAI_API_KEY,避免把其他服务的 key 发给火山。
npx -y @yanickxia/agent-vision generate \
--prompt "一只坐在窗边的橘猫,水彩风格" --output ./cat.jpg --size 2K使用方舟原生
/api/v3/images/generations参数,不发送 OpenAI 的n或quality;传--quality会在调用前报错,不静默忽略。--size可用模型支持的分辨率档位(例如2K)或宽x高,省略时采用模型默认值。保持服务端的单图、非流式默认行为,不发送部分 Seedream 模型不支持的组图/流式开关。
保留模型默认水印和输出格式;Seedream 通常返回 JPEG,文件扩展名不会触发转码。
能解析火山的 HTTP 错误及成功 HTTP 响应中的顶层/单图错误,保留错误码与消息。
协议依据:方舟官方图片生成 API。 本适配针对方舟 Seedream 文生图,不是“视觉智能/即梦”的 AK/SK 签名接口,不包含图生图、组图或图层拆分。
生图环境变量
也可以用环境变量配置生图:
变量 | 回退 / 说明 |
| 默认 |
| 生图模型名称,必须显式设置;不回退到识图模型 |
| OpenAI 协议回退到识图 URL 及其别名;方舟协议默认 |
| OpenAI 协议回退到识图 key 及其别名;方舟协议回退到 |
| 仅用于 |
| 生图专用额外请求头 JSON,覆盖通用环境变量请求头 |
文件中 generationProviders 优先于环境变量生图 target;provider 的 apiKey
省略时使用上述生图 key 回退,显式 "apiKey": "" 可关闭 key 回退。
文件内未指定 type 的 provider 始终是 openai-images,不受环境变量的 TYPE 影响。
请求头优先级:通用环境变量 → 生图环境变量 → 文件顶层 → provider。
通用选项
以下可选变量与对应的配置文件字段等价:
变量 | 默认值 | 说明 |
|
| 最大输出 tokens |
|
| 下载和 API 超时 |
|
| 输入和生成图片的大小上限 |
|
| 视频大小上限 |
|
| 额外请求头 JSON |
|
| 允许私网图片/视频来源及生图结果 URL |
| 自动发现 | 指定 ffmpeg 可执行文件;默认先用 PATH,再用 npm 内置版本 |
MCP 安装
npm(推荐)
{
"mcpServers": {
"agent-vision": {
"command": "npx",
"args": ["-y", "@yanickxia/agent-vision"]
}
}
}直接从 GitHub 运行
{
"mcpServers": {
"agent-vision": {
"command": "npx",
"args": ["-y", "github:yanickxia/agent-vision"]
}
}
}无需 env 块:配置从 ~/.config/agent-vision/settings.json 读取(见上文「配置」)。
也可以继续用环境变量,在 env / environment 块里传入。
Claude Desktop、Claude Code、Cursor、Cline 等使用上面的标准
mcpServers 格式。
OpenCode
{
"mcp": {
"agent-vision": {
"type": "local",
"command": ["npx", "-y", "@yanickxia/agent-vision"],
"enabled": true
}
}
}Agent Skill
npx skills add yanickxia/agent-vision --skill agent-vision -g -y也可以复制 skills/agent-vision/ 到 Agent 的 skills 目录。Skill 是能力自适应
的:Agent 自己能看图就直接看;看不了(或输入是视频)就由 Skill 指导 Agent
直接跑 CLI 完成分析。生图不受「能否看图」限制,任何 Agent 都可以使用。
MCP 和 Skill/CLI 都支持火山方舟生图,共用同一份 settings.json 中的
generationProviders 配置。Skill 直接执行 CLI,不依赖启动或安装 MCP Server。
入口 | 调用 | 输出 |
MCP |
| 图片内容和元数据,可选保存文件 |
Skill / CLI |
| 保存文件,stdout 返回绝对路径 |
从源码使用尚未发布的功能时,先在仓库中运行 npm run build,然后用
node /绝对路径/agent-vision/dist/cli.js generate ... 运行 CLI。
本地 MCP 配置也可使用 command: "node"、args: ["/绝对路径/agent-vision/dist/cli.js"],
不要加 generate 参数,无子命令时即启动 MCP stdio 服务。
CLI
npx -y @yanickxia/agent-vision image ./screenshot.png \
--prompt "读取报错并给出可能原因"
npx -y @yanickxia/agent-vision video ./demo.mp4 \
--prompt "按时间顺序总结 UI 操作" --frames 8
npx -y @yanickxia/agent-vision generate \
--prompt "一只坐在窗边的橘猫,水彩风格" --output ./cat.png --size 1024x1024
npx -y @yanickxia/agent-vision doctorgenerate 必须提供 --prompt(或一个带引号的位置参数)和 --output。
可选 --size 原样传给生图 API;--quality 仅用于 OpenAI-compatible 协议。
省略时采用 provider 默认值,支持值取决于模型。
输出文件的父目录必须存在,已有文件不会被覆盖;成功时 stdout 输出保存的绝对路径。
图片保留 API 返回的实际格式,不会因输出文件的扩展名而转码。
MCP 工具
analyze_image
{
"source": "/absolute/path/to/image.png",
"prompt": "这个页面有哪些可用性问题?"
}支持 JPEG、PNG、WebP、GIF、BMP。GIF 是否能体现动画取决于视觉模型;需要稳定的
时间线分析时请转为视频并使用 analyze_video。
analyze_video
{
"source": "/absolute/path/to/video.mp4",
"prompt": "概括操作流程和关键变化",
"max_frames": 8
}max_frames 范围 1–16,默认 8。实现会均匀抽取 JPEG 帧,并带时间点一起发给
模型。它不是逐帧转写,快速变化可能被漏掉。
generate_image
{
"prompt": "一只坐在窗边的橘猫,水彩风格",
"output_path": "/absolute/path/to/cat.png",
"size": "1024x1024"
}只有 prompt 必填。output_path 可选,不传时仅返回 MCP 图片内容,不写本地文件。
size、quality 可选。返回一个 text 内容块(JSON 元数据:模型、MIME 类型、字节数、
可选输出路径和上游改写的 prompt)以及一个 image 内容块。路径是 MCP Server 所在机器上的路径。
隐私与安全
图片会发送到你配置的视觉 API。
生图 prompt 会发送到你配置的生图 API;如果上游返回图片 URL,会下载该图片,但不会向图片下载地址转发生图 API key 或自定义请求头。
视频不会整体发给模型;远程视频先下载到本地临时目录,然后只发送抽出的 JPEG 帧。
临时目录在成功或失败后都会删除。
用户提供的远程图片/视频 URL 和上游返回的生图 URL 默认拒绝 localhost、私网、链路本地和云元数据地址。
生图失败会删除本次预留的输出文件;成功的生图文件会保留,不会作为临时文件清理。
stdio 模式不向 stdout 写日志,避免破坏 MCP 协议。
开发
npm install
npm run typecheck
npm test
npm pack --dry-runLicense
MIT。第三方依赖保留各自许可证,见 NOTICE。
Available Tools
3 toolsanalyze_imageAnalyze imageA
Analyze an image from a local path, HTTP(S) URL, or base64 data URI. Use ONLY when you cannot view the image yourself (no native multimodal/image input) — if you can see the image directly, answer from your own vision instead. For screenshots, UI, OCR, diagrams, photos, and visual debugging.
| Name | Required | Description | Default |
|---|---|---|---|
| prompt | No | The user's specific question about the image | |
| source | Yes | Local path, HTTP(S) URL, or data:image/... URI |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full behavioral burden. It discloses accepted input channels and the intended-use gate, but says nothing about return format, latency/cost, whether image data is sent to a remote service, or any permission requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, front-loaded with the capability, then the usage gate, then example use cases. No filler and nothing buried.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter tool this covers inputs and when-to-use well, but with no output schema and no annotations the description should not be silent on what comes back from the analysis or on the handling of the supplied image data.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so both parameters are already documented and 3 is the baseline. The description restates the source formats already in the schema and adds no guidance on the prompt parameter (e.g., how specific it should be).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ("Analyze an image") and enumerates the accepted source forms (local path, HTTP(S) URL, base64 data URI). It is trivially separable from its only sibling, analyze_video, by resource type, though it never explicitly contrasts with it.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit gating rule: use ONLY when you cannot view the image yourself, and answer from native vision otherwise. It also names concrete contexts (screenshots, UI, OCR, diagrams, photos, visual debugging), leaving no inference needed about when to reach for it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
analyze_videoAnalyze videoA
Analyze a local or remote video by extracting evenly spaced keyframes. Use this for screen recordings, UI flows, demos, and event summaries.
| Name | Required | Description | Default |
|---|---|---|---|
| prompt | No | The user's specific question about the video | |
| source | Yes | Local path or HTTP(S) URL | |
| max_frames | No | Number of frames to sample; default 8 |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the burden of behavioral disclosure. It transparently reveals the keyframe-extraction mechanism and source types (local/remote). It does not mention output format or potential limitations (e.g., loss of temporal context between frames), which prevents a top score, but the core behavior is well outlined.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is exceptionally concise: two sentences, no fluff, with the primary purpose front-loaded and supporting usage guidance in the second sentence. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has no output schema, so the description should clarify the return value. While the description explains the mechanism (extracting keyframes) and gives usage examples, it does not state what the analysis produces (e.g., text summary, labeled frames, etc.), leaving a notable gap for an AI agent selecting and invoking the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides detailed descriptions for all three parameters (source, prompt, max_frames), achieving 100% schema coverage. The description adds no extra parameter-level meaning beyond what the schema already states, so the baseline score of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Analyze a local or remote video by extracting evenly spaced keyframes.' This specific verb+resource+method distinguishes it from typical image tools. It also lists concrete use cases like 'screen recordings, UI flows, demos, and event summaries,' further clarifying its video-centric purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'Use this for screen recordings, UI flows, demos, and event summaries' offers clear, context-rich guidance on when to use the tool. However, it does not explicitly mention when not to use it or compare it with the sibling tool 'analyze_image,' so it stops short of full alternative guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
generate_imageGenerate imageA
Generate one image from a text prompt using configured OpenAI-compatible or Volcengine Ark (Seedream) providers. Uses the same settings as the standalone CLI. Returns image content and metadata; optionally saves to a new local file. This is image creation, not image analysis, and works regardless of native vision capability.
| Name | Required | Description | Default |
|---|---|---|---|
| size | No | Provider-supported image size, e.g. 1024x1024 or 2K for Ark; omitted uses provider default | |
| prompt | Yes | Description of the image to create | |
| quality | No | OpenAI-compatible quality; not supported by volcengine-ark, omit for Ark | |
| output_path | No | Optional local file path; parent must exist and file must not exist |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=false, openWorldHint=true, idempotentHint=false and destructiveHint=false. The description adds real value beyond that: it discloses the return shape (image content plus metadata) and the optional side effect of writing a new local file, which the annotations alone don't convey. It stops short of mentioning cost, rate limits, or provider auth requirements, which matters for a non-idempotent, open-world generation call.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with the core verb and resource, then the provider scope, then the return behavior and the disambiguation. Almost every clause earns its place, with only minor redundancy between 'Returns image content' and the later clarification.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description correctly fills the gap by describing the return (image content and metadata) and the optional file write. All parameters are schema-documented and safety traits are annotated. The remaining gap is operational context, such as which provider settings are read from configuration and whether generation costs or rate limits apply.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so all four parameters (prompt, size, quality, output_path) are documented in the schema, including provider-specific caveats like quality being unsupported by Ark. The description only echoes the optional file-saving behavior for output_path and adds no syntax, defaults, or provider-selection logic beyond the schema. Baseline 3 is appropriate when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Generate) and resource (one image from a text prompt), names the supported providers, and explicitly separates itself from the analyze_image sibling with 'This is image creation, not image analysis.' An agent can distinguish it from siblings without reading any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clearly establishes the context: image generation from prompts, provider-agnostic, and independent of native vision capability, with an explicit statement that it is not image analysis. It does not explicitly say when NOT to use it (e.g., editing an existing image) or name analyze_image directly as the alternative, which leaves a small inference gap.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
v0.3.0- Added
generate_image
2 tool updates
v0.1.0- First observed
analyze_image - First observed
analyze_video
TDQS
Scored across 3 tools
The three tools target clearly distinct actions and modalities: image generation, image analysis, and video analysis. The analyze_image description also explicitly clarifies when not to use it, reducing misselection risk.
All tools follow a consistent verb_noun pattern: generate_image, analyze_image, analyze_video. The convention is predictable and directly reflects each tool's purpose.
Three tools are well-scoped for a focused vision server: generation, image understanding, and video understanding. Each tool has a clear reason to exist, with no redundant surface area.
The core lifecycle for vision tasks is covered: create images, inspect images, and inspect videos. Minor gaps exist, such as image editing/transformation or batch analysis, but agents can work around them for most stated use cases.
Maintenance
Related MCP Connectors
MCP server for OpenAI API (chat completions, image generation, embeddings) via AceDataCloud
MCP server for Qwen Image 3 AI image generation
MCP server for Grok Imagine AI video generation
MCP server for Hailuo (MiniMax) AI video generation
Related MCP Servers
- AlicenseBqualityCmaintenanceA lightweight MCP server for image analysis using any OpenAI-compatible API endpoint, enabling AI agents to analyze images via a single tool.18 npmMIT
- AlicenseAqualityAmaintenanceA portable image-understanding MCP server that lets agents analyze local images, URLs, or base64 images via an OpenAI-compatible vision model.193 npm68MIT
- AlicenseAqualityDmaintenanceAn MCP server that gives AI agents the ability to observe and understand images via multi-provider vision, object detection, hierarchical analysis, and color extraction.444 npm2MIT
- FlicenseBqualityDmaintenanceA Python-based MCP server that adds image analysis capabilities to text-only LLMs via a single analyze_image tool, supporting local files, URLs, auto-scaling, and multiple OpenAI-compatible APIs.1-