Skip to main content
Glama

Local-MMCP

本地多模态 MCP Server — 为 Claude Code / Codex / Cline 等 AI 客户端提供视觉、文档、音频、视频、GUI 自动化能力

版本 Python 架构 测试

项目简介

Local-MMCP 是一个基于 MCP (Model Context Protocol) 的本地多模态服务器,通过 stdio transport 向 AI 客户端暴露 9 个多模态工具

核心设计理念:本地优先、隐私安全、优雅降级。所有数据处理在本地完成,无需将图片/文档/音频上传到第三方服务。

Related MCP server: lucid-apple-mcp

架构总览

MCP Client (Claude Code / Codex / Cline / Roo / OpenCode 等)
        │
        ▼
Local-MMCP MCP Server (stdio transport)
        │
        ├── oMLX 视觉/文本模型 (本地 Apple Silicon)
        │   ├── gemma-4-31B-it-Uncensored-MAX-MLX (视觉)
        │   └── Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-mlx-8bit (文本)
        │
        ├── MinerU / PaddleOCR (文档解析)
        ├── Qwen3-ASR / mlx-audio (语音转写)
        ├── ffmpeg (视频处理)
        └── Playwright / ADB / macOS Accessibility (GUI 自动化)

分层架构

Server (MCP Protocol Handler)
    └── Tools (9 个工具模块)
        ├── Clients (oMLX API 客户端 + 模型生命周期管理)
        ├── Adapters (外部工具适配器)
        └── Utils (图片处理 / 安全 / JSON 工具)

工具清单

工具

用途

依赖

需确认

health_check

检查所有组件可用性(支持 deep 模式验证 VLM)

vision_inspect

分析图片:截图理解、UI 分析、图表、错误诊断

oMLX VLM

vision_crop_verify

裁剪图片局部放大复核

oMLX VLM

vision_diff

比较两张图片差异(像素 + 语义)

oMLX VLM

doc_parse

文档转 Markdown/JSON(PDF/DOCX/PPTX/XLSX/图片)

MinerU 或 PaddleOCR

audio_transcribe

音频转写(WAV/MP3/M4A/FLAC)

mlx-audio (Qwen3-ASR)

video_index

视频时间线索引(抽帧 + ASR + VLM 理解)

ffmpeg + oMLX + mlx-audio

gui_observe

观察 GUI 状态(浏览器/Android/macOS)

Playwright / ADB / AppKit

gui_act

执行 GUI 动作(点击/输入/滑动等)

Playwright / ADB / AppKit

快速开始

1. 环境要求

  • 硬件: Mac with Apple Silicon (M1+), 推荐 128GB+ 统一内存

  • 系统: macOS 14+

  • Python: 3.11+

  • oMLX: omlx.ai 本地模型推理服务

  • 包管理: uv

2. 安装

git clone https://github.com/rorojiao/local-mmcp.git
cd local-mmcp
uv sync

# 可选依赖(按需安装)
uv pip install mlx-audio          # 语音转写
uv pip install pyautogui           # macOS 桌面自动化

3. 配置

cp config.example.yaml config.yaml
# 编辑 config.yaml,调整 omlx 端口、模型名、安全路径等

⚠️ config.yaml 含本地路径和 API 密钥,已在 .gitignore 中排除,不会被提交。

4. 配置 MCP Client

Claude Code

~/.claude/settings.json 中添加:

{
  "mcpServers": {
    "local-mmcp": {
      "command": "uv",
      "args": [
        "run",
        "--project",
        "/ABSOLUTE/PATH/local-mmcp",
        "python",
        "-m",
        "local_mmcp.server"
      ],
      "env": {
        "MMCP_CONFIG": "/ABSOLUTE/PATH/local-mmcp/config.yaml"
      }
    }
  }
}

其他 MCP Client

修改对应配置文件,使用相同 command/args/env 结构即可。

5. 运行检查

# 直接运行 MCP Server(测试用)
MMCP_CONFIG=config.yaml uv run python -m local_mmcp.server

# 运行诊断脚本
bash scripts/doctor.sh

安全设计

路径白名单

只允许访问 config.yamlsecurity.allowed_roots 列出的目录:

security:
  allowed_roots:
    - "~/Projects"
    - "~/Desktop"
    - "~/Downloads"
    - "~/working"
    - "~/Pictures"
    - "/tmp"
    - "~/.claude"

路径黑名单

自动拒绝包含以下模式的路径:.ssh.gnupgLibrary/Keychains.envid_rsaid_ed25519

确认令牌机制

高风险 GUI 操作(gui_actadb_install 等)需要一次性确认令牌,TTL 300 秒。

macOS 文件图标检测

当用户从 Finder 复制文件并粘贴到 AI 客户端时,系统可能传递文件图标而非真实图片内容。Local-MMCP 会:

  1. 检测:通过灰度 + alpha 多样性 + 尺寸综合评分识别文件图标

  2. 自动替换:三层策略找到原始文件(剪贴板路径 → 剪贴板图片数据 → 文件系统搜索)

  3. 不阻塞:检测失败时仅警告,不阻断分析流程

设计说明

详细的架构文档和设计说明请参考:

文档

内容

ARCHITECTURE.md

完整架构文档:数据流、模块详解、性能数据

local-mmcp-dev-doc.md

原始设计文档:功能定义、接口设计、实现计划

CHANGELOG.md

版本变更记录

核心设计模式

  1. 单一 image_source 参数:对齐 ZhiPu MCP 设计,自动识别 URL / base64 / 本地路径

  2. 模型生命周期管理ModelManager 单模型互斥 + 空闲超时自动卸载(300 秒)

  3. 优雅降级:VLM 不可用时降级为 OCR 模式,不会因单一组件崩溃

  4. 最小图像处理:不做 normalize/resize,直接传原始文件给 VLM API

测试

# 运行所有单元测试(57 个测试用例)
uv run pytest tests/ -v

# 运行特定测试
uv run pytest tests/test_all.py -v         # 全量集成测试
uv run pytest tests/test_model_manager.py  # 模型管理器测试

测试覆盖

测试类别

用例数

覆盖内容

health_check

9

所有组件状态检查

vision_inspect

3

本地路径/URL/base64

安全路径校验

12

合法路径/非法路径/deny 列表

确认令牌

6

创建/验证/重用/过期

文件图标检测

5

真实照片/海报/模拟图标

OmlxClient

10

初始化/JSON/data URL

vision_crop_verify

1

裁剪数学验证

vision_diff

1

像素+语义差异

错误路径

12

空参数/非法路径/不存在

已知限制

  1. Apple Silicon only — 依赖 omlx + MLX 框架,不支持 Intel Mac 或 Linux

  2. 内存需求高 — 视觉模型 (gemma-4-31B) 约 58GB,推荐 128GB+ 统一内存

  3. 模型加载延迟 — 冷启动加载模型约 20 秒,后续请求复用已加载模型

  4. MinerU/PaddleOCR — 需单独安装,未安装时 doc_parse 返回安装建议

  5. macOS 权限 — GUI 自动化需要辅助功能权限(System Preferences → Privacy → Accessibility)

  6. 剪贴板时效性 — macOS 文件图标替换依赖剪贴板内容,复制后需立即使用

项目结构

local-mmcp/
├── local_mmcp/
│   ├── server.py           # MCP 服务器入口
│   ├── config.py           # Pydantic 配置模型
│   ├── security.py         # 路径安全校验
│   ├── schemas.py          # 数据模型
│   ├── errors.py           # 错误处理
│   ├── tools/              # 9 个工具实现
│   │   ├── vision.py       # vision_inspect / crop_verify / diff
│   │   ├── document.py     # doc_parse
│   │   ├── audio.py        # audio_transcribe
│   │   ├── video.py        # video_index
│   │   ├── gui.py          # gui_observe / gui_act
│   │   └── health.py       # health_check
│   ├── clients/
│   │   └── omlx_client.py  # oMLX API 客户端 + ModelManager
│   ├── adapters/           # 外部工具适配器
│   │   ├── paddleocr_adapter.py
│   │   ├── mineru_adapter.py
│   │   ├── qwen_asr_adapter.py
│   │   ├── ffmpeg_adapter.py
│   │   ├── playwright_adapter.py
│   │   ├── adb_adapter.py
│   │   ├── macos_adapter.py
│   │   └── ui_tars_adapter.py
│   ├── utils/
│   │   ├── images.py       # 图片处理 + 图标检测/替换
│   │   ├── files.py        # 文件工具
│   │   ├── json_tools.py   # JSON 提取
│   │   └── subprocesses.py # 子进程管理
│   └── prompts/            # 工具 prompt 模板
├── tests/                  # 57 个测试用例
├── scripts/
│   ├── doctor.sh           # 诊断脚本
│   └── run_mcp.sh          # 运行脚本
├── config.example.yaml     # 配置模板
├── ARCHITECTURE.md         # 架构文档
├── local-mmcp-dev-doc.md   # 原始设计文档
└── pyproject.toml          # 项目配置

License

MIT

Available Tools

9 tools
audio_transcribeA

音频转写:支持 wav/mp3/m4a/flac 等格式。使用 Qwen3-ASR (mlx-audio)。

Args: audio_path: 音频文件路径 language: 语言 (auto|zh|en|ja|ko) timestamps: 是否输出时间戳 output_format: 输出格式 (json|txt)

ParametersJSON Schema
NameRequiredDescriptionDefault
languageNoauto
audio_pathYes
timestampsNo
output_formatNojson

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must disclose behavior. It mentions the model (Qwen3-ASR) and formats, but does not mention side effects, requirements, or return behavior. For a transcription tool, it's likely safe, but this is not explicitly stated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise: a short intro plus a well-formatted args list. Every sentence adds value (formats, model, parameter meanings) with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple, has an output schema, and the description covers core aspects like formats and model. It lacks explicit guidance on when to use vs alternatives, but overall it is adequate for a transcription tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, but the description lists all four parameters with meaningful explanations, including allowed values for language (auto|zh|en|ja|ko) and output_format (json|txt). This compensates well for the schema's minimal parameter descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states '音频转写' (audio transcription) with a specific verb and resource, and lists supported formats (wav/mp3/m4a/flac). It is unambiguous and obviously distinct from the sibling vision/GUI tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage by specifying supported audio formats and the transcription context. It does not explicitly state when not to use or alternatives, but the siblings are clearly different, making the intended use clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

doc_parseA

解析文档(PDF/DOCX/PPTX/XLSX/图片)为 Markdown/JSON。使用 MinerU 或 PaddleOCR。

Args: file_path: 文档文件路径 pages: 页码范围(如 "1-20"),默认全部 engine: 解析引擎 (auto|mineru|paddleocr) need_tables: 是否需要表格 need_formulas: 是否需要公式 need_images: 是否需要图片提取 output_format: 输出格式 (markdown|json|markdown+json)

ParametersJSON Schema
NameRequiredDescriptionDefault
pagesNo
engineNoauto
file_pathYes
need_imagesNo
need_tablesNo
need_formulasNo
output_formatNomarkdown+json

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description bears full responsibility for behavioral disclosure. It mentions the use of MinerU or PaddleOCR engines and output formats, but does not state whether the operation is read-only, if network access is required, or any error-handling behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured: a single purpose sentence followed by a parameter list. Every sentence earns its place, and the primary function is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers all parameters, supported input formats, engine options, and output formats. It lacks details on prerequisites or edge cases, but since an output schema exists, return-value details are not needed here.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

All 7 parameters are described with meanings and examples (e.g., pages '1-20', engine choices, boolean flags). This adds substantial value beyond the input schema, which only lists titles and defaults without descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening line clearly states the tool's function: 'Parse documents (PDF/DOCX/PPTX/XLSX/images) into Markdown/JSON'. This specifies a verb, resource types, and output formats, making it distinct from sibling tools like audio_transcribe or video_index.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description lists supported file formats (PDF/DOCX/PPTX/XLSX/images), giving clear context on when to use this tool. It does not explicitly mention alternatives or exclusions, but sibling tools cover other media types, making the intended use evident.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

gui_actA

执行 GUI 动作。高风险动作(点击、输入、提交等)需要确认令牌。

Args: target: 目标 (browser|android|macos) session_id: 会话 ID action: 动作 (click|type|press|swipe|back|home|wait|open_url) selector: CSS selector(浏览器用) x: X 坐标 y: Y 坐标 text: 输入文本 url: URL(open_url 动作) reason: 执行原因 confirm_token: 确认令牌(高风险动作必须)

ParametersJSON Schema
NameRequiredDescriptionDefault
xNo
yNo
urlNo
textNo
actionNo
reasonNo
targetNobrowser
selectorNo
session_idNodefault
confirm_tokenNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses that high-risk actions require a confirmation token, which is valuable behavioral context. However, it does not mention potential side effects (e.g., navigation, state changes) or prerequisites like session validity. The token requirement is the only explicit behavioral trait.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise: a one-line summary followed by a clean parameter list. Each item earns its place, and the structure is easy to scan. No fluff or redundant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 10 parameters and no annotations, the description covers all parameters and includes a safety note about high-risk actions. The output schema exists, so return values need not be described. Minor gaps remain, such as clarifying which actions are exactly 'high-risk' and whether a session must already exist, but overall it is sufficient for an agent to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description explicitly explains every parameter in the schema, including target, session_id, action, selector, x, y, text, url, reason, and confirm_token. It adds context by specifying allowed values for action and target, and clarifies that confirm_token is required for high-risk actions. This fully compensates for the schema's lack of descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states '执行 GUI 动作' (Execute GUI actions), with a specific verb and resource. It enumerates target types and action types, distinguishing it from the sibling tool gui_observe which is for observation. The scope is explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage by naming targets and actions, but provides no explicit guidance on when to choose this tool over alternatives (e.g., gui_observe). It mentions the confirm token requirement for high-risk actions but does not compare with sibling tools. Context is present but alternatives are not addressed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

gui_observeA

观察 GUI 目标:浏览器页面、安卓设备屏幕或 macOS 桌面。返回截图和状态信息。

Args: target: 观察目标 (browser|android|macos) session_id: 会话 ID include_screenshot: 是否包含截图 include_accessibility_tree: 是否包含无障碍树

ParametersJSON Schema
NameRequiredDescriptionDefault
targetNobrowser
session_idNodefault
include_screenshotNo
include_accessibility_treeNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description is the sole source of behavioral information. The verb '观察' (observe) implies a read-only operation, and the description states it returns a screenshot and status, which is useful. However, it does not explicitly confirm non-destructiveness, mention any side effects on the target system, or disclose prerequisites like requiring an active session. It adds some context beyond the schema but lacks full transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and well-structured: a one-sentence purpose followed by a concise ARGS list. Every line provides necessary information with no filler or repetition. It is appropriately sized for the tool's complexity and front-loaded with the primary behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given an output schema exists (though not shown), the description need not enumerate return fields in detail. It mentions the key return type (screenshot, status) and parameter effects. It could further explain session semantics or real-time capture behavior, but for a straightforward observation tool the coverage is adequate. Missing explicit mention of read-only behavior is slightly mitigated by the verb 'observe'.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides only titles and defaults with zero descriptions, so the description's Args section carries the full semantic weight. It explains each parameter's meaning, including the allowed values for 'target' (browser|android|macos), the purpose of session_id, and booleans for screenshot and accessibility tree inclusion. This is a strong complement to the schema, adding critical context where the schema is silent.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states it observes GUI targets (browser, Android, macOS) and returns screenshots and status info. It uses a specific verb 'observe' and names distinct resources, which distinguishes it from mutation tools like gui_act. However, it doesn't explicitly differentiate from vision-related sibling tools such as vision_inspect, so it's not fully distinct.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives no guidance on when to use this tool versus alternatives like vision_inspect or health_check. There are no exclusions or context cues for choosing this over other observers/monitors. The intended use is implied by the name and primary sentence, but explicit usage guidance is absent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

health_checkA

检查 oMLX、VLM、MinerU、PaddleOCR、Qwen3-ASR、ffmpeg、Playwright、ADB 等组件的可用性。

Args: deep: 是否进行深度检查(实际调用 API 验证视觉能力)

ParametersJSON Schema
NameRequiredDescriptionDefault
deepNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It does disclose that deep=true involves actual API calls to verify vision capability, which is a meaningful behavioral detail. However, it does not state whether the tool is read-only, what the return structure is, or whether network access is required. The transparency is moderate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, with a single front-loaded sentence stating the tool's purpose, followed by a clear Args section for the parameter. No unnecessary prose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers the core purpose and the one parameter well. Given the low complexity (one boolean), it is mostly adequate. However, it does not describe the expected output format or return values, and with no output schema or annotations available, the agent may remain uncertain about how to interpret the results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema only lists a boolean 'deep' with no description, and schema description coverage is 0%. The description compensates fully by explaining that deep controls whether to perform a deeper check involving actual API calls to verify visual abilities. This unambiguously clarifies the parameter's purpose.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states it checks the availability of a specific list of components (oMLX, VLM, MinerU, PaddleOCR, etc.), which satisfies 'specific verb+resource'. This clearly distinguishes health_check from sibling tools that focus on GUI observation, vision inspection, parsing, transcription, and actions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no explicit guidance on when to use this tool versus alternatives. It does not mention pre-flight validation scenarios or exclude cases where a deep check is unnecessary. The only hint is the 'deep' parameter explanation, which implies deeper verification but does not offer usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

video_indexC

视频索引:抽帧 + ASR 语音转写 + VLM 视觉分段理解。输出时间线。

Args: video_path: 视频文件路径 task: 分析任务描述 fps: 抽帧帧率 max_frames: 最大帧数 need_asr: 是否提取语音转写 need_ocr: 是否对帧做 OCR

ParametersJSON Schema
NameRequiredDescriptionDefault
fpsNo
taskNo分析视频内容
need_asrNo
need_ocrNo
max_framesNo
video_pathYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations are absent, so the description bears the full burden of behavioral disclosure. It lists the pipeline steps (frame extraction, ASR, VLM) but fails to mention computational cost, processing time, potential errors, or whether the video file is modified. The read-only nature is implied but not explicit.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact: a one-sentence summary followed by a parameter list. There is no fluff or redundancy beyond repeating parameter names from the schema, but the added Chinese annotations earn their place. The main functionality is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return value details are covered. The description explains the processing pipeline and all parameters at a basic level, but it omits important contextual details such as supported video formats, resource requirements, failure modes, or performance expectations. It is adequate for a basic invocation but not for robust selection and planning.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does add Chinese labels to all six parameters, which clarifies their meaning. However, the labels are terse; for example, 'task' is just 'analysis task description' with no details on expected task formats or examples, and the relationship between fps and max_frames is left ambiguous.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's composite function: frame extraction, ASR transcription, VLM visual understanding, and timeline output. This distinguishes it from sibling tools like audio_transcribe or vision_inspect by explicitly combining multiple modalities, though it doesn't name those alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives such as audio_transcribe for speech-only tasks or vision_inspect for image analysis. The description assumes the user already knows the appropriate context, offering no exclusions or prerequisite conditions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

vision_crop_verifyA

Crop a region of an image, enlarge it, and re-analyze with OCR/VLM for verification. Ideal for verifying small text, button states, etc. Use this tool ONLY when you need to zoom into and verify a specific region of an image. Do NOT use for: full image analysis (use vision_inspect), comparing two images (use vision_diff).

Args: image_source: Image source - local file path or remote URL (recommended, auto-detects format) image_path: Original image file path (backward compatible) bbox: Crop region [x1, y1, x2, y2] in original image pixel coordinates scale: Enlargement scale factor task: Verification task description use_paddleocr: Whether to use PaddleOCR image_data: Base64 encoded image data (backward compatible)

ParametersJSON Schema
NameRequiredDescriptionDefault
bboxNo
taskNo识别这块区域的文字和元素状态
scaleNo
image_dataNo
image_pathNo
image_sourceNo
use_paddleocrNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It accurately describes the multi-step process (crop, enlarge, re-analyze) and the intended use case (verifying small text, button states). It does not mention potential failure modes, authentication, or rate limits, but for a read-only analysis tool, the core behavior is transparent and aligns with what the tool does.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured: a concise opening sentence explaining the core action, a usage guideline sentence, a clear 'Do NOT use' sentence with alternatives, and a tidy list of parameters. It is appropriately sized with no filler or redundant text, and every sentence serves a purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (7 params, no annotations, no schema descriptions), the description covers all essential aspects: what it does, when to use it, what inputs are needed and their formats, and how it relates to sibling tools. The output schema exists, so the description need not detail return values. This is effectively a complete guide for an agent to select and invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 0%, so the description must fully compensate. It provides a separate Args section explaining every parameter, including serialization details ('bbox: Crop region [x1, y1, x2, y2] in original image pixel coordinates'), format choices ('image_source: local file path or remote URL... auto-detects format'), and backward-compatibility notes. This adds meaning far beyond the raw schema types.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Crop a region of an image, enlarge it, and re-analyze with OCR/VLM for verification.' It identifies a specific verb ('crop', 'enlarge', 're-analyze') and a resource (image region) and distinguishes itself from siblings by explicitly naming vision_inspect and vision_diff as alternatives for different tasks.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit when-to-use guidance: 'Use this tool ONLY when you need to zoom into and verify a specific region of an image.' It also gives clear exclusions with alternatives: 'Do NOT use for: full image analysis (use vision_inspect), comparing two images (use vision_diff).' This leaves no ambiguity about when to select this tool versus its siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

vision_diffA

Compare two UI screenshots or a design mockup with an implementation screenshot to identify visual differences. Use this tool ONLY when comparing an expected/reference UI with an actual implementation. Do NOT use for: analyzing a single image (use vision_inspect), error diagnosis, general image comparison.

Args: expected_image_source: Expected image source - local file path or remote URL (recommended) actual_image_source: Actual image source - local file path or remote URL (recommended) expected_image_path: Expected/design image path (backward compatible) actual_image_path: Actual/screenshot path (backward compatible) task: Comparison task description include_pixel_diff: Whether to include pixel-level diff visualization expected_image_data: Base64 encoded expected image data (backward compatible) actual_image_data: Base64 encoded actual image data (backward compatible)

ParametersJSON Schema
NameRequiredDescriptionDefault
taskNo比较两张图片,指出差异
actual_image_dataNo
actual_image_pathNo
include_pixel_diffNo
actual_image_sourceNo
expected_image_dataNo
expected_image_pathNo
expected_image_sourceNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It does disclose that the tool compares images and can produce pixel-level diffs (via include_pixel_diff), but it does not mention whether the tool is read-only, handles remote/local sources differently, or any potential side effects. Some behavioral context is present, but significant gaps remain.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured: a clear purpose sentence, explicit usage rules, then a compact arg list. Each line earns its place, and the layout makes it easy to scan. No redundancy or fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (8 params, no schema descriptions, no annotations) and the existence of an output schema, the description covers purpose, usage, and parameter semantics. It omits details like error handling or return format, but the output schema likely covers that. Minor gaps in edge-case behavior prevent a perfect score.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does thoroughly. It explains all 8 parameters, including the relationships between source, path, and data variants, and flags backward compatibility. This is far more useful than the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Compare two UI screenshots or a design mockup with an implementation screenshot to identify visual differences.' It clearly distinguishes this tool from siblings like vision_inspect, which is called out as the alternative for single-image analysis.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states when to use: 'Use this tool ONLY when comparing an expected/reference UI with an actual implementation.' It also lists exclusions and alternatives: 'Do NOT use for: analyzing a single image (use vision_inspect), error diagnosis, general image comparison.' This is exemplary usage guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

vision_inspectA

Analyze a single image: screenshot understanding, UI analysis, chart comprehension, error extraction. Use this tool ONLY when the user has an image and wants to understand its content, extract information, or analyze visual elements. Do NOT use for: comparing two images (use vision_diff), cropping/verifying a region (use vision_crop_verify).

Supports automatic detection of input format via image_source parameter (recommended):

  • Local file path (e.g. /path/to/image.png)

  • HTTP/HTTPS URL (e.g. https://example.com/image.png)

  • Base64 data URI (e.g. data:image/png;base64,...)

  • Plain base64 string (auto-detected by file header)

Args: image_source: Image source - local file path or remote URL (recommended, auto-detects format) image_path: Local image file path (backward compatible, prefer image_source) task: Analysis task description mode: Analysis mode (general|game_ui|web_ui|error|chart|diagram|design_to_code) need_ocr: Whether to enable OCR need_bbox: Whether to detect element bounding boxes detail_level: Detail level (normal|high) max_tokens: Maximum output tokens image_data: Base64 encoded image data (backward compatible)

ParametersJSON Schema
NameRequiredDescriptionDefault
modeNogeneral
taskNo分析这张图片
need_ocrNo
need_bboxNo
image_dataNo
image_pathNo
max_tokensNo
detail_levelNonormal
image_sourceNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses auto-detection of input formats via image_source, lists backward compatibility parameters, and explains the mode options. However, it does not mention side effects, error behavior, or safety profile (e.g., read-only nature), which would be relevant for a tool that processes user-provided images. Still, it adds meaningful behavioral context beyond the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured, front-loaded with the core purpose, followed by usage exclusions, input format guidelines, and an Args list. Every sentence serves a purpose, and the bullet-like format makes it easy to scan. Despite covering 9 parameters, it remains compact and free of fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (9 parameters, 0% schema coverage, no annotations), the description provides near-complete context: it covers all parameters, explains the recommended input source, lists mode values, and gives clear sibling distinctions. An output schema exists, so not explaining return values is acceptable. No major gaps remain.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does so thoroughly: every parameter is listed in the Args section with a concise explanation, and image_source gets extra detail with concrete format examples (local path, URL, data URI, plain base64). This adds substantial meaning beyond the bare schema titles.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description starts with a specific verb and resource: 'Analyze a single image' and lists concrete use cases (screenshot understanding, UI analysis, chart comprehension, error extraction). It explicitly distinguishes from siblings by naming vision_diff and vision_crop_verify as alternatives for excluded tasks, making the tool's purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description states clearly when to use ('Use this tool ONLY when the user has an image and wants to understand its content') and when not to use it, naming the alternative tools for two specific excluded scenarios. This provides explicit usage context and differentiation from siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 9 tool updatesv0.3.0
    • First observedaudio_transcribe
    • First observeddoc_parse
    • First observedgui_act
    • First observedgui_observe
    • First observedhealth_check
    • First observedvideo_index
    • First observedvision_crop_verify
    • First observedvision_diff
    • First observedvision_inspect

TDQS

A4/5.0

Scored across 9 tools

Disambiguation5/5

Each tool has a distinct purpose: gui_observe/gui_act for GUI control, vision_* tools for image analysis with explicit separation (inspect, crop_verify, diff), and doc_parse/audio_transcribe/video_index for media processing. The vision tools even include usage warnings to prevent misselection.

Naming Consistency5/5

All tools follow a consistent <domain>_<verb> pattern (e.g., gui_observe, vision_inspect, doc_parse, audio_transcribe, video_index, health_check). No mixed casing or verb-style inconsistency.

Tool Count5/5

9 tools is well within the ideal range for a multimodal toolkit, covering GUI automation, vision analysis, document parsing, audio transcription, video indexing, and health checks without unnecessary bloat.

Completeness4/5

The surface covers the core modalities (vision, audio, video, documents, GUI) and includes a health check utility. Minor gaps exist, such as no separate GUI session management tool, but the core workflows are well supported.

Maintenance

ActivityInactive
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    C
    maintenance
    Standalone MCP server that gives AI agents full GUI control over macOS — screenshots, mouse, keyboard, apps, clipboard, and multi-display — with zero private dependencies.
    19
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    MCP server that gives Claude and local LLMs access to Apple's on-device frameworks — Vision OCR, NSDataDetector, and Apple Intelligence FoundationModels. Everything runs on your Mac with zero data leaving.
    1
    MIT
  • A
    license
    B
    quality
    D
    maintenance
    MCP server that enables AI to fully control macOS — mouse, keyboard, terminal, screenshots, window management, UI element detection, and provides AI-optimized information reporting.
    36
    19 npm
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    MCP server that enables local Apple on-device Foundation Model access via any MCP client, supporting text generation, structured output, and multi-turn chat on macOS.
    2
    MIT