local-mmcp
Enables GUI automation for Android devices via ADB, supporting observation and execution of actions on Android screens.
Integrates FFmpeg for video processing, enabling video time indexing with frame extraction and ASR.
Provides GUI automation capabilities on macOS via accessibility APIs, allowing observation and execution of actions on native desktop applications.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@local-mmcp 分析这张截图里的UI元素"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Local-MMCP
本地多模态 MCP Server — 为 Claude Code / Codex / Cline 等 AI 客户端提供视觉、文档、音频、视频、GUI 自动化能力
项目简介
Local-MMCP 是一个基于 MCP (Model Context Protocol) 的本地多模态服务器,通过 stdio transport 向 AI 客户端暴露 9 个多模态工具。
核心设计理念:本地优先、隐私安全、优雅降级。所有数据处理在本地完成,无需将图片/文档/音频上传到第三方服务。
Related MCP server: lucid-apple-mcp
架构总览
MCP Client (Claude Code / Codex / Cline / Roo / OpenCode 等)
│
▼
Local-MMCP MCP Server (stdio transport)
│
├── oMLX 视觉/文本模型 (本地 Apple Silicon)
│ ├── gemma-4-31B-it-Uncensored-MAX-MLX (视觉)
│ └── Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-mlx-8bit (文本)
│
├── MinerU / PaddleOCR (文档解析)
├── Qwen3-ASR / mlx-audio (语音转写)
├── ffmpeg (视频处理)
└── Playwright / ADB / macOS Accessibility (GUI 自动化)分层架构
Server (MCP Protocol Handler)
└── Tools (9 个工具模块)
├── Clients (oMLX API 客户端 + 模型生命周期管理)
├── Adapters (外部工具适配器)
└── Utils (图片处理 / 安全 / JSON 工具)工具清单
工具 | 用途 | 依赖 | 需确认 |
| 检查所有组件可用性(支持 deep 模式验证 VLM) | 无 | ❌ |
| 分析图片:截图理解、UI 分析、图表、错误诊断 | oMLX VLM | ❌ |
| 裁剪图片局部放大复核 | oMLX VLM | ❌ |
| 比较两张图片差异(像素 + 语义) | oMLX VLM | ❌ |
| 文档转 Markdown/JSON(PDF/DOCX/PPTX/XLSX/图片) | MinerU 或 PaddleOCR | ❌ |
| 音频转写(WAV/MP3/M4A/FLAC) | mlx-audio (Qwen3-ASR) | ❌ |
| 视频时间线索引(抽帧 + ASR + VLM 理解) | ffmpeg + oMLX + mlx-audio | ❌ |
| 观察 GUI 状态(浏览器/Android/macOS) | Playwright / ADB / AppKit | ❌ |
| 执行 GUI 动作(点击/输入/滑动等) | Playwright / ADB / AppKit | ✅ |
快速开始
1. 环境要求
硬件: Mac with Apple Silicon (M1+), 推荐 128GB+ 统一内存
系统: macOS 14+
Python: 3.11+
oMLX: omlx.ai 本地模型推理服务
包管理: uv
2. 安装
git clone https://github.com/rorojiao/local-mmcp.git
cd local-mmcp
uv sync
# 可选依赖(按需安装)
uv pip install mlx-audio # 语音转写
uv pip install pyautogui # macOS 桌面自动化3. 配置
cp config.example.yaml config.yaml
# 编辑 config.yaml,调整 omlx 端口、模型名、安全路径等⚠️
config.yaml含本地路径和 API 密钥,已在.gitignore中排除,不会被提交。
4. 配置 MCP Client
Claude Code
在 ~/.claude/settings.json 中添加:
{
"mcpServers": {
"local-mmcp": {
"command": "uv",
"args": [
"run",
"--project",
"/ABSOLUTE/PATH/local-mmcp",
"python",
"-m",
"local_mmcp.server"
],
"env": {
"MMCP_CONFIG": "/ABSOLUTE/PATH/local-mmcp/config.yaml"
}
}
}
}其他 MCP Client
修改对应配置文件,使用相同 command/args/env 结构即可。
5. 运行检查
# 直接运行 MCP Server(测试用)
MMCP_CONFIG=config.yaml uv run python -m local_mmcp.server
# 运行诊断脚本
bash scripts/doctor.sh安全设计
路径白名单
只允许访问 config.yaml 中 security.allowed_roots 列出的目录:
security:
allowed_roots:
- "~/Projects"
- "~/Desktop"
- "~/Downloads"
- "~/working"
- "~/Pictures"
- "/tmp"
- "~/.claude"路径黑名单
自动拒绝包含以下模式的路径:.ssh、.gnupg、Library/Keychains、.env、id_rsa、id_ed25519
确认令牌机制
高风险 GUI 操作(gui_act、adb_install 等)需要一次性确认令牌,TTL 300 秒。
macOS 文件图标检测
当用户从 Finder 复制文件并粘贴到 AI 客户端时,系统可能传递文件图标而非真实图片内容。Local-MMCP 会:
检测:通过灰度 + alpha 多样性 + 尺寸综合评分识别文件图标
自动替换:三层策略找到原始文件(剪贴板路径 → 剪贴板图片数据 → 文件系统搜索)
不阻塞:检测失败时仅警告,不阻断分析流程
设计说明
详细的架构文档和设计说明请参考:
文档 | 内容 |
完整架构文档:数据流、模块详解、性能数据 | |
原始设计文档:功能定义、接口设计、实现计划 | |
版本变更记录 |
核心设计模式
单一 image_source 参数:对齐 ZhiPu MCP 设计,自动识别 URL / base64 / 本地路径
模型生命周期管理:
ModelManager单模型互斥 + 空闲超时自动卸载(300 秒)优雅降级:VLM 不可用时降级为 OCR 模式,不会因单一组件崩溃
最小图像处理:不做 normalize/resize,直接传原始文件给 VLM API
测试
# 运行所有单元测试(57 个测试用例)
uv run pytest tests/ -v
# 运行特定测试
uv run pytest tests/test_all.py -v # 全量集成测试
uv run pytest tests/test_model_manager.py # 模型管理器测试测试覆盖
测试类别 | 用例数 | 覆盖内容 |
health_check | 9 | 所有组件状态检查 |
vision_inspect | 3 | 本地路径/URL/base64 |
安全路径校验 | 12 | 合法路径/非法路径/deny 列表 |
确认令牌 | 6 | 创建/验证/重用/过期 |
文件图标检测 | 5 | 真实照片/海报/模拟图标 |
OmlxClient | 10 | 初始化/JSON/data URL |
vision_crop_verify | 1 | 裁剪数学验证 |
vision_diff | 1 | 像素+语义差异 |
错误路径 | 12 | 空参数/非法路径/不存在 |
已知限制
Apple Silicon only — 依赖 omlx + MLX 框架,不支持 Intel Mac 或 Linux
内存需求高 — 视觉模型 (gemma-4-31B) 约 58GB,推荐 128GB+ 统一内存
模型加载延迟 — 冷启动加载模型约 20 秒,后续请求复用已加载模型
MinerU/PaddleOCR — 需单独安装,未安装时 doc_parse 返回安装建议
macOS 权限 — GUI 自动化需要辅助功能权限(System Preferences → Privacy → Accessibility)
剪贴板时效性 — macOS 文件图标替换依赖剪贴板内容,复制后需立即使用
项目结构
local-mmcp/
├── local_mmcp/
│ ├── server.py # MCP 服务器入口
│ ├── config.py # Pydantic 配置模型
│ ├── security.py # 路径安全校验
│ ├── schemas.py # 数据模型
│ ├── errors.py # 错误处理
│ ├── tools/ # 9 个工具实现
│ │ ├── vision.py # vision_inspect / crop_verify / diff
│ │ ├── document.py # doc_parse
│ │ ├── audio.py # audio_transcribe
│ │ ├── video.py # video_index
│ │ ├── gui.py # gui_observe / gui_act
│ │ └── health.py # health_check
│ ├── clients/
│ │ └── omlx_client.py # oMLX API 客户端 + ModelManager
│ ├── adapters/ # 外部工具适配器
│ │ ├── paddleocr_adapter.py
│ │ ├── mineru_adapter.py
│ │ ├── qwen_asr_adapter.py
│ │ ├── ffmpeg_adapter.py
│ │ ├── playwright_adapter.py
│ │ ├── adb_adapter.py
│ │ ├── macos_adapter.py
│ │ └── ui_tars_adapter.py
│ ├── utils/
│ │ ├── images.py # 图片处理 + 图标检测/替换
│ │ ├── files.py # 文件工具
│ │ ├── json_tools.py # JSON 提取
│ │ └── subprocesses.py # 子进程管理
│ └── prompts/ # 工具 prompt 模板
├── tests/ # 57 个测试用例
├── scripts/
│ ├── doctor.sh # 诊断脚本
│ └── run_mcp.sh # 运行脚本
├── config.example.yaml # 配置模板
├── ARCHITECTURE.md # 架构文档
├── local-mmcp-dev-doc.md # 原始设计文档
└── pyproject.toml # 项目配置License
MIT
Available Tools
9 toolsaudio_transcribeA
音频转写:支持 wav/mp3/m4a/flac 等格式。使用 Qwen3-ASR (mlx-audio)。
Args: audio_path: 音频文件路径 language: 语言 (auto|zh|en|ja|ko) timestamps: 是否输出时间戳 output_format: 输出格式 (json|txt)
Inputs: audio_path (required), language, timestamps, output_format. Returns: result
doc_parseA
解析文档(PDF/DOCX/PPTX/XLSX/图片)为 Markdown/JSON。使用 MinerU 或 PaddleOCR。
Args: file_path: 文档文件路径 pages: 页码范围(如 "1-20"),默认全部 engine: 解析引擎 (auto|mineru|paddleocr) need_tables: 是否需要表格 need_formulas: 是否需要公式 need_images: 是否需要图片提取 output_format: 输出格式 (markdown|json|markdown+json)
Inputs: file_path (required), pages, engine, need_images, +3 more. Returns: result
gui_actA
执行 GUI 动作。高风险动作(点击、输入、提交等)需要确认令牌。
Args: target: 目标 (browser|android|macos) session_id: 会话 ID action: 动作 (click|type|press|swipe|back|home|wait|open_url) selector: CSS selector(浏览器用) x: X 坐标 y: Y 坐标 text: 输入文本 url: URL(open_url 动作) reason: 执行原因 confirm_token: 确认令牌(高风险动作必须)
Inputs: x, y, url, text, +6 more. Returns: result
gui_observeA
观察 GUI 目标:浏览器页面、安卓设备屏幕或 macOS 桌面。返回截图和状态信息。
Args: target: 观察目标 (browser|android|macos) session_id: 会话 ID include_screenshot: 是否包含截图 include_accessibility_tree: 是否包含无障碍树
Inputs: target, session_id, include_screenshot, include_accessibility_tree. Returns: result
health_checkA
检查 oMLX、VLM、MinerU、PaddleOCR、Qwen3-ASR、ffmpeg、Playwright、ADB 等组件的可用性。
Args: deep: 是否进行深度检查(实际调用 API 验证视觉能力)
Inputs: deep. Returns: result
video_indexC
视频索引:抽帧 + ASR 语音转写 + VLM 视觉分段理解。输出时间线。
Args: video_path: 视频文件路径 task: 分析任务描述 fps: 抽帧帧率 max_frames: 最大帧数 need_asr: 是否提取语音转写 need_ocr: 是否对帧做 OCR
Inputs: video_path (required), fps, task, need_asr, +2 more. Returns: result
vision_crop_verifyA
Crop a region of an image, enlarge it, and re-analyze with OCR/VLM for verification. Ideal for verifying small text, button states, etc. Use this tool ONLY when you need to zoom into and verify a specific region of an image. Do NOT use for: full image analysis (use vision_inspect), comparing two images (use vision_diff).
Args: image_source: Image source - local file path or remote URL (recommended, auto-detects format) image_path: Original image file path (backward compatible) bbox: Crop region [x1, y1, x2, y2] in original image pixel coordinates scale: Enlargement scale factor task: Verification task description use_paddleocr: Whether to use PaddleOCR image_data: Base64 encoded image data (backward compatible)
Inputs: bbox, task, scale, image_data, +3 more. Returns: result
vision_diffA
Compare two UI screenshots or a design mockup with an implementation screenshot to identify visual differences. Use this tool ONLY when comparing an expected/reference UI with an actual implementation. Do NOT use for: analyzing a single image (use vision_inspect), error diagnosis, general image comparison.
Args: expected_image_source: Expected image source - local file path or remote URL (recommended) actual_image_source: Actual image source - local file path or remote URL (recommended) expected_image_path: Expected/design image path (backward compatible) actual_image_path: Actual/screenshot path (backward compatible) task: Comparison task description include_pixel_diff: Whether to include pixel-level diff visualization expected_image_data: Base64 encoded expected image data (backward compatible) actual_image_data: Base64 encoded actual image data (backward compatible)
Inputs: task, actual_image_data, actual_image_path, include_pixel_diff, +4 more. Returns: result
vision_inspectA
Analyze a single image: screenshot understanding, UI analysis, chart comprehension, error extraction. Use this tool ONLY when the user has an image and wants to understand its content, extract information, or analyze visual elements. Do NOT use for: comparing two images (use vision_diff), cropping/verifying a region (use vision_crop_verify).
Supports automatic detection of input format via image_source parameter (recommended):
Local file path (e.g. /path/to/image.png)
HTTP/HTTPS URL (e.g. https://example.com/image.png)
Base64 data URI (e.g. data:image/png;base64,...)
Plain base64 string (auto-detected by file header)
Args: image_source: Image source - local file path or remote URL (recommended, auto-detects format) image_path: Local image file path (backward compatible, prefer image_source) task: Analysis task description mode: Analysis mode (general|game_ui|web_ui|error|chart|diagram|design_to_code) need_ocr: Whether to enable OCR need_bbox: Whether to detect element bounding boxes detail_level: Detail level (normal|high) max_tokens: Maximum output tokens image_data: Base64 encoded image data (backward compatible)
Inputs: mode, task, need_ocr, need_bbox, +5 more. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 9 tools
Each tool has a distinct purpose: gui_observe/gui_act for GUI control, vision_* tools for image analysis with explicit separation (inspect, crop_verify, diff), and doc_parse/audio_transcribe/video_index for media processing. The vision tools even include usage warnings to prevent misselection.
All tools follow a consistent <domain>_<verb> pattern (e.g., gui_observe, vision_inspect, doc_parse, audio_transcribe, video_index, health_check). No mixed casing or verb-style inconsistency.
9 tools is well within the ideal range for a multimodal toolkit, covering GUI automation, vision analysis, document parsing, audio transcription, video indexing, and health checks without unnecessary bloat.
The surface covers the core modalities (vision, audio, video, documents, GUI) and includes a health check utility. Minor gaps exist, such as no separate GUI session management tool, but the core workflows are well supported.
Related MCP Connectors
Chat, image, video, and audio models from many vendors behind one remote MCP server and one key.
A comprehensive Model Context Protocol (MCP) server that enables AI assistants to interact with yo…
OCR, transcription, file extraction, and image generation for AI agents via MCP.
MCP server that lets your AI assistant build and deploy apps, hosting each one at a live URL.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceStandalone MCP server that gives AI agents full GUI control over macOS — screenshots, mouse, keyboard, apps, clipboard, and multi-display — with zero private dependencies.20MIT
- AlicenseNot gradedqualityCmaintenanceMCP server that gives Claude and local LLMs access to Apple's on-device frameworks — Vision OCR, NSDataDetector, and Apple Intelligence FoundationModels. Everything runs on your Mac with zero data leaving.1MIT
- AlicenseBqualityDmaintenanceMCP server that enables AI to fully control macOS — mouse, keyboard, terminal, screenshots, window management, UI element detection, and provides AI-optimized information reporting.3621 npmMIT
- AlicenseNot gradedqualityDmaintenanceMCP server that enables local Apple on-device Foundation Model access via any MCP client, supporting text generation, structured output, and multi-turn chat on macOS.2MIT