Skip to main content
Glama

local-agent-senses

English

CI License: MIT Python Platform

Local vision, video and speech tools for text-only LLM agents. The default backend is Ollama on your own machine; media is not uploaded unless you explicitly configure an OpenAI-compatible endpoint.

中文说明见 README.zh-CN.md

Related MCP server: pic-id-mcp

中文

中文版本请见 README.zh-CN.md

Features

  • Image understanding and verbatim OCR/transcription.

  • PDF page rasterization for document OCR/analysis (poppler pdftoppm).

  • Video sampling for scenes, contact sheets, time windows and subtitles.

  • Optional FunASR speech-to-text with timestamps.

  • Windows screen and clipboard capture, plus platform adapters.

  • CLI for shell-based agents and MCP tools for Codex, Claude Desktop, Cursor, Cline and other MCP clients.

  • SSRF checks, download limits, media duration limits and no disk cache by default.

Quick start

Requirements: Python 3.10+, Ollama, and ffmpeg. Speech, video-site URL resolution, and PDF rasterization (poppler-utils pdftoppm) are optional extras.

git clone https://github.com/Scheme0/local-agent-senses.git
cd local-agent-senses
ollama pull haervwe/GLM-4.6V-Flash-9B
ollama pull qwen3.5:4b
python vision.py --check
python vision.py photo.png --prompt "Describe this image"
python vision.py document.png --mode text --transcribe
python vision.py paper.pdf --transcribe
python vision.py clip.mp4 --mode scenes --prompt "List each scene"

Install console commands with pip install .:

vision --check
vision-mcp
vision-adapters --all

Run vision --doctor for an offline dependency report. Add --json for machine-readable diagnostics. Service admission is limited by VISION_MAX_CONCURRENCY and VISION_SERVICE_TIMEOUT.

The adapter generator is implemented by scripts/generate_adapters.py and can also be invoked directly when integrating with another agent.

Optional dependencies:

pip install .[speech]
pip install .[ytdlp]
pip install .[test]

MCP

Run the stdio server with:

python extras/mcp_server.py

Register it in Codex, for example:

codex mcp add vision -- python /absolute/path/extras/mcp_server.py

Available tools are describe_image, transcribe, analyze_video, transcribe_audio, vision_status and vision_check.

Service results use a JSON envelope with text, kind, mode, metadata and warnings. Expected failures use { "code": ..., "message": ... }, which lets MCP clients distinguish invalid input from backend or media failures.

MCP validates image count, prompt size, frame count and FPS before invoking the CLI. The server uses an in-memory result cache. Disk caching is disabled by default because results may contain screenshots, transcripts or private documents. Enable it only when appropriate:

set VISION_MCP_CACHE=1       # Windows
export VISION_MCP_CACHE=1    # Linux/macOS

Remote media and security

HTTP and HTTPS media URLs are treated as untrusted input. Local, private, loopback, link-local, multicast, cloud-metadata and NAT64 addresses are blocked, and redirects are checked again. By default, remote media is downloaded through the checked Python path before ffmpeg sees it. This avoids the DNS re-resolution gap that can occur when ffmpeg opens a URL directly.

DNS rebinding is only mitigated, not eliminated. The stdlib downloader resolves DNS and connects in a single step, so a hostile hostname can in principle be re-resolved between the check and the connect. For strict SSRF requirements, put the tool behind an egress proxy / allowlist.

Direct ffmpeg URL streaming is available only as an explicit compatibility option (off by default) and should be used only with trusted URLs:

set VISION_DIRECT_URL_STREAM=1       # Windows
export VISION_DIRECT_URL_STREAM=1    # Linux/macOS

Remote OpenAI-compatible endpoints (api_base) must use https:// unless they point at localhost (http://localhost / http://127.0.0.1). Plain HTTP against a remote host is rejected, as are URLs without a scheme or with embedded credentials.

Set limits in vision-config.json or with environment variables:

Setting

Default

Purpose

VISION_MAX_IMAGE_MB

20

Image size cap

VISION_MAX_STDIN_MB

2000

Piped stdin media cap

VISION_MAX_PDF_PAGES

50

PDF page cap (0 = system hard ceiling of 200)

VISION_PDF_DPI

150

PDF rasterization DPI

VISION_MAX_DOWNLOAD_MB

500

Unknown/remote media cap

VISION_MAX_DURATION_H

6

Audio/video duration cap

VISION_MCP_CACHE

false

Persist MCP results to disk

VISION_DIRECT_URL_STREAM

false

Let ffmpeg open remote URLs directly

VISION_SERVICE_QUEUE_TIMEOUT

1800

Seconds to wait in the service queue

VISION_SERVICE_EXECUTION_TIMEOUT

1800

Execution budget for one request

The video thumbnail candidate count is capped by a fixed internal hard limit (MAX_THUMBNAIL_FRAMES, 1200) that cannot be disabled through configuration. Downloads and PDF pages also have system hard ceilings (8192 MB and 200 pages) that apply even when a configured limit is 0. Explicitly configured external tool paths (VISION_FFMPEG, VISION_SPEECH_PYTHON, OLLAMA_EXE, VISION_PDF_RENDERER, VISION_YTDLP) must be absolute paths to an existing regular file.

Read SECURITY.md before exposing the MCP server to another machine. Media content is untrusted data; do not follow instructions found in images, subtitles or model output.

Configuration

Copy vision-config.example.json to vision-config.json. Configuration priority is environment variables, then JSON config, then auto-detection. Important settings include text_model, quick_model, ollama_host, api_base, api_key, ffmpeg, speech_python, mcp_cache_dir and the limits listed above.

Limitations

  • Video understanding is frame-based rather than end-to-end temporal reasoning.

  • OCR accuracy depends on resolution, layout, font and the selected model.

  • FunASR is optional and may require a separate environment and model download.

  • HEIC, AVIF and other uncommon formats depend on the installed ffmpeg build.

  • Configuring a cloud endpoint sends model inputs to that endpoint; local sampling and audio processing remain local.

Development

python -m pytest tests/ -v
python -m build

See CONTRIBUTING.md, CHANGELOG.md and docs/COMPATIBILITY.md.

License

MIT. See LICENSE and THIRD_PARTY_NOTICES.md.

Available Tools

6 tools
analyze_videoA

Analyze a video with a local vision model: scenes, actions, time-window deep reading, contact sheets, etc. Returns JSON: {"text": ..., "mode": ..., "frames": [{"t": ..., "w": ..., "h": ...}], "duration": ...}.

ParametersJSON Schema
NameRequiredDescriptionDefault
toNowindow end
fpsNoSampling frame rate
fromNowindow/burst start, e.g. 1:20
modeNoauto
videoYesVideo path / URL
promptYesQuestion or instruction
max_framesNoFrame cap

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must carry the burden. It discloses the use of a 'local vision model' and provides the return JSON shape (text, mode, frames, duration). However, it does not explain behavior of different modes, potential limitations, or prerequisites. This is adequate but not rich.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded: two sentences. The first states the action and scope; the second gives the return format. No filler. The JSON example is dense but informative, and every element earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 7 parameters and no output schema, the description provides a return JSON structure, which is helpful. It lists capabilities and mentions modes. However, it does not explain the meaning of each mode or how frames are generated, leaving some gaps for a tool with this complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is high (86%), with descriptions for most parameters (e.g., 'Video path / URL', 'Sampling frame rate'). The description adds some context by mentioning 'time-window deep reading' and 'contact sheets' which map to modes, but it does not significantly enhance parameter understanding beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Analyze a video with a local vision model' and enumerates specific capabilities ('scenes, actions, time-window deep reading, contact sheets, etc.'). This distinguishes it from sibling tools like describe_image (image) and transcribe (audio), making it obvious this is for video analysis.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for video analysis but does not explicitly state when to use this tool over alternatives like describe_image or transcribe. There are no exclusions or direct comparisons to siblings. The context is clear for video tasks, but no explicit guidance is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

describe_imageA

Describe or understand one or more images with a local vision model (scene, objects, colors, UI, etc.). Returns JSON: {"text": ..., "mode": "image", "media": [...]}.

ParametersJSON Schema
NameRequiredDescriptionDefault
cropNoOptional WxH+X+Y crop
sizeNosmall=320px thumbnail, full=original size
imagesYesImage paths or URLs, at least one
promptYesQuestion or instruction

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden. It discloses the local vision model usage and return JSON structure (text, mode, media), which adds useful context. However, it does not mention any side effects, limitations, failure modes, or permission requirements. As a read-only description tool, the risk is low, but behavioral details are minimal.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences: the first conveys the core purpose, the second gives the output format. It is concise, front-loaded, and every word earns its place with no fluff or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the absence of annotations and output schema, the description still provides a clear purpose, output format, and parameter context via the schema. It is reasonably complete for a simple, read-only image description tool, but lacks usage guidance and edge-case information, so it is not fully complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 100% parameter coverage with clear descriptions (images, prompt, crop, size). The tool description adds no additional parameter semantics beyond stating the return JSON. Since the schema already documents each parameter thoroughly, the description is not required to compensate, hence a baseline score of 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Describe or understand one or more images with a local vision model' and enumerates content types (scene, objects, colors, UI). This is a specific verb+resource pairing that distinguishes it from siblings like transcribe or analyze_video, which target audio/video.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no explicit guidance on when to use this tool versus alternatives. It never mentions the sibling tools or conditions like 'use for still images; use analyze_video for video'. Usage is only implied by the tool name and image-focused phrasing, which is insufficient for clear tool selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

transcribeA

Verbatim transcription of text in images/documents/screenshots/video subtitles (no summary, judgment, or interpretation). Returns JSON with text and, for videos, frame timestamps.

ParametersJSON Schema
NameRequiredDescriptionDefault
mediaYesImage or video path / URL
max_framesNoFrame cap for video transcription (default 48)

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It discloses that the tool returns JSON with text and, for videos, frame timestamps. It also clarifies the verbatim nature and lack of interpretation. It does not mention potential limitations like language support or file format restrictions, but the core behavior is well described.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence that conveys the tool's essential purpose and return format with no wasted words. It packs key details (verbatim, media types, no interpretation, JSON output with timestamps) efficiently.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has a moderate complexity with 2 parameters and no output schema, so the description must explain return values. It does mention the JSON structure and frame timestamps. However, it could be slightly more complete with details like supported file formats or default behavior for max_frames, but it's largely sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema provides 100% coverage, with both parameters (media and max_frames) already described. The description adds minimal extra meaning beyond schema, only noting that videos get timestamps in the return JSON, which relates to max_frames but is not directly explained. Baseline 3 applies because the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool performs verbatim transcription of text from images, documents, screenshots, and video subtitles. It distinguishes itself from siblings by emphasizing 'no summary, judgment, or interpretation', which sets it apart from describe_image and analyze_video.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context on what the tool does and implicitly indicates when to use it (extracting text verbatim) versus when not (when analysis or interpretation is needed). However, it does not explicitly name alternative tools like 'transcribe_audio' or 'describe_image', so it's clear but lacks explicit exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

transcribe_audioA

Speech-to-text (FunASR SenseVoice, with timestamps; embedded subtitle tracks are preferred when available). Returns JSON: {"text": ..., "source": "asr"|"subtitle"}.

ParametersJSON Schema
NameRequiredDescriptionDefault
langNoauto/zh/en/yue/ja/koauto
mediaYesAudio/video file path / URL
asr_modelNosensevoice

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses that it uses SenseVoice, includes timestamps, prefers embedded subtitles when available, and returns a JSON with 'text' and 'source' fields. This is meaningful behavioral context, though it does not cover failure modes or rate limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the core function, and the second sentence provides the return format. Every word earns its place; there is no wasted text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Considering the complexity (3 params, no output schema, no annotations), the description gives the key output format and subtitle preference, which helps the agent understand the tool's behavior. Minor gaps remain: no clarification of how 'transcribe' differs, and no details on timestamps format or error handling. Overall, it is sufficiently complete for basic invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 67% (lang and media have descriptions; asr_model only has enum values). The description adds minimal parameter context by mentioning SenseVoice (the default model) and timestamps, but it does not explain asr_model options or lang choices beyond what the schema already provides. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool performs speech-to-text (FunASR SenseVoice, with timestamps), which is a specific verb and resource. It also mentions the output format, but it does not distinguish itself from the sibling tool 'transcribe', which could be ambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The tool implies usage for audio/video files needing transcription and notes that embedded subtitle tracks are preferred when available, hinting at when subtitles might be used. However, it does not explicitly state when to use this tool over the 'transcribe' sibling or provide any exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

vision_checkA

Run the environment health check (image reading, transcription, video sampling; can be slow).

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are present, so the description carries full responsibility. It usefully warns that the operation can be slow and lists the types of checks performed. However, it does not disclose output format, side effects, or failure behavior, which would be valuable for a diagnostic tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence: it states the action and then adds scope and a performance cue in a parenthetical. Every word earns its place, with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The scope is clear, but the description does not explain what the health check returns (e.g., a pass/fail status, detailed results, or metrics). Since there is no output schema, this leaves the agent uncertain about how to interpret the result. Still, for a zero-parameter tool, it is reasonably adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the schema trivially covers everything. The description adds no parameter-specific meaning, but the baseline of 4 for no-parameter tools is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb ('run') and resource ('environment health check'), and enumerates the covered capabilities (image reading, transcription, video sampling). This distinguishes it from the sibling tools, which each handle a single capability.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The usage is implied by the description: this is the aggregate health check tool. However, it does not explicitly state when to prefer this over the individual tools like describe_image or transcribe, nor does it mention any alternatives or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

vision_statusA

Show the vision backend status (models, backend type, ffmpeg, speech environment, GPU, watchdog).

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It discloses the tool shows status information and lists key aspects, implying a read-only operation, but it does not explicitly confirm no side effects, required permissions, or possible failure behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence with a concrete list of what the status includes. There is no redundant information or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter status tool with no output schema, the description adequately scopes the tool by naming the exact aspects (models, backend type, ffmpeg, speech environment, GPU, watchdog). It omits return format but this is not critical for a status check.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so there is no parameter semantics to clarify. Baseline 4 applies; the description correctly omits parameter details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb 'Show' and resource 'vision backend status', and enumerates the included components (models, ffmpeg, GPU). This clearly distinguishes it from sibling tools like describe_image or transcribe, which serve different purposes.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies a diagnostic use case for inspecting backend status but does not explicitly state when to use this tool vs alternatives like vision_check. No exclusions are mentioned, so guidance is only implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

TDQS

A3.8/5.0
Disambiguation4/5

Each tool targets a distinct medium or task: image understanding, text extraction, video analysis, audio transcription, and system diagnostics. Minor overlap exists between describe_image and analyze_video for video content, but descriptions clearly separate by media type.

Naming Consistency2/5

Tool names mix conventions: three use verb_noun (describe_image, analyze_video, transcribe_audio), one is a bare verb (transcribe), and two use a vision_ prefix (vision_status, vision_check). This inconsistency makes the set feel less predictable.

Tool Count5/5

Six tools is well-scoped for a multimodal sensing server. Each tool has a clear purpose and none feel redundant or unnecessary.

Completeness4/5

The set covers the core sensing lifecycle: image description, video analysis, audio/speech transcription, and health checks. Minor gaps like general audio analysis or dedicated frame extraction are absent but not essential for the stated purpose.

Maintenance

ActivityMaintained
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Scheme0/local-agent-senses'

If you have feedback or need assistance with the MCP directory API, please join our Discord server