FusionVisionMCP
FusionVisionMCP is a local MCP server that fuses multiple CPU/GPU vision models behind eleven tools for image understanding, analysis, and measurement.
OCR: Transcribe text from images or PDFs with per-span confidence, bounding boxes, and multi-column reading order (EasyOCR).
Captioning: Generate detailed prose descriptions of whole scenes, with optional verbatim text verification and auto-correction of close text misses (Florence-2).
Object detection & grounding: Locate named objects and return bounding boxes, center points, and labels; also caption all salient regions automatically (Florence-2).
Instance counting: Count occurrences of a named object with per-instance boxes and scores, consensus checks, silhouette-based estimates for collapsed overlaps, clip-art mode, and adaptive thresholds (Grounding DINO / Florence-2).
Visual question answering: Ask free-form questions about an image, with consistency checking and automatic routing to measurements for low-confidence answers (Moondream2).
Spatial relations: Measure touch, gaps, containment depth, and shape between named objects using segmentation plus geometry (SAM2 + Grounding DINO).
Aesthetic scoring: Rate images on a 1-10 scale, optionally with style/medium context, technical quality (MUSIQ), artistic judgment (Ollama), and calibrated like-with-like comparison against a reference image (CLIP + LAION).
Composition critique: Combine aesthetic scoring, rule-of-thirds/centeredness framing analysis, and low-score explanations into one critique (Florence-2 + CLIP/LAION + Moondream2).
Batch analysis: Run any of the above operations across many images in one call, with per-image failure isolation.
Raw model access: Run arbitrary Florence-2 task tokens as an escape hatch for tasks not covered by named tools.
Provides aesthetic scoring with technical quality and artistic judgment via Ollama models.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@FusionVisionMCPCount the number of people in this image."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
FusionVisionMCP: Multi-model Vision Server
🚧 Work in progress — see README_DETAILED.md for current limits and measured results before relying on this for anything important.
An MCP server that fuses local, CPU-capable vision models — Florence-2, Moondream2, SAM2, Grounding DINO, EasyOCR, IQA, and CLIP/LAION — behind eleven tools: OCR, captioning, object detection/grounding, instance counting, visual question answering, spatial measurement (touch/gap/containment), and aesthetic scoring (including technical quality and artistic judgment via Ollama).
Quick Navigation
Why FusionVisionMCP? | Tools | Installation | Memory Modes | Architecture
Why FusionVisionMCP?
I kept running computer-vision tasks through Claude's native (frontier-model) vision and burning through a 5-hour usage allocation in about an hour — CV work is token-hungry in a way that's easy to underestimate until you watch the budget disappear. Moving that work to my GPU wasn't an option either: the GPU was already committed to image generation, and running vision inference alongside those testing loops would have contended for the same VRAM.
What was sitting idle was CPU and system RAM. FusionVisionMCP runs Florence-2, Moondream2, SAM2, Grounding DINO
and CLIP entirely on CPU, so routine vision work — reading text, describing an image, counting objects, checking
whether two things touch — no longer has to spend a frontier model's own multimodal tokens. Building it out also
meant finding the gaps against Claude's native vision one by one and closing them; it's now close to that
capability for most everyday CV tasks, and a few tools here (spatial_relations' touch/gap/containment
measurement, count_objects' parallel-query instance counting) aren't things Claude's native vision does at all.
The remaining gaps are structural, not tunable — a caption head that paraphrases text, a small VQA model that can't be trusted for open-ended judgment, a detector with no notion of what separates a spot from a petal, an aesthetic score trained on photographs alone. Neither v0.6.0 nor v0.7.0 closes those (they're not closable locally). v0.6.0 added opt-in parameters to four tools that surface the failure instead of hiding it — a cross-checkable text span next to a caption's guess, a consistency flag on a VQA answer, a second opinion plus a separability flag on an ambiguous count, a medium classification alongside an aesthetic score. v0.7.0 turns each of those flags into an actionable result by combining tools already in the project, no new models: the caption gets an auto-corrected copy, a collapsed count gets an outline-based estimate, a low-confidence VQA answer routes to the measurement that actually answers the question, and the aesthetic score gains a calibrated like-with-like comparison mode. v0.7.1 fixed six further gaps that a second head-to-head pass against Claude's native vision turned up (a partially-occluded object recovered at the detector's existing default threshold, a segmentation occlusion hole no longer defeats containment, a false column split on sparse small text, clip-art counting, a blank-canvas false detection, and a "which is largest/smallest" VQA question routed to a real measurement) — each root-caused against the exact fixture that exposed it, not patched at the symptom. See README_DETAILED.md for what's verified against each case.
Capability | How it's provided |
OCR & document text | EasyOCR (CRAFT detection + CRNN recognition) — printed, stylized, cursive and low-contrast text through one tool, with a per-span confidence score |
Captioning | Florence-2, whole-scene or per-region |
Object detection / grounding | Florence-2's grounding head — boxes and center points for a named object |
Instance counting | Grounding DINO — parallel object queries, not a sequential emission, so overlapping-but-separate instances don't collapse into one |
Visual question answering | Moondream2 — free-form questions about image content |
Spatial measurement | SAM2 masks plus a from-scratch geometry module — touch, gap, containment depth, shape |
Aesthetic scoring | CLIP + a LAION-trained aesthetic head |
Memory footprint | Configurable idle-release timers, per model, chosen at install time |
Hardware | CPU by default; an NVIDIA GPU is opt-in at install time via the |
Related MCP server: vision-mcp
Tools
Tool | Model(s) | Description |
| EasyOCR | Transcribe any text in an image or PDF — printed, stylized, cursive or low-contrast alike — with a per-span confidence score. Multi-column pages are split by geometry and read in reading order. |
| Florence-2 | Describe what an image shows as one detailed prose caption of the whole scene. |
| Florence-2 | Locate a named object, returning bounding boxes, center points and labels. |
| Florence-2 | Caption every salient region of an image at once, without naming objects first. |
| Moondream2 | Ask a free-form question about an image (visual question answering). |
| Grounding DINO | Count how many instances of a named object an image contains. Use this, not |
| Grounding DINO + SAM2 | Measure contact, gaps, containment depth and shape between two named objects. |
| CLIP + LAION | Rate how aesthetically pleasing an image looks on a 1-10 scale. |
| Florence-2 + CLIP/LAION + Moondream2 | Check framing against the rule of thirds; for low-scoring shots, explain what looks off. Also supports |
| (routes to any tool above) | Run one operation across many images in a single call, isolating failures per image. |
| Florence-2 | Run a raw Florence-2 task token for tasks the named tools don't cover. |
Full argument reference, measured accuracy, and known limits for each tool: README_DETAILED.md.
Example
Once connected, an assistant calls these tools on its own when a request needs them — there's nothing to invoke
by hand. For example, asking "How many bolts are in this photo?" with an image attached routes to
count_objects(src=..., object_name="bolt"), which returns:
{
"count": 6,
"bboxes": [[102, 44, 138, 79], "... 5 more"],
"scores": [0.91, "... 5 more"],
"group_boxes_dropped": 0
}What's genuinely new here, not just a wrapper
Most tools above expose one underlying model's own capability directly. Two do not — no single model in the stack answers these on its own:
spatial_relationscombines Florence-2 boxes, SAM2 masks, and a from-scratch geometry module to measure whether two objects actually touch, contain, or overlap — a bounding box alone can't answer that, since boxes overlap the instant one object is merely in front of another.critique_compositioncombines Florence-2 localization, a from-scratch rule-of-thirds check, the CLIP/LAION aesthetic score, and — only for low-scoring images — a Moondream2 explanation, into one composition critique.
Installation
Claude Desktop
Download the latest MCP bundle
fusion-vision-mcp.mcpbfrom ReleasesOpen the downloaded
.mcpbfile, or drag it into Claude Desktop's Settings windowPick a Memory mode (or leave it on Standard and change it later)
Leave Compute device on cpu unless you have an NVIDIA GPU — see CPU or GPU
Also connectable from Cursor, Windsurf, VS Code, or any other MCP-compatible client via manual configuration below.
Manual Installation
Prerequisites
Python 3.12+
Git
8GB+ RAM recommended
Setup
git clone https://github.com/Whoawhen/FusionVisionMCP.git
cd FusionVisionMCP
uv sync --extra cpu --extra ocr-specialist --extra iqaUse --extra cu130 in place of --extra cpu for an NVIDIA GPU. Exactly one of the two is
required — see CPU or GPU for why, and why this needs uv rather than pip.
CPU or GPU
torch is not a plain dependency here. It sits behind two mutually exclusive extras, so the
build is chosen at install time:
Extra | Installs | Use it when |
|
| The default. Works everywhere, and downloads ~2 GB less. |
|
| You have an NVIDIA GPU and a driver supporting CUDA 13. |
Naming neither installs no torch, and the server will not load a model. That is the cost of
making the choice explicit; uv will not guess for you.
Why it is worth the friction: torch from the default index pulls 15 NVIDIA runtime packages
totalling 2.09 GB on Linux — cuDNN alone is 527 MB — into what is otherwise a CPU-only
server. Windows and macOS never saw this, because torch gates those packages on
sys_platform == 'linux', which is why the waste went unnoticed.
This requires uv, not pip. The index routing lives in [tool.uv.sources] and
[tool.uv.index], which pip ignores entirely — a pip install -e ".[cpu]" resolves torch
from PyPI and silently defeats the whole arrangement.
macOS takes its usual CPU/MPS build from PyPI either way: PyTorch's CPU index publishes only
manylinux_2_28_x86_64 and win_amd64 wheels, so the cpu extra deliberately does not apply
there. Use --extra cpu; you will get the right thing.
cu130 rather than cu128, for anyone wondering: the cu128 index stops at torch 2.9.1 and
this project requires torch>=2.13. Verified end to end on an RTX 5060 Laptop (Blackwell,
sm_120) — the wheel ships real sm_120 kernels, and fp16 matmul, cuDNN convolution and
scaled-dot-product attention all run correctly on it, which matters because every model wrapper
selects fp16 on a non-CPU device.
Configuration
{
"mcpServers": {
"fusionvision": {
"command": "uv",
"args": ["run", "fusion-vision-mcp", "--memory-mode", "standard"]
}
}
}Swap standard for aggressive, persistent, or any number of minutes.
System Requirements
RAM: 8GB minimum (16GB+ recommended)
Storage: 16GB free space for model weights (downloaded automatically on first use, then cached locally)
OS: Windows 10+, macOS 12+, or Linux
Memory Modes
Vision models are large. FusionVisionMCP lets you decide how long each one stays resident in memory after its last use, picked at install time — no config file required:
Mode | Models released | Best for |
Aggressive | After 5 minutes idle | Tight memory budgets, short bursts of work |
Standard (default) | After 10 minutes idle | Everyday use — fast during work, tidy afterwards |
Persistent | Never | Maximum speed on a dedicated machine |
Custom | After N minutes you set | Matching your own working rhythm |
Models reload automatically on the next request, so no setting can lose work — only time. Release is per model: a session that only captions never loads the segmentation or aesthetic models at all.
Architecture
Five models, each loaded on-demand and released on its own idle timer:
Florence-2 (Microsoft) — captioning, OCR, object detection/grounding, dense region captioning
Moondream2 (Vikhyat) — visual question answering
SAM2 (Meta) — segmentation masks, the basis for
spatial_relationsGrounding DINO (IDEA-Research) — open-vocabulary detection backing
count_objectsCLIP + LAION aesthetic head — aesthetic quality scoring
Runs on CPU by default. An NVIDIA GPU is opt-in at install time (the cu130 extra) and is then detected and used automatically; --device pins it either way. Because inference happens locally, no image
data leaves the machine, and the CPU/RAM budget it uses is generally idle capacity rather than resources
competing with a GPU-bound workload.
Originally derived from jkawamoto/mcp-florence2 (MIT), which
provides three tools — ocr, caption, process — against Florence-2 alone. This is no longer a fork: the
history was severed and the original mcp_florence2 package replaced outright. Parts of florence2.py are
still that project's code, so its copyright notice travels with them — see LICENSE.
License
This project is licensed under the MIT License — see the LICENSE file for details.
Available Tools
11 toolsbatch_analyze_imagesA
Run one operation across many images in a single call.
The batch form of caption, ocr, detect_objects, count_objects,
dense_region_caption and query_image -- pick which with operation. Use
it when the same question applies to a whole set of images, since it costs
one round trip instead of one per image and loads each model once for the
whole run.
Failures are isolated per image: a missing file or an unreachable URL is reported as its own {"src", "success": false, "error"} entry and the rest of the batch still runs. Results come back in the order given.
For a single image, call the named tool directly -- its arguments are
checked up front rather than depending on operation.
| Name | Required | Description | Default |
|---|---|---|---|
| srcs | Yes | File paths or URLs of the images to process. | |
| question | No | Required when operation is 'query'. | |
| operation | Yes | One of: 'caption', 'ocr', 'detect', 'count', 'dense_caption', 'query'. Use 'query' (with `question`) rather than 'ocr' for watermarks, logos, signage, or stylized/cursive text -- 'ocr' misreads that kind of text confidently. Use 'count' (with `object_name`) rather than 'detect' for 'how many' -- 'detect' returns regions, which are not a tally. | |
| object_name | No | Required when operation is 'detect' or 'count'. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden and handles it well. It discloses per-image failure isolation, the error entry shape, that remaining images still run, and that results preserve input order. These are non-obvious behaviors that materially affect how an agent uses the tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core action, then efficiently covers use cases, failure semantics, ordering, and the single-image alternative. Every sentence contributes distinct information without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the rich output schema and detailed parameter schema, the description covers everything an agent needs: what the tool does, when to use it, how failures behave, result ordering, and when to prefer a sibling. No critical operational context is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the schema already explains operation choices, the question requirement, and object_name requirements. The description mainly restates that operation selects the underlying behavior rather than adding new per-parameter meaning, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Run one operation across many images in a single call.' It explicitly positions the tool as the batch form of six named sibling tools and makes the selection mechanism ('pick which with operation') immediately clear.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit when-to-use guidance: when the same question applies to a whole set of images, to save round trips and model loading. It also gives an explicit when-not-to-use rule: for a single image, call the named tool directly, because its arguments are checked up front.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
captionA
Describe what an image shows, as one detailed prose caption.
The default choice for "what is this a picture of". Returns a single paragraph covering the scene as a whole, with no coordinates.
Reach for a different tool when the question is narrower: query_image
to ask something specific about the image, dense_region_caption to get
a separate caption and box for each thing in it, and ocr to transcribe
text rather than describe it. Returns one caption per page for a PDF.
Do not trust any text this quotes back. A caption that mentions a name,
brand or label is describing it, not transcribing it, and Florence-2
misspells text here that it reads correctly under ocr -- it rendered a
logo reading "FusionVisionMCP" as "FusionVisionMP" mid-caption while ocr
read the same image exactly. When a specific piece of text matters,
confirm it with ocr, which handles stylized and cursive text as well as
printed text, rather than quoting this.
Set verify_text=true to have this confirmation done for you: the tool
also runs the OCR-with-region head and returns each verbatim text span
alongside its box, so a name the caption quoted can be checked against
what was actually read without a second call. It then goes further and
corrects the close misses: any caption token that is similar to (but
not identical to) a verbatim OCR span is substituted with the verbatim
text in a caption_corrected copy, and every such change is listed in
corrections (with the quoted-in-caption token, the verbatim OCR text,
the box and the similarity) so the substitution is auditable. The
return shape becomes one dict per page
({caption, text_regions, corrections, caption_corrected}) when this
is set. The correction is best-effort and only fires for high-similarity
same-word matches; the raw corrections list is always present so a
caller can audit or reject any change.
Set auto_verify_text=true to automatically detect when a caption quotes
embedded text or signage and run specialist OCR (EasyOCR) on
upscaled crops of the text regions, providing consensus metrics (text_consensus)
and warnings (caption_text_warning).
| Name | Required | Description | Default |
|---|---|---|---|
| src | Yes | Local file path or http(s) URL of the image to process. PDFs are also accepted and are rendered one image per page, so tools that return a list return one entry per page. | |
| verify_text | No | When true, also run Florence-2's OCR-with-region head and return, alongside the caption, the verbatim text spans it read (`text_regions`: {text, box}[]), a `corrections` list of the close misses it fixed, and a `caption_corrected` copy with each verbatim OCR span substituted in. The caption head paraphrases text and misspells names/brands (it rendered this project's own 'FusionVisionMCP' logo as 'FusionVisionMP'); the OCR head transcribes verbatim, so any text the caption quotes can be confirmed against `text_regions` and the corrected caption used directly. Default false keeps the original list[str] return shape; true returns one dict per page ({caption, text_regions, corrections, caption_corrected}). | |
| auto_verify_text | No | When true, automatically detect embedded text, quotation, or signage tokens in the caption or image, running specialist OCR (EasyOCR) with small-text upscale crops to cross-check text verbatim (Spec 13 & 14). Returns `caption_text_warning`, `text_consensus`, and `caption_corrected` with consensus text fused across models. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full disclosure burden, and it excels: it honestly warns 'Do not trust any text this quotes back,' documents the Florence-2 paraphrasing/misspelling failure mode with a concrete example ('FusionVisionMCP' rendered as 'FusionVisionMP' while ocr read it exactly), and discloses that the verify_text correction is 'best-effort and only fires for high-similarity same-word matches' and therefore auditable via the corrections list. It also discloses return-shape changes per flag and PDF pagination. This exceeds what annotations would typically provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Although long, the description is front-loaded — purpose, scope, and return shape in the first sentences — and each subsequent paragraph earns its place: sibling routing, a concrete failure-mode example, and flag-by-flag behavior. The paragraph breaks map cleanly onto distinct concerns (what → when-not → risk → mitigations), and there is no intra-description redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 3 parameters, two behavior-changing booleans, an output schema, and zero annotations, the description covers everything an agent needs: purpose, sibling routing, failure modes, per-flag return shapes ({caption, text_regions, corrections, caption_corrected} per page), PDF handling, and correction auditability. The only minor omission — behavior when both verify_text and auto_verify_text are set simultaneously — is an edge case that does not materially impair correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with detailed parameter descriptions, so the baseline is 3; the description adds genuine value beyond the schema by explaining when to enable each flag ('When a specific piece of text matters'), the best-effort/auditability nuance of verify_text corrections, and the consensus-metric behavior of auto_verify_text. It slightly exceeds the baseline but the schema already does heavy lifting for src, verify_text, and auto_verify_text.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: 'Describe what an image shows, as one detailed prose caption,' and immediately scopes it as 'the default choice for what is this a picture of,' noting it returns a single paragraph with no coordinates. It explicitly distinguishes itself from query_image (specific questions), dense_region_caption (caption+box per thing), and ocr (transcription), so an agent can disambiguate without opening sibling schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit when-to-use and when-not-to-use guidance, naming each alternative tool and the condition that selects it: 'Reach for a different tool when the question is narrower: query_image to ask something specific... dense_region_caption to get a separate caption and box for each thing... ocr to transcribe text rather than describe it.' It also gives conditional guidance for the boolean flags — set verify_text 'when a specific piece of text matters' — and states the PDF page-per-caption behavior. Nothing is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
count_objectsA
Count how many instances of a named object an image contains.
Use this, not detect_objects, whenever the question is "how many".
detect_objects returns however many regions Florence-2's grounding head
emits, which is not a tally: it collapses several repeated, undifferentiated
parts into one box (every petal of a flower, both halves of a fused blade)
and conversely splits one object into overlapping sub-part boxes.
Returns count, plus bboxes/points/labels/scores for the instances
found, index-aligned and in the same pixel-space convention detect_objects
uses. scores are per-detection confidences, so a count resting on marginal
detections is visible rather than implied. group_boxes_dropped counts
detections that enclosed the whole arrangement rather than one instance.
Accuracy is measured, not assumed: against a fixture suite whose synthetic counts are exact by construction, this is right on 9 of 10 positive cases with a mean error of 0.1, while holding all 8 single-object controls. Separated, touching, dense, two-instance and awkwardly-named cases are reliable.
Two measured limits, worth checking before trusting a count:
Heavy overlap undercounts. Eight identical shapes in a ring count as 8 when separated or touching, but 6 once they overlap by roughly two-thirds of their width. A count below what you expect means "could not separate them", not a real tally -- and it will undercount rather than overcount.
Some evidence is simply not in the picture. On a photo of a paper flower whose
petals overlap, this returns 1 at every resolution from 128px to 768px, as does
every other approach tried -- other detectors, outline geometry, tiled crops,
and interior-colour analysis alike. That flower's outline is 98% convex and its
interior contrast is near zero, so nothing measurable distinguishes the petals.
Ask query_image for a count in that situation and treat it as an estimate.
When only one region is found, a silhouette block is added measuring how
many repeated lobes that region's outline contains. count and
silhouette.lobes come from different methods and neither overrides the
other: count: 1 beside silhouette.lobes: 8 means the detector could not
separate the instances while the outline shows eight cores. agreement: true
means a second, independent estimator matched it; shattered or clipped
mean the number should not be used at all.
Set consensus=true (the default) for two extra fields that surface
structural ambiguity instead of hiding it:
consensus:{detector_count, region_label_count, agree}-- a second count from Florence-2's dense region captioner (a different head), which tally how many region labels contain the object name. Agreement is real evidence; disagreement is a warning.separable:"yes"if the count is a real tally (the detector separated instances, or a single region the silhouette confirms is one lobe),"no"if the detector collapsed while the outline shows several lobes (the overlapping-petal case: nothing local can count it honestly),"unknown"when there's no silhouette check to confirm either way. Readcountwith this in mind: acount: 1withseparable: "no"is a collapse, not a tally.
When separable is "no" but the outline still carries the lobe pattern
(silhouette.by_radial > 1), an estimates block is added reporting that
outline count as an actionable number -- it's a measurement (angular
notches in the silhouette), not a judgment, so it stays within "measure,
don't judge". count is never overwritten; estimates.outline is the
number the outline supports, with a basis string saying so. Set
vqa_estimate=true to also ask Moondream2 "how many ?" and attach
Additionally, every result reports semantic_ambiguity (bool) and
count_semantics ("measured_tally", "minimum_visible_instances", or
"unverified_tally"). When semantic_ambiguity is true, an ambiguity
payload is attached detailing why the count represents a collapsed or
ambiguous structure rather than a confirmed discrete tally.
A maximally vague object_name ("object", "thing", "item", ...) is guarded
the same way detect_objects is: Grounding DINO has the same fallback-to-
the-whole-frame failure mode on a name with nothing distinctive to point to,
so a near-full-frame box is dropped rather than counted as a real instance.
clip_art=true gets the equivalent guard through Florence-2's own
exclude_full_frame. Only the exact generic vocabulary triggers this; a
specific noun that genuinely fills the frame is counted unfiltered.
| Name | Required | Description | Default |
|---|---|---|---|
| src | Yes | Local file path or http(s) URL of the image to process. PDFs are also accepted and are rendered one image per page, so tools that return a list return one entry per page. | |
| clip_art | No | When true, count with Florence-2's grounding head instead of Grounding DINO. Grounding DINO's training distribution is real photographs; on flat vector art and iconography it can over-detect (measured on a mixed clip-art scene: 6 instead of 2 trees, and a spurious 4th house past the true 3). Florence-2 got both exactly right on the same scene. Try it when the image is clip art / flat vector style and the default count looks wrong; `scores` are unavailable on this path (Florence-2's grounding head carries no per-box confidence) and `group_boxes_dropped` is not applicable. | |
| consensus | No | When true (default), also tally how many `dense_region_caption` labels match the object name as an independent second opinion, and report a `separable` flag. The dense region captioner is a different Florence-2 head than Grounding DINO, so agreement between the two is real evidence and disagreement is a visible warning. Cheap -- uses the already-loaded Florence-2, no new model. | |
| threshold | No | Box confidence floor passed to Grounding DINO (default 0.15, the value measured against benchmarks/ to hold every negative control -- see CLAUDE.md). Raise it for a visually cluttered scene with distractor shapes near the target: on one measured case (a target star among 18 muted-color distractors) the default threshold counted every distractor, while 0.5 separated the true target (score 0.90) from all of them (scores 0.19-0.32) for an exact count. This is a manual lever, not automatic -- raising it blindly on a normal scene can just as easily drop real instances, so use it once you suspect clutter, not by default. | |
| object_name | Yes | Name of the object to locate, e.g. 'person', 'car', 'face'. A short noun phrase works too ('the red mug'). More specific names ground more reliably than broad ones. | |
| vqa_estimate | No | When true and the detector has collapsed overlapping instances (separable is 'no'), also ask Moondream2 'how many <name>?' and attach its answer as estimates.vqa -- a judgment estimate, clearly marked, not a tally. Off by default so a session that never asks keeps Moondream unloaded. | |
| return_annotated | No | If true, saves an annotated image with counting boxes and returns the file path. | |
| verify_silhouette | No | When the detector finds only one region, segment it and measure how many repeated lobes its outline has. Costs one SAM2 load on the first such call. | |
| adaptive_threshold | No | When true, evaluates the raw detection distribution using kernel density estimation to automatically raise the threshold just above noise clusters if clear modes exist. Suppresses hundreds of false-positive sub-boxes in texture-heavy scenes while preserving real detections. Overrides manual threshold if the adaptive threshold is higher. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure, and it delivers extensively. It discloses measured failure modes (heavy overlap undercounts: 8 shapes count as 6 when overlapping; the paper flower returns 1 at every resolution), explains the silhouette block and that count and silhouette.lobes come from different methods and neither overrides the other, describes the consensus and separable fields, and explains the generic-object-name guard. It also discloses that scores are unavailable on the clip_art path and that group_boxes_dropped is not applicable. This is far beyond what annotations would have provided.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every section earns its place: it front-loads the core purpose and sibling differentiation, then layers behavioral caveats, output semantics, and parameter-specific guidance. The structure uses clear section breaks and bolded terms. It is not concise in word count, but it is dense with actionable information and well-organized. A 4 is appropriate because the length is justified by the tool's complexity, though it could be tightened in places (e.g., the paper-flower anecdote is illustrative but slightly verbose).
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (9 parameters, 2 required, no annotations, rich output schema), the description is remarkably complete. It covers when to use the tool, when not to, measured accuracy, failure modes, output field semantics, parameter effects, and even the generic-object-name guard. The output schema exists, so return values need not be fully re-explained, but the description goes beyond that to explain the meaning and reliability of the returned fields. Nothing an agent needs to call this tool correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3, but the description adds substantial meaning beyond the schema. It explains the semantic distinction between count and silhouette.lobes, the meaning of separable values, the consensus fields, the estimates block, and the semantic_ambiguity/count_semantics fields. It also explains the generic-object-name guard and the clip_art path's lack of scores. The description enriches the output semantics far beyond what the input schema alone provides, and it clarifies how parameters like consensus, vqa_estimate, and verify_silhouette affect the output structure.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Count how many instances of a named object an image contains.' It immediately distinguishes itself from the sibling detect_objects by explaining that detect_objects returns regions, not a tally, and that count_objects is the tool to use when the question is 'how many'. This is a textbook example of purpose clarity with sibling differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use this, not detect_objects, whenever the question is "how many"' and explains why detect_objects is unsuitable (it collapses repeated parts and splits objects). It also provides detailed when-to-use guidance for parameters like clip_art (flat vector art), threshold (cluttered scenes), consensus (independent second opinion), and vqa_estimate (when detector collapsed). It even gives measured examples of when the tool fails (heavy overlap, low-contrast petals) and advises checking query_image in those cases. This is comprehensive usage guidance with explicit alternatives and exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
critique_compositionA
Critique an image's composition: framing, and, for low-scoring images, why it looks off.
Combines score_aesthetics (numeric quality), a rule-of-thirds/centeredness
check on the main subject's bounding box, and — only when the aesthetic
score is below low_score_threshold — a Moondream2 VQA explanation of what
specifically looks unbalanced. Use it over score_aesthetics alone when you
need to know why a shot is weak and where its subject sits in the frame,
rather than just how it scores.
Returns image_size, subject_box, aesthetics, and framing (with
thirds_offset near 0 meaning the subject sits on a rule-of-thirds power
point, and center_offset near 0 meaning it is dead-center instead).
Pass target_subject whenever you know what the subject is — from your own
context or a prior caption call. Auto-detection picks the largest,
most central region and degrades on busy scenes that fill the frame, where
no single region is the subject. If nothing can be located, returns a
soft-failure shape (image size, aesthetic score, and a "note") rather than
raising. Only the first page of a PDF is assessed.
Set style_context=true to also get the image's medium (style plus
style_distribution), so the aesthetic score is read in the context of its
medium — the documented photography bias means a non-photographic medium
should not be judged by the raw score.
Set compare_with to a reference image to switch to relative mode: both
images are critiqued and the result becomes
{image, reference, delta, preferred} (tie when |delta| < 0.05), plus a
cross_medium_warning when style_context=true and the two media differ.
Same like-with-like framing as score_aesthetics' compare_with; the
absolute score is not recalibrated, the relative delta is.
| Name | Required | Description | Default |
|---|---|---|---|
| src | Yes | Local file path or http(s) URL of the image to process. PDFs are also accepted and are rendered one image per page, so tools that return a list return one entry per page. | |
| compare_with | No | Path or URL of a reference image to compare against. When set, both the image and the reference are critiqued and the result becomes {image, reference, delta, preferred} (plus cross_medium_warning when style_context=true and the media differ). Same like-with-like framing as score_aesthetics' compare_with. Omit (default) for the single-image critique. | |
| style_context | No | When true, also classify the image's medium/genre (photograph, oil painting, ...) using the already-loaded CLIP backbone and add it to the result as `style` and `style_distribution`. The aesthetic head was trained on photographs, so a non-photographic medium is the context the score is read in. Default false omits the classification. | |
| target_subject | No | Name of the main subject, e.g. 'the dog'. Omit to auto-detect it. | |
| low_score_threshold | No | Below this aesthetic score, ask Moondream2 to explain why. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and is highly disclosive. It states that the VQA explanation only triggers below `low_score_threshold`, that auto-detection degrades on busy scenes, that a soft-failure shape is returned rather than raising, that only the first PDF page is assessed, and that relative mode does not recalibrate the absolute score.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but densely structured, with each paragraph covering one distinct concern: core behavior, return shape, subject detection and PDF handling, style context, and relative mode. The core action and primary alternative are front-loaded, and there is no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 5-parameter tool with multiple modes, the description covers inputs, return semantics, edge cases, and failure behavior. It clarifies ambiguous return fields like `thirds_offset`, `center_offset`, `delta`, `preferred`, and `cross_medium_warning`, so nothing an agent needs to invoke it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds substantial meaning beyond the schema. It explains that `target_subject` can come from a prior `caption` call, that `low_score_threshold` gates the Moondream2 explanation, that `style_context` adds `style`/`style_distribution`, and that `compare_with` switches the result shape and defines the tie threshold.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource ('Critique an image's composition') and details the constituent checks: framing, rule-of-thirds/centeredness on the subject bounding box, and a conditional Moondream2 explanation. It explicitly differentiates itself from the sibling `score_aesthetics`, so an agent can tell them apart without inspecting schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says when to use this tool over `score_aesthetics` ('when you need to know *why* a shot is weak and where its subject sits in the frame, rather than just how it scores'). It also gives parameter-level guidance: pass `target_subject` if known, set `style_context` for medium-aware scoring, and set `compare_with` for relative mode.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
dense_region_captionA
Caption every salient region of an image at once, with bounding boxes.
Use this to inventory an image without knowing in advance what is in it --
it returns bboxes and labels for each region it finds, discovering the
objects itself. That is the difference from detect_objects, which needs
you to name the object you are looking for, and from caption, which
describes the whole scene in prose with no coordinates.
| Name | Required | Description | Default |
|---|---|---|---|
| src | Yes | Local file path or http(s) URL of the image to process. PDFs are also accepted and are rendered one image per page, so tools that return a list return one entry per page. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden itself. It discloses the key behavior: the tool discovers salient regions autonomously and returns bboxes and labels for each. It does not discuss edge cases like images with no salient regions, but the core operational behavior is clearly stated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is tight and front-loaded: one sentence states the purpose, then a short paragraph explains when to use it and how it differs from alternatives. Every sentence contributes useful selection and invocation information without repeating schema contents.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given a single well-documented parameter, an output schema, and a clear comparison to relevant siblings, the description provides everything needed to select and invoke the tool correctly. The mention of return contents (bboxes and labels) is useful despite the output schema existing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There is only one parameter, src, and the schema already documents it thoroughly, including local paths, http(s) URLs, PDF rendering, and per-page list behavior. The description adds no extra parameter-level guidance, so the baseline of 3 applies because the schema coverage is complete.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Caption every salient region of an image at once, with bounding boxes.' It clearly differentiates this tool from siblings by explaining that detect_objects requires a named object and caption gives whole-scene prose without coordinates.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit use case: inventory an image without knowing in advance what is in it. It also names the alternatives and exactly what makes them different, so an agent can decide when to use this tool instead of detect_objects or caption.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
detect_objectsA
Locate a named object in an image, as bounding boxes and center points.
Returns bboxes ([x1, y1, x2, y2] each), points (the center of each box)
and labels, all index-aligned -- so use this whether you want regions or
coordinates; the centers come free with the boxes.
If return_annotated is true, the return value becomes a dictionary containing
the regular results plus an annotated_image_path key.
The count of results is NOT a reliable count of objects on an ambiguous
class name: Florence-2 can return several overlapping results for one
physical object (a whole-animal box plus sub-part boxes all labelled
'wing'), or a single result spanning two touching instances (two fused
blades labelled once as 'sword blade'). Prefer a more specific
object_name, and treat results as candidates to inspect, not a tally.
Use count_objects when you actually need "how many" -- it runs a
detection head that emits one region per instance, which this head does
not. Neither can separate heavily overlapping instances, so a count of 1
from either means "could not separate", not "there is one".
Boxes cannot answer whether two objects actually touch or whether one is
inside another -- they overlap the moment one object is merely in front
of another. Use spatial_relations for that.
A maximally vague object_name ("object", "thing", "item", ...) on a
scene with nothing distinctive to point to gets a near-full-frame box
filtered out rather than returned as a match -- Florence-2's grounding
head has no explicit "nothing here" output for that case and falls back
to the whole image otherwise. This only applies to the exact generic
vocabulary itself: a specific noun that genuinely fills the frame (a
close-up of wood, asked for "wood") is returned unfiltered.
| Name | Required | Description | Default |
|---|---|---|---|
| src | Yes | Local file path or http(s) URL of the image to process. PDFs are also accepted and are rendered one image per page, so tools that return a list return one entry per page. | |
| object_name | Yes | Name of the object to locate, e.g. 'person', 'car', 'face'. A short noun phrase works too ('the red mug'). More specific names ground more reliably than broad ones. | |
| return_annotated | No | If true, returns a local temp file path to an annotated image with drawn bounding boxes. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden, and it delivers: it discloses that the count is unreliable under ambiguous names (Florence-2 overlapping sub-part boxes, fused instances), that boxes cannot answer touch/inside questions, that a vague object_name yields a near-full-frame box that gets filtered out, and that neither tool separates heavily overlapping instances. This is rich, honest behavioral disclosure with zero annotation support.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long — four substantial paragraphs — but it is not padded. It is front-loaded with the core purpose and return format, then each subsequent paragraph addresses a distinct footgun (count unreliability, spatial limitations, generic vocabulary). The length is proportionate to the tool's genuine behavioral complexity, and every sentence earns its place. Slightly long for a first read, hence not a 5.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
This is a complex tool with real failure modes, and an output schema exists to document return values. The description covers the return format, the return_annotated variant, count unreliability, sibling differentiation, spatial limitations, and the vague-name edge case. Nothing an agent needs to call it correctly is missing, given the rich schema and output schema that accompany it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the schema already documents src, object_name, and return_annotated well, so the baseline is 3. The description adds genuine interpretational value: it explains how object_name specificity affects grounding reliability and the return_annotated key, and it teaches the agent how to interpret the returned bboxes/points/labels (index-aligned, centers come free). This bumps it above baseline without fully re-documenting parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb-resource pair: 'Locate a named object in an image, as bounding boxes and center points.' It states exactly what is produced (bboxes, points, labels) and is clearly differentiated from siblings: it tells the agent to use count_objects for tallies and spatial_relations for touch/containment questions, so detect_objects cannot be confused with either.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit routing guidance: 'Use count_objects when you actually need "how many"' and 'Use spatial_relations for that' for overlap/containment questions. It also advises preferring a more specific object_name and warns against generic vocabulary. The when-to-use and when-not-to-use guidance is unambiguous and names the exact alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ocrA
Process an image file or URL using OCR to extract text.
Uses EasyOCR for robust scene-text extraction. Excels at photos, signage, watermarks, logos, and printed text.
This is the right tool for any text in an image. EasyOCR handles both
of the cases that used to need separate routing -- dense printed text and
stylized, cursive, logo or low-contrast text -- so there is no longer a
reason to send the second kind to query_image, which paraphrases rather
than transcribing.
A page laid out in side-by-side columns (a form, meeting notes, a resume) is detected automatically: each column is OCR'd separately and joined in reading order, so fields from different columns don't get interleaved.
An image with no text returns an empty result, and that is the correct answer rather than a failure -- do not re-ask a captioning or VQA model to produce text for it, which is how invented text gets into a payload.
Set detail=true to get confidence scores and bounding boxes. This is
highly recommended for checking generative image artifacts: if an image
contains gibberish text, the confidence scores will drop significantly.
| Name | Required | Description | Default |
|---|---|---|---|
| src | Yes | Local file path or http(s) URL of the image to process. PDFs are also accepted and are rendered one image per page, so tools that return a list return one entry per page. | |
| detail | No | When true, return verbatim text *and* the bounding box and confidence score for each text span (via EasyOCR). Each page becomes a dict with `text_regions` ({text, confidence, box}[], box=[x1,y1,x2,y2] in page coordinates) and the joined `text`. Use this when you need to know *where* a phrase is, or to evaluate OCR confidence scores for anomaly detection. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully carries the behavioral burden. It discloses that it uses EasyOCR, handles column layouts automatically, returns an empty result for no-text images rather than failing, and that detail=true exposes confidence scores and bounding boxes useful for detecting gibberish text. This is rich, non-obvious behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-organized and front-loaded with the core purpose. Each paragraph adds a distinct point: capabilities, routing, column handling, empty-result semantics, and detail guidance. It could be slightly tightened, but nothing is extraneous.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an OCR tool with a rich schema and output schema, the description covers selection, alternatives, edge cases, and advanced usage. It explains what happens with no text, how columns are handled, and when to use detail. No critical information needed to invoke the tool correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already describes both parameters at 100% coverage, so the baseline is 3. The description adds value beyond the schema by recommending detail=true for generative-artifact checking and explaining that low confidence indicates invented or gibberish text, which the schema does not convey.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence states a specific verb and resource: 'Process an image file or URL using OCR to extract text.' It further distinguishes itself from the sibling tool query_image by explaining that query_image paraphrases rather than transcribes, making the tool's unique role explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says this is the right tool for 'any text in an image' and explains why query_image is not appropriate. It also gives when-to-use guidance for detail=true and warns against re-asking a captioning model when no text is found, covering both use and non-use cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
processA
Run a raw Florence-2 task token against an image (escape hatch).
prompt must be a Florence-2 task token, not an instruction: '',
'', '' and the like. Passing plain
English ("describe this image") does not fail — it returns confident
nonsense, because the model has no such task and decodes the words as
one anyway.
Only for task tokens the named tools do not already cover. Prefer
caption, ocr, detect_objects and dense_region_caption: they wrap
the common tokens, parse the structured output into usable fields, and
document where each one misleads. This returns raw text either way.
| Name | Required | Description | Default |
|---|---|---|---|
| src | Yes | Local file path or http(s) URL of the image to process. PDFs are also accepted and are rendered one image per page, so tools that return a list return one entry per page. | |
| prompt | Yes | A Florence-2 task token, e.g. '<OD>', '<CAPTION>', '<REGION_PROPOSAL>'. Not a natural-language instruction -- plain English here produces garbage, not an answer. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure and meets it: it warns that plain English 'does not fail' but returns confident nonsense, and that the output is raw text either way. This exposes the tool's raw-token-only nature and its key failure mode, which is exactly the context an agent needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short paragraphs, each earning its place: a scope sentence, a critical failure-mode warning, and routing guidance. The caveat about confident nonsense is essential rather than filler, and the most important constraints are front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter escape-hatch tool, the description tells the agent what it does, what it does not do, when not to use it, what failure looks like, and what the return is. The schema covers parameter formats and PDF behavior, so nothing essential is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already describes both src and prompt at 100% coverage, so the baseline is 3. The description reinforces the token-vs-instruction distinction and the garbage-output risk, but it does not add a significant new parameter-level fact beyond what the schema already says. No compensation is needed here, so 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific action ('Run a raw Florence-2 task token against an image'), identifies the tool as an escape hatch, and distinguishes it from the sibling wrappers by stating it applies only to tokens those tools do not already cover. This is far more informative than the generic tool name 'process'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Only for task tokens the named tools do not already cover' and instructs the agent to prefer caption, ocr, detect_objects, and dense_region_caption, explaining why those are better (they parse structured output and document misleading cases). The routing decision is unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
query_imageA
Ask a free-form question about an image (visual question answering).
This is the right tool for reading photo watermarks, logos, signage,
or any cursive/stylized/low-contrast text — ask e.g. "What does the
text/watermark say, exactly?". The ocr tool misreads that kind of
text confidently; prefer this one for it instead.
Moondream2 is a small model and is documented to answer open-ended judgment
questions ("describe anything wrong in this image") with a flat "None" on
images that all had real visible defects, and to give the same yes/no answer
across genuinely different images -- a default response, not a real
observation. Set check_consistency=true to make that visible: the tool also
asks a rephrased control question and returns, per image,
{answer, control_answer, consistent, confidence}. confidence is "low"
in either of two failure modes: the two answers agree on a short
default-looking token ("None", "Yes", "Nothing", ...), which is the
signature of a flat default rather than a genuine observation; or they
substantively contradict each other, which makes either answer weaker
evidence than it looks alone. "normal" requires substantive answers that
agree. A low result on a judgment question means you should not trust the
answer without independent confirmation. When the answer is
low-confidence, the tool also tries to route to the measurement that
actually answers the question: it classifies the question's wording and,
if a measurable category applies and the object names parse from the
wording, runs that tool's measurement (spatial_relations for a
contact/containment question, count_objects for "how many", ocr for a
text-reading question, detect_objects for "which is largest/smallest")
and attaches it as cross_check. The "largest/smallest" case detects every
instance of the named object and picks the extremum by bounding-box area,
returning its box -- so "which circle is biggest" resolves to coordinates,
not a repeated guess. The cross-check is omitted when no measurement
applies or the names can't be parsed -- it never guesses.
| Name | Required | Description | Default |
|---|---|---|---|
| src | Yes | Local file path or http(s) URL of the image to process. PDFs are also accepted and are rendered one image per page, so tools that return a list return one entry per page. | |
| question | Yes | A free-form question to ask about the image. | |
| check_consistency | No | When true, also ask a rephrased control question and report whether the two answers agree. `confidence` is 'low' in two cases: the answers agree on a short default-looking token ('None', 'Yes', 'No', 'Nothing', ...), or they substantively contradict each other. Moondream2 is a small VLM that answers open-ended judgment questions ('describe anything wrong') with a flat 'None' on images that all had real visible defects -- this layer makes that default-answer behavior visible instead of presenting it as reliable. On a low-confidence answer it also routes to the measurement that actually answers the question when one applies (spatial_relations for a contact/containment question, count_objects for 'how many', ocr for a text-reading question) and attaches it as `cross_check`. Default false keeps the original list[str] return; true returns one {answer, control_answer, consistent, confidence, cross_check?} dict per image. | |
| structured_analysis | No | When true, bypasses the simple VQA return and returns structured Observation and Anomaly payloads per image. Corroborates VLM claims against physical Grounding DINO measurements to proactively detect anatomical and structural AI generation artifacts (e.g. extra/missing body parts). |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure, and it does so thoroughly. It discloses the model's documented failure modes (flat 'None' answers, same yes/no across different images), explains the confidence semantics, and describes the cross-check routing behavior including when it is omitted ('it never guesses'). This is exactly the kind of behavioral context an agent needs beyond the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every sentence earns its place: it covers purpose, sibling differentiation, failure modes, confidence semantics, and cross-check routing. It is front-loaded with the core purpose and the key alternative. It loses one point for density — the cross-check explanation is somewhat run-on and could be tightened — but it is not padded or redundant.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (4 params, no annotations, rich behavioral nuances, output schema present), the description is complete. It explains the return types, the failure modes, the confidence field, the cross-check routing, and the structured_analysis bypass. An agent has everything needed to decide when to call it and how to interpret results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds meaningful value beyond the schema by explaining the return-shape change when check_consistency=true (list[str] vs dict per image) and by detailing the cross-check routing logic that depends on the question wording. It also clarifies the src parameter's PDF behavior indirectly through the schema, but the description's main added value is on check_consistency semantics, which are already partially in the schema. The description does not add much beyond the schema for src or question, but the check_consistency elaboration justifies a 4.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Ask a free-form question about an image (visual question answering)') and immediately distinguishes itself from the ocr sibling by naming the exact use case (watermarks, logos, signage, stylized text). It clearly identifies what the tool does and what it is not for, so an agent can select it correctly without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says when to prefer this tool over ocr ('The ocr tool misreads that kind of text confidently; prefer this one for it instead'), and it gives concrete example questions. It also explains when to set check_consistency=true and when the cross-check routing applies, providing clear decision guidance for the agent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
score_aestheticsA
Rate how aesthetically pleasing an image looks, independent of its content.
Uses a CLIP-based predictor trained on human aesthetic ratings (the LAION "improved aesthetic predictor"). Reflects visual qualities like lighting, composition and clarity — not whether the subject matter is correct or matches a prompt. A technically accurate but flatly-lit, cluttered photo can score low; a blurry but beautifully lit one can score comparatively higher.
In v0.8.0, the returned payload is enriched. The legacy score is duplicated
as photographic_aesthetic. If a Reasoner is configured at the server level,
an artistic_judgment is generated. If the server is installed with the [iqa]
extra, a technical_quality score (0-100) is included via the MUSIQ ONNX model.
Returns one {"score": float, "rating": str} object per page/image. score
is roughly on a 1-10 scale; rating buckets it coarsely for quick triage —
read score for anything comparative.
Its training set was photographic, so it rates photographs, not fine art: celebrated paintings and illustrations score middling (Hokusai's "The Great Wave" comes back around 5.8) without that meaning anything is wrong with them. Use it to compare like with like — several shots of the same subject, or successive edits of one image — and do not read a single absolute score as a verdict on quality.
Set style_context=true to also get the medium the score is being read in.
The CLIP backbone (already loaded for scoring) classifies the image as a
photograph, oil painting, digital illustration, etc., and that style plus
its style_distribution are added to each result. This is the local-model
answer to the photography bias: it doesn't make the head understand fine art,
but it tells you that the score is for a non-photographic medium, so you
read it with the documented caveat instead of as an absolute verdict.
Set compare_with to a reference image to switch to relative mode: both
images are scored and the result becomes one entry per image page carrying
{image, reference, delta, preferred}, with preferred being "image" /
"reference" / "tie" (tie when |delta| < 0.05). This is the predictor's
calibrated use -- like-with-like comparison -- so it sidesteps the absolute
photography bias that makes a single score misleading across media. With
style_context=true, both media are classified and a cross_medium_warning
is added when they differ (cross-medium comparison is out of calibrated
scope). The absolute score is not recalibrated; the relative delta is the
actionable output.
| Name | Required | Description | Default |
|---|---|---|---|
| src | Yes | Local file path or http(s) URL of the image to process. PDFs are also accepted and are rendered one image per page, so tools that return a list return one entry per page. | |
| compare_with | No | Path or URL of a reference image to compare against. The predictor's documented valid use is like-with-like comparison (edits of one image, or several shots of one subject), so this routes you there instead of a single bias-affected absolute number: both images are scored and the result carries the per-image scores, the `delta`, and `preferred` ('image'/'reference'/'tie', tie when |delta| < 0.05). With style_context=true, both media are classified and a `cross_medium_warning` is added when they differ (cross-medium comparison is out of calibrated scope). Each image page is compared to the first page of the reference. Omit (default) for the original single-image scoring shape. | |
| style_context | No | When true, also classify the image's medium/genre (photograph, oil painting, digital illustration, ...) using the already-loaded CLIP backbone, and return it alongside the score. The aesthetic head was trained on photographs, so a non-photographic medium is the context the score must be read in -- an oil painting scoring ~5.8 is not 'wrong'. Default false keeps the original {score, rating} shape; true adds {style, style_distribution} to each result. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full behavioral burden, and it delivers thoroughly. It discloses the CLIP-based model, the photographic training bias, version-dependent payload changes, the tie threshold of |delta| < 0.05, and the cross_medium_warning behavior. There is no contradiction with any annotation, and no surprising side effects are hidden.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured and front-loaded with the core purpose, then moves through output, bias, and parameters. It is longer than necessary and repeats some parameter details that the schema already states, but every paragraph contributes a distinct piece of calibration or behavioral context.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with three parameters, two optional modes, model-specific caveats, and version-dependent output, the description is complete. It covers the output shape, score scale, photography bias, fine-art caveat, comparison semantics, style classification, and version-specific payload enrichment. An agent selecting or invoking this tool has everything it needs.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds conceptual rationale beyond the schema: compare_with is framed as the calibrated like-with-like route around photography bias, and style_context is framed as the answer for non-photographic media. The added value is real but modest because the schema parameter descriptions are already unusually rich and largely overlap with the prose.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb-plus-resource statement: "Rate how aesthetically pleasing an image looks, independent of its content." It explicitly scopes out content correctness and prompt matching, which distinguishes it from content-focused siblings like caption or detect_objects. It also signals the output shape and the two optional modes, making the tool's identity unmistakable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use guidance: compare like with like, such as several shots of the same subject or successive edits of one image. It also gives explicit when-not-to-use guidance: do not read a single absolute score as a quality verdict, and do not expect meaningful ratings for fine art. This is stronger than a typical 4 because it actively steers the agent away from common misuses.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spatial_relationsA
Measure how named objects in an image sit relative to one another.
Locates each object, segments it, and reports measurements that are hard to judge by eye: whether two things actually touch, how many pixels apart they are, how much of one lies inside the other and how deeply, plus each object's own elongation, straightness and end-to-end width profile.
This reports geometry, not verdicts — it does not decide what is wrong.
Interpret the numbers against what the scene ought to look like: a hand
and the grip it holds that come back separate with a large gap are not
in contact; a hand overlapping a shield with a_inside_b near 1.0 and a
large embed_depth is buried in the shield face rather than gripping its
rim; an elongated object whose end_symmetry is near 1.0 is equally wide
at both ends, unlike a blade that tapers to a point at one end only.
Useful for checking whether a generated or edited image holds together physically, for verifying that an object is where it should be relative to another, and for any question of contact, containment or clearance that a bounding box cannot answer — boxes overlap whenever one object is simply in front of another.
Takes the single best-scoring match per name, so this assumes one instance of
each named object. Asked for 'red circle'/'blue circle'/'green circle' on a
scene with one of each, the detector returned the same three boxes for every
query — color alone doesn't reliably discriminate same-shaped objects — but the
correctly-matching box scored highest every time, which is what this relies on.
For several instances of one kind of thing, give them distinguishing names, or
use count_objects for a tally instead.
A maximally vague name in objects ("object", "thing", "item", ...) is guarded
the same way detect_objects/count_objects are: a near-full-frame box from a
name with nothing distinctive to point to is dropped rather than treated as a
located match, so it doesn't get reported here as the "best-scoring" box for
that name.
| Name | Required | Description | Default |
|---|---|---|---|
| src | Yes | Local file path or http(s) URL of the image to process. PDFs are also accepted and are rendered one image per page, so tools that return a list return one entry per page. | |
| objects | Yes | Names of the objects to locate and compare, e.g. ['hand', 'sword', 'shield']. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden, and it does so richly: it states the tool locates and segments objects, does not render judgments, assumes one best-scoring match per name, and has a guard for maximally vague names. It even discloses the color-discrimination limitation observed with same-shaped objects, which is precisely the kind of behavioral nuance an agent needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but logically organized: purpose, interpretation guidance, use cases, assumptions, and caveats. It is not wasteful, though the red/blue/green circle example is more detailed than strictly necessary. Still, every section earns its place by preventing misinvocation.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter tool with a full output schema, the description is complete: it covers what the tool measures, what it deliberately does not decide, when to prefer alternatives, the one-instance assumption, and failure modes around vague names. The presence of an output schema means return-value details need not be restated here.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds real value for the objects parameter: it explains the one-instance assumption, gives naming guidance for multiple instances, and warns about vague names being dropped. It does not add anything specific to src, but the schema already fully describes it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Measure how named objects in an image sit relative to one another.' It then enumerates concrete measurements (touch, pixel distance, containment, elongation) and explicitly positions this as geometry rather than verdicts. This distinguishes it from siblings like detect_objects and count_objects, which only locate or tally.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use contexts: checking physical coherence, verifying placement, and answering contact/containment/clearance questions that bounding boxes cannot resolve. It also gives an explicit alternative for multiple instances ('use count_objects for a tally instead') and warns against vague object names, so an agent can route to the correct tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
- Changed
query_image1 field changed- changed
Input schema / properties / check_consistency / descriptionPrevious value: -"When true, also ask a rephrased control question and report whether the two answers agree, flagging short default-looking answers ('None', 'Yes', 'No', 'Nothing', ...) as low confidence. Moondream2 is a small VLM that answers open-ended judgment questions ('describe anything wrong') with a flat 'None' on images that all had real visible defects -- this layer makes that default-answer behavior visible instead of presenting it as reliable. On a low-confidence answer it also routes to the measurement that actually answers the question when one applies (spatial_relations for a contact/containment question, count_objects for 'how many', ocr for a text-reading question) and attaches it as `cross_check`. Default false keeps the original list[str] return; true returns one {answer, control_answer, consistent, confidence, cross_check?} dict per image."New value: +"When true, also ask a rephrased control question and report whether the two answers agree. `confidence` is 'low' in two cases: the answers agree on a short default-looking token ('None', 'Yes', 'No', 'Nothing', ...), or they substantively contradict each other. Moondream2 is a small VLM that answers open-ended judgment questions ('describe anything wrong') with a flat 'None' on images that all had real visible defects -- this layer makes that default-answer behavior visible instead of presenting it as reliable. On a low-confidence answer it also routes to the measurement that actually answers the question when one applies (spatial_relations for a contact/containment question, count_objects for 'how many', ocr for a text-reading question) and attaches it as `cross_check`. Default false keeps the original list[str] return; true returns one {answer, control_answer, consistent, confidence, cross_check?} dict per image."
11 tool updates
v0.8.2- First observed
batch_analyze_images - First observed
caption - First observed
count_objects - First observed
critique_composition - First observed
dense_region_caption - First observed
detect_objects - First observed
ocr - First observed
process - First observed
query_image - First observed
score_aesthetics - First observed
spatial_relations
TDQS
Scored across 11 tools
Most tools have clear boundaries (caption vs detect vs count vs spatial relations), but ocr and query_image directly contradict each other on which should read stylized/watermark text, and caption adds a third text-related caveat. An agent following the descriptions could route a text-reading request to the wrong model.
Names are uniformly lowercase snake_case and mostly readable, but conventions are mixed: verb_noun (detect_objects, score_aesthetics) sits alongside bare nouns (ocr, caption), an adjective_noun (spatial_relations), and a multi-part modifier (dense_region_caption). No single predictable pattern emerges.
11 tools is well within the ideal range and each fills a distinct role in the vision-understanding domain: OCR, captioning, VQA, detection, counting, spatial geometry, aesthetics, batch processing, and a raw escape hatch. The count feels deliberately scoped rather than padded.
The surface covers the major image-understanding operations—transcribe, describe, locate, count, measure relationships, and score aesthetics—and batch_analyze_images plus the raw process escape hatch close most workflow gaps. No obvious dead end or missing lifecycle step stands out for this domain.
Maintenance
Related MCP Connectors
OCR, transcription, file extraction, and image generation for AI agents via MCP.
Multimodal video analysis MCP — transcription, vision, and OCR for any video URL.
A paid remote MCP for HyperFrames, built to return verdicts, receipts, usage logs, and audit-ready J
Free OpenAI-compatible inference with signed provenance receipts and 3 focused MCP tools.
Related MCP Servers
- AlicenseAqualityBmaintenanceEnables any MCP client to perform image understanding and OCR via any OpenAI-compatible vision-language model. Supports local, private inference without images leaving the machine.215 npmMIT
- AlicenseNot gradedqualityCmaintenanceProvides vision capabilities to text-only LLMs via MCP, enabling image understanding, Q&A, OCR, and image processing through cloud multimodal APIs.MIT
- AlicenseAqualityCmaintenanceEnables any MCP-capable agent to perform vision tasks like describing images, answering questions, OCR, and comparing images using supported vision backends.5MIT
- AlicenseNot gradedqualityBmaintenanceProvides image understanding capabilities via local OCR and cloud VLM, enabling text extraction and visual description through MCP.17 npm2MIT