count_objects
Counts instances of a named object in an image, separating overlapping or touching items and flagging ambiguous tallies so results are reliable.
Instructions
Count how many instances of a named object an image contains.
Use this, not detect_objects, whenever the question is "how many".
detect_objects returns however many regions Florence-2's grounding head
emits, which is not a tally: it collapses several repeated, undifferentiated
parts into one box (every petal of a flower, both halves of a fused blade)
and conversely splits one object into overlapping sub-part boxes.
Returns count, plus bboxes/points/labels/scores for the instances
found, index-aligned and in the same pixel-space convention detect_objects
uses. scores are per-detection confidences, so a count resting on marginal
detections is visible rather than implied. group_boxes_dropped counts
detections that enclosed the whole arrangement rather than one instance.
Accuracy is measured, not assumed: against a fixture suite whose synthetic counts are exact by construction, this is right on 9 of 10 positive cases with a mean error of 0.1, while holding all 8 single-object controls. Separated, touching, dense, two-instance and awkwardly-named cases are reliable.
Two measured limits, worth checking before trusting a count:
Heavy overlap undercounts. Eight identical shapes in a ring count as 8 when separated or touching, but 6 once they overlap by roughly two-thirds of their width. A count below what you expect means "could not separate them", not a real tally -- and it will undercount rather than overcount.
Some evidence is simply not in the picture. On a photo of a paper flower whose
petals overlap, this returns 1 at every resolution from 128px to 768px, as does
every other approach tried -- other detectors, outline geometry, tiled crops,
and interior-colour analysis alike. That flower's outline is 98% convex and its
interior contrast is near zero, so nothing measurable distinguishes the petals.
Ask query_image for a count in that situation and treat it as an estimate.
When only one region is found, a silhouette block is added measuring how
many repeated lobes that region's outline contains. count and
silhouette.lobes come from different methods and neither overrides the
other: count: 1 beside silhouette.lobes: 8 means the detector could not
separate the instances while the outline shows eight cores. agreement: true
means a second, independent estimator matched it; shattered or clipped
mean the number should not be used at all.
Set consensus=true (the default) for two extra fields that surface
structural ambiguity instead of hiding it:
consensus:{detector_count, region_label_count, agree}-- a second count from Florence-2's dense region captioner (a different head), which tally how many region labels contain the object name. Agreement is real evidence; disagreement is a warning.separable:"yes"if the count is a real tally (the detector separated instances, or a single region the silhouette confirms is one lobe),"no"if the detector collapsed while the outline shows several lobes (the overlapping-petal case: nothing local can count it honestly),"unknown"when there's no silhouette check to confirm either way. Readcountwith this in mind: acount: 1withseparable: "no"is a collapse, not a tally.
When separable is "no" but the outline still carries the lobe pattern
(silhouette.by_radial > 1), an estimates block is added reporting that
outline count as an actionable number -- it's a measurement (angular
notches in the silhouette), not a judgment, so it stays within "measure,
don't judge". count is never overwritten; estimates.outline is the
number the outline supports, with a basis string saying so. Set
vqa_estimate=true to also ask Moondream2 "how many ?" and attach
Additionally, every result reports semantic_ambiguity (bool) and
count_semantics ("measured_tally", "minimum_visible_instances", or
"unverified_tally"). When semantic_ambiguity is true, an ambiguity
payload is attached detailing why the count represents a collapsed or
ambiguous structure rather than a confirmed discrete tally.
A maximally vague object_name ("object", "thing", "item", ...) is guarded
the same way detect_objects is: Grounding DINO has the same fallback-to-
the-whole-frame failure mode on a name with nothing distinctive to point to,
so a near-full-frame box is dropped rather than counted as a real instance.
clip_art=true gets the equivalent guard through Florence-2's own
exclude_full_frame. Only the exact generic vocabulary triggers this; a
specific noun that genuinely fills the frame is counted unfiltered.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| src | Yes | Local file path or http(s) URL of the image to process. PDFs are also accepted and are rendered one image per page, so tools that return a list return one entry per page. | |
| clip_art | No | When true, count with Florence-2's grounding head instead of Grounding DINO. Grounding DINO's training distribution is real photographs; on flat vector art and iconography it can over-detect (measured on a mixed clip-art scene: 6 instead of 2 trees, and a spurious 4th house past the true 3). Florence-2 got both exactly right on the same scene. Try it when the image is clip art / flat vector style and the default count looks wrong; `scores` are unavailable on this path (Florence-2's grounding head carries no per-box confidence) and `group_boxes_dropped` is not applicable. | |
| consensus | No | When true (default), also tally how many `dense_region_caption` labels match the object name as an independent second opinion, and report a `separable` flag. The dense region captioner is a different Florence-2 head than Grounding DINO, so agreement between the two is real evidence and disagreement is a visible warning. Cheap -- uses the already-loaded Florence-2, no new model. | |
| threshold | No | Box confidence floor passed to Grounding DINO (default 0.15, the value measured against benchmarks/ to hold every negative control -- see CLAUDE.md). Raise it for a visually cluttered scene with distractor shapes near the target: on one measured case (a target star among 18 muted-color distractors) the default threshold counted every distractor, while 0.5 separated the true target (score 0.90) from all of them (scores 0.19-0.32) for an exact count. This is a manual lever, not automatic -- raising it blindly on a normal scene can just as easily drop real instances, so use it once you suspect clutter, not by default. | |
| object_name | Yes | Name of the object to locate, e.g. 'person', 'car', 'face'. A short noun phrase works too ('the red mug'). More specific names ground more reliably than broad ones. | |
| vqa_estimate | No | When true and the detector has collapsed overlapping instances (separable is 'no'), also ask Moondream2 'how many <name>?' and attach its answer as estimates.vqa -- a judgment estimate, clearly marked, not a tally. Off by default so a session that never asks keeps Moondream unloaded. | |
| return_annotated | No | If true, saves an annotated image with counting boxes and returns the file path. | |
| verify_silhouette | No | When the detector finds only one region, segment it and measure how many repeated lobes its outline has. Costs one SAM2 load on the first such call. | |
| adaptive_threshold | No | When true, evaluates the raw detection distribution using kernel density estimation to automatically raise the threshold just above noise clusters if clear modes exist. Suppresses hundreds of false-positive sub-boxes in texture-heavy scenes while preserving real detections. Overrides manual threshold if the adaptive threshold is higher. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |