Shelf Auditor MCP
This server is an MCP toolset for auditing retail shelf photos with computer vision (YOLO-based detection and OCR), returning structured results and annotated images.
count_products: Count products on a shelf photo; returns per-label totals, per-item bounding boxes (absolute pixels and normalized 0β1 coordinates), confidence scores, and an annotated image.
detect_gaps: Detect empty/out-of-stock shelf regions grouped by row; each gap includes its bounding box, width ratio vs. row mean product width, and severity (minor/moderate/major).
read_price_tags: Read price labels via OCR, parse values/currency, and link each tag to the nearest detected product.
check_planogram: Compare the shelf against a planogram spec (slots with expected normalized regions and optional order); returns missing, misplaced, extra, and wrong-order deviations plus a compliance flag.
shelf_report: Run count, gap detection, and optionally a planogram check in one call; returns a combined JSON report and/or a Markdown summary, with optional annotated image output.
Customizable inference: Each vision tool accepts an optional detector confidence override; server supports environment-variable configuration for weights, confidence threshold, and inference resolution.
Flexible image input: Accepts local file paths, HTTP(S) URLs, or base64 data URIs.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Shelf Auditor MCP Run a shelf audit on this photo: count products, find gaps, check planogram"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
π Shelf Auditor MCP
Point an LLM agent at a photo of a retail shelf β it counts the products, finds the out-of-stock gaps, and checks the layout against a planogram.
A Model Context Protocol server for retail-shelf computer vision.
count_products on a shelf photo β every facing boxed, with pixel + normalised coordinates and a confidence.
Why this exists
Retail audit work β counting facings, catching stockouts, verifying planogram compliance β is repetitive visual grunt work. An LLM agent with the right tools can do it from a single phone photo. This server gives an agent those tools over MCP, so it drops straight into Claude Desktop, Cline, or any MCP client.
It is built as a portfolio piece: a clean, documented, demoable MVP with a real fine-tuned model behind it, not a pile of half-features.
Related MCP server: Vision MCP Server
What it can do
Tool | What it returns | Status |
| Per-label counts, every item's | β |
| Empty shelf regions grouped by row, each with a width ratio vs. the row's mean product and a severity ( | β |
|
| β |
| Meta-tool: runs count + gaps (+ planogram) and returns one combined JSON and a Markdown summary | β |
| Poll long-running jobs | β |
| Prices OCR'd ( | π§ͺ solid on readable labels; tiny / superscript-cent tags in wide shots need a dedicated tag detector |
The vision backends are pluggable behind one interface. read_price_tags needs
the optional ocr extra; without it the tool returns a clean not_implemented
status (the agent handles it, and shelf_report still runs the rest) instead of
crashing.
Resources: image://{id} Β· results://{job_id} Β· annotated://{job_id} Β· models://available
Prompts: count-items Β· detect-gaps Β· read-shelf Β· planogram-check
See it work
In an agent (Claude Desktop)
Ask in plain language β "How many products are on this shelf and are there any
gaps?" β and the agent picks shelf_report, runs it on the image, and comes
back with the counts, the gap list, and an annotated picture. Walkthrough:
examples/demo_walkthrough.md.
π₯ Screen-recording of the agent flow β coming shortly.
Why a retail-specific detector
This is a class-ontology comparison, not a recall one. Stock YOLOv8s has no
"product" class, so it forces every loaf into the nearest of its 80 COCO labels β
donut, cake, sandwich. Confident and useless: you can't audit facings from
a pile of "donuts". The SKU-110K fine-tune has one purpose-built class and reports
each facing as product.
Stock YOLOv8s β COCO classes | YOLOv8s fine-tuned on SKU-110K |
| one |
That confidence drop is real and expected: a German bakery case behind glass β
angled, warm-lit, reflective β is about as far from SKU-110K's evenly-lit US
grocery aisles as a shelf photo gets, so scores fall and the default
detector_conf of 0.2 filters some true loaves. On in-distribution shots like the
wine shelf above, the same weights box 73/73 facings cleanly. Closing that
domain gap is the "in-domain detector" item in the Roadmap; it does
not affect the benchmark number (0.938 mAP@0.5 on SKU-110K val).
shelf_report output
{
"count_products": { "total": 73, "counts": { "product": 73 } },
"detect_gaps": { "gap_count": 3, "rows_detected": 3,
"gaps": [{ "row": 0, "severity": "moderate", "width_ratio": 1.45 }, β¦] }
}# Shelf audit report
**Products detected:** 73
**Shelf gaps:** 3 across 3 row(s)
- row 0: moderate (Γ1.45 mean width)Architecture
The MCP layer (protocol, schema validation, job management) is kept strictly
separate from the vision backend. Every capability implements one interface β
BaseVisionModel.run(image, params) -> Result β so swapping YOLOv8 for RT-DETR,
or a local model for a cloud API, never touches MCP code.
flowchart LR
A[MCP client<br/>Claude Desktop Β· Cline] -->|tool call| B[FastMCP server]
B --> C[schemas.py<br/>strict Pydantic]
C --> D[tools/*]
D --> E[imaging.py<br/>load Β· resize Β· annotate]
D --> F[jobs.py Β· storage.py]
D --> G[backends/BaseVisionModel]
G --> H[YoloDetector<br/>SKU-110K fine-tune]
G --> I[OcrBackend<br/>stub]src/vision_mcp/
server.py FastMCP instance β registers tools, resources, prompts
schemas.py per-tool input/output models
config.py pydantic settings (env prefix VISION_MCP_)
imaging.py path / URL / base64 β RGB array; resize; annotate
jobs.py async job manager storage.py image + result cache
backends/ base.py + detector.py (YOLO) + ocr.py (stub) + matching.py
tools/ one module per tool
scripts/hpc/ fine-tune the detector on a SLURM clusterQuickstart
uv sync
uv run python scripts/download_models.py # COCO fallback weights β models/
uv run python -m vision_mcp.server # stdio MCP serverRuns out of the box on the COCO fallback. For the SKU-110K accuracy in the
numbers above, fetch the fine-tune β see scripts/hpc/README.md.
Wire it into a client with examples/claude_desktop_config.json (fix the path),
then walk through examples/demo_walkthrough.md.
Try it on any image β annotated PNGs + JSON land in out/:
uv run python scripts/try_image.py path/to/shelf.jpg
uv run python scripts/try_image.py shelf.jpg --slots planogram.jsonPoke the protocol with the MCP Inspector:
uv run mcp dev src/vision_mcp/server.pyThe detector
Default weights are a YOLOv8s fine-tuned on SKU-110K,
trained on the University of Twente GPU cluster (scripts/hpc/, one L40, 42 min).
Value | |
SKU-110K val mAP@0.5 | 0.938 |
SKU-110K val mAP@0.5:0.95 | 0.595 |
Precision / Recall | 0.916 / 0.886 |
Training | 30 epochs Β· imgsz 960 Β· 1Γ NVIDIA L40 Β· 42 min |
The detector runs class-agnostic, so its single object class is reported as
"product". If models/yolov8s_sku110k.pt is missing, it falls back to
auto-downloaded COCO yolov8n.pt and says so in the response notes. To fetch
the fine-tune, follow scripts/hpc/README.md then:
export VISION_MCP_DETECTOR_WEIGHTS=yolov8s_sku110k.ptDevelopment
uv run pytest # core suite is synthetic β no weights needed
uv run ruff check . && uv run mypy src
uv run python tests/fixtures/download_fixtures.py # + real-photo integration testsDocker
docker build -t shelf-auditor . # CPU
docker build -f Dockerfile.cuda -t shelf-auditor:cuda . # CUDA β run with --gpus allRoadmap
Price-tag OCR β
read_price_tagsrunspython-doctrand parses{value, currency}(works well on clear labels β$11,$0.25/100,1.19). Generic OCR misses small / distant tags and European superscript-cent formats (5β΄β°); a production version needs a price-tag region detector feeding per-crop OCR.In-domain detector. SKU-110K is dead-on, evenly-lit US grocery; on angled or dim store photos the fine-tune localises well but scores lower (hence
detector_conf0.2).train_sku110k.sbatchretrains on any labelled set (IMGSZ=1280for the tiniest facings).Deliberately out of scope: video / tracking, per-client model training, batch folders, cloud vision backends, multi-tenant auth.
Credits & licensing
Code: MIT β see
LICENSE.SKU-110K (Goldman et al., CVPR 2019) is released for academic / non-commercial use β the shipped fine-tuned weights inherit that restriction. Retrain on licensed or self-collected data before commercial deployment.
Demo photos are CC-BY from Wikimedia Commons: Alsatian wines in a supermarket by francois (CC BY 2.0); Krustenbrot for sale at supermarket by Maksym Kozlenko (CC BY-SA 4.0). Full list in
tests/fixtures/SOURCES.md.
Config reference
Environment variables, prefix VISION_MCP_ (see src/vision_mcp/config.py):
Var | Default | |
|
|
|
|
| file in |
|
| detection confidence floor |
|
| YOLO inference resolution |
|
| longest image edge before inference |
|
| above this, annotated images return as a reference |
Built by TrαΊ§n Quang ThΓ nh β AI Engineer specialising in Computer Vision & LLM agents. Available for freelance work.
Available Tools
7 toolscheck_planogramB
Compare the shelf against a planogram spec (list of slots with expected normalized regions + optional order). Lists missing / misplaced / extra / wrong-order deviations.
Inputs: params (required). Returns: image, notes, annotated, compliant, +2 more
count_productsB
Count products on a shelf photo. Returns per-label counts, per-item boxes (absolute + normalized) with confidence, and an annotated image.
Inputs: params (required). Returns: image, items, notes, total, +2 more
detect_gapsB
Find empty (out-of-stock) regions on the shelf, grouped by row, each with a width ratio versus the row's mean product width and a severity.
Inputs: params (required). Returns: gaps, image, notes, annotated, +2 more
get_job_resultC
Result of a finished async job.
Inputs: job_id (required).
get_job_statusC
Status of an async job created by a heavy tool call.
Inputs: job_id (required).
read_price_tagsB
Read price labels and link each to the nearest product. NOTE: the OCR backend is currently a stub and returns status 'not_implemented'.
Inputs: params (required). Returns: tags, image, notes, status
shelf_reportB
Run count, gap detection, and (optionally) a planogram check on one image and return a combined JSON and/or markdown report.
Inputs: params (required). Returns: image, report, markdown, annotated
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelogβ¦
TDQS
Scored across 7 tools
Each tool has a clearly distinct purpose: counting products, detecting gaps, reading price tags, checking planograms, and managing async jobs. The shelf_report tool is a composite but still distinct from the individual analysis tools.
All tool names follow a consistent verb_noun pattern in snake_case (e.g., count_products, detect_gaps, get_job_status). The naming is predictable and uniform.
With 7 tools, the set is well-scoped for a shelf analysis domain. It covers the main analysis functions plus job management without being excessive or sparse.
The toolset covers all core shelf analysis tasks: product counting, gap detection, price tag reading, planogram compliance, and combined reporting. Async job support is also included, providing a comprehensive surface.
Maintenance
Related MCP Connectors
Verified location-bound retail evidence for AI agents, delivered by human field workers.
Roboflow computer vision for AI agents: datasets, annotation, versioning, workflows, inference.
Give agents eyes on any web page: structured context, and changes explained in plain language.
Image & PDF tools for AI agents: compress, convert, resize, PDF, AI vision, pipeline.
Related MCP Servers
- FlicenseAqualityNot gradedmaintenanceEnables AI agents to analyze images through vision AI providers (Gemini, OpenAI, Claude), performing tasks like image description, object detection with bounding boxes, region-specific analysis, and precise color extraction without consuming context window with raw pixels.4-
- AlicenseAqualityDmaintenanceEnables AI agents to analyze images, extract text, compare images, and analyze video through any OpenAI-compatible vision model.4330 npm20MIT
- AlicenseNot gradedqualityFmaintenanceEnables AI agents to read images with metadata, OCR text, regions, and citeable evidence without relying on generative LLMs.66 npm2MIT
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to analyze images using any OpenAI-compatible vision API, providing tools for image analysis, OCR, error diagnosis, diagram understanding, and chart analysis.MIT