Skip to main content
Glama

πŸ›’ Shelf Auditor MCP

Point an LLM agent at a photo of a retail shelf β€” it counts the products, finds the out-of-stock gaps, and checks the layout against a planogram.

A Model Context Protocol server for retail-shelf computer vision.

Python MCP Detector License

count_products on a shelf photo β€” every facing boxed, with pixel + normalised coordinates and a confidence.


Why this exists

Retail audit work β€” counting facings, catching stockouts, verifying planogram compliance β€” is repetitive visual grunt work. An LLM agent with the right tools can do it from a single phone photo. This server gives an agent those tools over MCP, so it drops straight into Claude Desktop, Cline, or any MCP client.

It is built as a portfolio piece: a clean, documented, demoable MVP with a real fine-tuned model behind it, not a pile of half-features.

Related MCP server: Vision MCP Server

What it can do

Tool

What it returns

Status

count_products

Per-label counts, every item's bbox (pixels) + bbox_normalized (0–1) + confidence, and an annotated image

βœ…

detect_gaps

Empty shelf regions grouped by row, each with a width ratio vs. the row's mean product and a severity (minor / moderate / major)

βœ…

check_planogram

missing / misplaced / extra / wrong_order deviations against a slot spec, plus a compliant flag

βœ…

shelf_report

Meta-tool: runs count + gaps (+ planogram) and returns one combined JSON and a Markdown summary

βœ…

get_job_status Β· get_job_result

Poll long-running jobs

βœ…

read_price_tags

Prices OCR'd (python-doctr), parsed to {value, currency}, linked to the nearest product

πŸ§ͺ solid on readable labels; tiny / superscript-cent tags in wide shots need a dedicated tag detector

The vision backends are pluggable behind one interface. read_price_tags needs the optional ocr extra; without it the tool returns a clean not_implemented status (the agent handles it, and shelf_report still runs the rest) instead of crashing.

Resources: image://{id} Β· results://{job_id} Β· annotated://{job_id} Β· models://available Prompts: count-items Β· detect-gaps Β· read-shelf Β· planogram-check

See it work

In an agent (Claude Desktop)

Ask in plain language β€” "How many products are on this shelf and are there any gaps?" β€” and the agent picks shelf_report, runs it on the image, and comes back with the counts, the gap list, and an annotated picture. Walkthrough: examples/demo_walkthrough.md.

πŸŽ₯ Screen-recording of the agent flow β€” coming shortly.

Why a retail-specific detector

This is a class-ontology comparison, not a recall one. Stock YOLOv8s has no "product" class, so it forces every loaf into the nearest of its 80 COCO labels β€” donut, cake, sandwich. Confident and useless: you can't audit facings from a pile of "donuts". The SKU-110K fine-tune has one purpose-built class and reports each facing as product.

Stock YOLOv8s β€” COCO classes

YOLOv8s fine-tuned on SKU-110K

donut 0.78, cake, sandwich β€” wrong ontology

one product class per facing β€” right ontology, lower scores on this shot

That confidence drop is real and expected: a German bakery case behind glass β€” angled, warm-lit, reflective β€” is about as far from SKU-110K's evenly-lit US grocery aisles as a shelf photo gets, so scores fall and the default detector_conf of 0.2 filters some true loaves. On in-distribution shots like the wine shelf above, the same weights box 73/73 facings cleanly. Closing that domain gap is the "in-domain detector" item in the Roadmap; it does not affect the benchmark number (0.938 mAP@0.5 on SKU-110K val).

shelf_report output

{
  "count_products": { "total": 73, "counts": { "product": 73 } },
  "detect_gaps":    { "gap_count": 3, "rows_detected": 3,
                      "gaps": [{ "row": 0, "severity": "moderate", "width_ratio": 1.45 }, …] }
}
# Shelf audit report
**Products detected:** 73
**Shelf gaps:** 3 across 3 row(s)
- row 0: moderate (Γ—1.45 mean width)

Architecture

The MCP layer (protocol, schema validation, job management) is kept strictly separate from the vision backend. Every capability implements one interface β€” BaseVisionModel.run(image, params) -> Result β€” so swapping YOLOv8 for RT-DETR, or a local model for a cloud API, never touches MCP code.

flowchart LR
    A[MCP client<br/>Claude Desktop Β· Cline] -->|tool call| B[FastMCP server]
    B --> C[schemas.py<br/>strict Pydantic]
    C --> D[tools/*]
    D --> E[imaging.py<br/>load Β· resize Β· annotate]
    D --> F[jobs.py Β· storage.py]
    D --> G[backends/BaseVisionModel]
    G --> H[YoloDetector<br/>SKU-110K fine-tune]
    G --> I[OcrBackend<br/>stub]
src/vision_mcp/
  server.py     FastMCP instance β€” registers tools, resources, prompts
  schemas.py    per-tool input/output models
  config.py     pydantic settings (env prefix VISION_MCP_)
  imaging.py    path / URL / base64 β†’ RGB array; resize; annotate
  jobs.py       async job manager      storage.py   image + result cache
  backends/     base.py + detector.py (YOLO) + ocr.py (stub) + matching.py
  tools/        one module per tool
scripts/hpc/    fine-tune the detector on a SLURM cluster

Quickstart

uv sync
uv run python scripts/download_models.py     # COCO fallback weights β†’ models/
uv run python -m vision_mcp.server           # stdio MCP server

Runs out of the box on the COCO fallback. For the SKU-110K accuracy in the numbers above, fetch the fine-tune β€” see scripts/hpc/README.md.

Wire it into a client with examples/claude_desktop_config.json (fix the path), then walk through examples/demo_walkthrough.md.

Try it on any image β€” annotated PNGs + JSON land in out/:

uv run python scripts/try_image.py path/to/shelf.jpg
uv run python scripts/try_image.py shelf.jpg --slots planogram.json

Poke the protocol with the MCP Inspector:

uv run mcp dev src/vision_mcp/server.py

The detector

Default weights are a YOLOv8s fine-tuned on SKU-110K, trained on the University of Twente GPU cluster (scripts/hpc/, one L40, 42 min).

Value

SKU-110K val mAP@0.5

0.938

SKU-110K val mAP@0.5:0.95

0.595

Precision / Recall

0.916 / 0.886

Training

30 epochs Β· imgsz 960 Β· 1Γ— NVIDIA L40 Β· 42 min

The detector runs class-agnostic, so its single object class is reported as "product". If models/yolov8s_sku110k.pt is missing, it falls back to auto-downloaded COCO yolov8n.pt and says so in the response notes. To fetch the fine-tune, follow scripts/hpc/README.md then:

export VISION_MCP_DETECTOR_WEIGHTS=yolov8s_sku110k.pt

Development

uv run pytest                          # core suite is synthetic β€” no weights needed
uv run ruff check . && uv run mypy src

uv run python tests/fixtures/download_fixtures.py   # + real-photo integration tests

Docker

docker build -t shelf-auditor .                            # CPU
docker build -f Dockerfile.cuda -t shelf-auditor:cuda .    # CUDA β€” run with --gpus all

Roadmap

  • Price-tag OCR β€” read_price_tags runs python-doctr and parses {value, currency} (works well on clear labels β€” $11, $0.25/100, 1.19). Generic OCR misses small / distant tags and European superscript-cent formats (5⁴⁰); a production version needs a price-tag region detector feeding per-crop OCR.

  • In-domain detector. SKU-110K is dead-on, evenly-lit US grocery; on angled or dim store photos the fine-tune localises well but scores lower (hence detector_conf 0.2). train_sku110k.sbatch retrains on any labelled set (IMGSZ=1280 for the tiniest facings).

  • Deliberately out of scope: video / tracking, per-client model training, batch folders, cloud vision backends, multi-tenant auth.

Credits & licensing

  • Code: MIT β€” see LICENSE.

  • SKU-110K (Goldman et al., CVPR 2019) is released for academic / non-commercial use β€” the shipped fine-tuned weights inherit that restriction. Retrain on licensed or self-collected data before commercial deployment.

  • Demo photos are CC-BY from Wikimedia Commons: Alsatian wines in a supermarket by francois (CC BY 2.0); Krustenbrot for sale at supermarket by Maksym Kozlenko (CC BY-SA 4.0). Full list in tests/fixtures/SOURCES.md.

Config reference

Environment variables, prefix VISION_MCP_ (see src/vision_mcp/config.py):

Var

Default

DEVICE

auto

auto / cpu / cuda

DETECTOR_WEIGHTS

yolov8s_sku110k.pt

file in models/, or an ultralytics name

DETECTOR_CONF

0.2

detection confidence floor

DETECTOR_IMGSZ

960

YOLO inference resolution

MAX_EDGE_PX

1920

longest image edge before inference

MAX_INLINE_IMAGE_BYTES

4 MiB

above this, annotated images return as a reference


Built by TrαΊ§n Quang ThΓ nh β€” AI Engineer specialising in Computer Vision &amp; LLM agents. Available for freelance work.

Upwork Β· LinkedIn Β· GitHub

Available Tools

7 tools
check_planogramB

Compare the shelf against a planogram spec (list of slots with expected normalized regions + optional order). Lists missing / misplaced / extra / wrong-order deviations.

Inputs: params (required). Returns: image, notes, annotated, compliant, +2 more

count_productsB

Count products on a shelf photo. Returns per-label counts, per-item boxes (absolute + normalized) with confidence, and an annotated image.

Inputs: params (required). Returns: image, items, notes, total, +2 more

detect_gapsB

Find empty (out-of-stock) regions on the shelf, grouped by row, each with a width ratio versus the row's mean product width and a severity.

Inputs: params (required). Returns: gaps, image, notes, annotated, +2 more

get_job_resultC

Result of a finished async job.

Inputs: job_id (required).

get_job_statusC

Status of an async job created by a heavy tool call.

Inputs: job_id (required).

read_price_tagsB

Read price labels and link each to the nearest product. NOTE: the OCR backend is currently a stub and returns status 'not_implemented'.

Inputs: params (required). Returns: tags, image, notes, status

shelf_reportB

Run count, gap detection, and (optionally) a planogram check on one image and return a combined JSON and/or markdown report.

Inputs: params (required). Returns: image, report, markdown, annotated

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.6/5.0

    Scored across 7 tools

    Disambiguation5/5

    Each tool has a clearly distinct purpose: counting products, detecting gaps, reading price tags, checking planograms, and managing async jobs. The shelf_report tool is a composite but still distinct from the individual analysis tools.

    Naming Consistency5/5

    All tool names follow a consistent verb_noun pattern in snake_case (e.g., count_products, detect_gaps, get_job_status). The naming is predictable and uniform.

    Tool Count5/5

    With 7 tools, the set is well-scoped for a shelf analysis domain. It covers the main analysis functions plus job management without being excessive or sparse.

    Completeness5/5

    The toolset covers all core shelf analysis tasks: product counting, gap detection, price tag reading, planogram compliance, and combined reporting. Async job support is also included, providing a comprehensive surface.

    Maintenance

    ActivityStale
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      A
      quality
      Not graded
      maintenance
      Enables AI agents to analyze images through vision AI providers (Gemini, OpenAI, Claude), performing tasks like image description, object detection with bounding boxes, region-specific analysis, and precise color extraction without consuming context window with raw pixels.
      4
      -
    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI agents to analyze images using any OpenAI-compatible vision API, providing tools for image analysis, OCR, error diagnosis, diagram understanding, and chart analysis.
      MIT