Visual Product Search MCP Server
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Visual Product Search
Identify which catalogue product (SKU) is in a customer's photo, with no label, barcode or serial number, then check live stock. Image-to-image retrieval with an optional vision-LLM re-ranker, exposed as a FastAPI service and an MCP tool for agents, wrapped in the engineering that makes it safe to run: multi-tenancy, auth, versioned catalogue, eval gate, cost controls and full traceability.
flowchart LR
Q[customer photo] --> D1{door 1<br/>arguments}
D1 --> D2{door 2<br/>caller / token}
D2 --> D3{door 3<br/>tenant + data region}
D3 --> C[optional crop<br/>OWLv2]
C --> E[encoder<br/>DINOv2 / SigLIP2 / color_hist]
E --> S[vector search<br/>tenant-filtered, stamped index]
S --> P[pool top-k per SKU]
P --> R{close call?}
R -- yes, budget allows --> L[vision-LLM re-ranker<br/>returns an index, validated]
R -- no --> A
L --> A[answer + confidence<br/>abstain if below threshold]
A --> I[(inventory)]
A --> T[trail: stages, event line, usage row]Why image-to-image: describing the photo in text and searching with that text loses the fine visual detail that separates look-alike SKUs. Published comparisons put text-prompt matching far below image-embedding matching on the same products, so this design embeds images on both sides and only uses a language model to re-rank a short list.
1. Run it (offline: no download, no network)
pip install -r requirements.txt
cp .env.example .env (Windows: copy .env.example .env) -- keeps VPI_ENCODER=color_hist
python tasks.py test 34 tests, about 30 seconds
python tasks.py data draws a synthetic catalogue + "customer photos" (busy background, rotation, blur)
python tasks.py build puts the catalogue into the index
python tasks.py eval metrics + the miss listpython tasks.py with no target lists every target (a cross-platform stand-in for make).
Result on the synthetic set ( | Value |
recall@1 / recall@5 / recall@10 | 0.778 / 0.933 / 1.000 |
index self-match rate | 1.00 |
eval gate | FAIL (by design): the toy encoder cannot reject out-of-catalogue photos, and |
Related MCP server: BSN Inventory MCP
2. What to read, in this order
# | File | What you learn there |
1 | the shape of every answer, error ( | |
2 | every knob in one place; | |
3 | THE pipeline, steps 1–8; everything else serves this file | |
4 | the three doors: arguments → caller → tenant | |
5 | services/vision/ | photo → vector → nearest catalogue images |
6 | services/rerank/ | the optional AI judge, its validators, cache and budget breaker |
7 | catalogue versions: build, retire, withdraw, restore, status | |
8 | evals/evaluate.py, | measuring: metrics, threshold, PASS/FAIL, one knob per arm |
9 | shared/events.py, shared/usage.py, commands/trace.py, evals/usage_report.py | the trail, the usage rows, and their readers |
10 | the two surfaces: both are thin wrappers over guards.py | |
11 | each test is a small, runnable example of one rule |
3. Layout
shared/ settings, schemas, auth (door 2), tenancy (door 3 + data_region), ledger, events, usage, images (door 1)
services/vision/ encoders (image → vector), index (search, tenant filter), detector (optional crop)
services/rerank/ reranker (OpenRouter or local model, validators, cache), budget (router clocks + breaker)
services/inventory stock table lookup (deterministic, never cached)
services/api/ identify.py (the pipeline), guards.py (doors), runtime.py (load once), app.py (FastAPI)
services/mcp_server/ server.py: identify_product + check_inventory as MCP tools
evals/ sample_data, evaluate, calibrate, gate, ablate, usage_report
commands/ catalog (build/retire/withdraw/restore/status), trace, probe_index
smoke/ smoke_api, smoke_mcp: test RUNNING servers (success + refusal per door)
tests/ offline unit tests (toy encoder, fake re-ranker: no network)
data/synthetic/ everything the scripts write (one folder per dataset; delete it any time)4. The models
Setting | Download | Purpose |
| none | a toy model so you can run every step today; not accurate |
| ~90 MB | the real local default: learned shapes/textures without captions |
| ~375 MB | learned from image–caption pairs; the ablation compares it with DINOv2 |
| GBs | Colab only |
VPI_DETECTOR=owlv2 turns on cropping (~600 MB, slow on CPU). VPI_RERANKER is none,
openrouter:<model>:free (the photo leaves this machine, and only for tenants with data_region: any), or local:<hf id> (Colab GPU).
For the real encoders: pip install transformers==5.20.1. For OpenRouter: pip install openai==3.35.1 and set the key in .env.
5. Production practices and where they live
# | Practice | Here |
1 | one implementation behind every surface |
|
2 | tool = typed function + docstring |
|
3 | arguments → caller → tenant |
|
4 | errors as data |
|
5 | log refusals too |
|
6 | answer carries its trail |
|
7 | first broken mark = cause |
|
8 | named fallback events |
|
9 | one usage row per request, priced at write |
|
10 | reconcile additive measures |
|
11 | router + breaker, degrade never stop |
|
12 | replay, never touch live |
|
13 | ablation, one knob per arm |
|
14 | golden set + required rows + gate |
|
15 | versions by bytes; ledger; undo; status |
|
16 | stamp on every vector | `encoder@revision |
17 | model output = an index, validated |
|
18 | data_region per tenant, fail closed |
|
19 | check before spend; cache by hash |
|
20 | hashes, not content, in logs |
|
21 | smoke: success AND refusal per door |
|
22 | read-only probes |
|
23 | nothing left warm | models load on first use; no server starts unless you run |
24 | known-gaps box | section 8 below |
25 | verify checklist | section 7 below |
Note: the crop setting is recorded in stages.crop rather than in the index stamp, because only the customer
photo is cropped, never the catalogue.
6. Running the servers (only when you choose to)
python tasks.py serve API on http://localhost:8080/docs (Ctrl+C stops it)
python tasks.py smoke in a second terminal: 8 checks, success + refusal per door
pip install fastmcp==4.1.1
python tasks.py mcp MCP server on http://localhost:9000/mcp
python tasks.py smoke-mcp tools discovered, success, refusal at the caller doorTest tokens live in data/synthetic/dev_tokens.json: dev-token (API), dev-mcp-token (MCP), acme-token
(another tenant), and two that must be refused (no-email-token, other-service-token).
7. Verify it yourself (exact values from the synthetic set, VPI_ENCODER=color_hist)
Command | Pass value |
|
|
|
|
| first |
|
|
|
|
|
|
| self-match rate |
|
|
|
|
Why the gate fails with the toy model: the "not sure" threshold is still 0, so it answers every
out-of-catalogue photo (false-accept 1.0), and two required photos are wrong. calibrate then reports that no
threshold meets the targets. That is the harness doing its job: it refuses to bless a weak model. The next step
is to switch the encoder to dinov2_small and rerun build, ablate, calibrate and gate.
8. What this does not do yet (known gaps)
Accuracy is unproven. The synthetic set tests the code; accuracy has to be measured on real photos of the target catalogue (the same harness runs on Colab with the bigger encoders).
dinov2_small,siglip2_base,owlv2, OpenRouter and the local re-ranker paths are not exercised by the offline test suite: it uses a toy encoder and a fake re-ranker. The fake re-ranker in the tests covers the logic around them, not the model calls themselves.The MCP server and the smoke tests need
fastmcpand a running server; they are not part of the offline suite.The vector index is exact numpy search: fine to about 100k images. Swapping in a vector database only changes
index.py.Local dev tokens are not security; they imitate the checks a Google ID token gets.
VPI_AUTH_MODE=googleis written but untested.Budget counters live in memory per process (two processes = two counters). They are seeded from
usage.jsonlat start.Events and usage are JSONL files with no rotation. On Cloud Run they become Cloud Logging.
There is no UI here, and inventory is a local table: swap
services/inventory/stock.pyfor your system of record.A threshold calibrated on about 50 rows is a demonstration, not a decision.
9. Colab profile (later, once the code makes sense)
The same folder, uploaded to Colab, with .env:
VPI_PROFILE=colab
VPI_ENCODER=siglip2_so400m (or dinov2_large: whichever the 07 notebook bake-off picked)
VPI_DETECTOR=owlv2
VPI_DATA_DIR=/content/data/<dataset>Real data uses the same layout as data/synthetic/: catalog_images/<tenant>/<sku>/*.jpg, catalog.csv,
inventory.csv, golden.jsonl, tenants.json. Nothing in services/ changes.
License
MIT
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Live product search for AI agents across Shopify merchants and major retailers.
AI-agent product catalog: search, lookup & purchase routing over verified merchant data.
Verified location-bound retail evidence for AI agents, delivered by human field workers.
Search live store shelves, verify delivery windows, and run instant AI agent checkouts.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceEnables conversational product search and validation for e-commerce catalogs, with hybrid retrieval and live price/stock checks from a database.-
- FlicenseNot gradedqualityCmaintenanceEnables identification of BSN SKU/material and real-time inventory availability through natural language queries and vector search.-
- AlicenseNot gradedqualityBmaintenanceEnables agents to perform structured shopping actions and retrieve targeted product images for visual comparison, matching, and aesthetic recommendations.MIT
- AlicenseAqualityCmaintenanceEnables LLM agents to analyze retail shelf photos via computer vision, including product counting, gap detection, price tag reading, and planogram compliance checking.7MIT