Skip to main content
Glama
vishalhabib99

listing-claim-check

by vishalhabib99
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

listing-claim-check

Checks an AI-written marketplace listing against the seller's own item specifics before it's published, and returns PUBLISH or REVIEW with the exact words behind every unbacked claim.

Marketplaces let sellers generate listings with AI. A model writing sales copy fills gaps with plausible claims: "unlocked", "like new", "genuine leather", "includes the original box". When the item doesn't match, the buyer opens a not-as-described dispute. The item specifics the seller already entered are the source of truth. This checks the copy against them.

Try it in your browser → Runs locally, no account, no tracking.

Reference build by an AI product manager: PRD → eval plan with gates set first → build → blind evals, including a failed first run → design change → a fresh blind run. Not affiliated with any marketplace.

What it checks

Claim

Example it catches

High harm?

Condition

"Like new" or "barely worn" on an item listed as Used

Yes

Authenticity

"100% legit" with no authentication recorded

Yes

Model, storage, battery, carrier

"iPhone 14" for a 13 · "256GB" for 128 · "91% battery" for 76 · "works on any network" for a carrier-locked phone

Yes

Size, brand, warranty

"Size 11" for a 10 · a brand the specifics don't name · "under warranty" with none recorded

Yes

What's included

"Comes with the original box and charger" when only a cable is

Yes

Any number

A number no item specific backs up ("Tabby 26" for a Tabby 20, "bought in 2023")

Yes

Color, material

"Pink" for a blue phone · "leather" for a canvas bag

Low

Each claim is marked SUPPORTED, CONTRADICTED (the specifics say otherwise) or UNSUPPORTED (the specifics don't say). Anything but SUPPORTED → REVIEW. Negated mentions ("charger not included") aren't claims.

Deterministic, no model. It's cheap enough to run on every listing, and every flag points at exact words. The known cost: a claim phrased outside its vocabulary isn't seen, so PUBLISH means "nothing unbacked was found", not "verified true".

Related MCP server: GoBuy Product Trust MCP Server

Results

Gates were set before any code. Each held-out set was written by a separate agent that never saw the code or the vocabulary, and was committed before its first run.

Run

Top harm: high-harm claims marked PUBLISH (gate ≤ 10%)

Friction: clean listings marked REVIEW (gate ≤ 20%)

v0.1, held-out 1, first run

2 of 11 (18%): FAIL

1 of 16 (6%)

v0.2, held-out 2 (fresh, blind), first run

0 of 11: PASS

2 of 16 (12%): PASS

What changed between them: v0.1 missed "91% health" and "Tabby 26". Both are numbers phrased outside the vocabulary. Instead of adding a pattern for each miss, v0.2 changed the design: any number that no item specific backs up goes to REVIEW. That's the same principle as retirement-answer-check. Two bugs found in the same run were fixed too, which means held-out 1 isn't blind for v0.2, so its re-run isn't reported as a result. Logs: evals/heldout_first_run.txt, evals/heldout2_first_run.txt.

Known issue (v0.2): "no box or charger" isn't read as negating the charger, because "or" ends the negation. Both false REVIEWs on held-out 2 came from this. It's safe-direction friction, not a missed claim. It's left unfixed until a third blind set can measure the fix.

Limits: 60 blind cases in 3 categories (phones, sneakers, handbags), all written by one model family. It hasn't been tested on real listings. Handbag and sneaker model names aren't recognized as model claims (only their numbers are).

Use it

pip install git+https://github.com/vishalhabib99/listing-claim-check
from listing_claim_check import check
check({"brand": "Nike", "size": "10", "condition": "Used"}, "Nike Dunk Low size 11", "Barely worn.")
# {'decision': 'REVIEW', 'claims': [
#   {'attribute': 'condition', 'text': 'Barely worn', 'status': 'CONTRADICTED', 'reason': 'specifics: condition = Used', ...},
#   {'attribute': 'size', 'text': 'size 11', 'status': 'CONTRADICTED', 'reason': 'specifics: size = 10', ...}, ...]}

As an MCP tool, so an AI listing assistant can check its own draft before showing it to the seller:

claude mcp add listing-claim-check -- uvx --from git+https://github.com/vishalhabib99/listing-claim-check listing-claim-check

Run the evals: python evals/run.py evals/heldout2.jsonl

License

MIT

Available Tools

1 tool
check_listingA
Read-onlyIdempotent

Check an AI-written marketplace listing against the seller's item specifics before it's published.

Finds claims in the title and description (condition, authenticity, model, storage, carrier lock, battery health, size, brand, warranty, included items, color, material, and any number) and marks each one SUPPORTED, CONTRADICTED or UNSUPPORTED by the specifics. Returns {"decision": "PUBLISH" | "REVIEW", "claims": [{"attribute", "claimed", "text", "status", "harm", "reason"}]}. Any claim that isn't SUPPORTED means REVIEW; show the seller each claim's text and reason. Deterministic, no model: claims phrased outside its vocabulary aren't seen, so PUBLISH means "nothing unbacked was found", not "verified true".

Args: specifics: The seller's structured item specifics, e.g. {"category": "smartphone", "brand": "Apple", "model": "iPhone 13", "condition": "Used", "storage": "128GB", "carrier": "Unlocked", "battery_health": 88, "includes": ["cable"]}. "includes" is a list of lowercase item names. title: The AI-written listing title. description: The AI-written listing description.

ParametersJSON Schema
NameRequiredDescriptionDefault
titleNo
specificsYes
descriptionNo

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already carry readOnlyHint, idempotentHint, and destructiveHint, and the description adds significant behavioral context: 'Deterministic, no model', the vocabulary limitation, and the precise meaning of PUBLISH ('nothing unbacked was found, not verified true'). This goes far beyond the annotations and helps the agent set correct expectations. No contradiction with annotations exists.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The first sentence front-loads the core purpose, and every subsequent section earns its place: behavior, return shape, decision logic, and parameter details. The argument breakdown is cleanly separated from the behavioral notes, and there is no redundant or filler content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having no output schema, the description specifies the exact return structure with fields, the decision values (PUBLISH/REVIEW), and how to interpret the results. It also explains the determinism limitation and gives enough parameter guidance to call the tool correctly. For a read-only checker with no siblings, this is complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must carry the full semantic weight. It provides a concrete example of the 'specifics' object with realistic keys and values, explicitly notes that 'includes' is a lowercase item list, and defines 'title' and 'description' as the AI-written listing text. This fully compensates for the empty schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'Check an AI-written marketplace listing against the seller's item specifics before it's published.' It enumerates the exact claim categories checked and the statuses assigned, so the tool's purpose is unambiguous. No sibling tools exist, so differentiation is not needed, but the description still makes the scope clear.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly states the intended use case ('before it's published'), defines the decision rule ('Any claim that isn't SUPPORTED means REVIEW'), and warns against over-interpreting PUBLISH. It does not explicitly discuss when not to use the tool, but because there are no sibling tools, this is a minor gap rather than a critical one.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 1 tool updatev0.2.0
    • First observedcheck_listing
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.8/5.0

Scored across 1 tool

Disambiguation5/5

There is only one tool, so there is no possibility of confusion or misselection between overlapping tools. Its single responsibility is clearly described and easy to identify.

Naming Consistency5/5

With a single tool, the naming is trivially consistent: 'check_listing' follows a clear verb_noun pattern and aligns with the server's purpose.

Tool Count4/5

One tool is below the typical well-scoped range of 3-15, so it feels slightly thin, but the server's narrow purpose genuinely requires only this one operation.

Completeness5/5

The tool fully covers the stated domain: it checks listing claims against item specifics, classifies them, and returns a publish or review decision. No additional endpoints are needed for this focused workflow.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • GoBuy Product Trust MCP Server by gobuy-ai

    GoBuy Product Trust MCP Serverofficial

    gobuy-ai
    A
    license
    Not graded
    quality
    B
    maintenance
    Provides AI agents with 0–100 marketplace-evidence scores for retail product listings, enabling inspection and comparison of products across Amazon, Walmart, Target, and Best Buy using native retailer IDs.
    8 npm
    MIT

  • AgentGround Cross-Check

    Arafel187
    A
    license
    Not graded
    quality
    B
    maintenance
    Enables AI agents to cross-verify candidate claims against caller-supplied source texts, flagging hallucinations, numerical drift, entity mismatches, contradictions, and unverified assertions. It returns sentence-level verdicts with matched evidence snippets and machine-readable factual grounding confidence scores, exposed over MCP stdio, HTTP REST, and A2A discovery routes.
    MIT