Skip to main content
Glama
sanity-labs

mcp-see

by sanity-labs
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Hybrid

mcp-see

An MCP server that gives AI agents eyes - the ability to observe and understand images without stuffing raw pixels into their context window.

Features

  • Multi-provider vision: Describe images using Gemini, OpenAI, or Claude

  • Object detection: Find objects with bounding boxes (Gemini only - native bbox support)

  • Hierarchical analysis: Detect regions, then zoom in for detail

  • Precise color extraction: K-Means clustering in LAB color space (runs locally, no API needed)

  • Color naming: Human-readable color names via color.pizza API

  • URL support: Analyze images directly from the web (http/https)

TL;DR: A Gemini API key gives you full functionality. OpenAI/Claude are optional alternatives for image description only.

Related MCP server: VisionPower

Installation

Install from npm:

npx @sanity-labs/mcp-see

Or install globally:

npm install -g @sanity-labs/mcp-see

Or clone and build locally:

git clone https://github.com/sanity-labs/mcp-see.git
cd mcp-see
npm install
npm run build

MCP Client Configuration

Claude Desktop

Add to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "mcp-see": {
      "command": "npx",
      "args": ["@sanity-labs/mcp-see"],
      "env": {
        "GEMINI_API_KEY": "your-gemini-api-key"
      }
    }
  }
}

Get your Gemini API key from Google AI Studio.

With all providers (optional):

{
  "mcpServers": {
    "mcp-see": {
      "command": "npx",
      "args": ["@sanity-labs/mcp-see"],
      "env": {
        "GEMINI_API_KEY": "your-gemini-api-key",
        "OPENAI_API_KEY": "sk-...",
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}

Enterprise/Vertex AI users:

{
  "mcpServers": {
    "mcp-see": {
      "command": "npx",
      "args": ["@sanity-labs/mcp-see"],
      "env": {
        "GOOGLE_CLOUD_PROJECT": "your-gcp-project-id"
      }
    }
  }
}

Requires ADC setup: gcloud auth application-default login

Other MCP Clients

The server runs on stdio transport. Configure your client to spawn npx @sanity-labs/mcp-see.

Tools

describe

Get an AI-generated description of an image.

Input:

{
  "image": "/path/to/image.png or https://example.com/image.jpg",
  "prompt": "What is shown in this image?",
  "provider": "gemini",
  "detail": "detailed"
}

Example Output:

The image shows a vibrant and colorful salad bowl, viewed from directly above.
The bowl is made of a light brown, possibly biodegradable material. The salad
is composed of various ingredients arranged in distinct sections: two small
white peeled eggs, sliced red tomatoes topped with chopped green onions, cubed
seasoned tofu, bright green edamame beans, shredded purple cabbage, and
julienned carrots...

detect

Detect objects and return bounding boxes. Uses Gemini for native bbox support.

Input:

{
  "image": "/path/to/image.png",
  "prompt": "find all TV screens"
}

Example Output:

{
  "count": 3,
  "objects": [
    { "id": 1, "label": "television", "bbox": [178, 245, 433, 818] },
    { "id": 2, "label": "television", "bbox": [614, 518, 792, 898] },
    { "id": 3, "label": "television", "bbox": [617, 198, 792, 493] }
  ]
}

Coordinates are [ymin, xmin, ymax, xmax] normalized 0-1000.

describe_region

Crop to a bounding box and describe that region in detail.

Input:

{
  "image": "/path/to/image.png",
  "bbox": [200, 200, 800, 800],
  "prompt": "describe this in detail",
  "provider": "gemini"
}

Example Output:

{
  "bbox": [200, 200, 800, 800],
  "description": "The image showcases a vibrant and colorful salad bowl in close-up. The bowl contains fresh ingredients including cubed tofu with a seasoned exterior, bright green edamame, sliced tomatoes, and shredded purple cabbage..."
}

analyze_colors

Extract dominant colors from a region using K-Means clustering in LAB color space.

Input:

{
  "image": "/path/to/image.png",
  "bbox": [100, 200, 400, 600],
  "top": 5
}

Example Output:

{
  "dominant": [
    {
      "hex": "#e6e6e5",
      "rgb": [230, 230, 229],
      "hsl": { "h": 60, "s": 2, "l": 90 },
      "name": "Ambience White",
      "percentage": 75.91
    },
    {
      "hex": "#b16c39",
      "rgb": [177, 108, 57],
      "hsl": { "h": 26, "s": 51, "l": 46 },
      "name": "Ginger Dough",
      "percentage": 15.91
    }
  ],
  "average": {
    "hex": "#c4b8a8",
    "rgb": [196, 184, 168],
    "name": "Doeskin"
  },
  "confidence": "high",
  "region": {
    "bbox": [100, 200, 400, 600],
    "size": [200, 150],
    "totalPixels": 30000
  }
}

The confidence field indicates color precision:

  • high: Flat colors (UI elements) - clusters are tight

  • medium: Mixed content

  • low: Photographs/gradients - colors are approximate

Workflows

Hierarchical Image Understanding

The power of mcp-see is in combining tools for progressive analysis:

1. describe(image)
   → "A shelf displaying various vintage electronics and TVs"

2. detect(image, "find all screens")
   → [{label: "television", bbox: [178, 245, 433, 818]}, ...]

3. describe_region(image, [178, 245, 433, 818])
   → "A vintage CRT television with wood grain casing, displaying
      a test pattern. The screen shows horizontal color bars..."

4. analyze_colors(image, [178, 245, 433, 818])
   → dominant: ["#2b1810" Espresso Bean, "#c4a882" Sandcastle, ...]

Design Reference Analysis

Extract implementation-ready specs from design mockups:

1. describe(image, "explain this UI to a web developer")
   → Layout structure, component hierarchy, spacing patterns

2. detect(image, "find all buttons")
   → Bounding boxes for each button

3. For each button:
   - describe_region() → Button label, icon, state
   - analyze_colors() → Exact color tokens for CSS

API Keys

Quick Start: Gemini Only

For full functionality, you only need a Gemini API key:

Variable

Description

GEMINI_API_KEY

Get one from Google AI Studio

This gives you access to all tools: describe, detect, describe_region, and analyze_colors.

Tool Availability by Provider

Tool

Gemini

OpenAI

Claude

No API

describe

✅

✅

✅

describe_region

✅

✅

✅

detect

✅

❌

❌

analyze_colors

✅

  • detect (object detection with bounding boxes) requires Gemini - it's the only provider with native bounding box support

  • analyze_colors runs locally using K-Means clustering - no API key needed

All Environment Variables

Variable

Required

Description

GEMINI_API_KEY

Recommended

API key from Google AI Studio. Enables all tools.

GOOGLE_CLOUD_PROJECT

Alternative

GCP project ID for Vertex AI instead of Gemini API. Requires ADC setup (gcloud auth application-default login).

OPENAI_API_KEY

Optional

OpenAI API key for GPT-4o vision. Alternative provider for describe and describe_region.

ANTHROPIC_API_KEY

Optional

Anthropic API key for Claude vision. Alternative provider for describe and describe_region.

If both GEMINI_API_KEY and GOOGLE_CLOUD_PROJECT are set, GEMINI_API_KEY takes precedence.

Technical Details

Color Extraction Algorithm

The analyze_colors tool uses K-Means clustering in LAB color space:

  1. Convert pixels from RGB to LAB (perceptually uniform)

  2. Subsample to 50k pixels for performance

  3. K-Means++ initialization for better convergence

  4. Cluster centroids become dominant colors

  5. Convert back to RGB, name via color.pizza API

This approach groups perceptually similar colors together, working well for both flat UI colors and noisy photographs.

Bounding Box Format

All bounding boxes use [ymin, xmin, ymax, xmax] format with coordinates normalized to 0-1000. To convert to pixel coordinates:

const pixelX = (normalizedX / 1000) * imageWidth;
const pixelY = (normalizedY / 1000) * imageHeight;

License

MIT

Available Tools

4 tools
analyze_colorsA

Extract dominant colors from an image region using K-Means clustering in LAB color space. Returns colors sorted by frequency with human-readable names from color.pizza.

ParametersJSON Schema
NameRequiredDescriptionDefault
topNoNumber of dominant colors to return (default: 5)
bboxNoOptional bounding box as [ymin, xmin, ymax, xmax] normalized 0-1000. Defaults to full image.
imageYesPath to the image file or URL (http/https)

TDQS

A3.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does a fair job: it discloses the algorithm (K-Means), the color space (LAB), the ordering guarantee (sorted by frequency), and the naming source (color.pizza). It omits operational traits like cost/latency or behavior on unreadable images, but adds real value beyond what annotations would give.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences, zero filler, with the primary purpose and return contract front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, but the description compensates by explaining the return ordering and naming. Coverage of failure modes (invalid path, fetch errors) is missing, which is the main residual gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all three parameters (image, top, bbox) are already documented in the schema. The description adds nothing parameter-specific, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a precise verb+resource ('Extract dominant colors from an image region') and even names the method (K-Means in LAB). It clearly reads as distinct from 'detect' and 'describe' siblings, though it never names or contrasts them explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No when-to-use or when-not-to-use guidance, and no mention of the alternatives 'describe_region' or 'detect' that an agent would need to choose between. The only implied usage comes from the schema's default notes.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

describeC

Get an AI-generated description of an image. Supports multiple providers (Gemini, OpenAI, Claude).

ParametersJSON Schema
NameRequiredDescriptionDefault
imageYesPath to the image file or URL (http/https)
detailNoLevel of detail in the description (default: detailed)
promptNoOptional question or instruction for the description
providerNoVision provider to use (default: gemini)

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden, yet it discloses nothing beyond the feature set: no API-key/auth requirements, no cost or latency implications of calling an external vision model, no error behavior for unsupported URLs. It does name the supported providers, but that detail is already captured by the provider enum.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two compact sentences, front-loaded with the core purpose. The provider sentence is mildly redundant with the provider enum but is cheap and informative at scan time.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a four-parameter tool with full schema coverage and no output schema, the description is minimally adequate. It omits what is actually returned (a text description), whether credentials/keys are needed, and how failures surface — gaps an agent would want before invoking an external vision provider.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all four parameters are already documented with types, enums, and defaults in the schema. The description's mention of providers adds no syntax or semantic detail beyond what the schema states; baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Get an AI-generated description of an image.' Clear enough for an agent to identify it. However, it never distinguishes itself from its sibling describe_region (region-scoped) or detect, leaving the whole-image scope implicit rather than stated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use guidance, no exclusion criteria, and no mention of alternatives such as describe_region or detect. The agent must infer from the tool name alone which sibling to pick.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

describe_regionA

Crop an image to a bounding box and describe that region in detail. Use this after detect() to zoom in on specific objects.

ParametersJSON Schema
NameRequiredDescriptionDefault
bboxYesBounding box as [ymin, xmin, ymax, xmax] normalized 0-1000
imageYesPath to the image file or URL (http/https)
promptNoOptional question or instruction for the description
providerNoVision provider to use (default: gemini)

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses the crop-then-describe behavior and the detect() sequencing, but says nothing about the vision-provider dependency, latency/cost implications of provider selection, or what the response looks like. It's adequate but notably incomplete for an unannotated tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tightly written sentences: the action first, then the workflow placement. Zero waste and well front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 4-param tool with no output schema, the description conveys the essential behavior and the trigger condition, and 'describe that region in detail' implies a text return. It could briefly note the return type (a textual description) to fully close the gap without an output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with bbox format, image path, prompt, and provider enum all documented in the schema. The description adds no parameter-level detail beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('crop an image and describe that region'), clearly distinct from sibling detect() which only locates objects. It doesn't explicitly contrast with the generic describe() sibling, but the bounding-box framing makes the difference inferable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly positions the tool in a workflow: 'Use this after detect() to zoom in on specific objects.' This gives clear usage context and names the preceding step. It lacks when-not guidance (e.g., when to use plain describe instead), so it falls short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

detectA

Detect objects in an image and return bounding boxes. Uses Gemini for native bounding box support. Coordinates are normalized 0-1000 as [ymin, xmin, ymax, xmax].

ParametersJSON Schema
NameRequiredDescriptionDefault
imageYesPath to the image file or URL (http/https)
promptNoOptional: what to detect (e.g., 'find all buttons', 'detect UI elements')

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does well by disclosing the output format: normalized 0-1000 coordinates as [ymin, xmin, ymax, xmax]. It doesn't disclose error behavior (e.g., if no objects found), rate limits, or auth requirements, but the coordinate format is crucial and beyond schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three dense sentences, zero waste, front-loaded with purpose immediately followed by technology and output format. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description appropriately details the return value (bounding boxes and coordinate format). Missing are any behavioral traits like what happens when no objects are found or whether multiple objects are returned, but overall it is complete enough for an agent to invoke and interpret.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents both parameters including examples for the prompt. The description adds no parameter meaning beyond what the schema provides, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (detect) and resource (objects in an image) with the return artifact (bounding boxes). It doesn't differentiate from siblings like describe or describe_region, which also extract information from images, leaving the agent to infer when detection is preferable to description.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The mention of 'native bounding box support' via Gemini implies usage for spatial detection, but there is no explicit when-to-use guidance or comparison to siblings like describe_region (which may also locate regions). No when-not-to-use or alternatives are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 4 tool updatesv0.1.0
    • First observedanalyze_colors
    • First observeddescribe
    • First observeddescribe_region
    • First observeddetect
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A3.6/5.0

Scored across 4 tools

Disambiguation4/5

Each tool targets a distinct image analysis operation: full-image description, object detection, region-level description, and color extraction. The only slight overlap is between describe and describe_region, but the latter's description clarifies its use after detect().

Naming Consistency4/5

All names are snake_case and start with a verb, but the pattern is mixed: describe and detect are bare verbs while describe_region and analyze_colors follow verb_noun. This minor inconsistency is still readable.

Tool Count5/5

Four tools is a well-scoped set for an image analysis server, with each tool fulfilling a distinct role. The count is neither bloated nor thin.

Completeness4/5

The surface covers core image understanding: description, object detection, region zoom, and color analysis. However, notable gaps exist such as OCR/text extraction or image classification, which some users might expect from an 'mcp-see' server.

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers