# nvidia-vision-mcp

Give any **MCP-compatible coding agent** the ability to actually look at images — UI screenshots,
simulator captures, design references, diagrams, screenshots of errors — through **NVIDIA NIM**
vision models.

Works with ZCode, Claude Code, Cursor and any other MCP client, over local **stdio** or a
self-hosted **Streamable HTTP** deployment. One tool, one job: `analyze_image`.

## Why this exists

Most coding agents are text-only. When you are implementing a UI, that is a real limitation: you can
describe a screenshot in prose, or paste hex values by hand, but the agent cannot actually see the
thing it is building. This server closes that gap without switching to a different agent — you point
it at an image, and it returns a textual analysis the agent can act on.

It is deliberately small. No dashboard, no database, no cache layer, no agent framework. Two runtime
dependencies: the MCP SDK and Zod.

## Architecture

```text
MCP client  →  nvidia-vision-mcp  →  NVIDIA NIM vision model  →  analysis result
(ZCode, Claude       (stdio or          (minimaxai/minimax-m3      (text, back through
 Code, Cursor)        HTTP)              by default)                the same path)
```

Both transports share one code path. `src/server.ts` builds the MCP server and owns the vision
logic; `src/index.ts` connects it to stdio and `src/http.ts` serves it over Streamable HTTP. Nothing
is duplicated between the two modes.

## Requirements

- Node.js ≥ 20.6
- Your own NVIDIA API key from <https://build.nvidia.com/> (free tier available, starts with `nvapi-`)

**You must supply your own key.** This project is not a hosted service — there is no shared endpoint,
and nothing here calls NVIDIA on your behalf. Every user runs their own instance with their own key
and their own quota.

## Quick start (local)

```bash
git clone https://github.com/cyril-belin/nvidia-vision-mcp.git
cd nvidia-vision-mcp
npm install
cp .env.example .env       # then put your NVIDIA_API_KEY in .env
npm run build
```

Verify it works before wiring it into a client:

```bash
npm test
```

Then add it to your MCP client. Full walkthrough: **[docs/local-stdio.md](docs/local-stdio.md)**.

### ZCode

ZCode reads `~/.zcode/cli/config.json` and nests servers under `mcp.servers` — note that this is
**not** the flat `mcpServers` key most other clients use:

```json
{
  "mcp": {
    "servers": {
      "nvidia-vision": {
        "type": "stdio",
        "command": "/usr/local/bin/node",
        "args": ["/absolute/path/to/nvidia-vision-mcp/dist/index.js"],
        "timeoutMs": 300000
      }
    }
  }
}
```

Use an absolute `command` path (`which node`) — a GUI-launched ZCode does not inherit your shell
`PATH`. Keep `timeoutMs` high; a detailed analysis takes 30–60 s.

### Claude Code, Cursor, and other clients

```json
{
  "mcpServers": {
    "nvidia-vision": {
      "command": "node",
      "args": ["/absolute/path/to/nvidia-vision-mcp/dist/index.js"]
    }
  }
}
```

Ready-to-edit copies of both are in [`examples/`](examples/).

## Self-hosted remote deployment

If you want the server reachable over HTTP instead of running locally, it also speaks **Streamable
HTTP** — deploy your own instance to Vercel, Railway, or any container host, using your own NVIDIA
key.

```bash
npm run start:http    # listens on $PORT (default 3000), endpoint /mcp
```

A remote instance cannot see your laptop's disk, so send the image itself: use **`image_url`** for a
publicly reachable image, or **`image_base64`** to upload the bytes directly in the call. Both work
with images that exist only on the client machine. `image_path` still works remotely, but refers to
the *server's* filesystem.

**Railway** (recommended — always-on, no request duration cap): push your fork, create a project from
the repo, and Railway builds the included [`Dockerfile`](Dockerfile) automatically. Set
`NVIDIA_API_KEY` and `MCP_AUTH_TOKEN` under Variables, generate a domain, and your endpoint is
`https://your-app.up.railway.app/mcp`. If you deploy without Docker, set the start command to
`node dist/http.js` — the default `npm start` runs the stdio entry point and will exit immediately.

**Vercel**: import the repo and accept the detected settings; [`vercel.json`](vercel.json) already
configures the build and the `api/mcp.js` function. Add `NVIDIA_API_KEY` and `MCP_AUTH_TOKEN` under
Environment Variables. Two caveats — functions are capped at 60 s on Hobby (a slow analysis can be cut
off), and **Vercel limits request bodies to 4.5 MB**, which caps `image_base64` at roughly a 3.3 MB
image once base64 inflation is counted. For large screenshots over `image_base64`, prefer Railway or
another container host, or use `image_url` instead.

Full guide, including Docker and client configuration:
**[docs/remote-deployment.md](docs/remote-deployment.md)**.

## The tool

### `analyze_image`

| Argument       | Type   | Required        | Description                                                                |
| -------------- | ------ | --------------- | -------------------------------------------------------------------------- |
| `prompt`       | string | **yes**         | Any analysis instruction. Nothing about the prompt is hardcoded.            |
| `image_path`   | string | one source      | Path to an image on **the machine running the server**. Absolute, `~/…`, or relative to its cwd. |
| `image_url`    | string | one source      | Public `http`/`https` URL of an image to download.                          |
| `image_base64` | string | one source      | Base64 image data, with or without a `data:image/png;base64,` prefix.       |
| `image_mime`   | string | no              | Advisory MIME hint for `image_base64`. The real format is always sniffed.   |
| `model`        | string | no              | NVIDIA model id override for this single call.                              |

**Exactly one** of `image_path`, `image_url` or `image_base64` must be supplied. Zero sources or two
or more are rejected with a clear message.

Which one to use:

- **Local (stdio):** `image_path` — the server runs on your machine, so it can read your files.
- **Remote (HTTP):** `image_url` or `image_base64` — the server is elsewhere and cannot see your
  disk. `image_path` still works remotely but refers to the *server's* filesystem.

Returns the model's textual analysis. Failures come back as MCP tool errors (`isError: true`) with an
actionable message.

**Supported formats:** PNG, JPEG, GIF, WEBP. The format is detected from magic bytes rather than the
file extension, so a mislabelled screenshot still works and a renamed non-image is rejected before it
reaches the API.

Example prompts:

- "Analyze this UI screenshot in detail for Flutter implementation."
- "Compare this simulator screenshot with the reference design."
- "Identify layout, spacing, typography, colors, shadows and visual differences."
- "Inspect this screenshot and tell the coding agent exactly what needs to change."

## Default model

The default is **`minimaxai/minimax-m3`**, the most accurate image-capable model available on
`integrate.api.nvidia.com` at the time of writing.

It is fully configurable — set `NVIDIA_VISION_MODEL`, or pass `model` on a single call. Other
verified image-capable options:

| Model                            | Character                                            |
| -------------------------------- | ---------------------------------------------------- |
| `minimaxai/minimax-m3`           | Best detail and color accuracy. ~15–40 s. **Default** |
| `nvidia/nemotron-nano-12b-v2-vl` | Faster and lighter, less precise on exact colors.     |
| `stepfun-ai/step-3.7-flash`      | Reasoning model — needs a large `NVIDIA_MAX_TOKENS`.  |

Browse the catalog at <https://build.nvidia.com/explore/vision>. Model availability varies by
account, and NVIDIA retires models periodically, so if a model starts returning 404 or 410 just point
`NVIDIA_VISION_MODEL` at a current one — no code change needed.

## Rate limits

NVIDIA's free endpoints throttle, and an agent analyzing several screenshots will fire calls in
parallel — the quickest way to get a 429. This is handled for you:

- **Requests are queued.** At most one NVIDIA request is in flight per instance, regardless of how
  many calls arrive at once. Image validation still happens immediately, so a bad path fails fast
  instead of waiting behind a slow analysis.
- **429s are retried automatically**, up to `NVIDIA_MAX_ATTEMPTS` (3 by default).
- **`Retry-After` is honoured** when NVIDIA sends it, in both the seconds and HTTP-date forms.
- **Otherwise it backs off exponentially** — 2 s, then 4 s, with full jitter.
- **Retries log to stderr only**, so the stdio protocol stream is never corrupted.

Two deliberate limits: a `Retry-After` longer than `NVIDIA_RETRY_MAX_DELAY_MS` (60 s) returns the
error immediately rather than stalling the queue behind it, and only 429 is retried — a 403 or 404
would return the same answer on the second attempt.

## Configuration

All configuration is environment variables, read from `.env` in the project directory or from the
client's `env` block.

| Variable                    | Default                                                | Notes                              |
| --------------------------- | ------------------------------------------------------ | ---------------------------------- |
| `NVIDIA_API_KEY`            | —                                                      | **Required.**                      |
| `NVIDIA_VISION_MODEL`       | `minimaxai/minimax-m3`                                 | Any image-capable NIM model.       |
| `NVIDIA_BASE_URL`           | `https://integrate.api.nvidia.com/v1/chat/completions` | Point at a self-hosted NIM if any. |
| `NVIDIA_TIMEOUT_MS`         | `180000`                                               | Per-request timeout.               |
| `NVIDIA_MAX_TOKENS`         | `4096`                                                 | Response budget.                   |
| `NVIDIA_TEMPERATURE`        | `0.2`                                                  | Low = more literal descriptions.   |
| `NVIDIA_MAX_IMAGE_BYTES`    | `20971520` (20 MB)                                     | Applies to all three image sources. |
| `NVIDIA_DOWNLOAD_TIMEOUT_MS`| `30000`                                                | Budget for fetching `image_url`.   |
| `NVIDIA_ALLOW_PRIVATE_URLS` | `false`                                                | Allow `image_url` to reach private/loopback addresses. Leave off unless you need it. |
| `NVIDIA_MAX_ATTEMPTS`       | `3`                                                    | Total tries per 429. `1` disables. |
| `NVIDIA_RETRY_BASE_MS`      | `2000`                                                 | First backoff, doubles per retry.  |
| `NVIDIA_RETRY_MAX_DELAY_MS` | `60000`                                                | Ceiling on any single wait.        |

HTTP mode only:

| Variable         | Default | Notes                                                          |
| ---------------- | ------- | -------------------------------------------------------------- |
| `PORT`           | `3000`  | Set automatically by most platforms.                            |
| `HOST`           | `0.0.0.0` | Bind address.                                                 |
| `MCP_PATH`       | `/mcp`  | Endpoint path.                                                  |
| `MCP_AUTH_TOKEN` | unset   | Bearer token. **Set this before exposing the server publicly.** |

## Security

- **Your API key stays yours.** It is read from your environment at runtime and sent only to NVIDIA.
  Nothing is proxied through a third party, and the project has no telemetry.
- **Never commit your key.** `.env` is gitignored; `.env.example` holds placeholders only. Use your
  platform's secret store for deployments rather than baking keys into an image or `vercel.json`.
- **Protect remote deployments.** An unauthenticated public endpoint lets anyone spend your NVIDIA
  quota. Set `MCP_AUTH_TOKEN`; the server warns at startup if you have not.
- **`image_path` reads the server's filesystem.** Under a remote deployment, anyone who can reach the
  endpoint can ask it to read image files the process has access to. This is another reason to set
  `MCP_AUTH_TOKEN`.
- **`image_url` is guarded against SSRF.** Only `http`/`https` are accepted, and hosts resolving to
  loopback, private, link-local (including cloud metadata at `169.254.169.254`), CGNAT or unique-local
  addresses are blocked by default. Redirects are followed manually, and every hop is re-validated so
  a public URL cannot bounce into your internal network. Set `NVIDIA_ALLOW_PRIVATE_URLS=true` only if
  you intentionally need internal URLs.
- **Downloads and payloads are capped.** `NVIDIA_MAX_IMAGE_BYTES` is enforced on `image_url`
  downloads (aborted mid-stream if a server omits or lies about `Content-Length`) and on
  `image_base64` (checked before decoding, so an oversized payload is never allocated).
- **Image contents are never logged.** Only sizes and redacted URLs (origin plus path, query string
  stripped) appear in diagnostics.
- If a key is ever exposed, rotate it at <https://build.nvidia.com/>.

## Testing

Everything that does not need the network — protocol, validation, rate limiting, and the HTTP
transport. No API key required:

```bash
npm test
```

Individual suites:

```bash
npm run test:ratelimit
```

```bash
npm run test:http
```

```bash
npm run test:sources
```

A real end-to-end call against NVIDIA (needs a valid `NVIDIA_API_KEY`):

```bash
npm run smoke
```

### Current test status

| Suite                      | Checks | Needs a key | Covers                                                                     |
| -------------------------- | ------ | ----------- | -------------------------------------------------------------------------- |
| `smoke-test.mjs --offline` | 12     | no          | stdio handshake, `tools/list`, schema, error paths                          |
| `rate-limit-test.mjs`      | 32     | no          | queueing, 429 retries, `Retry-After`, backoff, exhaustion                    |
| `http-test.mjs`            | 17     | no          | Streamable HTTP transport, shared queue, bearer auth, health                 |
| `image-source-test.mjs`    | 39     | no          | all three image sources, exactly-one rule, SSRF guard, size/timeout/format limits |
| `smoke-test.mjs` (full)    | 15     | **yes**     | all stdio checks plus a real NVIDIA vision call                              |

**100 checks pass without a key**; the full smoke test passes against the live API. The tests drive
the server as a real MCP client over its real transports, and use local mocks of the NVIDIA endpoint
and of an image host where a live call would be slow or costly.

## Troubleshooting

**The server does not appear in my client.**
Check that you pointed it at `dist/index.js` (not `src/index.ts`) and that you ran `npm run build`.
Use an absolute path for both the node binary and the script.

**`NVIDIA_API_KEY is not set`.**
The server starts and lists tools even without a key, so you get this as a tool error rather than a
crash. Confirm `.env` exists in the project root and contains the key, or set it in the client's
`env` block.

**403 Forbidden.**
NVIDIA returns 403 for both an invalid key and a valid key lacking entitlement for that model.
Verify the key at <https://build.nvidia.com/>, then try the default model.

**404 or 410 for the model.**
The model id is wrong, unavailable to your account, or retired. Pick a current one from
<https://build.nvidia.com/explore/vision> and set `NVIDIA_VISION_MODEL`.

**Requests time out in the client.**
Raise the client's own timeout — ZCode's `timeoutMs`, for example. A detailed analysis legitimately
takes 30–60 s, which is longer than many default client timeouts.

**Fine print in a screenshot is misread.**
NVIDIA downscales images server-side, so text that is small relative to a full 3x screenshot can be
lost. Crop to the region of interest and send that instead — it is also faster.

**Repeated 429s.**
You are hitting your quota rather than a burst limit. Retries will not help; wait, or check your
account at <https://build.nvidia.com/>.

## Project layout

```text
├── src/
│   ├── server.ts       MCP server + analyze_image tool (shared by both transports)
│   ├── index.ts        local entry point — stdio
│   ├── http.ts         remote entry point — Streamable HTTP
│   ├── config.ts       environment / .env handling
│   ├── image.ts        local file resolution, validation, MIME sniffing, base64
│   ├── sources.ts      image_url download + image_base64 decode, exactly-one-source rule
│   ├── nvidia.ts       NIM request, 429 retry/backoff, HTTP error mapping
│   └── queue.ts        serial queue — one NVIDIA request at a time
├── api/mcp.js          Vercel serverless adapter (delegates to src/http.ts)
├── docs/
│   ├── local-stdio.md
│   └── remote-deployment.md
├── examples/           sample image + client configuration files
└── scripts/            test suites
```

## License

MIT — see [LICENSE](LICENSE).