Image Annotate MCP
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Image Annotate MCP Let me draw boxes on this screenshot and get the coordinates."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
image-annotate-mcp
Human-in-the-loop image annotation for MCP agents. The user draws a box on a screenshot; the agent gets back pixel coordinates, a name, and a cropped template image — the three things a template-matching / GUI-automation agent actually needs.

Why this exists
Existing image-annotation MCP servers all work the same direction: the agent
decides the coordinates, the server draws them (annotation-mcp,
mcp-screenshot-server, markitup-mcp-server). That is useless when the human is
the one who knows where the button is.
image-annotate-mcp flips it: the human points, the agent receives.
One drag gives you three things at once:
Output | Example |
exact pixel coordinates on the original image |
|
the semantic name the human typed |
|
a ready-to-use cropped template |
|
Irregular shapes work too: the polygon tool records a vertex list and masks the template to that outline, so a diamond-shaped skill button does not drag half the background along with it.
Related MCP server: MCP Polygon Renderer
Install
pip install image-annotate-mcp
# optional: live screen capture (otherwise pass a file path / base64)
pip install image-annotate-mcp[screen]
# or run it without installing
uvx --from image-annotate-mcp annotate-mcpRequires Python 3.10+. Hard dependencies: mcp, pillow.
Configure your client
{
"mcpServers": {
"image-annotate": {
"command": "annotate-mcp",
"args": [],
"timeout": 600000
}
}
}timeout is in milliseconds and is only understood by some clients (Claude
Code, Cursor, …). It exists because of the blocking tool below — see
Blocking, and the timeout trap.
Useful flags: --data-dir <path> (default ~/.image-annotate-mcp),
--base-port 8777, --no-open (do not launch a browser, just return the URL).
Tools
Tool | What it does |
| Open a session for one or more images, return a |
| Block until the user hits 保存; returns coordinates + names + template filenames |
| Non-blocking peek: |
| Close a session and its web server; wakes anyone waiting |
| Annotations already on disk, newest first |
| Full structured payload of one saved annotation |
|
|
| The numbered overlay as image content, so a multimodal agent can read the labels itself |
Typical flow
annotate_image(source="screen", name="board_ui")
→ {"session": "s1", "url": "http://127.0.0.1:8777/?s=s1", ...}
... the human draws two boxes and one point, types names, clicks 保存 ...
annotate_wait(session="s1", timeout_s=240)
→ {"status": "saved", "rev": 1, "images": [{"items": [
{"index": 1, "name": "晶矿", "type": "point", "point": [1614, 398]},
{"index": 2, "name": "出战按钮", "type": "box", "box": [1671, 713, 240, 70],
"template": "board_ui__img1__chuzhan.png"}]}]}
annotate_export(name="board_ui", fmt="py")
→ BOARD_UI_IMG1_CHUZHAN = (1671, 713, 240, 70) # 出战按钮 · x, y, w, hThe annotation page
Single self-contained HTML file: no CDN, no external fonts, no ES modules. It works
offline and is served from 127.0.0.1 only.
框选 (
1) drag a rectangle → becomes a template crop点选 (
2) single click → becomes a click coordinate多边形 (
3) click each vertex of an irregular shape; close it by clicking the first vertex, double-clicking, orEnter.Backspacedrops the last vertex,Escabandons the outline.selected boxes can be moved and resized with 8 handles; polygon vertices can be dragged one at a time
hidden layers: the eye button on a row hides that single item,
Hhides all of them so you can see the screenshot underneath. Hidden items are still saved, still get a template, and are left out offmt="png"output.wheel zoom (on by default): zooms around the cursor, 1x-16x. Stroke widths and label sizes stay constant on screen instead of growing with the image — you zoom in to read pixels, not to inflate the annotation boxes. Hold
Spaceand drag, or drag with the middle button, to pan;Ctrl+0resets. The small bar in the bottom-left can turn wheel zoom off and reset the viewloupe (off by default,
Ltoggles): a draggable floating window showing an N x N image pixel region around the cursor, magnified (N = 40 / 60 / 80 / 120), with the exact pixel under the cursor outlined. Drag it elsewhere to pin it, click the button in its title bar to resume following. It magnifies original pixels, independent of the current zoom levelreal undo / redo (
Ctrl+Z/Ctrl+Shift+Z),Deleteremoves the selectionmultiple images in one session → tab strip at the top
coordinates are always in original image pixels, and the canvas honours
devicePixelRatioso HiDPI screens do not shift them by half a pixel. Zoom and pan only change the view, never that mapping: clicking a pixel at 8x saves the same coordinate as clicking it at 1x (tests/manual/e2e_view.pyguards this)
Data model
{
"schema": 2,
"name": "board_ui",
"created_at": "2026-11-15 16:32:04",
"saved_at": "2026-11-15 16:40:11",
"rev": 3,
"images": [
{
"id": "img1",
"name": "主界面",
"source": "file:/path/a.png",
"size": [1920, 1080],
"snapshot": "board_ui__img1.png",
"items": [
{"index": 1, "name": "晶矿", "type": "point", "point": [1614, 398], "note": ""},
{"index": 2, "name": "出战按钮", "type": "box", "box": [1671, 713, 240, 70],
"note": "", "template": "board_ui__img1__chuzhan.png"},
{"index": 3, "name": "技能环", "type": "polygon",
"points": [[820, 300], [1010, 340], [960, 520], [790, 470]],
"bbox": [790, 300, 220, 220], "hidden": true,
"note": "", "template": "board_ui__img1__jineng.png"}
]
}
]
}bbox is the polygon's vertex bounding rectangle in the same [x, y, w, h]
shape as a box, so a consumer that only understands boxes still has something
to work with. hidden is present only when the item is hidden.
Files written to --data-dir (default ~/.image-annotate-mcp):
annotations/<slug>.json structured, source of truth
annotations/<slug>.md markdown table, quicker for an agent to skim
templates/<slug>__<img>__<item>.png one crop per item (polygons masked, RGBA)
sources/<slug>__<img>.png copy of the base image, so exports survive deletion
exports/ annotated overlays and generated constantsTemplate filenames are ASCII-only (item1, chuzhan, …) because they end up
inside generated code; the original name — Chinese included — is kept in the JSON,
the Markdown and the generated comments.
Blocking, and the timeout trap
annotate_wait is a human-in-the-loop tool: it deliberately blocks while a
person draws. Two consequences:
A blocking call is not a hang. The page shows 未保存 / 已保存 so the human can see the state; the agent sees nothing until the save lands.
MCP clients have their own tool timeout, often 60 s. If the client kills the call first you get something like
MCP error -32001: Request timed out.
So the server default is timeout_s=240 and a timeout is not an error:
{"status": "pending", "rev": 0, "elapsed_s": 240.0,
"hint": "用户还没点保存(已等 240.0s)。可以再调一次 annotate_wait 继续等…"}The rule is: server timeout < client timeout, and on pending just call
annotate_wait again (with since_rev set to the rev you already have). If your
client supports a timeout field, raise it well above 240 s.
Screen capture
source="screen" (or omitting source) grabs the primary monitor; "screen:0"
grabs the whole virtual desktop, "screen:2" a specific monitor.
mssis used when installed (pip install image-annotate-mcp[screen]) — best multi-monitor behaviour.Otherwise
PIL.ImageGrabis tried, which works on Windows and macOS but usually fails on Linux (no display server helper). The error tells you what to install.Without any backend,
annotate_imagestill accepts file paths, data URLs, raw base64 andhttp(s)URLs.
The image is not resampled by default, so coordinates match the original file
exactly. resize_to=[w, h] is opt-in and the result reports both sizes.
Troubleshooting
Symptom | Cause / fix |
Page loads but | You are looking at a stale tab. Use the exact URL from |
| The port belongs to a different session id. Re-open the URL you were given. |
| Pass |
| A local proxy (Clash etc.) is intercepting loopback. Use |
| Client timeout fired. Raise the client |
Template files look stale | Re-saving the same name overwrites the JSON/MD and the templates of items that still exist; templates of deleted items are left on disk on purpose. |
A box is clamped to the image edge | Expected: boxes are clipped to the image, and the |
Saving fails with | The polygon's vertices are all collinear, so its crop would be fully transparent. Re-click the outline, or use 框选 instead. |
Development
python -m venv .venv
.venv/Scripts/activate # Windows; source .venv/bin/activate elsewhere
pip install -e ".[dev]"
pytest # 73 tests
ruff check . && ruff format --check .What the tests cover: coordinate clamping and crop correctness (the crop must be the
exact pixels under the box), EXIF rotation, the blocking handshake and its timeout
path, since_rev, cancel-wakes-waiter, port roll-over, atomic save (an agent must
never see items without template), file layout, v1 compatibility, all four export
formats, every HTTP endpoint, polygon vertex clamping + masked crops + the
refusal of zero-area outlines, the hidden flag, and a real JSON-RPC round trip
over stdio.
Two manual checks need a browser and are not part of pytest:
python tests/manual/e2e.py --two # prints a URL, waits for the save
node tests/manual/verify_page.mjs <url> # drives Edge, compares aimed vs saved coordinates
python tests/manual/e2e_polygon.py # polygon + hidden layers, 22 checks, self-contained
node tests/manual/make_gif.mjs <url> && python tests/manual/make_gif.pyLicense
MIT. See LICENSE.
Available Tools
8 toolsannotate_cancelA
Close a session and its local web server. Anyone blocked in annotate_wait wakes up as cancelled.
Inputs: session (required).
annotate_exportA
Export a saved annotation. fmt='png' renders the base image with numbered outlines (for a multimodal agent to look at), 'py' emits coordinate constants plus a template map, 'json' / 'yaml' dump the payload. Returns file paths and, for 'py', the generated source.
Inputs: name (required), fmt, out_dir, image_id.
annotate_imageA
Open an annotation session for one or more images and return a localhost URL. The user opens that URL and draws boxes (templates), points (click targets) or free-form polygons joined by straight segments, names them, then hits 保存. Then call annotate_wait to collect the result.
source: local file path | data URL | raw base64 | 'screen' (live screen capture) | a list of any of those for a multi-image session. Omit it to capture the screen. resize_to: optional [width, height]; by default the image is NOT resampled, so coordinates always match the original file.
Inputs: name, source, resize_to, open_browser.
annotate_listB
List annotations saved on disk, newest first.
Inputs: limit.
annotate_loadA
Load one saved annotation by name (or file path) and return the full structured payload: schema 2 with images[] -> items[] and template filenames.
Inputs: name (required).
annotate_renderA
Render the numbered annotation picture for one saved annotation and return it as image content, so you can read the labels yourself instead of asking the user to type names. Needs annotate_export(fmt='png') to have been run first, or it will render on the fly.
Inputs: name (required), image_id.
annotate_statusB
Non-blocking status of one session: pending / saved / cancelled, current rev, how many items so far, and how long it has been open.
Inputs: session (required).
annotate_waitA
Block until the user saves the annotation (or the timeout fires), then return the annotations: per image, a list of {index, name, type, box|point|points, template, hidden?}.
A status of 'pending' means the user has not saved yet -- that is NOT an error, just call annotate_wait again. Pass since_rev to only wait for saves newer than what you already have. Keep timeout_s below your MCP client's own tool timeout.
Inputs: session (required), since_rev, timeout_s.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 8 tools
Each tool maps to a distinct phase of the annotation lifecycle: open session, block, poll, cancel, list, load, export, render. The only real overlap is annotate_export(fmt='png') versus annotate_render, both of which produce the numbered outline image, and the descriptions explicitly note the render-on-the-fly fallback, so the boundary is workable though slightly blurry.
All eight tools share a consistent snake_case 'annotate_' prefix, which makes the set easy to scan. Minor deviation: some suffixes are verbs (load, wait, cancel, list, export, render) while others are nouns (image, status), so the verb_noun pattern isn't perfectly uniform.
Eight tools is well-scoped for an interactive annotation server: session open/wait/status/cancel, persistence via list/load, and output via export/render. No tool looks redundant or bolted on.
The session lifecycle and persistence read path are fully covered, including a non-blocking status escape hatch and multiple export formats. Gaps are minor: there is no delete, rename, or update operation for saved annotations, so housekeeping of accumulated files must happen outside the tool surface.
Maintenance
Related MCP Connectors
Agent-Native design tool - create and edit visual designs with agent assistance
Design canvas for agents: create, edit, render and export Figma-shaped designs over MCP.
PDF, image, video, OCR, screenshot, SQL, QR and text tools for agents. No API key, no signup.
A real browser for your agent: screenshots, page checks, Markdown and PDFs.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables drawing annotations directly on local web pages and sending screenshots, DOM data, and annotation context to your IDE agent for precise code edits.49 npm3MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to overlay labeled grids, define image regions by grid cells or pixel coordinates, and render polygons or automatically detected regions using OpenCV, returning images for visualization.MIT
- AlicenseNot gradedqualityBmaintenanceEnables users to capture any screen or URL, box the region they mean, attach notes, and deliver that annotated feedback—including cropped images, numbered boxes, and the underlying DOM elements or component names—directly to an AI coding agent over MCP. It also lets agents open the annotator, capture a URL, and wait for or retrieve feedback bundles from earlier sessions.MIT
- AlicenseAqualityBmaintenanceEnables AI assistants such as Claude to list and upload images on an image annotation platform, add, replace, or remove pins, polygons, circles, rectangles, and arrows, edit image metadata, and export annotations as COCO JSON or YOLO training data. It can run over stdio or Streamable HTTP so multiple clients can share one server, with an included LangGraph terminal agent.23167 PyPIMIT