Drawing Archive MCP Server
README.md
<!-- pixeltable-example-app: 20261118-drawing-archive-mcp -->
# Serve a multimodal table to agents as an MCP server in Python
**Short answer:** keep the drawings in one Pixeltable table with a `pxt.Image` column, let computed columns derive the thumbnail, size metadata, a vision-model caption and a text embedding index, then put a small MCP server in front of it whose tools just call the table (`t.search_text.similarity(...)`). Claude, Cursor or any MCP client can then search the archive. The same app also goes the other way: `pxt.mcp_udfs(url)` turns an MCP server's tools into UDFs, and `openai.invoke_tools` lets a `Questions` table answer questions by calling them. No separate object store, vector database or hand-written glue.
**Replaces:** a hand-written MCP server + S3 + a vector DB
**Use case:** [pixeltable.com/use-cases/ai-agents-mcp](https://pixeltable.com/use-cases/ai-agents-mcp)

[](https://codespaces.new/pierrebrunelle/pixeltable-drawing-archive-mcp?quickstart=1)
[](https://pixeltable.com)
[](https://pypi.org/project/pixeltable/)
[](https://github.com/pixeltable/pixeltable)
[](LICENSE)
`Sheets` holds one row per drawing sheet: the `image` (`pxt.Image`), a sheet number, title and project. Computed columns derive the discipline from the sheet-number prefix (a UDF), a 320x213 thumbnail, the image metadata and a one-sentence `gpt-4o-mini` caption (low-detail vision); an OpenAI `text-embedding-3-small` `EmbeddingIndex` over title + caption powers `find_sheets`, an `@pxt.query` function. `mcp_server.py` is an MCP server (streamable HTTP) with two tools, `search_drawings` and `get_drawing`, that read the table. `Questions` is the agent side: its `response` column calls `gpt-4o-mini` with the tools loaded by `pxt.mcp_udfs`, and `invoke_tools` runs whichever tool the model picked. One `FastAPIRouter` serves multipart sheet uploads, the searches, thumbnails and `/ask`.
[Pixeltable](https://pixeltable.com) is open-source, Python-native **multimodal AI data infrastructure**: tables, incremental computed columns, UDFs, indexes and serving in one library, running locally or on Pixeltable Cloud.
> โญ **Like this example?** Star [pixeltable/pixeltable](https://github.com/pixeltable/pixeltable) on GitHub. It helps other developers find it.
## What this example shows
- **FastAPI serving**: one `FastAPIRouter` turns tables and `@pxt.query` functions into typed REST routes (insert, update, delete, compute and query) with OpenAPI docs
- **Incremental computed columns** powered by plain Python UDFs (`@pxt.udf`)
- **B-tree indexes** declared on the model (`__indexes__`) back the lookup queries
- **Multimodal media columns** with configurable media destinations
- **Agents and tool calling**: Pixeltable functions exposed as LLM tools
- **Importable UDF module**: UDFs live in `udfs.py`; tables, queries and routes live together in `app.py` (Pixeltable resolves UDFs by module path)
- **`pixeltable.toml`** declares a local database and a **Pixeltable Cloud** database, so the same code deploys with `pxt db update`
## What's inside
| File | What it is |
|------|------------|
| `.devcontainer/devcontainer.json` | GitHub Codespaces / Dev Container config: Python 3.12, installs `requirements.txt`, forwards port 8000 |
| `.github/social-preview.png` | Social preview image (1280x640) |
| `CITATION.cff` | Citation metadata (authors, license, release date, keywords) |
| `app.py` | The app: tables declared as Python classes, `@pxt.query` functions, and the `FastAPIRouter` routes |
| `data/sheets/A-101.png` | Sample data |
| `data/sheets/A-102.png` | Sample data |
| `data/sheets/A-201.png` | Sample data |
| `data/sheets/A-202.png` | Sample data |
| `data/sheets/A-301.png` | Sample data |
| `data/sheets/A-501.png` | Sample data |
| `data/sheets/M-101.png` | Sample data |
| `data/sheets/S-101.png` | Sample data |
| `make_data.py` | Draw eight synthetic drawing sheets (floor plans, elevations, sections) with PIL into data/sheets/ |
| `mcp_server.py` | MCP server over the drawing archive: two tools that read the Pixeltable `sheets` table |
| `pixeltable.toml` | Project config: the local database plus a Pixeltable Cloud database (sizing, deploy excludes) |
| `seed.py` | Load the eight demo sheets (one gpt-4o-mini low-detail vision call + one embedding per sheet), then ask two questions |
| `udfs.py` | Pixeltable UDFs (`@pxt.udf`) in their own importable module, imported by `app.py` |
| `requirements.txt` / `pyproject.toml` | Dependencies (`pixeltable[serve]>=0.7.24`) |
**Tables**
| Table | Stored columns | Computed columns |
|-------|---------|------------------|
| `sheets` | `sheet_no`, `title`, `project`, `image` | `discipline`, `thumb`, `meta`, `vision`, `caption` |
| `questions` | `qid`, `question` | `response`, `tool_results`, `answer` |
**API routes** (service `archive`)
| Method | Path | Kind | Backed by | Notes |
|--------|------|------|-----------|-------|
| `POST` | `/sheets` | insert | `Sheets` | |
| `POST` | `/sheets/delete` | delete | `Sheets` | |
| `POST` | `/ask` | insert | `Questions` | |
| `GET` | `/search` | query | `find_sheets` | |
| `GET` | `/discipline` | query | `sheets_in` | |
| `GET` | `/sheets/thumb` | query | `sheet_thumb` | |
## Run in your browser (GitHub Codespaces)
[](https://codespaces.new/pierrebrunelle/pixeltable-drawing-archive-mcp?quickstart=1)
1. Click **Open in GitHub Codespaces** above (or [this link](https://codespaces.new/pierrebrunelle/pixeltable-drawing-archive-mcp?quickstart=1)). The dev container installs Python 3.12 and `pixeltable[serve]>=0.7.24` from `requirements.txt`.
2. In the codespace terminal, create the tables, seed them and start the API:
```bash
pxt schema update app.py archive
export OPENAI_API_KEY=... # captions (1 low-detail vision call per sheet), embeddings and the agent
python make_data.py # draws 8 demo drawing sheets
python mcp_server.py & # MCP server at http://localhost:8765/mcp (point Claude Desktop or Cursor at it)
pxt schema update app.py archive
python seed.py # 8 sheets and 2 agent questions
pxt service run app.py archive --port 8000 # open http://localhost:8000/docs
```
3. Codespaces forwards port 8000: open it from the **Ports** tab (or the pop-up) and add `/docs` to the URL for the interactive OpenAPI docs.
## Quickstart
Requires Python 3.11+ and `pixeltable[serve]>=0.7.24`.
```bash
git clone https://github.com/pierrebrunelle/pixeltable-drawing-archive-mcp.git
cd pixeltable-drawing-archive-mcp
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# Create the tables in a local catalog directory named `archive`
pxt schema update app.py archive
export OPENAI_API_KEY=... # captions (1 low-detail vision call per sheet), embeddings and the agent
python make_data.py # draws 8 demo drawing sheets
python mcp_server.py & # MCP server at http://localhost:8765/mcp (point Claude Desktop or Cursor at it)
pxt schema update app.py archive
python seed.py # 8 sheets and 2 agent questions
pxt service run app.py archive --port 8000 # open http://localhost:8000/docs
```
Try it:
```bash
curl -s 'localhost:8000/search?q=staircase&k=2'
curl -s 'localhost:8000/discipline?discipline=Structural'
curl -s -X POST localhost:8000/sheets -F sheet_no=A-601 -F title='Door schedule' -F project='Harbor Library' -F image=@data/sheets/A-501.png
curl -s -X POST localhost:8000/ask -H 'content-type: application/json' -d '{"qid":3,"question":"List the structural sheets"}'
curl -s -o thumb.jpg 'localhost:8000/sheets/thumb?sheet_no=A-601'
```
## Deploy to Pixeltable Cloud
The same `app.py` runs on [Pixeltable Cloud](https://pixeltable.com). Sign in (or get a free trial database with `pxt new`), point the second database entry in `pixeltable.toml` at your own database, then deploy:
```bash
pxt login # or: export PIXELTABLE_API_KEY=<your-api-key>
# edit pixeltable.toml: name = 'pxt://<your-org>:<your-db>'
pxt db update pxt://<your-org>:<your-db> # build the image and upload the project
pxt schema update app.py pxt://<your-org>:<your-db> # create the tables at the database root (no per-app folder)
pxt service update app.py pxt://<your-org>:<your-db> # start the API there
pxt service list pxt://<your-org>:<your-db> # list hosted services
```
Hosted routes require an API key: send it in the `X-api-key` header (for example `-H "X-api-key: $PIXELTABLE_API_KEY"`). Keep keys in environment variables or `pxt secret set`, never in code.
## Code walkthrough
**1. Business logic is plain Python, in `udfs.py`.** A `@pxt.udf` function can be used as a column expression. Pixeltable records UDFs by module path (`udfs.discipline_of`), so they live in their own importable module rather than inline in the app: the daemon, serving workers and Pixeltable Cloud import it again by that path.
```python
# udfs.py
@pxt.udf
def discipline_of(sheet_no: str) -> str:
"""Discipline name from the sheet-number prefix ('A-101' -> 'Architectural')."""
return _DISCIPLINES.get(sheet_no.strip().upper()[:1], 'Other')
```
**2. Tables are Python classes (`app.py`).** Annotated attributes are stored columns; attributes assigned an expression are **computed columns** (`discipline`, `thumb`, `meta`, `vision`, `caption`), evaluated incrementally on every insert or update and recomputed when their inputs change. Indexes live next to the columns.
```python
# app.py
class Sheets(TableModel, name='sheets'):
"""One drawing sheet."""
sheet_no = pxt.Column(type=pxt.String, primary_key=True)
title: pxt.String
project: pxt.String | None
image: pxt.Image
discipline = discipline_of(sheet_no)
thumb = image.resize([320, 213])
meta = image.get_metadata()
vision = openai.chat_completions(
vision_messages(CAPTION_PROMPT, thumb), model=CAPTION_MODEL, model_kwargs={'max_tokens': 60, 'temperature': 0}
)
caption = vision.choices[0].message.content.astype(pxt.String)
search_text = title + '. ' + caption
__indexes__ = [
pxt.EmbeddingIndex(search_text, embedding=openai.embeddings.using(model=EMBED_MODEL), name='sheet_idx'),
pxt.BtreeIndex(discipline),
]
```
**3. Queries are functions (`app.py`).** `@pxt.query` wraps a Pixeltable query so it can be called from Python or exposed as a route:
```python
# app.py
@pxt.query
def find_sheets(q: str, k: int = 3):
"""Sheets whose title and caption best match a text description."""
sim = Sheets.search_text.similarity(string=q, idx='sheet_idx')
return Sheets.order_by(sim, asc=False).limit(k).select(
sheet_no=Sheets.sheet_no, title=Sheets.title, discipline=Sheets.discipline, caption=Sheets.caption, score=sim
)
```
**4. One router, a full REST API.** `FastAPIRouter` generates request/response models from the column types, validates input, and publishes OpenAPI docs at `/docs`:
```python
# app.py
archive = FastAPIRouter(name='archive')
archive.add_insert_route(
Sheets,
path='/sheets',
inputs=[Sheets.sheet_no, Sheets.title, Sheets.project],
uploadfile_inputs=[Sheets.image],
outputs=[Sheets.sheet_no, Sheets.discipline, Sheets.caption, Sheets.meta],
)
archive.add_delete_route(Sheets, path='/sheets/delete')
archive.add_insert_route(Questions, path='/ask', inputs=[Questions.qid, Questions.question], outputs=[Questions.answer])
archive.add_query_route(path='/search', query=find_sheets, method='get')
archive.add_query_route(path='/discipline', query=sheets_in, method='get')
archive.add_query_route(path='/sheets/thumb', query=sheet_thumb, method='get', return_fileresponse=True)
```
## FAQ
**How do I serve images and documents to Claude or Cursor as an MCP server in Python?**
Store them in a Pixeltable table with media columns, add computed columns for what an agent needs (caption, embedding index), and wrap the table in an MCP server whose tools call `similarity()` and plain `where()` queries. `mcp_server.py` in this repo is about 40 lines.
**How does a Pixeltable app call tools from an MCP server?**
`pxt.mcp_udfs(url)` returns one UDF per tool; bundle them with `pxt.tools(...)`, pass them to `openai.chat_completions(tools=...)` and run the chosen tool with `openai.invoke_tools`. Give MCP tool arguments no defaults: a tool argument that has a default is treated as required, and a model that leaves it out gets a null result.
**Do I need a separate vector database?**
No. The embedding index is declared on the table (`__indexes__`) and stays in sync on insert; `similarity()` is just a query.
## Learn more
- ๐ Website: https://pixeltable.com
- ๐ Docs: https://docs.pixeltable.com
- ๐ป Source: https://github.com/pixeltable/pixeltable (โญ star it if Pixeltable is useful to you)
- ๐ฆ PyPI: https://pypi.org/project/pixeltable/
- ๐ฏ Use case: https://pixeltable.com/use-cases/ai-agents-mcp
- ๐งฉ More example apps: https://pierrebrunelle.github.io/awesome-pixeltable-apps/
**[More Pixeltable example apps โ](https://pierrebrunelle.github.io/awesome-pixeltable-apps/)**
---
<sub>Built as part of a daily series of Pixeltable example apps ยท Pixeltable 0.7.24 ยท Python, FastAPI, incremental computed columns ยท Licensed under Apache-2.0.</sub>
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues