Skip to main content
Glama
Drawing Archive MCP Server

Drawing Archive MCP Server

README.md
<!-- pixeltable-example-app: 20261118-drawing-archive-mcp -->
# Serve a multimodal table to agents as an MCP server in Python

**Short answer:** keep the drawings in one Pixeltable table with a `pxt.Image` column, let computed columns derive the thumbnail, size metadata, a vision-model caption and a text embedding index, then put a small MCP server in front of it whose tools just call the table (`t.search_text.similarity(...)`). Claude, Cursor or any MCP client can then search the archive. The same app also goes the other way: `pxt.mcp_udfs(url)` turns an MCP server's tools into UDFs, and `openai.invoke_tools` lets a `Questions` table answer questions by calling them. No separate object store, vector database or hand-written glue.

**Replaces:** a hand-written MCP server + S3 + a vector DB

**Use case:** [pixeltable.com/use-cases/ai-agents-mcp](https://pixeltable.com/use-cases/ai-agents-mcp)

![architecture-firm drawing archive API built with Pixeltable](.github/social-preview.png)

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/pierrebrunelle/pixeltable-drawing-archive-mcp?quickstart=1)
[![Built with Pixeltable](https://img.shields.io/badge/built%20with-Pixeltable-5b4bff)](https://pixeltable.com)
[![PyPI - pixeltable](https://img.shields.io/pypi/v/pixeltable?label=pixeltable)](https://pypi.org/project/pixeltable/)
[![GitHub stars](https://img.shields.io/github/stars/pixeltable/pixeltable?style=social)](https://github.com/pixeltable/pixeltable)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)

`Sheets` holds one row per drawing sheet: the `image` (`pxt.Image`), a sheet number, title and project. Computed columns derive the discipline from the sheet-number prefix (a UDF), a 320x213 thumbnail, the image metadata and a one-sentence `gpt-4o-mini` caption (low-detail vision); an OpenAI `text-embedding-3-small` `EmbeddingIndex` over title + caption powers `find_sheets`, an `@pxt.query` function. `mcp_server.py` is an MCP server (streamable HTTP) with two tools, `search_drawings` and `get_drawing`, that read the table. `Questions` is the agent side: its `response` column calls `gpt-4o-mini` with the tools loaded by `pxt.mcp_udfs`, and `invoke_tools` runs whichever tool the model picked. One `FastAPIRouter` serves multipart sheet uploads, the searches, thumbnails and `/ask`.

[Pixeltable](https://pixeltable.com) is open-source, Python-native **multimodal AI data infrastructure**: tables, incremental computed columns, UDFs, indexes and serving in one library, running locally or on Pixeltable Cloud.

> โญ **Like this example?** Star [pixeltable/pixeltable](https://github.com/pixeltable/pixeltable) on GitHub. It helps other developers find it.

## What this example shows

- **FastAPI serving**: one `FastAPIRouter` turns tables and `@pxt.query` functions into typed REST routes (insert, update, delete, compute and query) with OpenAPI docs
- **Incremental computed columns** powered by plain Python UDFs (`@pxt.udf`)
- **B-tree indexes** declared on the model (`__indexes__`) back the lookup queries
- **Multimodal media columns** with configurable media destinations
- **Agents and tool calling**: Pixeltable functions exposed as LLM tools
- **Importable UDF module**: UDFs live in `udfs.py`; tables, queries and routes live together in `app.py` (Pixeltable resolves UDFs by module path)
- **`pixeltable.toml`** declares a local database and a **Pixeltable Cloud** database, so the same code deploys with `pxt db update`



## What's inside

| File | What it is |
|------|------------|
| `.devcontainer/devcontainer.json` | GitHub Codespaces / Dev Container config: Python 3.12, installs `requirements.txt`, forwards port 8000 |
| `.github/social-preview.png` | Social preview image (1280x640) |
| `CITATION.cff` | Citation metadata (authors, license, release date, keywords) |
| `app.py` | The app: tables declared as Python classes, `@pxt.query` functions, and the `FastAPIRouter` routes |
| `data/sheets/A-101.png` | Sample data |
| `data/sheets/A-102.png` | Sample data |
| `data/sheets/A-201.png` | Sample data |
| `data/sheets/A-202.png` | Sample data |
| `data/sheets/A-301.png` | Sample data |
| `data/sheets/A-501.png` | Sample data |
| `data/sheets/M-101.png` | Sample data |
| `data/sheets/S-101.png` | Sample data |
| `make_data.py` | Draw eight synthetic drawing sheets (floor plans, elevations, sections) with PIL into data/sheets/ |
| `mcp_server.py` | MCP server over the drawing archive: two tools that read the Pixeltable `sheets` table |
| `pixeltable.toml` | Project config: the local database plus a Pixeltable Cloud database (sizing, deploy excludes) |
| `seed.py` | Load the eight demo sheets (one gpt-4o-mini low-detail vision call + one embedding per sheet), then ask two questions |
| `udfs.py` | Pixeltable UDFs (`@pxt.udf`) in their own importable module, imported by `app.py` |
| `requirements.txt` / `pyproject.toml` | Dependencies (`pixeltable[serve]>=0.7.24`) |

**Tables**

| Table | Stored columns | Computed columns |
|-------|---------|------------------|
| `sheets` | `sheet_no`, `title`, `project`, `image` | `discipline`, `thumb`, `meta`, `vision`, `caption` |
| `questions` | `qid`, `question` | `response`, `tool_results`, `answer` |

**API routes** (service `archive`)

| Method | Path | Kind | Backed by | Notes |
|--------|------|------|-----------|-------|
| `POST` | `/sheets` | insert | `Sheets` |  |
| `POST` | `/sheets/delete` | delete | `Sheets` |  |
| `POST` | `/ask` | insert | `Questions` |  |
| `GET` | `/search` | query | `find_sheets` |  |
| `GET` | `/discipline` | query | `sheets_in` |  |
| `GET` | `/sheets/thumb` | query | `sheet_thumb` |  |

## Run in your browser (GitHub Codespaces)

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/pierrebrunelle/pixeltable-drawing-archive-mcp?quickstart=1)

1. Click **Open in GitHub Codespaces** above (or [this link](https://codespaces.new/pierrebrunelle/pixeltable-drawing-archive-mcp?quickstart=1)). The dev container installs Python 3.12 and `pixeltable[serve]>=0.7.24` from `requirements.txt`.
2. In the codespace terminal, create the tables, seed them and start the API:

   ```bash
   pxt schema update app.py archive
   export OPENAI_API_KEY=...   # captions (1 low-detail vision call per sheet), embeddings and the agent
   python make_data.py          # draws 8 demo drawing sheets
   python mcp_server.py &       # MCP server at http://localhost:8765/mcp (point Claude Desktop or Cursor at it)
   pxt schema update app.py archive
   python seed.py               # 8 sheets and 2 agent questions
   pxt service run app.py archive --port 8000   # open http://localhost:8000/docs
   ```

3. Codespaces forwards port 8000: open it from the **Ports** tab (or the pop-up) and add `/docs` to the URL for the interactive OpenAPI docs.

## Quickstart

Requires Python 3.11+ and `pixeltable[serve]>=0.7.24`.

```bash
git clone https://github.com/pierrebrunelle/pixeltable-drawing-archive-mcp.git
cd pixeltable-drawing-archive-mcp
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Create the tables in a local catalog directory named `archive`
pxt schema update app.py archive

export OPENAI_API_KEY=...   # captions (1 low-detail vision call per sheet), embeddings and the agent
python make_data.py          # draws 8 demo drawing sheets
python mcp_server.py &       # MCP server at http://localhost:8765/mcp (point Claude Desktop or Cursor at it)
pxt schema update app.py archive
python seed.py               # 8 sheets and 2 agent questions
pxt service run app.py archive --port 8000   # open http://localhost:8000/docs
```

Try it:

```bash
curl -s 'localhost:8000/search?q=staircase&k=2'
curl -s 'localhost:8000/discipline?discipline=Structural'
curl -s -X POST localhost:8000/sheets -F sheet_no=A-601 -F title='Door schedule' -F project='Harbor Library' -F image=@data/sheets/A-501.png
curl -s -X POST localhost:8000/ask -H 'content-type: application/json' -d '{"qid":3,"question":"List the structural sheets"}'
curl -s -o thumb.jpg 'localhost:8000/sheets/thumb?sheet_no=A-601'
```

## Deploy to Pixeltable Cloud

The same `app.py` runs on [Pixeltable Cloud](https://pixeltable.com). Sign in (or get a free trial database with `pxt new`), point the second database entry in `pixeltable.toml` at your own database, then deploy:

```bash
pxt login                       # or: export PIXELTABLE_API_KEY=<your-api-key>
# edit pixeltable.toml: name = 'pxt://<your-org>:<your-db>'
pxt db update pxt://<your-org>:<your-db>                 # build the image and upload the project
pxt schema update app.py pxt://<your-org>:<your-db>       # create the tables at the database root (no per-app folder)
pxt service update app.py pxt://<your-org>:<your-db>      # start the API there
pxt service list pxt://<your-org>:<your-db>              # list hosted services
```

Hosted routes require an API key: send it in the `X-api-key` header (for example `-H "X-api-key: $PIXELTABLE_API_KEY"`). Keep keys in environment variables or `pxt secret set`, never in code.

## Code walkthrough

**1. Business logic is plain Python, in `udfs.py`.** A `@pxt.udf` function can be used as a column expression. Pixeltable records UDFs by module path (`udfs.discipline_of`), so they live in their own importable module rather than inline in the app: the daemon, serving workers and Pixeltable Cloud import it again by that path.

```python
# udfs.py
@pxt.udf
def discipline_of(sheet_no: str) -> str:
    """Discipline name from the sheet-number prefix ('A-101' -> 'Architectural')."""
    return _DISCIPLINES.get(sheet_no.strip().upper()[:1], 'Other')
```

**2. Tables are Python classes (`app.py`).** Annotated attributes are stored columns; attributes assigned an expression are **computed columns** (`discipline`, `thumb`, `meta`, `vision`, `caption`), evaluated incrementally on every insert or update and recomputed when their inputs change. Indexes live next to the columns.

```python
# app.py
class Sheets(TableModel, name='sheets'):
    """One drawing sheet."""

    sheet_no = pxt.Column(type=pxt.String, primary_key=True)
    title: pxt.String
    project: pxt.String | None
    image: pxt.Image

    discipline = discipline_of(sheet_no)
    thumb = image.resize([320, 213])
    meta = image.get_metadata()
    vision = openai.chat_completions(
        vision_messages(CAPTION_PROMPT, thumb), model=CAPTION_MODEL, model_kwargs={'max_tokens': 60, 'temperature': 0}
    )
    caption = vision.choices[0].message.content.astype(pxt.String)
    search_text = title + '. ' + caption

    __indexes__ = [
        pxt.EmbeddingIndex(search_text, embedding=openai.embeddings.using(model=EMBED_MODEL), name='sheet_idx'),
        pxt.BtreeIndex(discipline),
    ]
```

**3. Queries are functions (`app.py`).** `@pxt.query` wraps a Pixeltable query so it can be called from Python or exposed as a route:

```python
# app.py
@pxt.query
def find_sheets(q: str, k: int = 3):
    """Sheets whose title and caption best match a text description."""
    sim = Sheets.search_text.similarity(string=q, idx='sheet_idx')
    return Sheets.order_by(sim, asc=False).limit(k).select(
        sheet_no=Sheets.sheet_no, title=Sheets.title, discipline=Sheets.discipline, caption=Sheets.caption, score=sim
    )
```

**4. One router, a full REST API.** `FastAPIRouter` generates request/response models from the column types, validates input, and publishes OpenAPI docs at `/docs`:

```python
# app.py
archive = FastAPIRouter(name='archive')
archive.add_insert_route(
    Sheets,
    path='/sheets',
    inputs=[Sheets.sheet_no, Sheets.title, Sheets.project],
    uploadfile_inputs=[Sheets.image],
    outputs=[Sheets.sheet_no, Sheets.discipline, Sheets.caption, Sheets.meta],
)
archive.add_delete_route(Sheets, path='/sheets/delete')
archive.add_insert_route(Questions, path='/ask', inputs=[Questions.qid, Questions.question], outputs=[Questions.answer])
archive.add_query_route(path='/search', query=find_sheets, method='get')
archive.add_query_route(path='/discipline', query=sheets_in, method='get')
archive.add_query_route(path='/sheets/thumb', query=sheet_thumb, method='get', return_fileresponse=True)
```

## FAQ

**How do I serve images and documents to Claude or Cursor as an MCP server in Python?**

Store them in a Pixeltable table with media columns, add computed columns for what an agent needs (caption, embedding index), and wrap the table in an MCP server whose tools call `similarity()` and plain `where()` queries. `mcp_server.py` in this repo is about 40 lines.

**How does a Pixeltable app call tools from an MCP server?**

`pxt.mcp_udfs(url)` returns one UDF per tool; bundle them with `pxt.tools(...)`, pass them to `openai.chat_completions(tools=...)` and run the chosen tool with `openai.invoke_tools`. Give MCP tool arguments no defaults: a tool argument that has a default is treated as required, and a model that leaves it out gets a null result.

**Do I need a separate vector database?**

No. The embedding index is declared on the table (`__indexes__`) and stays in sync on insert; `similarity()` is just a query.

## Learn more

- ๐ŸŒ Website: https://pixeltable.com
- ๐Ÿ“š Docs: https://docs.pixeltable.com
- ๐Ÿ’ป Source: https://github.com/pixeltable/pixeltable (โญ star it if Pixeltable is useful to you)
- ๐Ÿ“ฆ PyPI: https://pypi.org/project/pixeltable/
- ๐ŸŽฏ Use case: https://pixeltable.com/use-cases/ai-agents-mcp
- ๐Ÿงฉ More example apps: https://pierrebrunelle.github.io/awesome-pixeltable-apps/

**[More Pixeltable example apps โ†’](https://pierrebrunelle.github.io/awesome-pixeltable-apps/)**

---

<sub>Built as part of a daily series of Pixeltable example apps ยท Pixeltable 0.7.24 ยท Python, FastAPI, incremental computed columns ยท Licensed under Apache-2.0.</sub>