# ClosetScan

Turn a closet walkthrough video into a wardrobe your AI assistant can use:
source frames, AI-generated front/back flat-lays, attributes, and optional
narration, handed to any MCP client by a local server and the included wardrobe
skill. Ask what goes with the navy jacket, what you said about it while you
were filming, or what you have not worn since spring — against your own
clothes, catalogued from one video, on your own machine.

By Will Gao. A mention is appreciated, but not required. MIT licensed;
retain the copyright and license notice as required by MIT.

![The ClosetScan demo: one phone walkthrough beside the catalogue generated from it](docs/images/demo-overview.png)

## See it first

**[Open the live demo](https://wgaostudio.github.io/closetscan-demo/)** —
one 5-minute-49-second phone recording beside the 34-garment catalogue built
from it. Nothing to install, no API key, no clone.

The demo is not part of this repository, so cloning stays small. To run it
offline, download `closetscan-demo.zip` from the
[latest release](https://github.com/wgaostudio/closetscan/releases/latest),
unzip it, and open `index.html`. Keep the folder together; the page loads its
images from `catalogue/`.

<img src="docs/images/demo-garment-detail.png" alt="A garment detail view showing generated front and back plates beside extracted attributes" width="720">

That catalogue was generated from the included recording using DINOv2 and the
paid AI stages; reruns can produce different results. It contains 34 garments
with 68 generated front/back images. Images marked as plates are generated
reconstructions; source frames remain available for comparison.

## Your closet, in your assistant

One MCP server and one skill, both local. Point the server at a catalogue —
one you build with the pipeline below, or the published demo's, which needs no
pipeline run at all — and your assistant can answer from your own clothes:

- *What goes with the grey wool trousers?*
- *What did I say about the jacket I bought in Kyoto?*
- *What have I not worn since March?*

Answers are only as good as the catalogue behind them; see
[Limits](#limits-and-contributions).

The MCP server uses only Python's standard library. It opens no sockets and makes
no outbound requests. Install with `python -m pip install -e .`, or run from a
clone without installation, pointing it at any catalogue directory:

```bash
python -m closetscan.mcp_server out
```

To query the published demo, unzip `closetscan-demo.zip` and point the server at
its `catalogue` directory.

This waits for newline-delimited JSON-RPC on stdin; it is not an interactive CLI.
Use [mcp/config.example.json](mcp/config.example.json) in your MCP client's server
configuration, replacing both absolute paths. Install the package into the Python
environment named by `command`. Copy [skills/wardrobe](skills/wardrobe) into your
agent's skill directory. The skill uses the seven tools exposed by this server:

- `list_attributes`, `list_garments`, `get_garment`
- `search_garments`, `search_narration`
- `log_wear`, `get_wear_history`

The server supports MCP protocol `2024-11-05` over stdio and advertises that version
at initialization; clients must support it. See the [MCP transport specification](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports).
Catalogue location: positional argument, then `CLOSETSCAN_CATALOGUE`, then `./out`.
Only `log_wear` writes, appending `wear_log.jsonl`. Back it up with its catalogue:
garment IDs can change when grouping is rerun. Search is lexical, not semantic.
`log_wear` requires a nonempty garment ID; invalid IDs never create or append
to the log. Corrupt or unreadable catalogue JSON raises an error. When
`dedup.json` is absent, the source-only candidate preview ignores existing
attributes, narration, and generated images because their grouping IDs
cannot safely be matched to candidate rows.
Image paths require local file access in the client. No pipeline tool is exposed.

## One notebook: Colab or local Jupyter

[Open `closetscan_colab.ipynb` in Colab](https://colab.research.google.com/github/wgaostudio/closetscan/blob/main/closetscan_colab.ipynb).
The notebook installs the package and processes the demo video or one you
upload. The same notebook also runs in local Jupyter: open it from a clone, or
set `PROJECT_DIR` in its options cell. For a local custom video, set
`USE_DEMO=False` and `VIDEO_PATH`. Python 3.10+ and FFmpeg/ffprobe on PATH are
required locally.

With `USE_DEMO=True` the notebook downloads the demo walkthrough from the
release assets on first use and caches it in the workspace.

Extraction defaults to a local histogram method;
choose DINOv2 for learned embeddings. Enable the optional paid stages and supply
OpenRouter model IDs to create the complete catalogue. Every command stops on failure.

Video lives in the Colab runtime. Paid stages send images and optionally transcript
text to OpenRouter and its providers. Whisper transcribes within the runtime on CPU.
Costs and model availability vary; there is no fixed-cost promise.

The histogram default tests execution, not recognition quality. The notebook
checks its manifest, images, HTML, MCP query, and ZIP before export. Local
Jupyter prints the archive path; Colab downloads it.

## Recording a walkthrough

**Talk through the closet while you film it.** Speech is not a bonus track; it
is the pipeline's strongest segmentation signal. The background of a closet is
constant — same wall, same rail — so frame-to-frame visual change follows your
hands more than it follows which garment you are holding, while the pauses
between utterances land almost exactly on the handoffs. A silent recording still
processes, but neighbouring garments merge into one.

- Say a few words about each item. Anything works: the timing carries the
  segmentation, and the words themselves become searchable narration.
- Pause between items, and hold each garment still while you talk about it.
- Show both sides; front and back are catalogued separately.
- Keep the microphone live, and keep the audio track when trimming or
  re-encoding a clip before processing.

`python -m closetscan.run` uses that audio by default and warns when it finds no
narration boundaries; `--no-audio` segments on visual change alone. Transcribing
what was said — what a fabric is, where an item came from — is the separate
optional `closetscan.narration` stage below.

## Local pipeline

Python 3.10+ and FFmpeg/ffprobe on PATH are required for video processing.
From a clone of this repository:

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[pipeline,vision,narration]'
python -m closetscan.run path/to/walkthrough.mp4 --out out --embedder dinov2 --html
```

To use the same recording as the published demo, take `phone-walkthrough.mp4`
from `closetscan-demo.zip` in the
[latest release](https://github.com/wgaostudio/closetscan/releases/latest).

For extraction without model downloads, install `.[pipeline]` and use
`--embedder hist`. For a complete catalogue, set `OPENROUTER_API_KEY`,
`VISION_MODEL` (vision input + text output) and `IMAGE_MODEL` (image input +
text/image output) in your shell, then run:

```bash
python -m closetscan.dedup out --include-views --model "$VISION_MODEL"
python -m closetscan.product_shots out --model "$IMAGE_MODEL"
python -m closetscan.attributes out --model "$VISION_MODEL"
python -m closetscan.narration out --clip path/to/walkthrough.mp4 --model "$VISION_MODEL"
python -m closetscan.html_export out
```

Narration is optional and currently processes one clip per invocation; use one
walkthrough for the complete workflow. Individual stages can be rerun; retain
previous outputs before replacing them. Local key-file fallback is
`~/.config/openrouter/key`. Do not commit credentials or private catalogues.
The CLI retains the original experiment's model defaults; pass explicit model IDs
available to your account. See [OpenRouter multimodal documentation](https://openrouter.ai/docs/guides/overview/multimodal/overview).

## Limits and contributions

Capture can miss garments, and a silent or poorly narrated recording misses
more of them; see [Recording a walkthrough](#recording-a-walkthrough).
Generated images and attributes can invent details. Narration can be mistranscribed
or attached to the wrong item. Missing catalogue entries do not establish ownership,
and missing wear logs do not mean clothes were never worn.

Bug reports and pull requests are welcome. Include Python version, command,
and a small reproducible example; remove keys and personal data first.
Run `python -m unittest discover -s tests -v` before submitting code changes.
For the full suite install `.[pipeline]`; without it four tests skip. Tests read
the small catalogue in [tests/fixtures](tests/fixtures), not the published demo,
so they run on a fresh clone. CI runs the same checks on Python 3.10 and 3.12.

[MIT license](LICENSE) covers the project and the published demo to the extent the
author holds rights. Dependency and model licenses remain their own; visible brand
marks in the demo do not imply endorsement or grant trademark rights.