# idml-engine

**Headless InDesign document composition.** Parse a designer's IDML, convert
editorial content to styled ICML stories, and compose complete newspaper
IDML documents — no InDesign required at build time.

Built to replace two things at a working newspaper (The Wayland Post):

- **DocsFlow** (the commercial Google Docs → InDesign plugin) for one-way,
  batch content placement, and
- a Scripts-panel ExtendScript handoff that needed a human to click a button
  inside InDesign for every build.

An issue that took a staffed InDesign session now composes in about one
second, headlessly, and the editor opens a finished IDML to refine.

## How it works

```
content (Google Docs / DOCX / Markdown / HTML / JSON)
   │  ingest.py — normalize to typed blocks (headline, byline, body, cutline…)
   ▼
typed blocks ──► story_converter.py ──► ICML stories   (Pandoc, style-mapped)
   │
   ▼
composer.py — "donor package" composition
   • a real published-issue IDML (or purpose-built template) donates
     Resources/ (all styles, fonts, colors) and MasterSpreads/ (the grid)
   • generates fresh Spreads/ + Stories/: standing head, headline frame,
     image grid with cutline captions, threaded body legs, bottom ad well
   ▼
complete .idml — opens in InDesign with the paper's exact design system
```

`template_parser.py` is the intelligence layer: it reads any IDML into a
manifest — style catalog, master geometry with computed column ranges, every
frame with threading, story-role classification (headline/body/caption/jump),
and DocsFlow live-links (story → Google Doc id, decoded merge base).

## Quickstart

```bash
pip install idml-engine[all]     # needs Pandoc on the machine for conversion

# understand a template or issue
idml-engine-parse issue.idml --summary

# convert a story
idml-engine-convert article.docx -o article.icml

# compose an issue (donor supplies styles/masters)
idml-engine-compose --donor published-issue.idml --manifest manifest.json --out new-issue.idml
```

Python API:

```python
from idml_engine import composer, ingest, story_converter, template_parser

manifest = template_parser.parse_template("issue.idml")
blocks   = ingest.from_file("article.docx")
story_converter.blocks_to_icml(blocks, "article.icml")
composer.compose("donor.idml", articles, "out.idml", ads=ads)
```

## Layout priors — "trained" on your own paper

The composer doesn't guess layout metrics; it learns them from your archive.
`layout_learner.py` mines a corpus of published-issue IDMLs into
`layout-priors.json` — measured body-text density (chars/pt², drives column
capacity), headline-frame heights, caption heights, image sizes, ad-band
heights, and stories-per-page (quartile distributions). The composer
auto-loads priors (env `IDML_ENGINE_PRIORS`, or the file next to the
package) and falls back to model constants without them.

```bash
python -m idml_engine.layout_learner "path\to\published issues" -o layout-priors.json
```

At The Wayland Post: 27 issues mined → 729 body-story density samples,
937 headline frames, 1,200 placed images. The measured density (~31
chars/line) corrected the hand-tuned model's 40 by 25%.

## MCP server (AI-drivable)

Every capability is exposed as MCP tools, so Claude (or any MCP client) can
parse templates and compose issues conversationally:

```bash
claude mcp add idml-engine -- python -m idml_engine.mcp_server
```

Tools: `parse_template`, `list_template_styles`, `get_template_geometry`,
`validate_template`, `ingest_content`, `convert_story`, `compose_issue`,
`compose_articles`.

## DocsBridge panel — the DocsFlow-style UI

`uxp-panel/` is an InDesign UXP panel that replicates DocsFlow's workflow on
this engine: load an issue → per-article **Place** (drops the styled ICML
story and live-links it via `storyTitle="docx:<google-doc-id>"` — DocsFlow's
own convention) → **Update** re-places into the linked story's first frame,
preserving the frame chain. Status badges (placed / not placed / no copy)
survive save/reopen because the link rides in the document.

Backed by `idml_engine/panel_service.py` (localhost:3100 — manifest
listing, on-demand blocks→ICML, Drive refetch). Load the panel once via the
Adobe UXP Developer Tool (`uxp-panel/manifest.json`).

## Commercial REST API

`commercial/api_server.py` wraps the same engine with Bearer-key auth and
per-key usage metering for hosted deployment (FastAPI):

```bash
uvicorn commercial.api_server:app --port 8080
# POST /api/compose  /api/parse  /api/convert   (Authorization: Bearer <key>)
# GET  /api/usage
```

Keys + metering are file-backed stubs; production wires them to Stripe
metered billing (see `commercial/README.md`).

## Findings that made this possible

- **ICML style mapping:** Pandoc's ICML writer (stable since 2014) plus a
  post-pass that strips its `" > Paragraph"` style-name suffix yields stories
  whose styles match the template by exact name — InDesign adopts the
  template's formatting on Place. Trailing-space style names (`"Byline "`)
  survive round-trip.
- **Donor-package composition:** a finished issue's IDML already carries the
  paper's whole design system. Cloning its resources and generating only
  spreads + stories produces documents InDesign opens cleanly.
- **DocsFlow link format:** DocsFlow stores its Google Doc live-links in
  plain sight — `StoryTitle="docx:<file-id>"` plus a merge-base blob that is
  just base64 → raw DEFLATE → JSON. A future release can read, preserve, and
  write these links for drop-in compatibility.

See `docs/TEMPLATE-STRUCTURE.md` for the full IDML anatomy notes.

## Status

Alpha. Working end-to-end at one real newspaper. Not yet implemented:
multi-article pages, jump lines ("Continued on…"), text wrap around
images, DocsFlow-compatible link writing, IDML→PDF export (open in
InDesign, or pair with InDesign Server / the UXP MCP bridge).

## License

BSD-3-Clause (this library). The commercial API layer and hosted service
are separate products built on top.