scrollback-mcp

Unofficial community project. Not affiliated with, endorsed by, or sponsored by Xiaomi.

Record your IDE scrolling through a project → get the project back as files. CLI + MCP server: point it at a screen recording of IntelliJ IDEA and it rebuilds the source tree (pom.xml, src/main/java/..., resources), checks the syntax, and gives you a short list of lines worth a second look — with the video frame for each.

recording.mp4 ─► keyframes ─► line number + text per row ─► vote per line ─► project/ + REPORT.md
                 (screen at rest)  (gutter = exact line index)    (all frames)

v0.2 is a rewrite. The 0.1 screen-control tools (click/type) are gone; this version is about recovering code from recordings.

How it works

• The gutter is the index. Each row is keyed by the line number printed next to it, so stitching scrolled frames is exact, blank lines are explicit, lines nobody scrolled to are reported by number, and Code Vision rows ("3 个用法", author hints) are skipped for free.

• Every row is read on its own. Text detection misses tiny tokens (}, */, single-digit line numbers), so rows are cut out by their ink and sent straight to the recogniser (PP-OCRv6 via RapidOCR; DirectML GPU on Windows).

• Spaces and indentation come from pixels. In a monospace font a gap of k cells is k spaces. Full-width Chinese punctuation is told apart from ASCII by its width.

• Every frame votes, and code-aware fixes repair typical slips (11 注释 → // 注释, S1f4j → Slf4j, import java.util.list → List) — only where the misread form is not valid code, and flagged for review.

• Names come from the code: a Java file is named by its public type and placed by its package; IDEA's navigation bar gives the module directory and non-Java paths.

Measured (v0.2)

Synthetic IDEA recordings rendered from a real multi-module Spring Boot project (苍穹外卖 day01 skeleton: Java with Chinese comments, pom.xml, YAML, MyBatis XML), 1080p, Darcula, Code Vision on, page-down + wheel scrolling (tools/make_video.py), RTX 4060 Laptop with DirectML:

¹ measured one iteration before the last fixes; remaining errors are mostly Chinese punctuation and // markers in comments, plus a few letter-case slips.

Related MCP server: Framesleuth

Recording checklist

1. Line numbers on (default) — they make stitching exact.

2. Navigation bar visible (View → Appearance → Navigation Bar, or the status-bar breadcrumb in the new UI) — gives each file its module and directory.

3. Expand all folds before scrolling: Ctrl+Shift+NumPad + (imports are folded by default).

4. Inlay hints off (Settings → Editor → Inlay Hints): parameter-name hints sit inside a line.

5. Font 14 pt or larger, soft wraps off, one editor split, native resolution.

6. Open each file at the top and PageDown to the end, pausing ~0.5 s per page.

Install

pip install "scrollback-mcp[gpu,yaml] @ git+https://github.com/SorenKang/scrollback-mcp.git"
scrollback-mcp check            # ocr_backend should be dml (Windows) or cuda
scrollback-mcp check --fix-gpu  # if it says cpu although onnxruntime-directml is installed

Java syntax checking needs a JDK (javac on PATH or JAVA_HOME).

CLI

scrollback-mcp rebuild recording.mp4 -o F:/replica     # resumable
scrollback-mcp report F:/replica                        # summary + review queue
scrollback-mcp crop F:/replica <file> 40 52             # PNG of those lines from the video
scrollback-mcp set  F:/replica <file> 47 "        return x;"
scrollback-mcp verify F:/replica [--build]              # javac/XML/YAML (+ mvn compile)

Bookkeeping lives in F:/replica/.xocr/: reads.jsonl (frame cache), state.json (per-line results), overrides.json (your fixes — re-applied on every rebuild), REPORT.md.

MCP

{ "mcpServers": { "scrollback": { "command": "scrollback-mcp" } } }

An agent with vision closes the loop: review_queue → view_lines → compare → set_lines → verify, until the queue is empty.

Testing with synthetic recordings

pip install pillow
python tools/make_video.py path/to/project -o demo.mp4 --code-vision
scrollback-mcp rebuild demo.mp4 -o out
python tools/evaluate.py path/to/project out --files demo.files.txt

Limits

• Built for IntelliJ IDEA; other editors work if they show line numbers in a left gutter.

• Lines never shown on screen cannot be recovered — they are listed by number.

• Review flags are still noisy (many single/cjk flags on correct lines).