🗜️ Sarup (สรุป)

Thai-first context compression for Claude Code. Shrink the text you feed an LLM by 50–88% — while the original stays 100% recoverable, byte-for-byte.

สรุป means "to summarize." Headroom routes Thai through noop (0% savings) because its whitespace tokenizer can't find Thai word boundaries. Sarup uses PyThaiNLP segmentation, so it compresses Thai as well as English — and caches every original so nothing is ever lost.

Contents

• Highlights

• Why it's safe — the two-tier guarantee

• How it works

• Tools

• Compression modes

• Measured results

• Install

• Register with Claude Code

• Auto-compression hook

• Configuration

• Project structure

• Tech stack & techniques

• Testing

• Roadmap

• License

Related MCP server: cctx

Highlights

• 🇹🇭 Real Thai compression — PyThaiNLP newmm word segmentation, not whitespace.

• ♻️ Lossless by guarantee — every compress caches the original; verified: true proves a byte-for-byte round-trip.

• 🎚️ Five modes — from offline 1 ms TF-IDF to an 88%-savings cascade.

• 🧠 Optional local LLM — embeddings + rewrite via Ollama, with automatic offline fallback.

• ⚙️ Auto mode — a PostToolUse hook compresses large tool outputs with zero manual calls.

• 📏 Honest metrics — token counts from a real tokenizer (tiktoken), not byte guesses.

• 🔌 Routing — JSON compaction, log dedup, and verbatim code-fence preservation built in.

Why it's safe — the two-tier guarantee

Aggressive lossy compression is safe because the original is always one sarup_retrieve(hash) away. This is how "maximum savings" and "100% accuracy" coexist — they live in different tiers.

How it works

Two entry points feed one engine: a cheap compressed view the model reads, and a lossless retrieval store that can restore the original byte-for-byte.

flowchart TD
    M["🧑 Manual<br/>sarup_compress()"]:::entry --> R
    A["⚙️ Automatic<br/>PostToolUse hook<br/>(Read · Bash · Grep)"]:::entry --> R

    R{"Sarup compress<br/>extractive · semantic · abstractive · pipeline"}:::engine
    R -- "compressed view<br/>50–88% fewer tokens" --> V["📄 Model context"]:::lossy
    R -. "cache original" .-> S[("🗄️ Retrieval store<br/>hash → original")]:::lossless

    V -. "need full detail?" .-> RET["🔑 sarup_retrieve(hash)"]:::lossless
    RET --> S
    S == "byte-for-byte ✓" ==> V

    classDef entry fill:#e0e7ff,stroke:#6366f1,color:#111
    classDef engine fill:#fde68a,stroke:#d97706,color:#111
    classDef lossy fill:#fef3c7,stroke:#f59e0b,color:#111
    classDef lossless fill:#bbf7d0,stroke:#16a34a,color:#111

• Manual — the model calls sarup_compress / sarup_retrieve itself.

• Automatic — the hook intercepts large tool outputs, caches the original to SARUP_DB_PATH, and substitutes the compressed view + a retrieval hash. Source code is skipped; small outputs pass through untouched.

Tools

sarup_compress arguments

Compression modes

¹ Measured on a 10-sentence Thai paragraph (522 tokens). Every mode stays 100% recoverable via the store; Ollama modes degrade gracefully to extractive when the backend is down.

Measured results

$ .\.venv\Scripts\python.exe bench\benchmark.py

sample                      before   after   savings   verify
Thai prose                     522     257    50.8%       OK
Thai prose (aggressive)        522     217    58.4%       OK
English prose                  105      54    48.6%       OK
JSON (lossless)                 67      44    34.3%       OK
Logs                           563     300    46.7%       OK
TOTAL                         1779     872    51.0%    ALL OK   → 100% recoverable

Mode comparison (Thai prose, 522 tok):
  extractive 50.8% (1ms) · auto 64.6% (~90ms) · semantic 64.6% (2.1s)
  abstractive 51.1% (8s) · pipeline 88.1% (2.3s)        ← all verified recoverable

Token counts via tiktoken cl100k_base — a real tokenizer, not a byte heuristic.

Install

py -3.11 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"

Optional local-LLM modes (semantic / abstractive / pipeline) need Ollama:

ollama pull nomic-embed-text     # embeddings → semantic mode
ollama pull gemma3:12b           # rewrite → abstractive / pipeline (Thai-validated)

Register with Claude Code

One-command setup (recommended). Detects this machine's paths, probes Ollama (picks the best mode + models), and merges into .mcp.json / .claude/settings.json without clobbering anything already there (a .bak is written first):

.\.venv\Scripts\python.exe scripts\install.py --with-hook --pull

• No Ollama? It configures offline extractive mode — still fully works.

• Ollama up? It auto-selects nomic-embed-text (semantic) + gemma3:12b (rewrite) and sets the hook to auto. --pull fetches any missing models.

• Idempotent — safe to re-run; --global writes to ~/.claude instead.

Manual — or add it yourself to your MCP config (e.g. .mcp.json or ~/.claude.json):

{
  "mcpServers": {
    "sarup": {
      "command": "d:\\WORK\\Sarup\\.venv\\Scripts\\python.exe",
      "args": ["-m", "sarup.server"],
      "env": { "SARUP_DB_PATH": "d:\\WORK\\Sarup\\.sarup-cache.db" }
    }
  }
}

Or run it directly over stdio:

.\.venv\Scripts\python.exe -m sarup.server

Auto-compression hook

Skip manual tool calls entirely: install the PostToolUse hook and large Read/Bash/Grep outputs are compressed before they enter context, with the original cached for retrieval. Source-code reads are skipped for safety. Full setup in hooks/README.md.

Experimental: the hook emits a valid updatedToolOutput, but current Claude Code builds (2.1.167 / 2.1.193) don't yet apply it, so it's a no-op in-session today. Use the manual sarup_compress tool meanwhile — the hook is ready for when Claude Code honors the field.

{
  "hooks": {
    "PostToolUse": [
      { "matcher": "Read|Bash|Grep",
        "hooks": [{ "type": "command",
          "command": "d:\\WORK\\Sarup\\.venv\\Scripts\\python.exe d:\\WORK\\Sarup\\hooks\\sarup_hook.py" }] }
    ]
  },
  "env": { "SARUP_DB_PATH": "d:\\WORK\\Sarup\\.sarup-cache.db" }
}

Privacy & data

To guarantee recovery, Sarup caches the original content in the store. Two things to know:

• With SARUP_DB_PATH set, originals are written to that SQLite file in plaintext (no encryption). Treat it like a cache of whatever you compressed.

• If you compress tool outputs that contain secrets (e.g. a .env dump or credentials in a log), those land in the cache too. The auto-hook skips source-code/config file reads, but Bash output is fair game — review what you point it at.

*.db is git-ignored, so the cache never gets committed. For zero on-disk footprint, leave SARUP_DB_PATH unset (memory-only; the MCP server then loses the cache on restart, and the hook will not substitute — see the hook docs).

Configuration

Project structure

sarup/
├── src/sarup/
│   ├── server.py       # MCP stdio server — 3 tools
│   ├── compressor.py   # router + modes (extractive/semantic/abstractive/pipeline/auto)
│   ├── thai.py         # PyThaiNLP tokenization, sentence split, TF-IDF
│   ├── semantic.py     # embedding centrality + cosine dedup
│   ├── llm.py          # optional Ollama backend (generate + embed)
│   ├── tokens.py       # real token counting (tiktoken)
│   └── store.py        # CCR store: hash → original (memory + SQLite)
├── hooks/
│   ├── sarup_hook.py   # PostToolUse auto-compression hook
│   └── README.md       # hook install guide
├── bench/benchmark.py  # before/after measurement
├── tests/              # 50 tests (test_thai, test_mcp, test_hook)
├── README.md
└── STACK.md            # full stack + techniques

Tech stack & techniques

Python 3.11 · MCP · PyThaiNLP newmm · tiktoken · Ollama (optional) · SQLite · hatchling · pytest.

The technique behind each mode — TF-IDF scoring, embedding centrality, cascade pipeline, content routing, and graceful degradation — is documented in STACK.md.

Testing

.\.venv\Scripts\python.exe -m pytest tests/ -q

50 tests cover Thai NLP, the MCP tool contracts, every mode (including Ollama-fallback paths), the roundtrip-verify guarantee, and the auto-compression hook (incl. cross-process retrieval).

Roadmap

• Make auto the default mode for sarup_compress (currently extractive).

• Optional Typhoon 2.1 abstractive (blocked on an Ollama template fix).

• Per-content adaptive target_ratio.

• Published PyPI package.

License

MIT