<p align="center">
  <img src="docs/logo.svg" width="120" alt="squeeze logo">
</p>

# squeeze

[![ci](https://github.com/arcnosixta/squeeze/actions/workflows/ci.yml/badge.svg)](https://github.com/arcnosixta/squeeze/actions/workflows/ci.yml)
[![node](https://img.shields.io/badge/node-%E2%89%A5%2022.18-brightgreen)](https://nodejs.org)
[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![zero dependencies](https://img.shields.io/badge/dependencies-none-2ea44f.svg)](#install)

**Your context window is full of the same error repeated 200 times.** squeeze
cuts that noise — up to 70% on real compiler output, 45% on real failing test
runs — and never deletes a byte.

A drop-in proxy that sits between your coding agent and the LLM API, compresses
tool output before it reaches the context window, and keeps every original byte
retrievable — by the model itself, over MCP. No code changes. No dependencies.
One environment variable.

```
real tsc output:    25,166 chars  ->   7,517 chars   -70.1%
real pytest output: 20,110 chars  ->  11,040 chars   -45.1%
```

Measured on captured tool runs, not hand-written fixtures: `scripts/collect-real.sh`
re-runs tsc, npm, pytest, node --test and this repository's own CI log, then the
`bench real` command reports the honest numbers.

<p align="center">
  <img alt="squeeze: self-test, real benchmark and inspect output in a terminal"
       src="docs/demo.png" width="680">
</p>

## Contents

- [Features](#features)
- [Why](#why)
- [Install](#install)
- [Use](#use)
- [Benchmark](#benchmark)
- [MCP](#mcp)
- [Levels](#levels)
- [How it works](#how-it-works)
- [Limits](#limits)
- [What it does not touch](#what-it-does-not-touch)
- [CLI](#cli)
- [Environment](#environment)
- [Tests](#tests)
- [Status](#status)
- [Provenance](#provenance)
- [License](#license)

## Features

- **Lossless by design** — squeeze never deletes. Every collapsed span is stored
  in a content-addressed ref store, so `expand()` reconstructs any document
  byte-for-byte.
- **Reversible by the model** — the `squeeze_fetch` MCP tool lets the agent pull
  the real text back on demand, instead of guessing what a ref meant.
- **Drop-in** — point `*_BASE_URL` at the proxy. No code, no config file, and
  zero runtime dependencies beyond Node itself.
- **Honest** — it compresses noise, never meaning. Source code and JSON are
  detected and left alone, and the benchmark reports `0.0%` for them on purpose.

## Why

Your context window is not full of your code. It is full of 200 identical
`TS2322` errors, 400 identical test-pass lines, and heartbeat logs that arrived
at 09:00:11. The model reads all of it, every turn, and you pay for all of it.

Existing token compressors delete. If the model later needs the deleted lines,
it cannot get them back — and you find out when it hallucinates a fix.

squeeze never deletes. It clusters, counts, and hands back a short ref:

```
src/mod0.ts(0,7): error TS2322: type mismatch  ⟨×119 more, ref=83a63083475a87f0⟩
```

That single line replaces 120. When the model decides it needs the real list, it
calls `squeeze_fetch` and gets every original line back:

```
  83a63083475a87f0 ->  120 lines, 6,412 chars
```

Refs are content-addressed and persisted to disk, so they survive a proxy
restart. A ref printed into a transcript last week still resolves today.

```bash
curl "http://127.0.0.1:8899/squeeze/expand?ref=83a63083475a87f0"
```

## Install

Requires Node 22.18+. No runtime dependencies, no build step for you.

**From npm:**

```bash
npm install -g @arcnosixta/squeeze
squeeze doctor
```

The npm name `squeeze` belongs to someone else, so this package lives under its
own scope. The tarball ships precompiled so the install works anywhere.

**From the source** (same thing, no install needed — Node runs the TypeScript
directly):

```bash
git clone https://github.com/arcnosixta/squeeze
cd squeeze
node src/cli/main.ts doctor
```

## Use

**As a proxy** (works with Claude Code, Codex, Cursor, anything speaking the
Anthropic API or OpenAI's Chat Completions and Responses APIs):

```bash
node src/cli/main.ts serve

export ANTHROPIC_BASE_URL=http://127.0.0.1:8899
export OPENAI_BASE_URL=http://127.0.0.1:8899
```

**On a file:**

```bash
node src/cli/main.ts compress build.log --aggressive > small.log
node src/cli/main.ts inspect build.log --aggressive
```

## Benchmark

Two sets of numbers, both reproducible on your machine.

**Real tool output** — files captured from actual runs, regenerated by
`scripts/collect-real.sh`:

```bash
node src/cli/main.ts bench real aggressive
```

| artifact (what it is) | original | after | saved |
| --- | ---: | ---: | ---: |
| `tsc-errors` — real `tsc 7.0` output, 60 generated files with real type errors | 25,166 | 7,517 | 70.1% |
| `pytest` — real `pytest 8.3` run, 42 failing of 190 tests | 20,110 | 11,040 | 45.1% |
| `github-actions` — this repo's real CI job log | 96,300 | 80,306 | 16.6% |
| `npm-install` — real `npm 11` install `--verbose` | 46,297 | 44,838 | 3.2% |
| `dpkg-log` — 5,000 lines of a real Debian package log | 344,601 | 325,792 | 5.5% |
| `node-test-run` — this repo's real `node --test` pass | 1,747 | 1,747 | 0.0% |
| **total by bytes** | **534,221** | **471,240** | **11.8%** |
| **plain average of artifacts** | — | — | **23.4%** |

The byte total is dragged down by `dpkg-log` — 64% of the corpus, near-unique
lines — while a plain average tells the other story: the artifacts that actually
fill an agent's context, compiler errors and failing test runs, are where
squeeze does its best work. The `0.0%` row is part of the point: a passing test
run is already noise, and squeeze leaves documents it cannot help alone.

**Synthetic fixtures** — the original hand-written corpus, kept for reference:

```bash
node src/cli/main.ts bench aggressive
```

| fixture | original | after | saved |
| --- | ---: | ---: | ---: |
| `npm-install-verbose` | 18,156 | 7,716 | 57.5% |
| `tsc-errors` | 11,387 | 125 | 98.9% |
| `json-dump` | 13,609 | 13,609 | 0.0% |
| `test-run` | 15,275 | 67 | 99.6% |
| `heartbeat-logs` | 4,759 | 85 | 98.2% |
| `source-file` | 4,820 | 4,818 | 0.0% |
| `blank-heavy` | 1,299 | 1,181 | 9.1% |
| `dup-blocks` | 2,939 | 55 | 98.1% |
| **total** | **72,244** | **27,656** | **61.7%** |

The 0.0% rows are the point: `json-dump` and `source-file` are read line by line
by the model, and squeeze leaves them alone rather than pretending otherwise. It
compresses noise, never meaning.

Reproduce the end-to-end claim quickly:

```bash
node src/cli/main.ts selftest    # 9,779 -> 80 chars, ref verified byte-exact
```

## MCP

Give the model a way to undo the compression on its own terms. This is the part
that makes losslessness real: without it, a ref is a promise nobody can keep.

```bash
node src/cli/main.ts mcp
```

```json
{
  "mcpServers": {
    "squeeze": {
      "command": "squeeze-mcp"
    }
  }
}
```

If you cloned instead of installing, point at the source file instead:

```json
{
  "mcpServers": {
    "squeeze": {
      "command": "node",
      "args": ["/path/to/squeeze/src/mcp/server.ts"]
    }
  }
}
```

Two tools:

| tool | purpose |
| --- | --- |
| `squeeze_fetch` | expand one ref back to its exact original text |
| `squeeze_compress` | compress noisy text before it enters context |

`squeeze_fetch` is the one that matters. The model reads
`⟨×119 more, ref=83a6…⟩`, decides the collapsed lines matter, and pulls them.

## Levels

| level | behaviour | reversible |
| --- | --- | --- |
| `none` | passthrough, byte-identical | n/a |
| `safe` | strips ANSI colour codes, collapses blank-line runs | no, see below |
| `aggressive` | safe + clusters repeated lines into counted refs | clusters yes |

```bash
node src/cli/main.ts serve            # defaults to aggressive
SQUEEZE_LEVEL=safe node src/cli/main.ts serve
```

`aggressive` is the default: on real tool output `safe` saves almost nothing
(no ANSI codes, few blank runs), while `aggressive` is where everything comes
from. The two exceptions to reversibility are the same in both modes: ANSI
codes and blank-line runs are discarded and keep no ref. Those bytes carry no
information a model needs, but calling them "lossless" would be wrong. Every
handle-based replacement — the `⟨ref=…⟩`, `⟨×N more, ref=…⟩` and
`⟨repeat of lines …; ref=…⟩` markers — stores its exact original span, and
`expand()` resolves every one back in place, so aggregate output reconstructs
byte-for-byte (the tests assert exactly that).

## How it works

<p align="center">
  <img alt="Architecture: coding agent <-> squeeze proxy <-> LLM API, with a ref store and squeeze_fetch over MCP"
       src="docs/architecture.png" width="720">
</p>

squeeze intercepts the request before it leaves the machine. Tool output is
compressed in place, the original spans go into a content-addressed store, and
the model sees a small prompt with short refs instead of a wall of repetition.
When it needs the real lines, it calls `squeeze_fetch` and gets them back.

Five passes over each text block:

1. **ANSI strip** — colour codes carry no meaning to a model.
2. **Template clustering** — digit runs collapse to `#` regardless of
   neighbouring word characters, so `suite1`, `module12` and `(3,14)` all match
   their siblings. Timestamps normalize for free, so adjacent log lines group.
3. **Block dedup** — runs of 8+ identical lines collapse to a range reference.
4. **Document-wide dedup** — real output rarely repeats *adjacent* lines: an
   error list names a different file and line number on every row. This pass
   keys on the whole document, so `row9.ts(15,5)` and `row10.ts(16,5)` collapse
   to the same handle even separated by other lines. Every line keeps its own
   slot — the original is swapped for a handle in place, never hoisted. Source
   code and JSON are detected and skipped, because there the repetition is the
   meaning.
5. **Blank-run collapse** — trailing newlines are preserved exactly, since tools
   diff on them.

Every collapse writes its original text into the store and emits a ref. Nothing
clustering-related is unrecoverable: `expand(input, store)` replaces each handle
with its stored span, in place.

Clustering happens in place. Hoisting log lines out of the transcript to group
them better would squeeze a few percent more, but it would silently reorder the
document — the model would read a reshuffled version of the user's history whose
original sequence no longer exists anywhere. That is not lossless, so the tests
enforce order preservation:

```
test/aggressive preserves document order when interleaving logs and content
```

A ref only ever holds a contiguous slice of the real input, and
`expand()` reconstructs the original document byte-for-byte. Both properties are
asserted, not assumed.

## Limits

`squeeze` compresses the *text payload* of tool results. It does not:

- Touch images, audio, or binary tool results
- Scope the store per project — refs from two projects share one store, which is
  fine for expansion (keys are content hashes) but means no per-project eviction
- Guarantee savings on source code or JSON. Those are irreducible, and the
  benchmark reports 0.0% for them on purpose

If your workload is mostly `cat`-ing source files, this will not help you. If it
is mostly test runs, compiler errors, and build logs, it will.

## What it does not touch

- Your system prompt, instructions, or `max_tokens`
- Reasoning blocks in either spelling — Anthropic `type: "thinking"` /
  `"redacted_thinking"`, Responses API `"reasoning_text"` / `"summary_text"` —
  because rewriting them changes reasoning semantics, so they pass through
  byte-identical
- `tool_use_id`, `call_id` and every structural field
- Your input object is cloned, never mutated

## CLI

```
squeeze bench [safe|aggressive|none]   run the benchmark on synthetic fixtures
squeeze bench real [level]             benchmark real collected tool output
squeeze bench all [level]              synthetic and real side by side
squeeze serve                          start the proxy
squeeze compress <file|--level>        compress a file or stdin
squeeze expand <file|--level>          resolve every ref back to the original
squeeze inspect <file|--level>         preview changes, write nothing
squeeze doctor                         check environment and store
squeeze selftest                       verify compression and ref durability
squeeze mcp                            run the MCP server on stdio
```

`--level` accepts `none`, `safe`, `aggressive`, or the alias `--aggressive`.
`SQUEEZE_LEVEL` sets the default when the flag is absent.

## Environment

| variable | default | meaning |
| --- | --- | --- |
| `SQUEEZE_LEVEL` | `aggressive` | `none`, `safe`, `aggressive` |
| `SQUEEZE_PORT` | `8899` | proxy port |
| `SQUEEZE_HOST` | `127.0.0.1` | bind address |
| `SQUEEZE_UPSTREAM_URL` | `https://api.openai.com` | forward target |
| `SQUEEZE_UPSTREAM` | `openai` | `openai` or `anthropic` |
| `SQUEEZE_STORE` | `./.squeeze/store.jsonl` | where refs are persisted |

Endpoints: `/squeeze/health`, `/squeeze/expand?ref=…`

## Tests

```bash
npm test                      # 31 unit tests
npm run test:proxy            # end-to-end proxy, real HTTP, all 3 levels
npm run test:mcp              # MCP over a real stdio JSON-RPC session
npm run test:persistence      # ref survives a full proxy restart
```

The unit suite covers the machinery directly: `looksStructured` protects JSON
and source files, document-wide dedup collapses non-adjacent repeats while
keeping every line's slot, and `expand()` reconstructs the original document
byte-for-byte.

The proxy test asserts the full round trip: compress a nested Anthropic
`tool_result`, confirm the forwarded payload shrank, confirm the untouched fields
survived, then expand the ref and count all 120 original lines back.

The persistence test is the important one. It kills the proxy outright, starts a
fresh one, and expands a ref created before the kill. That is the failure mode
that would silently corrupt a long-running agent session.

## Status

Working and tested: proxy, CLI, MCP server, persistent store, document-wide
dedup, byte-exact `expand()`, and all three request shapes (Anthropic
`messages`, OpenAI chat `messages`, OpenAI Responses `input`).

Not yet: per-project store scoping, and a proxy mode for deploying squeeze as a
separate service rather than on the agent's own host.

## Provenance

The real fixtures in `fixtures/real/` and the terminal screenshot in the README
are both generated, never hand-typed. `scripts/collect-real.sh` regenerates the
fixtures from the same tools that built the machine (tsc, npm, pytest,
node --test, this repo's public CI log, and the system dpkg log), and
`scripts/make-assets.py` re-renders the README images from live CLI output.
Where a tool is missing on your machine the scripts say so and keep the
checked-in copy.

## License

MIT