tooltrim

Drop-in compression for LLM agent tool outputs. Shrink bloated tool results — fetched web pages, paginated JSON, log dumps, CSV exports, long documents — before they re-enter your agent's context window. Keep the facts the model needs, drop the boilerplate, and keep the full output one expand() away.

from tooltrim import compressed_tool

@compressed_tool(max_tokens=400)
def web_fetch(url: str) -> str:
    ...                      # returns a 3,000-token HTML page
# your agent now receives a compact, on-topic extract instead

• Zero dependencies in the core. Pure-stdlib, deterministic, reproducible.

• Provider-agnostic. Works with OpenAI, Anthropic, local models, LangChain, LlamaIndex, OpenAI-Agents, MCP, raw function-calling — anything. It compresses strings, not APIs.

• Lossless by reference. Compression is extractive, and the full output stays retrievable via a short ref — so it's compression plus retrieval, not blind truncation.

• Content-aware. Separate compressors for HTML, JSON, tabular data, logs, and free text. Optionally query-aware (BM25) to keep what the agent is actually looking for.

• Faithfulness-tested. A built-in harness measures whether the model still answers correctly on compressed output (with Wilson 95% CIs) — not just how many tokens you saved.

• Deploy as a proxy. An OpenAI-compatible compression proxy trims role:"tool" messages in flight, so any app/language adopts it with zero code changes — just a base_url.

Why

In a real agent loop, the prompt isn't what blows up your context — tool outputs are. A single web_fetch returns thousands of tokens of nav bars and footers; a REST call returns a 300-item paginated array; a log tool dumps 10,000 lines of INFO heartbeat. And because the agent's transcript is replayed on every turn, you pay for that bloat again and again — slower responses, higher bills, and a model that loses the thread.

Routers, caches, and prompt compressors don't touch this. tooltrim targets the tool output directly, at the exact point it enters context.

Related MCP server: Refract

Benchmark

Realistic tool outputs compressed to a 400-token budget, exact tiktoken (cl100k_base) counts. Each output contains one planted fact ("needle") that the agent needs; tooltrim is given the task as its relevance query. Reproduce with benchmark.py.

39,575 → 1,111 tokens — a 35.6× smaller context, with the relevant fact kept in every case. (HTML/text collapse to the matching passage when the query pinpoints it; structured types keep a representative, schema-preserving sample.)

Does compression lose information? (it can help)

Throwing away 99% of the tokens is only safe if the model still answers correctly. We measure that directly: for 62 curated (tool output, question, gold answer) cases across all five content types — including multi-fact cases (the answer needs several facts from different parts of the output) and distractor cases (a deprecated value sits next to the current one) — a model is asked the question twice: once on the full output, once on the tooltrim-compressed output. Accuracy is reported with Wilson 95% confidence intervals. Reproduce with run_faithfulness.py — it runs offline by default (no API key) and has adapters for Claude / OpenAI / Groq / Ollama.

On small local models, compression doesn't just preserve accuracy — it improves it, because the model is no longer distracted by thousands of tokens of noise. The effect reproduces across two independent model families:

The compressed intervals don't overlap the full-context intervals — at n=62 this is a significant improvement for both models, not noise. Full provenance, per-case answers, and the cross-model table are saved as citable artifacts under benchmarks/runs/ and benchmarks/COMPARISON.md.

Stated plainly: these are small 7–8B models. A frontier long-context model handles the full context far better, so its baseline is higher and the accuracy uplift shrinks — but the token/cost savings remain. The uplift is largest for smaller/cheaper models and longer contexts. The harness is wired so a frontier run (--model claude) drops a new row into the same table when an API key is available; n=62 is a pilot, which is why the CIs are reported.

How does it compare to truncation and RAG?

Preserving accuracy vs full context only matters if it beats the obvious alternatives. run_baselines.py scores tooltrim against naive truncation, query-aware RAG top-k, RAG-embed, and LLMLingua-2 on the same cases and budgets, with a paired McNemar significance test. Retention (accuracy ÷ full-context accuracy), offline judge:

Query-aware compression retains 100% of accuracy while cutting 94–99% of tokens; blind truncation drops the needed fact and collapses (p < 0.001 at every budget). The offline judge is itself lexical, so RAG top-k ties tooltrim here — tooltrim's content-type structure advantage surfaces with a real-LLM judge on structured output. Details, caveats, and the full grid: benchmarks/BASELINES.md.

End-to-end: does it preserve task success? (tau-bench, multi-step)

Single-turn faithfulness isn't the whole story — in a real agent loop a compressor can drop a field the agent only needs three turns later. run_taubench.py measures that directly: it wraps tau-bench's own environment so every tool observation is compressed before it re-enters the agent's context, while tau-bench's reward function, LLM user simulator, and agent stay untouched. A compressor that shreds an output the agent needs later shows up as lower task success — the outcome metric, not a proxy.

The harness reports task success with Wilson 95% CIs over all (task, trial) observations and a paired McNemar test vs tooltrim, sweeps multiple token budgets to trace the accuracy-vs-budget curve, and emits a reproducibility manifest (pinned tau-bench commit, resolved model snapshot, seeds) alongside raw per-task JSON so every number is re-derivable offline without re-spending on the API. It runs against any litellm-supported model for both the agent and the user simulator.

Status: harness implemented and under pilot on tau-bench retail with gpt-4o-mini; results land in benchmarks/TAUBENCH.md. One design note surfaced by the pilot: retail's native tool outputs are modest (~130–650 tokens), so runs use a tight budget (≈128 tokens) where compression actually engages rather than passing through.

Install

pip install tooltrim          # zero-dependency core (heuristic token counts)
pip install tooltrim[tokens]  # add tiktoken for exact token counts

Extras: tooltrim[langchain], tooltrim[redis], tooltrim[s3].

CLI

tooltrim demo                                   # 10-second self-contained savings tour
cat big.json | tooltrim compress -q "refund status" --stats   # pipe in, compressed out
tooltrim compress page.html -q "rate limits" -m 400
tooltrim proxy --upstream https://api.openai.com/v1           # run the proxy

Usage (library)

1. Decorate a tool

from tooltrim import compressed_tool

@compressed_tool(max_tokens=400)
def read_file(path: str) -> str:
    return open(path).read()

2. Make it query-aware

Pull the relevance query from the call arguments…

@compressed_tool(max_tokens=400, query_from=lambda query, **_: query)
def web_search(query: str) -> str:
    ...

…or set the agent's current goal ambiently, so every tool call this turn keeps what's relevant to it:

from tooltrim import query_scope

with query_scope("find the customer's refund status"):
    result = run_agent_step()   # all @compressed_tool calls inside use this query

3. Imperative API + expand-on-demand

from tooltrim import ToolCompressor

tc = ToolCompressor(max_tokens=400)
res = tc.compress(huge_json_response, query="refund status for customer C-1007")

res.text             # compact text to feed back to the model
res.saved_tokens     # e.g. 14794
res.saved_ratio      # e.g. 0.979
res.ref              # e.g. "a1b2c3d4"

full = tc.expand(res.ref)                    # get the original back
slice_ = tc.expand(res.ref, start=0, length=2000)

By default the compressed output ends with a small footer the model can act on:

…compressed extract…

[tooltrim: compressed 15119->325 tokens (saved 14794); full output ref=a1b2c3d4]

Expose an expand(ref) tool to your agent and it can pull the full output back whenever the extract isn't enough — turning aggressive compression into a safe default. tooltrim hands you both the tool schema and the handler:

tools = my_tools + [tc.expand_tool_spec(style="openai")]   # or style="anthropic"

# when the model calls expand_tool_output(ref=..., start=..., length=...):
result_text = tc.handle_expand(ref, start=start, length=length)   # paged, safe

See examples/04_expand_tool.py for a full wiring. Extractive compressors also keep neighbor context (a line/sentence around each match) so the model gets context, not just the bare matching line.

4. Optional: LLM distillation (any provider)

The deterministic compressors need no LLM. When you want summarization instead of extraction, plug in any model with a one-line completion function — use a small/cheap one; distilling 15k → 300 tokens once saves your expensive model from re-reading the blob every turn.

from tooltrim import LLMDistiller

def complete(prompt: str) -> str:
    # wrap OpenAI / Anthropic / local — your choice
    return my_client.responses(prompt)

distiller = LLMDistiller(complete, max_tokens=300)
summary = distiller.compress(huge_output, query="refund status")

5. Drop into LangChain — one line per tool

Already have LangChain tools? Wrap any of them and you get back a tool with the same name, description, and argument schema, so the agent calls it unchanged — but its (string) output is compressed before it lands in the scratchpad. The relevance query comes from the tool's own arguments.

pip install tooltrim[langchain]

from tooltrim.integrations import compress_langchain_tool, compress_langchain_tools

fetch = compress_langchain_tool(my_tool, max_tokens=400,
                                query_from=lambda query, **_: query)

# or wrap the whole toolset at once (sharing one compressor + expand store):
tools = compress_langchain_tools(my_tools, max_tokens=400)

See examples/03_langchain_tool.py.

6. Or LlamaIndex — same one-liner

pip install tooltrim[llamaindex]

from tooltrim.integrations import compress_llamaindex_tool, compress_llamaindex_tools

fetch = compress_llamaindex_tool(my_tool, max_tokens=400,
                                 query_from=lambda topic: topic)
tools = compress_llamaindex_tools(my_tools, max_tokens=400)

A LlamaIndex tool returns a ToolOutput; only its content (what the LLM reads) is compressed — the structured raw_output is preserved. See examples/05_llamaindex_tool.py.

7. Or the OpenAI Agents SDK — same one-liner

pip install tooltrim[openai-agents]

from tooltrim.integrations import compress_openai_agents_tool, compress_openai_agents_tools

fetch = compress_openai_agents_tool(my_tool, max_tokens=400,
                                    query_from=lambda url: url)
tools = compress_openai_agents_tools(my_tools, max_tokens=400)

Only the tool's on_invoke_tool is wrapped — name, JSON schema, and guardrails are preserved. See examples/06_openai_agents_tool.py.

8. Or at the MCP boundary — a compressing gateway

MCP tool results (tools/call) are exactly the bloated outputs tooltrim targets. Run a gateway in front of any MCP server and point your MCP client (Claude Desktop, an IDE, an agent) at it — every result is compressed in flight, no code change:

pip install tooltrim[mcp]
tooltrim mcp -- npx -y @modelcontextprotocol/server-filesystem /path

Or wrap the result-handling in your own server:

from tooltrim.integrations import compressing_call_tool, compress_tool_result

# wrap an upstream call_tool coroutine...
call = compressing_call_tool(session.call_tool, max_tokens=400)
# ...or compress a single CallToolResult (errors / non-text pass through)
result = compress_tool_result(result, compressor=tc, query=query)

See examples/08_mcp_gateway.py.

8b. Or expose tooltrim itself as an MCP server

Want compression as a tool your agent can call directly — no upstream server to front? Run tooltrim as a standalone MCP server. It exposes two tools over stdio: compress(text, query=None, max_tokens=None) and expand_tool_output(ref).

pip install tooltrim[mcp]
tooltrim serve

It's published to the MCP Registry as io.github.nac7/tooltrim, so an MCP-aware client can launch it with no clone:

uvx --from tooltrim[mcp] tooltrim serve

9. Or run it as a proxy — zero code changes

Point your client at the tooltrim proxy; every tool result is compressed (using the latest user message as the relevance query) before being forwarded upstream. Both wire formats are understood, routed by request path — you only change base_url.

python run_proxy.py --upstream https://api.openai.com/v1     # OpenAI-compatible
python run_proxy.py --upstream https://api.anthropic.com/v1  # Claude

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8800/v1", api_key="<upstream key>")

from anthropic import Anthropic
client = Anthropic(base_url="http://127.0.0.1:8800")

/v1/chat/completions compresses OpenAI role:"tool" messages; /v1/messages compresses Anthropic tool_result blocks. The proxy is stdlib-only and fails open: if anything goes wrong it forwards the original request untouched, so it never breaks a production call.

Online, it also keeps you under provider rate limits. Against a live hosted model (Groq free tier, 6,000-tokens-per-request cap), 45% of raw tool outputs are rejected (HTTP 413) but 100% of tooltrim-compressed calls fit — a 14,415-token result is compressed to 26 tokens in flight and the call succeeds. See benchmarks/ONLINE_GROQ.md.

10. Scale out — shared expand-store + metrics

The default expand-store is in-process, fine for one worker. To run multiple workers/replicas behind a load balancer, the store must be shared — otherwise a ref minted by one worker can't be expanded by another. Swap in a backend (all are content-addressed, so writes dedup automatically):

from tooltrim import ToolCompressor, FileStore, RedisStore, S3Store

tc = ToolCompressor(store=FileStore("/mnt/shared/tooltrim"))         # zero-dep, shared volume
tc = ToolCompressor(store=RedisStore(url="redis://cache:6379/0",     # pip install tooltrim[redis]
                                     ttl_seconds=86_400))
tc = ToolCompressor(store=S3Store(bucket="my-bucket"))               # pip install tooltrim[s3]

The proxy exposes Prometheus metrics at GET /metrics (tokens in/out/saved, messages compressed, fail-open count, upstream errors, latency) — scrape it to quantify savings fleet-wide:

tooltrim_tokens_saved_total 14389
tooltrim_messages_compressed_total 1
tooltrim_fail_open_total 0

How it works

1. Pass-through if the output already fits the budget (zero overhead).

2. Detect the content type (JSON / HTML / tabular / logs / text).

3. Compress with a type-specific strategy:

   • JSON — preserve structure; sample arrays (keeping the key schema), note (+N more items), truncate long strings; tighten until it fits.

   • HTML — extract readable text (drop script/style/nav/footer), then fit the budget.

   • Tabular — keep the header + a sample of rows + (+N more rows).

   • Logs — collapse repeated lines (x42), always keep errors/warnings, fill with head/tail context.

   • Text — query-aware extractive selection (BM25 or embeddings), […] elisions.

4. Stash the full output under a content-addressed ref for expand().

With a query, every compressor keeps the most relevant parts; without one, it falls back to structure-preserving head/tail selection.

Semantic relevance (optional)

Scoring defaults to lexical BM25 (zero-dependency). For semantic matching — so a query for "car" keeps a chunk about "automobiles" — pass an EmbeddingScorer. It's provider-agnostic: give it any embed(texts) -> vectors callable (OpenAI, Cohere, local), or let it load sentence-transformers (pip install tooltrim[embeddings]). The scorer threads through every content type:

from tooltrim import ToolCompressor, EmbeddingScorer

tc = ToolCompressor(max_tokens=400,
                    scorer=EmbeddingScorer(embed=my_client.embed))

Streaming (bounded memory)

Some outputs are too big to hold in memory — a multi-GB log, a subprocess's stdout, an HTTP stream. compress_stream consumes an iterable incrementally with constant memory (bounded head/tail/top-K/important-line buffers), then fits the survivors to the budget:

from tooltrim import compress_stream

text = compress_stream(open("huge.log"), max_tokens=400, query="disk error")

How it's different

tooltrim targets the tool-output boundary — the largest and most-ignored token sink in agentic apps — and works alongside all of the above.

Status

v0.2 — deterministic zero-dependency core, 104-test suite, reproducible token + faithfulness benchmarks (with Wilson CIs, cross-model), a proxy speaking both OpenAI and Anthropic wire formats with Prometheus /metrics, LangChain, LlamaIndex, and OpenAI-Agents adapters, an MCP compressing gateway, pluggable File/Redis/S3 expand-stores for horizontal scale, optional embedding-based relevance, streaming compression for outputs too big to hold in memory, a tooltrim CLI, a multi-step task-success harness that compresses tool outputs inside tau-bench's own agent loop (in-loop, McNemar vs baselines, reproducibility manifest), and citable run artifacts under benchmarks/. Published on PyPI.

Roadmap: frontier-model faithfulness runs, the scaled tau-bench task-success sweep (multi-budget, multi-trial) and its benchmark release, and native streaming passthrough in the proxy.

Contributions and benchmark cases welcome. MIT licensed.