Skip to main content
Glama
workflow-replay

workflow-replay

by owentw23

Reflo

Workflow replay for AI agents

Reflo learns recurring read-only workflows from your agent’s tool calls. When a similar request comes in, it runs the saved steps with fresh inputs and gives the model a compact result to answer from.

Node Claude Code Codex MCP Tests Savings License

What it does · How it works · Results · Quick start · Usage · Safety · Docs

NOTE

Reflo pays off onrecurring read-only lookups such as order status checks, account summaries, and ticket counts. It does little for open-ended work such as writing code. Run the built-in analyzer on your own history before you deploy it.

Contents

Related MCP server: Pilot Browse MCP

What it does

Agents spend most of their tokens on tool calls: deciding what to call, calling it, then reading the full raw response back in. Each extra turn re-sends the whole conversation. When the same request comes back with different values ("status of order #W123", then "#W456"), the agent does all of that again from scratch.

Reflo sits between the agent and its tools.

  • Learns workflows automatically from the agent's own successful runs: which calls, in what order, with which values coming from the request and which from earlier results. It handles loops, pagination, filtered counts and totals.

  • Verifies each learned workflow by replaying it against past runs, then runs it in shadow mode next to the real agent before it goes live.

  • Replays matching requests before the model starts working, or mid-turn when the model's first tool call matches a known workflow, and hands the model a compact receipt (item_count=5; total=3819.12) to answer from.

  • Keeps the model in charge. By default the model still writes every answer, using the receipt and the conversation, so answers stay context-aware and get checked. Unusual results are flagged to it.

  • Never writes on its own. Workflows that change data only prepare the exact action; it runs once, after your Claude Code permissions allow it or you approve it.

How it works

flowchart TB
  H1["Claude Code or Codex"] --> R["Router"]
  H3["Your own agent"] --> R
  H2["MCP clients"] --> G
  R -- "match" --> X["Run saved plan"]
  R -- "no match" --> A["Normal agent turn"]
  A --> G["Gateway"]
  X --> G
  G --> U["Your tools and APIs"]
  G --> L["Learning loop"]
  L --> C[("Playbook")]
  C --> R

Part

What it does

Router

Finds a matching workflow, fills in its inputs, runs the safety gate

Run saved plan

Replays the learned calls with the new inputs

Gateway

Trims tool results to the needed fields and records every call

Learning loop

Mines repeated runs, verifies them, merges duplicates, shadows new ones

Playbook

Local SQLite store of learned workflows, shared by all sessions

First time a task is seen: the agent works normally. Every call goes through the gateway, which trims results to the fields that matter and records the run.

After a few similar runs: the miner groups runs by call shape, works out which values are inputs, and compiles a plan. It's checked against past runs, then shadowed against the live agent. Routine consolidation merges near-duplicate plans: two plans that differ only in a fixed value become one plan with a placeholder.

When a similar request comes back:

sequenceDiagram
  participant U as User
  participant H as Hook
  participant R as Router
  participant X as Saved plan
  participant M as Model
  U->>H: Where is order W456?
  H->>R: request text
  R->>R: find match, fill inputs, safety gate
  alt verified match
    R->>X: run with order W456
    X-->>H: status delivered, tracking 1Z99
    H->>M: request plus compact receipt
    M-->>U: answer, no tool calls
  else no safe match
    H->>M: normal turn with trimmed tools
    M-->>U: answer
  end
  • Every input value must appear in the request (or in a result carried from earlier in the conversation).

  • Words that change the question ("first" instead of "latest", "unavailable", "cancel", "zip" instead of "city") decline the match unless the workflow was learned with them.

  • A learned filter ("delivered" orders) must be mentioned in the request.

  • Terms that belong only to a competing workflow decline the match.

  • Mid-turn replays (the compiled run_workflow tool and gateway speculation) pass the same gate as pre-model routing.

Results

Measured on real records from the public τ²-bench retail and airline databases (500 customers, 1,000 orders, 2,000 reservations), with DeepSeek V4.1 Flash. There are 69 test requests × 2 repeats × 3 arms, run in random order. Learning happened first, on different customers.

Plain agent

Reflo (model writes every answer)

Accuracy

99.3%

100%

Tokens per request

4,026

2,489 (−38%)

Estimated cost*

—

−17%

Model calls per request

2.97

2.62

Median latency

2.6 s

2.0 s

Wrong answers caused by a replay

—

0 of 44 tricky controls

* Cost units weight uncached input 1×, cached input 0.1× and output 2×.

xychart-beta
  title "Tokens per request (bars: plain agent, line: Reflo)"
  x-axis ["Exact repeat", "Reworded repeat", "Unusual result", "Similar but different", "New task"]
  y-axis "Tokens" 0 --> 5500
  bar [3761, 3833, 5298, 3613, 2476]
  line [191, 1191, 651, 4363, 3339]

Request type

Plain agent

Reflo

Change

Repeat, same wording

3,761

191

−95%

Repeat, new wording

3,833

1,191

−69%

Repeat with an unusual result (flagged to the model)

5,298

651

−88%

Similar wording, different question

3,613

4,363

+21%

Never-seen task

2,476

3,339

+35%

TIP

Repeats get much cheaper and everything else gets a bit more expensive, sobreak-even is roughly 35% repeat traffic. If you let the harness skip the model on clean hits (opt-in), repeats cost nothing and the totals go to −42% tokens and −30% cost.

Full method, every run (including the ones that went wrong and what fixed them) and limits: docs/v2-results.md.

Who benefits

You run…

Expect

A support bot or internal assistant answering the same kinds of lookups all day

Large savings on the repeated share, faster answers

Claude Code or Codex against company APIs (ERP, CRM, ticketing) where you ask the same questions repeatedly

Good savings on those lookups; the model still writes every answer

Claude Code for everyday coding

Little to none. The analyzer found 0.5% replayable turns in 596 coding turns

One-off research or writing

None; nothing repeats

Quick start

Prerequisites: Node.js 24 or later. Claude Code and/or Codex if you want the automatic hooks.

git clone https://github.com/owentw23/workflow-replay.git
cd workflow-replay && npm ci

1. Measure first

npm run analyze

This scans your local Claude Code transcripts and prints only totals: how many turns were read-only lookups that repeat, their share of your tokens, and a savings ceiling. Nothing leaves your machine.

2. Create a profile for your service

A profile tells the harness which operations exist and whether each one reads or writes. Generate one from what you already have:

npm run wrap -- mcp my-service --read-only -- node path/to/your-mcp-server.mjs
npm run wrap -- openapi my-api --spec openapi.json --base https://api.example.com --token-env MY_API_TOKEN

Each operation is labelled read or write, with the reason. Unclear operations are labelled write, so they need approval. --read-only drops write operations entirely, which is the right choice for systems that must never be changed. Review the generated file in ~/.workflow-replay/v2/profiles/.

3. Install into Claude Code and Codex

npm run setup -- --profile my-service

One command adds the hooks and the workflow-replay MCP server to every host it finds, backing up each file first. Preview with --dry-run, pick hosts with --target claude|codex, or install for one project with --scope project --project ..

npm run setup -- --remove

That removes everything the installer added; your profiles and playbooks are kept. Start a new Claude Code session afterwards, and in Codex, review and trust the hooks in /hooks.

4. Use your agent as usual

Reflo records gateway calls, learns every few turns, shadows new workflows, and promotes read-only ones after they agree with the agent 5 times. Use the status command to inspect the playbook.

node src/v2/hosts/cli.mjs my-service status

Usage

Answer modes

Mode

What happens on a hit

Set with

Inject (default)

The model gets the compact receipt and writes the answer: no tool calls, full context

—

Show-and-carry

The hook answers immediately with zero model calls; the result is carried into your next message so follow-ups work

--mode carry at install

Direct (own agents)

Your app returns the result without a model call

policy: { answerMode: 'direct' }

Unusual results (missing values, unseen statuses, unexpected zeros) always go to the model, with a note on what's unusual.

Your own agent

import { createHarness } from './src/v2/harness.mjs';
import { openStore } from './src/v2/store.mjs';

const harness = createHarness({ registry, upstream, project, extract, store: openStore('playbook.sqlite') });

// Before the model: replay a known workflow, or get prefetched data for a related request.
const hit = await harness.handle(userText, { session });
if (hit.mode === 'inject') {
  // One lean model call: a short instruction + conversation + hit.receipt, no tool list.
}

// During the model's turn: route its tool calls through the gateway so they are trimmed and recorded.
const gate = harness.gateway({ request: userText, context: hit.context });
const result = await gate.call(toolName, args);
gate.finish(finalAnswer, { success: true });

await harness.learn(); // mine, verify, consolidate; run periodically

registry maps operation names to { description, args, optional?, effect: 'read' | 'write' }. upstream(op, args) performs the real call. extract is an optional small-model call the router uses when wording differs from past examples.

Command

Purpose

npm run analyze

Measure the replayable share of your past transcripts

npm run wrap -- mcp NAME -- <cmd>

Profile from an MCP server

npm run wrap -- openapi NAME --spec F --base URL

Profile from an OpenAPI (JSON) spec

npm run setup -- --profile NAME

Install hooks and MCP server into Claude Code and Codex

npm run setup -- --remove

Uninstall

cli.mjs NAME status

Learned workflows and their states

cli.mjs NAME learn

Learn now instead of waiting for the next round

cli.mjs NAME approve ROUTE

Promote a workflow that passed shadow mode (required for write workflows)

cli.mjs NAME ask "request"

See what the harness would do with a request

cli.mjs NAME feedback EPISODE good|bad

Rate a recorded run

cli.mjs NAME commit PROPOSAL.json

Review and approve a prepared write in the terminal

Safety

  • Reads replay; writes never do. A learned workflow that ends in a write prepares the exact call. In Claude Code, commit_workflow_action checks your permission settings for that write tool. A deny rule refuses it, an allow rule runs it once, and anything else asks you first. Every proposal runs at most once, expires after 10 minutes, and is tied to the user who requested it.

  • Each user's own access. Every call, including replays, prefetches and verification runs, carries the requesting user (the principal) to your upstream, which uses that user's credentials. An optional authorize(principal, op, args) must allow each call. Workflows store structure, never data, so a shared playbook can't leak one user's results to another.

  • Data stays data. Results passed to the model are sanitised: invisible characters removed, long text cut, and anything that reads like instructions to a model ("ignore previous instructions…") withheld and flagged to the user. They are wrapped in <data> markers.

  • Verified before use, watched after. Each workflow is replayed against past runs, then must agree with the live agent in shadow mode; a single disagreement retires it. Once live, every 20th hit runs the ordinary agent as a canary, and a workflow goes back to shadow mode if the canary disagrees, a returned field changes type, or most recent results look unusual.

  • Declines when unsure. Misses, ambiguities and near-misses fall back to the ordinary agent. Mid-turn replays pass the same gate.

  • Local data. Profiles, playbooks and recorded runs stay in ~/.workflow-replay (or the folder you set in WORKFLOW_REPLAY_HOME). Recorded runs contain fetched data, are deleted after 30 days, and should stay out of git. Workflow descriptions never include input values.

  • Your config is respected. The installer backs up every file it touches and leaves other hooks and MCP servers untouched.

WARNING

Profiles are JavaScript you run with your own credentials. Only use profiles you wrote or reviewed, and keep systems that must never be changed on--read-only profiles.

Reproduce the benchmark

npm run bench:v2:fetch
npm test

Put a BytePlus ModelArk key in cred.env (ARK_API_KEY=…, never commit it), then:

npm run bench:v2 -- --repeats 2 --workers 4

The run writes prompts, traces, usage and a report to .replay/v2-*/. A full run takes about 6 minutes and roughly 1M tokens.

Project layout

src/v2/                 core: plan format, miner, router, gateway, carry store, harness
src/v2/hosts/           Claude Code / Codex hooks, MCP gateway, installer, wrap, analyzer, permissions, CLI
eval/v2/                benchmark: τ²-bench service, tasks, agent loop, report
test/                   unit and integration tests (node --test)
docs/                   architecture, build spec, results
src/, eval/ (top level) v0.1: hand-written workflows, prompt hook, remote connector

Limitations

  • Plans are declarative: lookups, loops, pagination, filters and totals. Branching logic and nested totals fall back to the agent.

  • The default retrieval uses TF-IDF plus a local hashed embedding; plug in a real embedding model through embed for synonyms that share no letters.

  • Codex hooks are installed but have not been tested in a live Codex session. Claude Code hooks, show-and-carry, follow-ups and permission-checked writes were verified with Claude Code 2.1.319.

  • The benchmark uses one model, authored phrasings and a fixed task mix. Measure on your own traffic.

  • OpenAPI wrapping reads JSON specs only.

Documentation

Acknowledgements

Benchmark data comes from τ²-bench (Sierra Research, MIT), downloaded at a pinned commit and not redistributed here. Earlier evaluation prompts adapt items from BFCL and IBM's BFCL-FC-robustness.

License

MIT

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      D
      maintenance
      Enables efficient AI workflow orchestration by chaining multi-step LLM operations while keeping intermediate results out of the context window, reducing token usage by 90%+ and supporting multiple AI providers.
      7
      41 npm
      1
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI agents to save solutions as repeatable workflows that run automatically with new data via email, Google Drive, or schedule, eliminating token costs for recurring tasks.
      MIT