TRACE — Telemetry Reconciliation And Correctness Engine

An MCP server that traces synthetic events through a toy ingestion pipeline. Built to demonstrate the exact tool PostHog's own Q3 2026 objectives name as unbuilt: "Ship an ingestion MCP for tracing events through the pipeline."

The pipeline mirrors PostHog's documented architecture (Capture → Kafka → ingestion worker → ClickHouse) with lighter local equivalents — Redis Streams for Kafka, Postgres for ClickHouse — so the whole thing runs in Docker Compose with no cloud dependency.

producer.ts --fault-flags-->  Redis Stream "raw-events"
                                      |
                              processor.ts (consumer group "ingest-worker")
                                      |
                         [received] -> [enriched] -> [written]
                                      |
                              PostgreSQL "sink_events" table
                                      |
                         Every stage transition is logged to
                         PostgreSQL "event_trace_log" table
                         (append-only, never mutated)

mcp-server.ts reads from sink_events + event_trace_log to answer tool calls.
It never touches the Redis stream directly (except pipeline_health, which
reports broker-level length/lag) — it queries derived state only.

The core idea

producer_log is the ground truth (what the producer tried to do, including which faults it deliberately applied). event_trace_log + sink_events is what the pipeline observed and did. Every correctness claim in this repo is a comparison of those two independently-written records — never inferred from one side alone.

Related MCP server: otel-mcp

Prerequisites

Docker + Node 22.

Demo walkthrough (the 2-minute version)

# 1. Start everything. This must genuinely work from a clean clone.
docker compose up -d
npm install
npx tsx src/migrate.ts

# 2. Emit 1000 synthetic events with known faults: 5% dropped, 3% duplicated.
npx tsx src/producer.ts --events 1000 --drop 5 --duplicate 3

# 3. Run the ingestion worker (drains the stream, then exits).
npx tsx src/processor.ts --once

Now connect an MCP client to the server. With Claude Desktop, add this to claude_desktop_config.json:

{
  "mcpServers": {
    "trace": {
      "command": "npx",
      "args": ["tsx", "/absolute/path/to/TRACE/src/mcp-server.ts"],
      "env": {
        "DATABASE_URL": "postgres://trace:trace@localhost:5432/trace",
        "REDIS_URL": "redis://localhost:6379"
      }
    }
  }
}

(Or point any MCP Inspector-style client at npx tsx src/mcp-server.ts, which uses the stdio transport. Note: some clients launch servers with an unfamiliar working directory — if tsx isn't resolved, replace "command": "npx" with the absolute path to your node binary and "args" with ["/absolute/path/to/TRACE/node_modules/tsx/dist/cli.mjs", "/absolute/path/to/TRACE/src/mcp-server.ts"].)

Then ask:

"What got dropped?"

The agent calls find_dropped_events() and gets back the exact set of dropped event IDs, each marked expected: true (deliberate --drop fault) or expected: false (genuinely lost — a real pipeline bug). Ask "Were there duplicates, and did the pipeline handle them?" → find_duplicate_events() shows each twice-sent event collapsed to exactly one sink row. Ask "What happened to event X?" → trace_event(X) returns the producer record, the ordered received → enriched → written stage history, and a final_status of written | dropped | in_flight | missing_unexpectedly. Ask "Is the pipeline healthy?" → pipeline_health() reports stream length, consumer lag, throughput, and error rate.

Fault injection

npx tsx src/producer.ts --events 1000 --drop 5 --duplicate 3 --delay 2 --delay-ms 4000

MCP tools

Validation

npm test   # Vitest: unit + property-based (fast-check) + MCP protocol integration

• Ground-truth reconciliation (tests/reconciliation.test.ts): with --events 1000 --drop 5 --duplicate 3, the tool-reported drop/duplicate sets are compared element-for-element against producer_log. Zero false positives, zero false negatives — or nothing else matters.

• Property-based (tests/properties.test.ts): no silent loss, no phantom success, duplicate collapse, trace ordering (+ a pure model check on fault planning across 100 seeds).

• MCP protocol integration (tests/mcp-integration.test.ts): spins up the real server process, connects a real client over stdio, calls all four tools.

• Load sanity (tests/load.test.ts): health numbers track an independent Redis XLEN measurement at increasing volumes.

• Delay ground truth (tests/delay.test.ts): delayed events' received timestamps lag emitted_at by the full hold.

Project structure

docker-compose.yml          Redis + Postgres, one-command up
src/
  schema.sql                producer_log / event_trace_log / sink_events
  db.ts                     pg pool + shared constants
  migrate.ts                applies schema.sql
  producer.ts               synthetic producer CLI with --drop/--duplicate/--delay
  processor.ts              consumer-group worker: received → enriched → written
  tools.ts                  the four tool implementations (Postgres only)
  mcp-server.ts             stdio MCP server exposing the tools
tests/
  producer.unit.test.ts     arg parsing + fault planning model
  reconciliation.test.ts    §6.1 exact-set ground-truth proof (1000 events)
  properties.test.ts        §6.2 fast-check invariants
  mcp-integration.test.ts   §6.3 real server + real client over stdio
  load.test.ts              §6.4 health vs independent XLEN
  delay.test.ts             delayed-fault timestamp ground truth

Known spec deviation

The spec's DDL declares producer_log.event_id as PRIMARY KEY and its fault table describes a second producer_log row for duplicates — these two can't both hold. The DDL wins: the producer upserts a single row per event (fault_applied='duplicated', fault_meta.duplicate_of = self). All ground-truth queries key on fault_applied, never on row counts.

Non-scope

No web UI, no multi-provider adapters, no real Kafka, no real ClickHouse, no auth/multi-tenancy, no rules engine. Small, tightly-scoped proof of one capability.