# TraceDelta

**A shared, source-verifiable evidence-memory layer for AI operations — over SigNoz.**

Built for the [WeMakeDevs "Agents of SigNoz" hackathon](https://www.wemakedevs.org/hackathons/signoz).

---

Today, multiple AI agents and engineers investigating the same production incident
each re-query the same logs, traces, and metrics — wasting LLM context, telemetry
work, and reaching inconsistent conclusions. TraceDelta sits **between AI agents
and SigNoz** and turns a bounded investigation into a **versioned, source-linked
evidence capsule**:

- incident scope (env, services, time range, symptom) as a canonical identity;
- **typed** claims — observed facts, comparisons, correlations, hypotheses,
  action-candidates, and explicit **known-unknowns** (never one confident blob);
- evidence references (trace ids, span ids, log clusters, metric queries, alerts);
- freshness timestamps and a **delta** showing what changed since the last version;
- a **savings ledger**: reused evidence, duplicate queries avoided, context reduction.

> It is **not** a generic "AI summary of logs" and **not** another dashboard. It is
> a reusable, evidence-linked, freshness-aware investigation artifact that routes
> back to the exact telemetry.

## Validation status (honest)

Validated against **live, self-hosted SigNoz v0.134.0 (not SigNoz Cloud)**, deployed
by **Foundry** — full detail and evidence in [`docs/validation.md`](docs/validation.md)
and [`docs/live-evidence/`](docs/live-evidence).

- ✅ **Foundry installs SigNoz + its MCP server** (`spec.mcp.spec.enabled: true` →
  forged `signoz-mcp`); stack healthy, ran 11h+.
- ✅ **Real ingestion from the demo** — `checkout-service` / `payment-service` with
  100% error rate, real 502 error logs and traces (root spans `POST /checkout` ~1.25s).
- ✅ **TraceDelta ↔ live SigNoz (REST adapter, v0.134)** — full flow on real data:
  `start_investigation` (100% error rate, p95 5.9× baseline) → reuse → delta →
  `verify_claim` (supported) → `drill_down` (live trace + deep link) → savings
  (**30.6× context compression, 6 duplicate queries avoided**).
- ✅ **TraceDelta's own OTel in SigNoz** — `tracedelta-mcp` appears as a service with
  `start_investigation` / `refresh_capsule` spans (the closed loop).
- ✅ **Official SigNoz MCP server** completes an `initialize` handshake at `:8000/mcp`.
- ✅ **Fixture tier** — 24 unit tests; `make demo` (25.5× compression); image builds.
- ✅ **Screenshots** (headless Chromium, real) — SigNoz Services + Traces of the live
  incident, and the TraceDelta capsule / delta / evidence-drawer / savings surfaces.
  See [`docs/live-evidence/screenshots/`](docs/live-evidence/screenshots).

The default `FixtureSigNoz` backend reproduces the incident shape so the demo/tests
run with no backend — the blueprint-sanctioned adapter fallback (§15).

## The 30-second proof

```bash
make install     # uv venv + deps (Python 3.12)
make test        # 24 tests: matching, versioning, delta, redaction, evidence links
make demo        # two agents investigate one incident, end-to-end
make web         # http://127.0.0.1:8700 — inspect the capsule visually
```

`make demo` shows: Agent A opens a capsule → Agent B **reuses** it (no re-query) →
`refresh` returns a **delta** (retries increased, no new signature) → `verify_claim`
routes to source evidence → `drill_down` returns a **redacted** raw trace →
`report_savings` reports duplicate queries avoided and ~20×+ context compression.

## MCP tools

| Tool | What it does | Safety |
|---|---|---|
| `start_investigation` | Begin or **reuse** a bounded investigation → capsule | read-only |
| `find_related_capsules` | Candidates for a scope | never auto-merges |
| `get_capsule` | Situation, typed claims, evidence, freshness, consumers | respects policy |
| `refresh_capsule` | Change-only **delta** + new version | read-only |
| `verify_claim` | Supporting + contradicting source evidence | shows query context |
| `drill_down` | Bounded **raw** evidence | redaction + field allow-list |
| `compare_capsules` | Structured version diff | no causal inference |
| `report_savings` | Reuse / avoided queries / compression | estimates labelled |

Connect Claude Code with the shipped `.mcp.json` (`/mcp` → `tracedelta`), or over
HTTP with `make mcp-http`. See [`docs/mcp-config.md`](docs/mcp-config.md).

## Architecture

Full diagram and request flow in [`docs/architecture.md`](docs/architecture.md).

```
AI clients / humans  ─►  TraceDelta MCP server  ─►  SigNoz (OTel: traces·logs·metrics·alerts)
                          │ resolver (deterministic canonical key)
                          │ evidence compiler (refs → typed claims)
                          │ delta engine · policy+redaction · savings ledger
                          └ SQLite (capsules · claims · evidence · deltas)
        TraceDelta also emits its OWN OpenTelemetry back into SigNoz (the closed loop)
```

- **Deterministic matching** — a human-inspectable canonical key
  (`backend | org | env | service-set | time-window | symptom-kind | signature | intent`),
  not an LLM guess. Related-but-not-identical investigations are surfaced, never merged.
- **Object model** — Investigation 1─N Capsule versions, Capsule 1─N Claims,
  Claim N─N Evidence, Capsule 1─N Deltas, Capsule 1─1 Policy.
- **Persistence** — SQLite (local-first, inspectable), full version history.

## SigNoz integration (and the Foundry deployment)

The final deployment target is **self-hosted SigNoz** (not SigNoz Cloud), installed
by **Foundry**. `casting.yaml` enables the MCP component, so `foundryctl` installs
and runs **both SigNoz and the official SigNoz MCP server** (`signoz-mcp` on
`signoz-network:8000`):

```yaml
spec:
  deployment: { flavor: compose, mode: docker }
  mcp:
    spec:
      enabled: true      # Foundry installs signoz/signoz-mcp-server alongside SigNoz
```

TraceDelta consumes SigNoz through documented **SigNoz REST APIs behind a clean
adapter** (blueprint §15); the Foundry-managed `signoz-mcp` is available for agents
to call directly at `http://localhost:8000/mcp`. Both live behind one
`SigNozClient` interface, and a deterministic `FixtureSigNoz` implements the same
interface so the demo/tests need no running backend.

```bash
make gauge        # foundryctl gauge — validate tools
make signoz-up    # foundryctl forge (→ casting.yaml.lock + pours/) + docker compose up (SigNoz + MCP)
make stack-up     # overlay: demo app + TraceDelta MCP + web (wired to self-hosted SigNoz)
```

`casting.yaml` and `casting.yaml.lock` are committed; `pours/` (the generated
compose stack + configs) is regenerable with `make forge`. Full-stack endpoints and
the incident load generator are documented in [`DEMO.md`](DEMO.md).

## Demo incident

`checkout-service` → `payment-service` → simulated external `stripe` provider
(`demo/`). The load generator runs baseline traffic, emits a **deploy marker**
(`checkout-service@v2.4.1`), then flips the provider into timeout mode: checkout
502s rise, retries climb, timeout spans dominate. TraceDelta correctly reports the
deploy as **correlation, not causation**, and names the safe next verification step.

## Security & trust

- **Read-only** MVP — no writes to production systems.
- **Redaction before storage/display** — emails, bearer tokens, API keys, JWTs,
  card- and secret-like values are stripped before they reach SQLite; trace/span
  ids are preserved as references.
- **Evidence-backed** — every material claim carries ≥1 source reference.
- **Honest gaps** — missing evidence or an unreachable backend is stated plainly;
  TraceDelta never invents a root cause.
- **Typed uncertainty** — facts, correlations, hypotheses, and unknowns stay distinct.

## Tests

```bash
make test
```

Covers deterministic matching, capsule versioning, delta generation, redaction
(incl. that raw secrets never reach the store), and claim↔evidence links. A sample
capsule is in [`docs/sample_capsule.json`](docs/sample_capsule.json).

## Project layout

```
tracedelta/        core: identity · redaction · models · db · resolver · compiler
                        · delta · savings · service · mcp_server · web · otel
  signoz/          SigNozClient: base · fixture (deterministic) · api (real REST)
demo/              checkout-service · payment-service · loadgen (OTel-instrumented)
scripts/           demo.py (end-to-end) · gen_sample.py
tests/             matching · versioning · delta · redaction · evidence links
deploy/            tracedelta.compose.yaml (overlay onto Foundry's signoz-network)
casting.yaml(.lock)  SigNoz deployment via Foundry
docs/              architecture.md · mcp-config.md · validation.md · live-evidence/
```

> The product blueprint that guided this build (vision, security model, spec) is
> kept private and is not part of this repository.

## Limitations (MVP, by design)

Deferred per blueprint §11: universal semantic matching of arbitrary questions,
autonomous remediation/writes, multi-tenant sharing, exact provider-billing
reconciliation, and automatic root-cause claims. The real SigNoz REST adapter is
best-effort and version-sensitive; the fixture backend is the guaranteed demo path.

## AI-assistance disclosure

See [`DISCLOSURE.md`](DISCLOSURE.md).