TraceDelta

A shared, source-verifiable evidence-memory layer for AI operations — over SigNoz.

Built for the WeMakeDevs "Agents of SigNoz" hackathon.

Today, multiple AI agents and engineers investigating the same production incident each re-query the same logs, traces, and metrics — wasting LLM context, telemetry work, and reaching inconsistent conclusions. TraceDelta sits between AI agents and SigNoz and turns a bounded investigation into a versioned, source-linked evidence capsule:

• incident scope (env, services, time range, symptom) as a canonical identity;

• typed claims — observed facts, comparisons, correlations, hypotheses, action-candidates, and explicit known-unknowns (never one confident blob);

• evidence references (trace ids, span ids, log clusters, metric queries, alerts);

• freshness timestamps and a delta showing what changed since the last version;

• a savings ledger: reused evidence, duplicate queries avoided, context reduction.

It is not a generic "AI summary of logs" and not another dashboard. It is a reusable, evidence-linked, freshness-aware investigation artifact that routes back to the exact telemetry.

Validation status (honest)

Validated against live, self-hosted SigNoz v0.134.0 (not SigNoz Cloud), deployed by Foundry — full detail and evidence in docs/validation.md and docs/live-evidence/.

• ✅ Foundry installs SigNoz + its MCP server (spec.mcp.spec.enabled: true → forged signoz-mcp); stack healthy, ran 11h+.

• ✅ Real ingestion from the demo — checkout-service / payment-service with 100% error rate, real 502 error logs and traces (root spans POST /checkout ~1.25s).

• ✅ TraceDelta ↔ live SigNoz (REST adapter, v0.134) — full flow on real data: start_investigation (100% error rate, p95 5.9× baseline) → reuse → delta → verify_claim (supported) → drill_down (live trace + deep link) → savings (30.6× context compression, 6 duplicate queries avoided).

• ✅ TraceDelta's own OTel in SigNoz — tracedelta-mcp appears as a service with start_investigation / refresh_capsule spans (the closed loop).

• ✅ Official SigNoz MCP server completes an initialize handshake at :8000/mcp.

• ✅ Fixture tier — 24 unit tests; make demo (25.5× compression); image builds.

• ✅ Screenshots (headless Chromium, real) — SigNoz Services + Traces of the live incident, and the TraceDelta capsule / delta / evidence-drawer / savings surfaces. See docs/live-evidence/screenshots/.

The default FixtureSigNoz backend reproduces the incident shape so the demo/tests run with no backend — the blueprint-sanctioned adapter fallback (§15).

Related MCP server: agent-debugger

The 30-second proof

make install     # uv venv + deps (Python 3.12)
make test        # 24 tests: matching, versioning, delta, redaction, evidence links
make demo        # two agents investigate one incident, end-to-end
make web         # http://127.0.0.1:8700 — inspect the capsule visually

make demo shows: Agent A opens a capsule → Agent B reuses it (no re-query) → refresh returns a delta (retries increased, no new signature) → verify_claim routes to source evidence → drill_down returns a redacted raw trace → report_savings reports duplicate queries avoided and ~20×+ context compression.

MCP tools

Connect Claude Code with the shipped .mcp.json (/mcp → tracedelta), or over HTTP with make mcp-http. See docs/mcp-config.md.

Architecture

Full diagram and request flow in docs/architecture.md.

AI clients / humans  ─►  TraceDelta MCP server  ─►  SigNoz (OTel: traces·logs·metrics·alerts)
                          │ resolver (deterministic canonical key)
                          │ evidence compiler (refs → typed claims)
                          │ delta engine · policy+redaction · savings ledger
                          └ SQLite (capsules · claims · evidence · deltas)
        TraceDelta also emits its OWN OpenTelemetry back into SigNoz (the closed loop)

• Deterministic matching — a human-inspectable canonical key (backend | org | env | service-set | time-window | symptom-kind | signature | intent), not an LLM guess. Related-but-not-identical investigations are surfaced, never merged.

• Object model — Investigation 1─N Capsule versions, Capsule 1─N Claims, Claim N─N Evidence, Capsule 1─N Deltas, Capsule 1─1 Policy.

• Persistence — SQLite (local-first, inspectable), full version history.

SigNoz integration (and the Foundry deployment)

The final deployment target is self-hosted SigNoz (not SigNoz Cloud), installed by Foundry. casting.yaml enables the MCP component, so foundryctl installs and runs both SigNoz and the official SigNoz MCP server (signoz-mcp on signoz-network:8000):

spec:
  deployment: { flavor: compose, mode: docker }
  mcp:
    spec:
      enabled: true      # Foundry installs signoz/signoz-mcp-server alongside SigNoz

TraceDelta consumes SigNoz through documented SigNoz REST APIs behind a clean adapter (blueprint §15); the Foundry-managed signoz-mcp is available for agents to call directly at http://localhost:8000/mcp. Both live behind one SigNozClient interface, and a deterministic FixtureSigNoz implements the same interface so the demo/tests need no running backend.

make gauge        # foundryctl gauge — validate tools
make signoz-up    # foundryctl forge (→ casting.yaml.lock + pours/) + docker compose up (SigNoz + MCP)
make stack-up     # overlay: demo app + TraceDelta MCP + web (wired to self-hosted SigNoz)

casting.yaml and casting.yaml.lock are committed; pours/ (the generated compose stack + configs) is regenerable with make forge. Full-stack endpoints and the incident load generator are documented in DEMO.md.

Demo incident

checkout-service → payment-service → simulated external stripe provider (demo/). The load generator runs baseline traffic, emits a deploy marker (checkout-service@v2.4.1), then flips the provider into timeout mode: checkout 502s rise, retries climb, timeout spans dominate. TraceDelta correctly reports the deploy as correlation, not causation, and names the safe next verification step.

Security & trust

• Read-only MVP — no writes to production systems.

• Redaction before storage/display — emails, bearer tokens, API keys, JWTs, card- and secret-like values are stripped before they reach SQLite; trace/span ids are preserved as references.

• Evidence-backed — every material claim carries ≥1 source reference.

• Honest gaps — missing evidence or an unreachable backend is stated plainly; TraceDelta never invents a root cause.

• Typed uncertainty — facts, correlations, hypotheses, and unknowns stay distinct.

Tests

make test

Covers deterministic matching, capsule versioning, delta generation, redaction (incl. that raw secrets never reach the store), and claim↔evidence links. A sample capsule is in docs/sample_capsule.json.

Project layout

tracedelta/        core: identity · redaction · models · db · resolver · compiler
                        · delta · savings · service · mcp_server · web · otel
  signoz/          SigNozClient: base · fixture (deterministic) · api (real REST)
demo/              checkout-service · payment-service · loadgen (OTel-instrumented)
scripts/           demo.py (end-to-end) · gen_sample.py
tests/             matching · versioning · delta · redaction · evidence links
deploy/            tracedelta.compose.yaml (overlay onto Foundry's signoz-network)
casting.yaml(.lock)  SigNoz deployment via Foundry
docs/              architecture.md · mcp-config.md · validation.md · live-evidence/

The product blueprint that guided this build (vision, security model, spec) is kept private and is not part of this repository.

Limitations (MVP, by design)

Deferred per blueprint §11: universal semantic matching of arbitrary questions, autonomous remediation/writes, multi-tenant sharing, exact provider-billing reconciliation, and automatic root-cause claims. The real SigNoz REST adapter is best-effort and version-sensitive; the fixture backend is the guaranteed demo path.

AI-assistance disclosure

See DISCLOSURE.md.