Veritas

An AI-powered data investigation workspace for Product Managers, combining structured investigation workflows with a collectively maintained three-tier data catalog.

What This Is

Two things working together:

1. Investigation MCP Server -- a custom MCP server that provides investigation lifecycle tools (start, continue, capture, close), three-tier catalog management (local/community/curated), and a self-improving memory system

2. Data Investigation Framework -- a structured workflow for running multi-session analyses with persistent context, findings tracking, and institutional knowledge that compounds over time

3. Agent Memory System -- behavioral learning that enables the AI agent to remember corrections, preferences, and workflow patterns across sessions, with a promotion pipeline for turning recurring patterns into permanent rule improvements

Data access (SQL queries, schema discovery, profiling) is handled by the official Databricks MCP server, which runs alongside the investigation MCP server. This separation means adding new data sources (Dynatrace, Amplitude, etc.) is just adding another MCP server entry.

Related MCP server: KDD Conversacional (mcpDataAnalysis)

Quick Start

Setup takes ~5 minutes. See GETTING_STARTED.md for the full guide with screenshots, or follow the quick version:

1. Clone and run setup

git clone https://github.com/hertzcorp/htzd-databricks-mcp.git
cd htzd-databricks-mcp

Mac / Linux: ./setup.sh | Windows (PowerShell): .\setup.ps1

The script checks prerequisites, installs dependencies, then asks you a few questions: your name, email, Databricks HTTP path, and a personal access token. It shows you the exact links to copy each value. One run — no editing files.

2. Restart Cursor and approve MCP servers

Restart Cursor (Mac: Cmd+Q | Windows: Alt+F4, then reopen). When it starts, click "Start" on the two MCP server prompts at the bottom of the screen. This is a one-time approval -- all tools are pre-approved after that.

3. Start asking questions

Open the Chat panel (Mac: Cmd+Shift+I | Windows: Ctrl+Shift+I) and try:

• "What tables do we know about?" -- browse the team's data catalog

• "Start investigation" -- begin a tracked multi-session analysis

• "What data do we have on billing?" -- search the catalog for relevant knowledge

• "Deep investigate: why is contact rate rising?" -- autonomous recursive investigation

• "Deep index" -- crawl Databricks lineage to build the table knowledge base

See docs/ONE-PAGER.md for a quick reference of all features with end-to-end examples.

Project Structure

.
├── src/investigation_mcp/          # Investigation MCP server
│   ├── server.py                   # MCP server entry point (24 tools)
│   └── tools/                      # Tool implementations
│       ├── investigation.py        # Start, continue, capture, close investigations
│       ├── catalog_index.py        # Catalog reading: index, entries, health, context
│       ├── catalog_local.py        # Local entry capture + preferences
│       ├── catalog_sharing.py      # Share local entries to community via PR
│       ├── catalog_curation.py     # Curator tools: review, approve, reject
│       ├── memory.py               # Agent memory: capture, load, reinforce, review, apply
│       ├── _git_helpers.py         # Shared git utilities
│       └── catalog_management.py   # Legacy catalog tools (deprecated)
├── data-catalog/                       # Three-tier data catalog
│   ├── curated/                        # Reviewed, authoritative knowledge (team-shared)
│   ├── community/                      # Unvetted contributions from team members (team-shared)
│   ├── local/                          # Personal notes, auto-captured (gitignored)
│   ├── _deep_index/                    # Pre-computed table knowledge base (team-shared)
│   ├── _schema.yaml                    # Entry type definitions
│   └── _curators.yaml                  # Approved catalog curators
├── investigations/                     # Tracked analyses (gitignored except templates + index)
│   └── _templates/                     # Templates for new investigations
├── docs/
│   ├── hertzcorp-architecture/         # Hertz system architecture references
│   ├── adoption/                       # Adoption strategy and growth plans
│   ├── workshop/                       # Workshop facilitator guide and slide assets
│   ├── ONE-PAGER.md                    # Quick feature reference
│   ├── SPEC.md                         # Technical specification
│   └── PATTERNS.md                     # Architectural patterns
├── .cursor/
│   ├── memory/                         # Agent behavioral memory (gitignored)
│   │   ├── corrections/                # Behavioral corrections (YAML)
│   │   ├── preferences/                # Style and output preferences (YAML)
│   │   ├── patterns/                   # Detected workflow patterns (YAML)
│   │   └── reflections/                # Session retros (Markdown)
│   ├── rules/                          # Cursor rules
│   │   ├── analysis-principles.mdc     # 8 data analysis principles
│   │   ├── memory.mdc                  # Agent memory behavior (when to capture/load/cite)
│   │   ├── pm-guardrails.mdc           # Prevents AI from modifying protected files
│   │   ├── first-run.mdc              # Welcome experience for new users
│   │   └── ...                         # Development workflow rules
│   ├── hooks/
│   │   └── memory-flush.py             # Advisory: remind to capture learnings at session end
│   └── skills/
│       └── data-investigation/         # Investigation workflow skill
├── ideation/                       # Product concepts and drafts (gitignored)
├── scripts/                        # Standalone analysis scripts
├── .github/
│   └── workflows/
│       └── catalog-validation.yml  # CI: validates catalog entries on PR
├── setup.sh                        # Onboarding wizard
├── GETTING_STARTED.md              # PM-friendly setup guide
└── pyproject.toml                  # Package: investigation-toolkit

Three-Tier Data Catalog

The catalog is the institutional knowledge layer. It grows with every investigation.

Knowledge lifecycle:

Investigation discovers knowledge
  -> AI auto-captures to local/ (continuous, silent)
  -> PM selects what to share ("share this with the team")
  -> share_to_catalog creates PR to community/ (CI validates, approver merges)
  -> Curator reviews community entries (review_pending)
  -> Curator promotes to curated/ (approve_entry) or rejects (reject_entry)

The AI reads all three tiers with trust ordering: curated (high confidence) > community (hints, verified before use) > local (personal recall).

Investigation Framework

How It Works

Say "start investigation" in Cursor chat to begin a tracked analysis. The framework:

1. Discovery -- loads the catalog, refines the question, finds the right tables

2. Analysis -- runs queries via Databricks MCP, auto-captures discoveries to local catalog

3. Validation -- samples raw records, confirms field semantics, checks system behavior

4. Capture -- saves findings, presents unshared catalog entries for sharing

5. Close -- marks complete, prompts for catalog sharing

Investigations persist across sessions. Say "continue investigation" to resume where you left off.

Agent Memory System

The agent learns from your corrections and preferences over time through two paths:

Soft path (immediate): Corrections and preferences are saved to .cursor/memory/ and loaded into context at the start of future sessions. The agent adjusts its behavior based on past learnings without any structural changes.

Hard path (permanent): When a correction recurs 3+ times across sessions, the agent detects the pattern and proposes a permanent change to the workflow rules or investigation skill. You approve before anything changes. Platform-scope proposals create a PR for team review.

Memory is personal and gitignored. Each PM's agent learns independently. Improvements that benefit everyone flow through the PR-based promotion pipeline.

Analysis Principles

The .cursor/rules/analysis-principles.mdc rule enforces eight principles:

1. Ground truth first -- sample raw records before aggregating

2. Profile before joining -- verify field contents, don't assume from names

3. Ask about system behavior early -- domain knowledge shapes data interpretation

4. Correlation is not causation -- prevalence on a record does not equal cause

5. Explicit assumptions -- every volume estimate needs a stated deflection rate

6. Hypotheses before conclusions -- present as "data suggests" until validated

7. Flag observability gaps -- generic error codes need decomposition, not sizing

8. Signal catalog trust level -- indicate whether knowledge is curated, community, or local

Commands Reference

Investigation Commands

Catalog Commands

Curator Commands

MCP Tools Reference

Investigation MCP (Custom)

Official Databricks MCP (Data Access)

Data access tools are provided by the official Databricks MCP server:

Tips for Better Results

At the start of an investigation:

• The AI loads the catalog automatically. For new domains without curated entries, tell the AI what tables to use.

• Share system context early -- how key systems behave, known data quirks, business rules.

• State a hypothesis: "I think X is caused by Y" is more productive than "Tell me about X."

During analysis:

• Challenge aggregate findings. Ask "Have you looked at actual records?"

• When the AI presents a volume estimate, ask about the assumption: "What deflection rate?"

• The AI auto-captures gotchas, joins, and terms to your local catalog as it discovers them.

At the end of a session:

• Say "capture findings" to save progress. The AI presents unshared catalog entries for sharing.

• Choose which discoveries to share with the team. The AI handles the PR.

For executive deliverables:

• Ask the AI to create a separate brief rather than editing findings.md.

• Push back on oversimplified narratives. Ask for sub-problems.

• Request an assumptions and limitations section.

Extending with Additional MCP Servers

Adding a new data source is just adding an entry to .cursor/mcp.json.template:

{
  "mcpServers": {
    "dynatrace": {
      "command": "npx",
      "args": ["-y", "@dynatrace-oss/dynatrace-mcp-server@latest"],
      "env": {
        "DT_URL": "{{DT_URL}}",
        "DT_TOKEN": "{{DT_TOKEN}}"
      }
    }
  }
}

The investigation MCP and catalog are source-agnostic. The catalog's system field tracks which data source knowledge came from.

Running Scripts

Legacy standalone analysis scripts can be run from the repo root. Note: some older scripts may reference the previous databricks_mcp package which has been renamed to investigation_mcp. See the scripts directory for details.

Troubleshooting

"MCP server not showing in Cursor"

• Run ./setup.sh --test-only to validate configuration

• Restart Cursor completely (Cmd+Shift+P > "Reload Window")

• Check .cursor/mcp.json exists and has correct paths

"Setup fails at connection test"

• Verify your Databricks workspace URL includes https://

• Check that your personal access token hasn't expired

• Ensure your SQL Warehouse is running

• Try ./setup.sh --no-test to skip validation, fix credentials in .env, then ./setup.sh --test-only

"Table not found"

• Check the full table path (catalog.schema.table)

• Use load_catalog_index to see what the team knows about

• Verify you have access to the table in Databricks

Security Notes

• Databricks tokens stored in .env (gitignored, never committed)

• Local catalog entries are gitignored (personal to each user)

• Community catalog entries go through PR review before merging

• CI scans catalog PRs for secrets and PII patterns

• PM guardrails rule prevents AI from modifying protected files (rules, skills, curated catalog) -- exception: apply_proposal during approved learning reviews

• Agent memory files are gitignored (personal behavioral learnings never pushed to remote)

• Memory promotions to rule files require explicit human approval; platform-scope changes go through PR review

• Curator authorization enforced programmatically via _curators.yaml