# AI Due Diligence Copilot

AI Due Diligence Copilot is an end-to-end financial document analysis system that ingests corporate filings (10-K / 10-Q), extracts structured financial information, retrieves supporting evidence, and answers analyst questions using the most reliable source available.

Unlike many AI applications that rely entirely on an LLM, this system intelligently routes each question to either:

- Structured financial data stored in PostgreSQL
- Retrieved filing evidence through a Retrieval-Augmented Generation (RAG) pipeline
- A combination of both

To improve trustworthiness, generated answers can be evaluated using a PyTorch-based groundedness classifier that checks whether claims are supported by retrieved evidence.

---

## Why This Project?

Financial analysts frequently need answers that are:

- Factually accurate
- Explainable
- Traceable to source documents

Traditional LLM-based assistants can hallucinate numbers or provide unsupported claims.

This project addresses that problem by:

- Using SQL for quantitative reasoning
- Using RAG for qualitative document understanding
- Using groundedness evaluation to assess evidence support
- Supporting optional Claude-based answer synthesis while remaining fully functional without any paid API

For deeper implementation details and design decisions, see:

```text
ARCHITECTURE.md
```

---

## Features

### Document Ingestion

- Upload financial filings (`.txt`, `.pdf`, `.md`)
- Automatic document cleaning and chunking
- Filing metadata tracking
- Persistent PostgreSQL storage

### Structured Financial Metric Extraction

Currently extracts and stores:

- Revenue
- Operating Margin
- Net Income
- R&D Expense
- Cash & Cash Equivalents

These metrics are stored in PostgreSQL and can be queried directly for quantitative analysis.

### Intelligent Query Routing

The system automatically determines whether a question should be answered through:

- Structured SQL retrieval
- Document retrieval (RAG)
- SQL + RAG

Examples:

| Question | Route |
|-----------|--------|
| What was the change in operating margin? | SQL |
| What supplier risks does the company face? | RAG |
| How did margins change and why? | SQL + RAG |

### MCP Tool Integration

Implements Model Context Protocol (MCP) tools:

- Company Lookup
- Financial Ratio Calculator
- Document Search

### Groundedness Evaluation

A lightweight PyTorch classifier evaluates whether generated claims are supported by retrieved evidence.

Outputs include:

- Groundedness scores
- Claim-level support classification
- Confidence indicators

### Evaluation Harness

Built-in evaluation framework measuring:

- Routing accuracy
- Retrieval relevance
- Groundedness
- Latency

### Interactive Dashboard

Web interface for:

- Filing uploads
- Question answering
- Viewing routing decisions
- Viewing tool usage
- Viewing groundedness scores

---

# System Architecture

```text
                    Financial Filing
                           │
                           ▼
                  Document Ingestion
                           │
                           ▼
                      Chunking
                           │
                           ▼
       Metric Extraction + Vectorization
                 (TF-IDF + SVD)
                 │              │
                 ▼              ▼
          PostgreSQL      Vector Store
                 ▲              ▲
                 │              │
User Query ───► Query Router ───┘
                 │
      ┌──────────┴──────────┐
      ▼                     ▼

 SQL Financial Route     RAG Retrieval

      ▼                     ▼

 Financial Metrics     Evidence Search

      └──────────┬──────────┘
                 ▼

         Answer Generation

                 ▼

      Groundedness Evaluation

                 ▼

          Final Response
```

---

# Answer Generation Modes

## 1. Offline / Local Mode (Default)

No API keys required.

The system answers questions using:

- PostgreSQL financial data
- RAG retrieval
- Extractive answer generation
- PyTorch groundedness evaluation

This mode was used during development and testing.

---

## 2. Claude-Assisted Mode (Optional)

If an Anthropic API key is provided:

```bash
ANTHROPIC_API_KEY=your_api_key_here
```

retrieved evidence can be passed to Claude for natural-language answer generation.

Pipeline:

```text
User Query
    ↓
Retrieval / SQL
    ↓
Claude
    ↓
Groundedness Evaluation
    ↓
Final Response
```

Claude is only used for answer synthesis.

The system does **not** depend on Claude for:

- Retrieval
- Query routing
- Financial calculations
- Groundedness scoring

If no API key is present, the application automatically falls back to the local extractive pipeline.

---

# Tech Stack

## Backend

| Technology | Purpose |
|------------|----------|
| FastAPI | API Framework |
| Uvicorn | ASGI Server |
| Pydantic | Data Validation |

### Database

| Technology | Purpose |
|------------|----------|
| PostgreSQL | Primary Database |
| SQLAlchemy | ORM |

### Machine Learning

| Technology | Purpose |
|------------|----------|
| PyTorch | Groundedness Classifier |
| Scikit-Learn | Retrieval Pipeline |
| NumPy | Numerical Operations |

### Retrieval

| Component | Purpose |
|------------|----------|
| TF-IDF | Document Vectorization |
| Truncated SVD | Dense Semantic Representation |
| Cosine Similarity | Retrieval Ranking |

### AI Tooling

| Component | Purpose |
|------------|----------|
| MCP Tools | Structured Tool Access |
| Query Router | Route Selection |
| Groundedness Evaluator | Evidence Validation |

---

# Project Structure

```text
diligence-copilot/
│
├── app/
│   ├── db/
│   ├── ingestion/
│   ├── ml/
│   ├── rag/
│   ├── mcp_tools/
│   └── main.py
│
├── data/
│
├── scripts/
│   └── setup_postgres.sql
│
├── tests/
│
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── ARCHITECTURE.md
└── README.md
```

---

# Installation

## Prerequisites

- Python 3.11+
- PostgreSQL 16+ (tested on PostgreSQL 17.11)
- Git

---

## 1. Clone Repository

```bash
git clone <repository-url>
cd diligence-copilot
```

---

## 2. Create Virtual Environment

### Windows

```powershell
python -m venv venv
.\venv\Scripts\Activate.ps1
```

### Linux / macOS

```bash
python -m venv venv
source venv/bin/activate
```

---

## 3. Install Dependencies

Install CPU-only PyTorch:

```bash
pip install torch --index-url https://download.pytorch.org/whl/cpu
```

Install project requirements:

```bash
pip install -r requirements.txt
```

---

## 4. Configure PostgreSQL

Run:

```bash
psql -U postgres -f scripts/setup_postgres.sql
```

This creates:

- `diligence_copilot` database
- `diligence_app` user
- Required permissions

---

## 5. Build Groundedness Dataset

```bash
python -m app.ml.build_dataset
```

---

## 6. Train Groundedness Classifier

```bash
python -m app.ml.groundedness
```

---

## 7. Start Application

```bash
uvicorn app.main:app --reload
```

Application:

```text
http://localhost:8000
```

API Documentation:

```text
http://localhost:8000/docs
```

---

## Docker

```bash
docker-compose up --build
```

---

# Verification Walkthrough

## Create Test Filing

```text
Total revenue for fiscal year 2024 was $500 million.

Operating margin for fiscal year 2024 was 12.5%, compared to 10.1% in fiscal year 2023.

The company faces significant risk from a single supplier located in Vietnam.
```

---

## Upload Filing

Use:

- Ticker: TEST
- Company: Test Company
- Fiscal Period: FY2024

Example output:

```text
Done: 1 chunks, 3 financial metrics extracted.
```

---

## Test Financial Reasoning

Question:

```text
What was the change in operating margin?
```

Example output:

```text
operating_margin moved from 10.1 (FY2023)
to 12.5 (FY2024), a change of 23.76%.
```

Route:

```text
SQL
```

Tool:

```text
financial_ratio_calculator
```

---

## Test Retrieval

Question:

```text
What supplier risks does the company face?
```

Example output:

```text
The company faces significant risk from a single supplier located in Vietnam.
```

Note: For very small filings that fit into a single chunk, retrieval may return the entire chunk rather than a single sentence.

Route:

```text
RAG
```

---

## Run Evaluation

```bash
python -m app.eval.run_eval
```

---

# Results

Validated end-to-end on:

- Document ingestion
- Financial metric extraction
- PostgreSQL persistence
- Query routing
- SQL-based financial reasoning
- Retrieval-based risk analysis
- Groundedness evaluation
- FastAPI deployment

### Sample Evaluation Results

| Metric | Value |
|----------|----------|
| Route Accuracy | 1.00 |
| Average Retrieval Relevance | 0.67 |
| Average Latency | ~30ms |

### Groundedness Classifier

| Metric | Value |
|----------|----------|
| Accuracy | 0.70 – 0.90 |
| Recall | 0.75 – 1.00 |

Results vary slightly because the evaluation dataset is intentionally small.

---

# Limitations

- Groundedness classifier trained on 38 labeled examples
- Financial metric extraction currently uses rule-based patterns
- Retrieval uses TF-IDF + SVD instead of transformer embeddings
- Vector index is in-memory and optimized for demonstration-scale workloads
- Claude integration is optional and not required for core functionality

---

# Future Improvements

- Transformer-based embeddings
- Hybrid search (keyword + vector)
- Larger groundedness datasets
- LLM-based extraction fallback
- Multi-step agent workflows
- pgvector integration
- Multi-document comparison
- Automated analyst report generation