# FinSight Copilot: Financial Research & Analysis RAG Agent

**FinSight Copilot** is an end-to-end, AI-powered financial research assistant for financial analysts. Built on top of the Anthropic Claude Messages API, PostgreSQL with `pgvector`, real-time financial market tools and the Model Context Protocol (MCP), FinSight Copilot provides cited earnings report analysis, deterministic portfolio risk metrics, and guardrail-enforced financial research capabilities.

---

## Key Capabilities & Features

* **Analyst Copilot Engine:** Powered by Claude Messages API with XML-structured prompts, Chain-of-Thought (CoT) reasoning, structured output parsing, and real-time response handling.
* **Vector RAG with In-line Citations:** Grounded analysis over SEC filings (10-Ks, earnings disclosures, and transcripts) using `pgvector`. Every claim includes strict document and section citations.
* **Live Financial Tools:** Real-time financial market lookups (quotes, P/E ratios, trading volumes) integrated via external financial data APIs (Yahoo Finance / `yfinance`).
* **Deterministic Portfolio Math:** Value at Risk (VaR) calculations combined with enforced constitutional disclaimers.
* **Model Context Protocol (MCP):** Exposes financial tools as standardized MCP endpoints using `fastmcp`.
* **Optimization & Caching:** Leverages active Anthropic Prompt Caching (`cache_control={"type": "ephemeral"}`) to reduce input token overhead by up to ~90% on repeated RAG queries.
* **Evaluation & Defense Harness:** Includes an automated test suite verifying grounding accuracy, prompt injection defense, guardrail adherence, and execution cost tracking.

---

## Project Structure

```text
finsight-copilot/
│
├── data/
│   ├── 10k_reports/             # Seed Data: Raw PDF filings (e.g., Tesla, Nvidia, Amazon 10-Ks)
│   ├── research_notes/          # Supplementary RAG context (e.g., tech_sector_2024.txt)
│   └── transcripts/             # Adversarial testing data (e.g., malicious_hack.txt for prompt injection defenses)
│
├── src/
│   ├── agent.py                 # Autonomous State Machine loop (e.g., FETCH -> FLAG_RISK -> SUMMARIZE)
│   ├── copilot.py               # Main agent logic, tool routing, guardrails, and prompt caching
│   ├── eval.py                  # Automated evaluation suite testing accuracy, security, and token costs
│   ├── mcp_server.py            # Model Context Protocol (MCP) server exposing tools via fastmcp
│   ├── rag.py                   # Ingestion pipeline, PDF sliding-window chunker, and pgvector engine
│   ├── skills.py                # Modular AI skill definitions and instructions used by the agent/server
│   └── tools.py                 # Live yfinance integration and portfolio risk (VaR) math functions
│
├── .env                         # Local environment variables (Git ignored)
├── .env.example                 # Template for required environment variables
├── requirements.txt             # Python project dependencies
└── README.md                    # Project documentation
```

---

## Prerequisites

Ensure you have the following installed on your machine before setup:

* **Python:** `v3.10` or higher
* **Docker Desktop:** Required for running the PostgreSQL + pgvector container (or local PostgreSQL v15+ with pgvector)
* **Anthropic API Key:** Active key from the Anthropic Console

---

## Installation & Setup Guide

### Step 1: Clone Project Directory

```bash
git clone https://github.com/ayadav6/finsight-copilot.git
cd finsight-copilot
```

### Step 2: Set Up Python Virtual Environment (`venv`)

Create and activate an isolated Python virtual environment:

#### macOS / Linux:

```bash
python3 -m venv venv
source venv/bin/activate
```

#### Windows (Command Prompt):

```dos
python -m venv venv
venv\Scripts\activate.bat
```

#### Windows (PowerShell):

```powershell
python -m venv venv
.\venv\Scripts\Activate.ps1
```

### Step 3: Install Project Dependencies

```bash
pip install --upgrade pip
pip install -r requirements.txt
```

(Note: `fastmcp` is included in requirements.txt for the MCP server functionality).

### Step 4: Configure Environment Variables

Create a `.env` file in the root directory:

```bash
touch .env
```

Add your API credentials and database connection details to `.env`:

```plaintext
# Anthropic API Configuration
ANTHROPIC_API_KEY=your_anthropic_api_key_here

# PostgreSQL database connection string (Default for local Docker pgvector)
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres
```

### Step 5: Start & Configure PostgreSQL Vector Database

Start a containerized PostgreSQL instance with pgvector enabled using Docker:

```bash
docker run --name pgvector-finsight \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=postgres \
  -p 5432:5432 \
  -d pgvector/pgvector:pg16
```

---

## Execution Guide

### Step 1: Seed Data & Ingest Financial Disclosures

Place all target SEC Form 10-K PDFs (e.g., `Tesla_2025_10K.pdf`, `Nvidia_2025_10K.pdf`, `Amazon_2025_10K.pdf`) into the `data/10k_reports/` directory.

Run the RAG document ingestion script to parse disclosures, chunk text using sliding context windows, compute embeddings, and populate PostgreSQL:

```bash
python src/rag.py
```

(Note: Ingesting multi-page PDF filings locally on CPU can take between 5 to 15 minutes. The script contains an automated document count check to bypass redundant re-ingestion on subsequent executions).

### Step 2: Running FinSight Copilot

Once the vector database is populated with financial disclosures, run the main interactive copilot application:

```bash
python src/copilot.py
```

#### Supported Operations in `copilot.py`:

* **Live Market Lookups:** Fetches current stock prices, volume, and P/E ratios (e.g., "What is NVDA's current stock price and P/E ratio?").
* **Portfolio Risk Calculations:** Computes deterministic Value at Risk (VaR) metrics for user portfolios while appending mandatory disclaimers.
* **Document Grounded RAG:** Inquires about historical financial data (e.g., "Summarize Tesla's related party transactions with xAI in 2025 and 2026.") with document citations.

### Step 3: Running the Automated Evaluation Harness

To run adversarial compliance tests, verify grounded citations, check guardrails, and generate an execution token micro-cost report:

```bash
python src/eval.py
```

### Step 4: Running the MCP Server

FinSight's tools are packaged into an MCP server. Run the MCP server inspector using:

```bash
fastmcp dev inspector src/mcp_server.py
```

This command launches a local web server (usually `http://localhost:5173`) where you can visually inspect, connect, and test the `get_stock_quote` and `compute_portfolio_risk` tools directly via a web UI.

---

## Engineering & Architectural Highlights

| Feature                   | Description                                                                                                                                                                          |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Prompt Caching            | Utilizes `cache_control={"type": "ephemeral"}` to lower input token overhead from ~4,000 tokens down to 1 token on repeated system prompts, reducing latency and costs by up to 90%. |
| Context Windowing         | Overlapping sliding window text chunking (1500 characters, 300-character overlap) prevents splitting financial figures, tables, or key sentences across boundaries.                  |
| Constitutional Guardrails | System instructions enforce strict non-personalized advice policies and mandate liability disclaimers on portfolio risk outputs.                                                     |