# Data Science Agent

Autonomous AI agent for data analysis — custom ReAct loop, sandboxed Python/SQL execution, hypothesis testing, MCP protocol, skill system, real-time streaming WebUI.

## Features

### Core Agent
- **Custom ReAct Loop** — No LangChain dependency. Full control over planning, tool dispatch, and error recovery
- **Dynamic Tool Registry** — Pluggable tools with JSON Schema definitions. Add new tools in one line
- **Planner** — Decomposes vague requests into structured analysis plans
- **Streaming WebSocket** — Real-time thought → tool → result flow

### Analysis Capabilities
- **Python Sandbox** — Isolated subprocess execution with auto-imported pandas/numpy/matplotlib/seaborn/scipy
- **SQL Executor** — Auto-loads uploaded files into SQLite tables for JOIN/GROUP BY queries
- **Hypothesis Testing Engine** — 8 statistical tests (t-test, chi-square, ANOVA, Mann-Whitney U, Pearson/Spearman) with effect sizes (Cohen's d, η², Cramer's V)
- **Multi-file Analysis** — Cross-table JOIN and correlation analysis across multiple uploaded files
- **Chart Generation** — Auto-captured matplotlib/seaborn charts displayed inline
- **Data Lineage Tracking** — Trace any conclusion back to its source data and tool calls

### Skill System
- **5 Pre-built Analysis Templates**: Financial Analysis, User Segmentation, Anomaly Detection, Correlation Analysis, Time Series Analysis
- **Multi-select** — Combine multiple skills for comprehensive analysis
- **Extensible** — Add custom skills as JSON files in `skills/`

### MCP Protocol
- **JSON-RPC 2.0 endpoint** at `/mcp` — tools/list, tools/call, resources/list
- Compatible with any MCP client
- 6 tools exposed: file_reader, python_executor, sql_executor, hypothesis_test, skill_loader, finish

### Memory & Persistence
- **SQLite-backed sessions** — Survives restarts and page refreshes
- **Auto-titling** — Sessions named after uploaded files
- **Cleanup on delete** — Removing a session deletes its files, reports, and database

### Export
- **Markdown / HTML / PDF / Jupyter Notebook (.ipynb)** — full analysis pipeline as executable notebook
- **Streaming chat UI** with real-time step visualization
- **Configurable LLM** — Set API key/base URL/model via UI settings panel

## Architecture

```
User → WebUI (Next.js) → FastAPI
                          ├── ReAct Agent Core
                          ├── Tool Registry (6 tools)
                          ├── MCP Server (/mcp)
                          ├── Skill Registry (5 templates)
                          ├── Python Sandbox (subprocess)
                          ├── SQL Executor (SQLite per session)
                          └── Memory (SQLite persistence)
```

## Quick Start

### Prerequisites
- Python 3.9+
- Node.js 22+
- LLM API key (OpenAI, DeepSeek, OpenRouter, etc.)

### Setup

```bash
git clone <repo-url>
cd data-science-agent

# Backend
cp .env.example .env
# Edit .env with your API key
pip install -r requirements.txt
python -m uvicorn server.main:app --host 0.0.0.0 --port 8000

# Frontend (new terminal)
cd web
npm install
npm run dev
```

Open http://localhost:3000.

### Usage Guide

**1. 配置 API**
点击右上角齿轮图标 → 填入 DeepSeek / OpenAI / OpenRouter 的 API Key、Base URL、Model → 保存

**2. 上传数据**
左侧「数据文件」区域拖拽或点击上传 CSV/Excel/JSON 文件。支持多个文件做关联分析。

**3. 选择技能（可选）**
聊天区顶部选择预置技能模板（财务分析、异常检测、相关性分析等），可多选组合。

**4. 提问分析**
输入框输入问题，发送。Agent 自动：探查数据 → 写代码 → 出图表 → 统计检验 → 生成报告

**5. 查看结果**
- 分析步骤和图表实时展示
- 完成后点「查看报告」看完整报告
- 点「数据血缘」追溯数据来源
- 下载 PDF / Markdown / Jupyter Notebook

### Docker

```bash
docker compose up --build
```

## CLI Usage

```bash
# Single analysis
python cli.py "Analyze sales trends by region" -f data.csv -o report.md

# Interactive mode
python cli.py -i
```

## API

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/api/sessions` | Create session |
| `POST` | `/api/sessions/{id}/upload` | Upload file |
| `POST` | `/api/sessions/{id}/chat` | Send message |
| `WS` | `/ws/{id}` | Streaming chat |
| `GET` | `/api/sessions/{id}/report` | Download report (md/html/pdf) |
| `POST` | `/mcp` | MCP JSON-RPC endpoint |
| `GET` | `/api/skills` | List analysis skills |

## Project Structure

```
├── agent/
│   ├── core/          # ReAct loop, planner
│   ├── llm/           # LLM client (OpenAI-compatible)
│   ├── tools/builtin/ # file_reader, python_executor, sql_executor,
│   │                  # hypothesis_test, skill_loader, finish
│   ├── memory/        # Session context + SQLite store
│   ├── sandbox/       # Subprocess & Docker sandbox
│   ├── mcp/           # MCP protocol server
│   ├── skills/        # Skill registry
│   └── reporter/      # Markdown report generator
├── server/            # FastAPI app
├── web/               # Next.js frontend
├── skills/            # Skill template JSON files
├── cli.py             # CLI entry point
└── docker-compose.yml
```

## Key Design Decisions

1. **No LangChain** — Full control over agent loop and tool dispatch
2. **MCP-native** — Tools exposed via standard protocol, not proprietary API
3. **Hypothesis-driven** — Beyond descriptive stats; every conclusion backed by statistical tests
4. **Lineage tracking** — Every data point traceable to its source
5. **Skill templates** — Reusable, validated analysis workflows

## License

MIT