# StudyPilot

StudyPilot is a course-material learning agent project. It uses an MCP server
to expose course documents as resources and provide search/read tools.

Project references: [architecture](docs/ARCHITECTURE.md),
[evaluation protocol](docs/EVALUATION.md), [demo script](docs/DEMO.md), and
[release checklist](docs/RELEASE_CHECKLIST.md).

## Current capabilities

- Import local Markdown, text, and text-based PDF documents.
- Preserve one-based PDF page numbers in search results and citation details.
- Store material chunks and their Qwen3-Embedding-0.6B vectors in a persistent FAISS `IndexFlatIP` index (exact cosine similarity after L2 normalization).
- Retrieve with hybrid search: dense vector candidates plus an in-process lexical rank, fused with reciprocal-rank fusion (RRF).
- Keep study tasks and quiz results in PostgreSQL; material retrieval no longer depends on PostgreSQL full-text search.
- Read a complete document by its stable document ID.
- Expose the same operations as MCP tools and resources.
- Create, list, and update persistent study tasks through MCP tools.
- Provide a source-grounded quiz prompt through MCP.
- Let DeepSeek discover and call MCP tools in a bounded Agent loop.
- Route requests between a Tutor Agent and Planner Agent with isolated MCP tool permissions.
- Select DeepSeek or a release-gated local Ollama model for Tutor requests while Planner remains on DeepSeek.
- Persist the assistant display name per browser conversation in PostgreSQL.

## Multi-Agent routing

StudyPilot uses deterministic routing instead of spending another model call on a
Supervisor Agent. Ordinary course questions go directly to the Tutor Agent. Task-only
commands go directly to the Planner Agent. Composite requests such as "analyze my weak
points from the materials and schedule revision" run Tutor first, then pass its cited
diagnosis to Planner as a structured handoff.

- Tutor tools: `search_materials`, `get_document`
- Planner tools: `create_study_task`, `list_study_tasks`, `update_study_task_status`
- Single-purpose requests invoke one Agent; only composite requests invoke both.

Each Agent run is bounded by both model steps and total tool calls. Composite
requests pass a JSON Tutor handoff containing the original request, grounded
diagnosis, and extracted citation IDs to the Planner.

Within one Agent run, an identical mutating tool call is executed at most once.
This protects task creation and status updates from repeated model tool calls;
read-only tools may still be called again when needed. Web Agent task creation
also receives a persistent idempotency key derived from the conversation, turn,
and normalized business arguments. Concurrent or retried creation with that key
returns the original task. Direct MCP/CLI creation without a key keeps normal
create-each-time behavior.

The Web client stores a random `conversation_id` in `localStorage`. Explicit renames such
as "以后你叫不亮" are stored in PostgreSQL and injected into both Agent prompts on later
turns. Clearing the chat creates a new conversation and resets the display name.

## Persistent conversation context

Every user and assistant message is stored in PostgreSQL with its `conversation_id`,
`turn_id`, role, timestamp, and executing Agent. The browser restores the complete saved
history through `GET /api/conversations/{conversation_id}` after a refresh. Full history
remains in the database; only the latest 20 messages within a 12,000-character budget are
sent to the model on each turn.

If a completed `conversation_id` and `turn_id` is submitted again with the same
user message, both chat endpoints replay the persisted assistant response without
running the Agent again. Reusing a turn ID for different content returns HTTP 409.
This covers completed-request retries; it is not a lock for two concurrent requests
that arrive before the first response is persisted. Persistent task idempotency
still prevents identical task creation within that concurrent turn.

StudyPilot records user preferences only from explicit statements, including preferred
name, language, response detail, and step-by-step teaching style. Its learning profile
separates explicit goals and weak points from observed activity such as retrieval queries,
source documents, and study tasks. Observed activity is derived from real MCP traces, not
model guesses. Clearing the chat starts a new conversation but does not delete the old
history from PostgreSQL.

## Setup

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"
```

## Run the MCP server

Start PostgreSQL for study-task persistence, migrate the schema, and index sample materials into FAISS. The first indexing run downloads `Qwen/Qwen3-Embedding-0.6B` from Hugging Face.

```powershell
docker compose up -d postgres
.venv\Scripts\alembic.exe upgrade head
.venv\Scripts\studypilot-index.exe
```

FAISS persists the index and chunk metadata under `data/faiss/`. Existing Chroma
cache files are not read after this migration, so run `studypilot-index` once
after upgrading to rebuild the index from `data/materials/`.
Set `FAISS_DIR` only when the index must live elsewhere, such as an isolated test
or deployment data volume; the default remains `data/faiss/`.

Document IDs are derived from normalized file stems and must be unique. Files
such as `Course Notes.md` and `course-notes.txt` conflict; indexing rejects the
set and reports both names instead of silently merging their chunks or producing
ambiguous citations.

Blank-line paragraph boundaries remain stable for ordinary material. A single
paragraph longer than 1,200 characters is split into bounded chunks, preferring
sentence endings in the final 40 percent of each window. This rule does not add
overlap because complete documents are reconstructed from stored chunks. Re-run
`studypilot-index` after upgrading from an index built before this rule.

The default transport is `stdio`, which is convenient for a local Agent host:

```powershell
studypilot-mcp
```

To inspect it interactively:

```powershell
mcp dev app/studypilot/mcp_server.py
```

## Test

```powershell
.venv\Scripts\python.exe -m pytest
```

The default suite uses local fakes for model calls and an isolated repository
temporary directory. PostgreSQL integration tests are skipped when the test
database is unavailable; a green local run therefore does not by itself prove
that PostgreSQL or a real model endpoint is reachable.

## Fixed retrieval evaluation

The fixed set is [`data/eval/retrieval_cases.json`](data/eval/retrieval_cases.json). It uses source chunk IDs as relevance judgments and writes a reproducible JSON report with Recall@1, Recall@3, Recall@5, MRR, and per-case bad cases.

```powershell
.venv\Scripts\studypilot-eval.exe --k 1 3 5
```

To score answers and citation faithfulness as well, provide real Agent output in this format (one record per fixed case):

```json
[
  {
    "id": "round-robin",
    "answer": "时间片过小会造成频繁进程切换，增加系统开销。",
    "citations": ["operating-systems:2"],
    "refused": false,
    "latency_ms": 840.5,
    "total_tokens": 320,
    "error": null
  }
]
```

```powershell
.venv\Scripts\studypilot-eval.exe --answers .\agent_answers.json
```

`data/eval/latest_report.json` records every retrieval list and identifies failures in
`bad_cases`. Cases marked `should_refuse` evaluate refusal separately from retrieval.
When answer records include runtime fields, the report also calculates P50/P95 latency,
mean token count, and failure rate. It never fabricates metrics when real Agent data is
not supplied.

## T2Ranking fixed rerank set

The project uses the official [THUIR/T2Ranking dev split](https://huggingface.co/datasets/THUIR/T2Ranking) as its external Chinese rerank benchmark. The checked-in fixed subset contains 200 query IDs selected from dev using a documented SHA-256 seed, retrieval qrels, graded rerank qrels, and hashes of all three official source files.

Build or verify the subset after downloading the official small metadata files into `data/eval/t2ranking/source/`:

```powershell
.venv\Scripts\python.exe .\tools\build_t2ranking_subset.py
```

The fixed query/qrels set alone is not enough to report rerank quality. A valid `Recall@3` comparison also needs a persisted Top-K candidate pool and the corresponding passages from the same `collection.tsv` revision. The baseline and reranker must reorder the same candidates. Until those artifacts and both run outputs exist, this project does not claim any `Recall@3` improvement.

The official `dev.bm25.tsv` candidate list produces a persisted 100-candidate pool per fixed query. Its measured baseline is **Recall@3 0.5250**, **MRR@10 0.4268**, and **nDCG@10 0.3050**. A completed local `BAAI/bge-reranker-v2-m3` run reranked the first 20 of the same 100 candidates for all 200 queries in one 4,000-pair batched call. The measured rerank result is **Recall@3 0.6000**, **MRR@10 0.4812**, and **nDCG@10 0.3495**, for deltas of **+0.0750**, **+0.0544**, and **+0.0445** respectively. The checked-in report and manifest identify the exact artifacts. To reproduce it, run:

```powershell
.venv\Scripts\python.exe .\tools\extract_t2ranking_passages.py
.venv\Scripts\python.exe .\tools\rerank_t2ranking.py --model .\data\models\bge-reranker-v2-m3 --top-n 20 --batch-size 16 --progress
.venv\Scripts\python.exe .\tools\evaluate_t2ranking.py --rerank .\data\eval\t2ranking\fixed-200\rerank.cross_encoder.tsv --report .\data\eval\t2ranking\fixed-200\rerank.report.json
```

The evaluator rejects a rerank file if it changes any query's candidate set. It therefore reports a genuine reranking comparison, not a candidate-generation comparison.

## Ask the Agent

Configure `DEEPSEEK_API_KEY` and `DEEPSEEK_MODEL` in `.env`, then run:

```powershell
.venv\Scripts\studypilot-chat.exe "时间片为什么不能设置得太小？"
```

## Model reliability and observability

`DeepSeekChatModel` enforces a per-attempt timeout, retries transient timeout,
connection, rate-limit, and server errors with exponential backoff, then tries
the explicitly configured fallback models. It does not silently switch to an
invented provider or model.

For streaming responses, retry and fallback are allowed only before the first
chunk is emitted. Every wait for the next chunk uses the same per-attempt
timeout. If a stream fails after partial content has reached the client, the
request ends with an error instead of replaying duplicated text.

Configure the primary and fallback sequence in `.env`:

```dotenv
DEEPSEEK_MODEL=deepseek-v4-flash
DEEPSEEK_FALLBACK_MODELS=your-confirmed-fallback-model
DEEPSEEK_TIMEOUT_SECONDS=30
DEEPSEEK_MAX_RETRIES=2
DEEPSEEK_RETRY_BASE_SECONDS=0.5
```

The system prompt carries the code-defined version `studypilot-2026-07-30.2`.
Every model attempt is appended to `data/logs/model_calls.jsonl` with its
provider, model, model artifact hash (when configured), prompt version,
retry/fallback position, latency, token usage, outcome, and error type. Web calls
also include `request_id`, `conversation_id`, and `turn_id`
so an HTTP response can be correlated with its model attempts. These identifiers
are propagated through both JSON and NDJSON responses. Prompt text, user questions,
API keys, and material content are not written to this log. To calculate cost,
provide verified per-million-token rates
for the exact model names; without them `estimated_cost_usd` remains `null`.

```dotenv
MODEL_PRICING_USD_PER_MILLION_JSON={"deepseek-v4-flash":{"input":0.0,"output":0.0}}
```

## Run the web workspace

```powershell
.venv\Scripts\studypilot-web.exe
```

Open <http://127.0.0.1:8787> to chat, import materials, inspect sources, and
manage study tasks.

For a containerized application process, keep Ollama on the host and run:

```powershell
docker compose build app
docker compose run --rm app alembic upgrade head
docker compose run --rm app studypilot-index
docker compose up -d app
```

The app container reaches Ollama through `host.docker.internal`. PostgreSQL,
FAISS data, Hugging Face cache, and application health have separate Compose
volumes/checks. The native and containerized Web processes both use port 8787,
so run only one of them at a time.

`GET /api/health` reports the application, PostgreSQL connection, and FAISS
material-index state separately. An index can be `ready`, `empty`,
`inconsistent`, or `unavailable`; `empty` remains usable because documents can
still be imported, while inconsistent or unavailable indexes make the top-level
health status false. The health payload also reports the current index-format
version. Metadata from an older or malformed format is rejected with an explicit
`studypilot-index` rebuild instruction.

## Hybrid retrieval and streaming chat

`search_materials` keeps its existing MCP tool name and result shape. Internally it
fetches vector candidates from FAISS, ranks all chunks for lexical term matches,
then uses RRF to combine the two lists. This improves exact terminology matches
without removing semantic retrieval.

The workspace now uses `POST /api/chat/stream`. It returns
`application/x-ndjson` with `status`, `delta`, `done`, and `error` events. The UI
renders each `delta` as it arrives and reports when the bounded Agent is calling an
MCP tool. `POST /api/chat` remains available for existing non-streaming clients.

## Optional Qwen3.5-4B tutor fine-tuning (deferred)

This release intentionally stops before adapter training, GGUF conversion, and
four-model quality comparison. The checked-in corpus, isolated indexes,
trajectory compiler, dataset validator, and CPU preflight are preparation
artifacts only; no local-model quality improvement is claimed. DeepSeek remains
the default Tutor and the local Ollama option stays release-gated.

The current app-scope release audit is:

```powershell
.venv\Scripts\python.exe .\tools\audit_release.py --scope app
```

The local tutor fine-tuning path is pinned to the official
`Qwen/Qwen3.5-4B` revision
`851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a`, which matches the `qwen35`
4.7B architecture used by the local Ollama `qwen3.5:4b` model. Ollama's
quantized GGUF file is for inference; training uses the original Safetensors
checkpoint.

Install the optional training dependencies:

```powershell
.venv\Scripts\python.exe -m pip install -e ".[train,dev]"
```

Formal data uses complete `system/user/assistant.tool_calls/tool/assistant`
trajectories and the frozen `data/finetune/tutor_tools.v1.json` schema. Only
assistant spans contribute to loss. System prompts, user questions, and tool
responses are masked. The checked-in eight-row file remains a response-only
smoke sample and cannot pass the formal 300-train/60-evaluation gate.

```powershell
.venv\Scripts\python.exe .\tools\fetch_public_corpus.py --inspect-rights
.venv\Scripts\python.exe .\tools\fetch_public_corpus.py --accepted-policy .\data\corpus\accepted-rights.v1.json
.venv\Scripts\python.exe .\tools\build_corpus_indexes.py
.venv\Scripts\python.exe .\tools\create_tutor_review_worklist.py --split train --output .\data\finetune\tutor_review.train.jsonl
.venv\Scripts\python.exe .\tools\create_tutor_review_worklist.py --split eval --output .\data\finetune\tutor_review.eval.jsonl
.venv\Scripts\python.exe .\tools\compile_tutor_trajectories.py --worklist .\data\finetune\tutor_review.train.jsonl --index-dir .\data\indexes\wikimedia_cs_v1\train --output .\data\finetune\tutor_sft.train.jsonl
.venv\Scripts\python.exe .\tools\compile_tutor_trajectories.py --worklist .\data\finetune\tutor_review.eval.jsonl --index-dir .\data\indexes\wikimedia_cs_v1\eval --output .\data\finetune\tutor_sft.eval.jsonl
.venv\Scripts\python.exe .\tools\validate_tutor_dataset.py --train .\data\finetune\tutor_sft.train.jsonl --eval .\data\finetune\tutor_sft.eval.jsonl --report .\data\finetune\tutor_dataset.validation.json
# Formal QLoRA training is deferred in this release.
```

The 8 GB profile uses a 768-token maximum, NF4 double quantization, rank-8 LoRA, batch size 1,
gradient accumulation, BF16 compute, and gradient checkpointing. LoRA targets
are discovered only under `model.language_model.layers`; the visual tower stays
frozen. The current validation report records exactly 300/60 rows, semantic
cross-split validation, and `review_mode=ai_assisted_authorized`. Formal
training remains deferred; no adapter or loss result is part of this release.

After a future training run, merge on CPU, convert to GGUF `Q4_K_M`, and run the
same holdout through base, adapter, DeepSeek, and deployed Ollama candidates
only when the recorded release gates pass:

```powershell
.venv\Scripts\python.exe .\tools\merge_tutor_lora.py
.venv\Scripts\python.exe .\tools\convert_tutor_to_gguf.py
.venv\Scripts\python.exe .\tools\run_tutor_evaluation.py --cases .\data\finetune\tutor_sft.eval.jsonl --output .\data\eval\tutor\runs.jsonl --candidate qwen3.5-4b-base=transformers --candidate studypilot-tutor-4b-adapter=peft:data/models/studypilot-tutor-4b-lora --candidate deepseek=deepseek --candidate studypilot-tutor-4b-q4_k_m=ollama:studypilot-tutor:4b
.venv\Scripts\python.exe .\tools\score_tutor_evaluation.py --cases .\data\finetune\tutor_sft.eval.jsonl --runs .\data\eval\tutor\runs.jsonl --output .\data\eval\tutor\report.json
.venv\Scripts\python.exe .\tools\create_ollama_tutor.py --gguf .\data\models\studypilot-tutor-4b-q4_k_m.gguf --evaluation-report .\data\eval\tutor\report.json --name studypilot-tutor:4b
.venv\Scripts\python.exe .\tools\audit_release.py --artifact .\data\models\studypilot-tutor-4b-q4_k_m.gguf
```

The Web API accepts `tutor_provider=deepseek|ollama`, and the page stores the
selection per browser conversation. Planner always uses DeepSeek. The local
choice is disabled until the verified artifact hash, matching deployment
manifest, and `OLLAMA_RELEASE_GATES_PASSED=1` are configured;
`ENABLE_EXPERIMENTAL_LOCAL_TUTOR=1` is an explicit non-release bypass.
See [fine-tuning and deployment](docs/FINETUNING.md) for data contracts, scoring,
failure evidence, and exact promotion rules.