Skip to main content
Glama
AI Research Paper Assistant

AI Research Paper Assistant

by guntaasaneja

AI Research Paper Assistant

A Retrieval-Augmented Generation (RAG) system that answers questions about foundational AI/ML research papers, with an agentic reasoning layer, automated evaluation, and an MCP (Model Context Protocol) server so any MCP-compatible LLM client can use it as a tool.


What it does

Ask questions like:

  • "What is the attention mechanism in Transformers?"

  • "How does RAG combine retrieval with generation?"

  • "Compare how the Transformer paper and ReAct paper approach reasoning"

The system retrieves relevant passages from three indexed papers — Attention Is All You Need, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, and ReAct: Synergizing Reasoning and Acting in Language Models — and generates a grounded answer, citing the source paper and page.

For complex, multi-part questions, an agent layer breaks the question into sub-questions, retrieves context for each, and synthesizes a combined answer — rather than relying on a single retrieval pass.


Related MCP server: Research Citations MCP Server

Architecture

┌─────────────┐
│   papers/   │  Raw PDFs (downloaded from arXiv)
└──────┬──────┘
       │  ingest.py: load → chunk → embed
       ▼
┌─────────────┐
│  chroma_db/ │  Vector store (291 chunks, all-MiniLM-L6-v2 embeddings)
└──────┬──────┘
       │
       ├─────────────────┬─────────────────────┐
       ▼                 ▼                      ▼
┌─────────────┐  ┌──────────────┐   ┌────────────────────┐
│ rag_chain.py│  │  agent.py    │   │  evaluation.py      │
│ Simple LCEL │  │  LangGraph:  │   │  RAGAS metrics:      │
│ retrieve→   │  │  plan→       │   │  faithfulness,        │
│ generate    │  │  retrieve→   │   │  answer relevancy,    │
│             │  │  synthesize  │   │  context precision    │
└──────┬──────┘  └──────┬───────┘   └─────────────────────┘
       │                │
       └───────┬────────┘
               ▼
      ┌─────────────────┐
      │  mcp_server.py   │  Exposes both as MCP tools
      │  + app.py (CLI)  │  for external LLM clients
      └─────────────────┘

Why this shape: all shared setup (embeddings, vector store connection, LLM client) lives in one place, src/config.py. Every other module imports from it instead of redefining it — so changing the embedding model or LLM provider means editing one file, not four.


Tech stack and why each piece was chosen

Component

Choice

Why

Orchestration

LangChain (LCEL)

Industry-standard way to compose retrieval + prompt + LLM into a pipeline

Agent framework

LangGraph

Lets the system plan multi-step retrieval instead of one fixed pass — needed for comparison-style questions

Vector store

Chroma

Local, free, simple to set up and inspect for a single-developer project

Embeddings

HuggingFace all-MiniLM-L6-v2

Runs locally on CPU, free, no API key or vendor lock-in — deliberate alternative to OpenAI embeddings

LLM

Groq (Llama 3.3 70B)

Free tier, very fast inference, good enough quality for this scope

Evaluation

RAGAS

Automated, LLM-judged scoring instead of just eyeballing answers

Tool exposure

MCP (Model Context Protocol)

Turns the pipeline into a callable tool for any MCP client (e.g. Claude Desktop), not just a terminal script


Evaluation results

Measured with RAGAS on a 3-question test set against known ground-truth answers:

Metric

Score

What it measures

Faithfulness

0.87

Does the answer stick to what's actually in the retrieved context, or hallucinate?

Answer Relevancy

0.93

Does the answer actually address the question asked?

Context Precision

0.94

Were the most relevant chunks ranked near the top of retrieval?


Project structure

ai-research-paper-assistant/
├── src/
│   ├── config.py       # Single source of truth: embeddings, vectorstore, LLM setup
│   ├── ingest.py        # PDF loading, chunking, embedding into Chroma
│   ├── rag_chain.py     # Simple retrieve-then-generate LCEL chain
│   ├── agent.py          # LangGraph agent: plan → retrieve → synthesize
│   └── evaluation.py    # RAGAS evaluation harness
├── mcp_server.py         # MCP tool wrapper around rag_chain and agent
├── app.py                 # CLI entry point (ingest / ask / agent / evaluate)
├── download_papers.py     # One-time script to pull papers from arXiv
├── requirements.txt
└── eval_results.txt        # Saved RAGAS output

Setup

# 1. Clone and enter the project
git clone <your-repo-url>
cd ai-research-paper-assistant

# 2. Create and activate a virtual environment
python -m venv venv
source venv/bin/activate      # Windows: venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Add your Groq API key (free at console.groq.com)
echo "GROQ_API_KEY=your_key_here" > .env

# 5. Download the papers and build the vector store
python download_papers.py
python app.py ingest

Known dependency issue: ragas==0.3.9 ships with a broken import (langchain_community.chat_models.vertexai, a module path that's since moved). If you hit ModuleNotFoundError on that import when running evaluate, it's a known upstream bug, not a problem with this code — open an issue or check for a patched ragas release. A workaround is to wrap that one import line in the installed package in a try/except ImportError, since this project doesn't use VertexAI at all.


Usage

python app.py ask         # Simple RAG: ask direct questions, one retrieval pass
python app.py agent       # Agentic RAG: handles multi-part/comparison questions
python app.py evaluate    # Re-run RAGAS scoring, saves to eval_results.txt
python mcp_server.py      # Start as an MCP server for external clients (e.g. Claude Desktop)

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    • Query your own PDFs and documents from any MCP client. Every answer cites the page it came from.

      https://mcp.docs2mcp.com/mcp
    • Agent-native MCP server over the public saagarpatel.dev corpus. Read-only, stateless.

      https://mcp.saagarpatel.dev/mcp
    • Agentic search over your Dewey document collections from any MCP-compatible client.

      https://mcp.meetdewey.com/mcp
    • Rivul is a free, read-only remote MCP server for searching scholarly literature. It runs semantic search over more than 300 million paper records from a dated OpenAlex snapshot, so an agent can find papers by meaning rather than exact keywords, then open the full record for any result. Tools: - search_papers: semantic search with optional filters for publication year range, minimum citation count, open access only, topic and field. Sort by relevance, most cited, newest or oldest. - get_paper_details: the full record for one work by its OpenAlex ID: title, every author, venue, work type, topic, field, publication date, citation count, DOI and open-access links. - usage_status: the workspace's MCP usage for the current day. Results come only from the index, never from a model. Each result carries a stable OpenAlex ID so the source can be checked, and every response states the snapshot date. Useful for literature reviews, scoping reviews, finding highly cited or open-access papers on a topic, citation chasing, and checking that a cited paper exists. Free for every account, with no plans or payment. Sign in with Google through OAuth 2.1 (PKCE, dynamic client registration), or use an MCP key from rivul.ai. Endpoint: https://rivul.ai/mcp (Streamable HTTP). Setup and docs: https://rivul.ai/features/atlas

      https://rivul.ai/mcp

    Related MCP Servers