Search Engine with RAG and MCP
A powerful search engine that combines LangChain, Model Context Protocol (MCP), Retrieval-Augmented Generation (RAG), and Ollama to create an agentic AI system capable of searching the web, retrieving information, and providing relevant answers.
Features
- Web search capabilities using the Exa API
- Web content retrieval using FireCrawl
- RAG (Retrieval-Augmented Generation) for more relevant information extraction
- MCP (Model Context Protocol) server for standardized tool invocation
- Support for both local LLMs via Ollama and cloud-based LLMs via OpenAI
- Flexible architecture supporting direct search, agent-based search, or server mode
- Comprehensive error handling and graceful fallbacks
- Python 3.13+ with type hints
- Asynchronous processing for efficient web operations
Architecture
This project integrates several key components:
- Search Module: Uses Exa API to search the web and FireCrawl to retrieve content
- RAG Module: Embeds documents, chunks them, and stores them in a FAISS vector store
- MCP Server: Provides a standardized protocol for tool invocation
- Agent: LangChain-based agent that uses the search and RAG capabilities
Project Structure
search-engine-with-rag-and-mcp/
├── LICENSE # MIT License
├── README.md # Project documentation
├── data/ # Data directories
├── docs/ # Documentation
│ └── env_template.md # Environment variables documentation
├── logs/ # Log files directory (auto-created)
├── src/ # Main package (source code)
│ ├── __init__.py
│ ├── core/ # Core functionality
│ │ ├── __init__.py
│ │ ├── main.py # Main entry point
│ │ ├── search.py # Web search module
│ │ ├── rag.py # RAG implementation
│ │ ├── agent.py # LangChain agent
│ │ └── mcp_server.py # MCP server implementation
│ └── utils/ # Utility modules
│ ├── __init__.py
│ ├── env.py # Environment variable loading
│ └── logger.py # Logging configuration
├── pyproject.toml # Poetry configuration
├── requirements.txt # Project dependencies
└── tests/ # Test directory
Getting Started
Prerequisites
- Python 3.13+
- Poetry (optional, for development)
- API keys for Exa and FireCrawl
- (Optional) Ollama installed locally
- (Optional) OpenAI API key
Installation
- Clone the repository
git clone https://github.com/yourusername/search-engine-with-rag-and-mcp.git
cd search-engine-with-rag-and-mcp
- Install dependencies
# Using pip
pip install -r requirements.txt
# Or using poetry
poetry install
- Create a
.env file (use docs/env_template.md as a reference)
Usage
The application has three main modes of operation:
1. Direct Search Mode (Default)
# Using pip
python -m src.core.main "your search query"
# Or using poetry
poetry run python -m src.core.main "your search query"
2. Agent Mode
python -m src.core.main --agent "your search query"
3. MCP Server Mode
python -m src.core.main --server
You can also specify custom host and port:
python -m src.core.main --server --host 0.0.0.0 --port 8080
Using Ollama (Optional)
To use Ollama for local embeddings and LLM capabilities:
- Install Ollama: https://ollama.ai/
- Pull a model:
ollama pull mistral:latest
- Set the appropriate environment variables in your
.env file:
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=mistral:latest
Development
This project follows these best practices:
- Code formatting: Black and isort for consistent code style
- Type checking: mypy for static type checking
- Linting: flake8 for code quality
- Testing: pytest for unit and integration tests
- Environment Management: python-dotenv for managing environment variables
- Logging: Structured logging to both console and file
License
This project is licensed under the MIT License - see the LICENSE file for details.
Acknowledgements