Integrates with python-dotenv for managing environment variables, facilitating the secure storage and access of API keys and configuration settings.

Provides repository access and version control integration through GitHub, allowing users to clone and contribute to the project.

Integrates with LangChain to provide agent-based search capabilities and facilitate the creation of agentic AI systems that can search the web and retrieve information.

Provides support for local LLM capabilities, allowing the server to use locally hosted models for embeddings and text generation rather than cloud-based solutions.

Enables the use of cloud-based LLMs via OpenAI's API as an alternative to local models, providing flexibility in model selection.

Supports Poetry for dependency management and packaging, providing an alternative installation method for development purposes.

Utilizes pytest for unit and integration testing to ensure reliability and functionality of the search and RAG capabilities.

Built on Python 3.13+ with type hints, leveraging Python's ecosystem for developing the search engine with RAG capabilities.

Search Engine with RAG and MCP

A powerful search engine that combines LangChain, Model Context Protocol (MCP), Retrieval-Augmented Generation (RAG), and Ollama to create an agentic AI system capable of searching the web, retrieving information, and providing relevant answers.

Features

Web search capabilities using the Exa API
Web content retrieval using FireCrawl
RAG (Retrieval-Augmented Generation) for more relevant information extraction
MCP (Model Context Protocol) server for standardized tool invocation
Support for both local LLMs via Ollama and cloud-based LLMs via OpenAI
Flexible architecture supporting direct search, agent-based search, or server mode
Comprehensive error handling and graceful fallbacks
Python 3.13+ with type hints
Asynchronous processing for efficient web operations

Related MCP server: WebSearch-MCP

Architecture

This project integrates several key components:

Search Module: Uses Exa API to search the web and FireCrawl to retrieve content
RAG Module: Embeds documents, chunks them, and stores them in a FAISS vector store
MCP Server: Provides a standardized protocol for tool invocation
Agent: LangChain-based agent that uses the search and RAG capabilities

Project Structure

search-engine-with-rag-and-mcp/ ├── LICENSE # MIT License ├── README.md # Project documentation ├── data/ # Data directories ├── docs/ # Documentation │ └── env_template.md # Environment variables documentation ├── logs/ # Log files directory (auto-created) ├── src/ # Main package (source code) │ ├── __init__.py │ ├── core/ # Core functionality │ │ ├── __init__.py │ │ ├── main.py # Main entry point │ │ ├── search.py # Web search module │ │ ├── rag.py # RAG implementation │ │ ├── agent.py # LangChain agent │ │ └── mcp_server.py # MCP server implementation │ └── utils/ # Utility modules │ ├── __init__.py │ ├── env.py # Environment variable loading │ └── logger.py # Logging configuration ├── pyproject.toml # Poetry configuration ├── requirements.txt # Project dependencies └── tests/ # Test directory

Getting Started

Prerequisites

Python 3.13+
Poetry (optional, for development)
API keys for Exa and FireCrawl
(Optional) Ollama installed locally
(Optional) OpenAI API key

Installation

Clone the repository

git clone https://github.com/yourusername/search-engine-with-rag-and-mcp.git cd search-engine-with-rag-and-mcp

Install dependencies

# Using pip pip install -r requirements.txt # Or using poetry poetry install

Create a .env file (use docs/env_template.md as a reference)

Usage

The application has three main modes of operation:

1. Direct Search Mode (Default)

# Using pip python -m src.core.main "your search query" # Or using poetry poetry run python -m src.core.main "your search query"

2. Agent Mode

python -m src.core.main --agent "your search query"

3. MCP Server Mode

python -m src.core.main --server

You can also specify custom host and port:

python -m src.core.main --server --host 0.0.0.0 --port 8080

Using Ollama (Optional)

To use Ollama for local embeddings and LLM capabilities:

Install Ollama: https://ollama.ai/
Pull a model:

ollama pull mistral:latest

Set the appropriate environment variables in your .env file:

OLLAMA_BASE_URL=http://localhost:11434 OLLAMA_MODEL=mistral:latest

Development

This project follows these best practices:

Code formatting: Black and isort for consistent code style
Type checking: mypy for static type checking
Linting: flake8 for code quality
Testing: pytest for unit and integration tests
Environment Management: python-dotenv for managing environment variables
Logging: Structured logging to both console and file

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgements

LangChain for the agent framework
Model Context Protocol for the standardized tool invocation
Ollama for local LLM capabilities
Exa for web search capabilities
FireCrawl for web content retrieval

Model Context Protocol Server