-
securityF
license-
qualityEnables seamless AI integration via Ollama's Deepseek model, providing protocol compliance and automatic configuration for clean AI-driven interactions.
Integrates with python-dotenv for managing environment variables, facilitating the secure storage and access of API keys and configuration settings.
Provides repository access and version control integration through GitHub, allowing users to clone and contribute to the project.
Integrates with LangChain to provide agent-based search capabilities and facilitate the creation of agentic AI systems that can search the web and retrieve information.
Provides support for local LLM capabilities, allowing the server to use locally hosted models for embeddings and text generation rather than cloud-based solutions.
Enables the use of cloud-based LLMs via OpenAI's API as an alternative to local models, providing flexibility in model selection.
Supports Poetry for dependency management and packaging, providing an alternative installation method for development purposes.
Utilizes pytest for unit and integration testing to ensure reliability and functionality of the search and RAG capabilities.
Built on Python 3.13+ with type hints, leveraging Python's ecosystem for developing the search engine with RAG capabilities.
Search Engine with RAG and MCP
A powerful search engine that combines LangChain, Model Context Protocol (MCP), Retrieval-Augmented Generation (RAG), and Ollama to create an agentic AI system capable of searching the web, retrieving information, and providing relevant answers.
Features
Web search capabilities using the Exa API
Web content retrieval using FireCrawl
RAG (Retrieval-Augmented Generation) for more relevant information extraction
MCP (Model Context Protocol) server for standardized tool invocation
Support for both local LLMs via Ollama and cloud-based LLMs via OpenAI
Flexible architecture supporting direct search, agent-based search, or server mode
Comprehensive error handling and graceful fallbacks
Python 3.13+ with type hints
Asynchronous processing for efficient web operations
Architecture
This project integrates several key components:
Search Module: Uses Exa API to search the web and FireCrawl to retrieve content
RAG Module: Embeds documents, chunks them, and stores them in a FAISS vector store
MCP Server: Provides a standardized protocol for tool invocation
Agent: LangChain-based agent that uses the search and RAG capabilities
Project Structure
search-engine-with-rag-and-mcp/
├── LICENSE # MIT License
├── README.md # Project documentation
├── data/ # Data directories
├── docs/ # Documentation
│ └── env_template.md # Environment variables documentation
├── logs/ # Log files directory (auto-created)
├── src/ # Main package (source code)
│ ├── __init__.py
│ ├── core/ # Core functionality
│ │ ├── __init__.py
│ │ ├── main.py # Main entry point
│ │ ├── search.py # Web search module
│ │ ├── rag.py # RAG implementation
│ │ ├── agent.py # LangChain agent
│ │ └── mcp_server.py # MCP server implementation
│ └── utils/ # Utility modules
│ ├── __init__.py
│ ├── env.py # Environment variable loading
│ └── logger.py # Logging configuration
├── pyproject.toml # Poetry configuration
├── requirements.txt # Project dependencies
└── tests/ # Test directory
Getting Started
Prerequisites
Python 3.13+
Poetry (optional, for development)
API keys for Exa and FireCrawl
(Optional) Ollama installed locally
(Optional) OpenAI API key
Installation
Clone the repository
git clone https://github.com/yourusername/search-engine-with-rag-and-mcp.git
cd search-engine-with-rag-and-mcp
Install dependencies
# Using pip
pip install -r requirements.txt
# Or using poetry
poetry install
Create a
.envfile (use docs/env_template.md as a reference)
Usage
The application has three main modes of operation:
1. Direct Search Mode (Default)
# Using pip
python -m src.core.main "your search query"
# Or using poetry
poetry run python -m src.core.main "your search query"
2. Agent Mode
python -m src.core.main --agent "your search query"
3. MCP Server Mode
python -m src.core.main --server
You can also specify custom host and port:
python -m src.core.main --server --host 0.0.0.0 --port 8080
Using Ollama (Optional)
To use Ollama for local embeddings and LLM capabilities:
Install Ollama: https://ollama.ai/
Pull a model:
ollama pull mistral:latest
Set the appropriate environment variables in your
.envfile:
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=mistral:latest
Development
This project follows these best practices:
Code formatting: Black and isort for consistent code style
Type checking: mypy for static type checking
Linting: flake8 for code quality
Testing: pytest for unit and integration tests
Environment Management: python-dotenv for managing environment variables
Logging: Structured logging to both console and file
License
This project is licensed under the MIT License - see the LICENSE file for details.
Acknowledgements
LangChain for the agent framework
Model Context Protocol for the standardized tool invocation
Ollama for local LLM capabilities
Exa for web search capabilities
FireCrawl for web content retrieval
This server cannot be installed
hybrid server
The server is able to function both locally and remotely, depending on the configuration or use case.
Provides a standardized protocol for tool invocation, enabling an AI system to search the web, retrieve information, and provide relevant answers through integration with LangChain, RAG, and Ollama.
Related MCP Servers
- AsecurityFlicenseAqualityA Model Context Protocol server that enables AI assistants to perform real-time web searches, retrieving up-to-date information from the internet via a Crawler API.Last updated -124217
- -securityFlicense-qualityA simple Model Context Protocol server that enables searching and retrieving relevant documentation snippets from Langchain, Llama Index, and OpenAI official documentation.Last updated -
- -securityFlicense-qualityAn integration that enables AI assistants to interact with network data through a standardized protocol, providing AI-ready tools and interfaces for network automation and management.Last updated -15