MCP Server

# MCP Server A server that generates Master Content Plans (MCPs) based on topics. The server aggregates resources from the web and organizes them into structured learning paths. ## Features - Generate learning paths for any topic (not just technology topics) - Find relevant resources using web search and scraping - Organize resources into a logical sequence with customizable number of nodes - Support for multiple languages with focus on Portuguese - Performance optimizations for Render's free tier - Caching system for faster responses - Return a standardized JSON structure for consumption by client applications - **NEW**: TF-IDF based resource relevance filtering to ensure resources match the requested topic - **NEW**: Strategic quiz distribution across learning trees for balanced learning experiences - **NEW**: YouTube integration to include relevant videos in learning paths - **NEW**: Category system to generate more specific content for different types of topics - **NEW**: Asynchronous task system with real-time progress feedback to improve user experience and avoid timeouts - **NEW**: Enhanced caching system for improved performance and faster response times - **NEW**: Optimized web scraping techniques for better resource utilization - **NEW**: Adaptive scraping system that automatically chooses the most efficient method for each website - **NEW**: Puppeteer instance pool for efficient browser reuse and reduced memory usage ## Tech Stack - Python 3.9+ - FastAPI - Pyppeteer for JavaScript-heavy web scraping - Pyppeteer-stealth for avoiding detection during web scraping - Puppeteer instance pool for efficient browser reuse - DuckDuckGo Search API - BeautifulSoup for HTML parsing - scikit-learn for TF-IDF based resource relevance filtering - yt-dlp for YouTube video search and metadata extraction - Redis (optional) for distributed caching - msgpack for efficient data serialization - cachetools for advanced in-memory caching ## Installation 1. Clone the repository: ``` git clone https://github.com/yourusername/mcp_server.git cd mcp_server ``` 2. Create a virtual environment: ``` python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate ``` 3. Install Python dependencies: ``` pip install -r requirements.txt ``` 4. Install Node.js dependencies (for the optimized scraping system): ``` npm install ``` 5. Install Chrome/Chromium for Pyppeteer (if not already installed) ## Usage ### Running Locally 1. Start the server using the provided batch file (Windows): ``` run_local.bat ``` Or manually with uvicorn: ``` uvicorn main:app --reload --host 0.0.0.0 --port 8000 ``` 2. Access the API at `http://localhost:8000` 3. Generate an MCP by making a GET request to: ``` GET /generate_mcp?topic=your_topic ``` 4. Check the API documentation at `http://localhost:8000/docs` ### Production URL The production server is available at: ``` https://reunemacacada.onrender.com ``` All endpoints documented in this README are available at both the local and production URLs. ### Testing the Caching System 1. Make a first request to generate an MCP (this will populate the cache): ``` GET /generate_mcp?topic=python&num_nodes=15&language=pt ``` 2. Make a second request with the same parameters (this should use the cache): ``` GET /generate_mcp?topic=python&num_nodes=15&language=pt ``` The second request should be significantly faster as the result will be retrieved from the cache. ## Documentation Detailed documentation is available in the `docs` folder: - [API Reference](docs/api_reference.md) - Detailed API documentation - [Endpoints Reference](docs/endpoints_reference.md) - Complete reference of all endpoints - [Flutter Integration](docs/flutter_integration.md) - Guide for integrating with Flutter apps - [Async Tasks System](docs/async_tasks_system.md) - Documentation for the asynchronous task system - [Performance Improvements](docs/performance_improvements.md) - Overview of performance optimizations - [Caching System](docs/caching_system.md) - Documentation for the caching system - [Web Scraping Optimization](docs/web_scraping_optimization.md) - Details on web scraping optimizations ## API Endpoints - `GET /health` - Health check endpoint - `GET /generate_mcp?topic={topic}&max_resources={max_resources}&num_nodes={num_nodes}&min_width={min_width}&max_width={max_width}&min_height={min_height}&max_height={max_height}&language={language}&category={category}` - Generate an MCP for the specified topic synchronously - `topic` (required): The topic to generate an MCP for (minimum 3 characters) - `max_resources` (optional): Maximum number of resources to include (default: 15, min: 5, max: 30) - `num_nodes` (optional): Number of nodes to include in the learning path (default: 15, min: 10, max: 30) - `min_width` (optional): Minimum width of the tree (nodes at first level) (default: 3, min: 2, max: 10) - `max_width` (optional): Maximum width at any level of the tree (default: 5, min: 3, max: 15) - `min_height` (optional): Minimum height of the tree (depth) (default: 3, min: 2, max: 8) - `max_height` (optional): Maximum height of the tree (depth) (default: 7, min: 3, max: 12) - `language` (optional): Language for resources (default: "pt") - `category` (optional): Category for the topic (e.g., "technology", "finance", "health"). If not provided, it will be detected automatically. - `POST /generate_mcp_async?topic={topic}&max_resources={max_resources}&num_nodes={num_nodes}&min_width={min_width}&max_width={max_width}&min_height={min_height}&max_height={max_height}&language={language}&category={category}` - Start asynchronous generation of an MCP - `GET /status/{task_id}` - Check the status of an asynchronous task - `GET /tasks` - List all tasks - `POST /clear_cache?pattern={pattern}&clear_domain_cache={clear_domain_cache}` - Clear the cache based on a pattern - `pattern` (optional): Pattern to match cache keys (default: "\*" for all) - `clear_domain_cache` (optional): Whether to also clear the domain method cache (default: false) - `GET /cache_stats` - Get statistics about the cache, including information about the domain method cache ## Examples ### Basic usage (Portuguese) ``` GET /generate_mcp?topic=python ``` ### Custom number of nodes ``` GET /generate_mcp?topic=machine+learning&num_nodes=20 ``` ### English language ``` GET /generate_mcp?topic=javascript&language=en ``` ### Specify category manually ``` GET /generate_mcp?topic=python&category=technology ``` ### Full customization ``` GET /generate_mcp?topic=história+do+brasil&max_resources=20&num_nodes=25&min_width=4&max_width=8&min_height=4&max_height=8&language=pt ``` ### Control tree structure ``` GET /generate_mcp?topic=machine+learning&min_width=2&max_width=4&min_height=5&max_height=10 ``` ### Asynchronous generation ``` POST /generate_mcp_async?topic=inteligência+artificial&category=technology ``` ### Check task status ``` GET /status/550e8400-e29b-41d4-a716-446655440000 ``` ### Clear cache ``` POST /clear_cache ``` ### Clear specific cache ``` POST /clear_cache?pattern=mcp:* ``` ## Performance Improvements The MCP Server includes several performance optimizations: - **Caching System**: Results are cached to improve response times for repeated queries - **Asynchronous Task System**: Long-running operations are handled asynchronously - **Resource Filtering**: TF-IDF based filtering to select the most relevant resources - **Optimized Web Scraping**: Efficient web scraping techniques to reduce resource usage - **Adaptive Scraping System**: Automatically chooses the most efficient scraping method for each website - **Puppeteer Instance Pool**: Reuses browser instances to reduce memory usage and startup time - **Domain Method Cache**: Remembers which scraping method works best for each domain - **Resource Blocking**: Blocks unnecessary resources (images, stylesheets, fonts) during scraping ### Performance Gains - **60-80% reduction in response time** for topics already in cache - **30-50% reduction in response time** for new topics - **40-60% reduction in memory usage** during web scraping - **3-5x increase in throughput** for simultaneous requests ## Deployment The server can be deployed to various platforms: ### Using Docker ``` docker build -t mcp-server . docker run -p 8080:8080 mcp-server ``` ### Deploying to Render, Fly.io, or other platforms Follow the platform-specific instructions for deploying a Docker container or a Python application. ## License **Proprietary Software - All Rights Reserved** This software is proprietary and confidential. Unauthorized copying, distribution, modification, public display, or public performance of this software is strictly prohibited. This software is intended for use under a paid subscription model only. © 2024 ReuneMacacada. All rights reserved. Last commit: v1.1.2 - Correção de problemas com DuckDuckGo rate limit e Puppeteer