Integrations
Uses the DuckDuckGo Search API to find relevant resources for generating learning paths
Built with FastAPI to provide a robust API framework for generating Master Content Plans
Provides integration guidance for Flutter applications to consume the generated MCPs
MCP Server
A server that generates Master Content Plans (MCPs) based on topics. The server aggregates resources from the web and organizes them into structured learning paths.
Features
- Generate learning paths for any topic (not just technology topics)
- Find relevant resources using web search and scraping
- Organize resources into a logical sequence with customizable number of nodes
- Support for multiple languages with focus on Portuguese
- Performance optimizations for Render's free tier
- Caching system for faster responses
- Return a standardized JSON structure for consumption by client applications
- NEW: TF-IDF based resource relevance filtering to ensure resources match the requested topic
- NEW: Strategic quiz distribution across learning trees for balanced learning experiences
- NEW: YouTube integration to include relevant videos in learning paths
- NEW: Category system to generate more specific content for different types of topics
- NEW: Asynchronous task system with real-time progress feedback to improve user experience and avoid timeouts
- NEW: Enhanced caching system for improved performance and faster response times
- NEW: Optimized web scraping techniques for better resource utilization
- NEW: Adaptive scraping system that automatically chooses the most efficient method for each website
- NEW: Puppeteer instance pool for efficient browser reuse and reduced memory usage
Tech Stack
- Python 3.9+
- FastAPI
- Pyppeteer for JavaScript-heavy web scraping
- Pyppeteer-stealth for avoiding detection during web scraping
- Puppeteer instance pool for efficient browser reuse
- DuckDuckGo Search API
- BeautifulSoup for HTML parsing
- scikit-learn for TF-IDF based resource relevance filtering
- yt-dlp for YouTube video search and metadata extraction
- Redis (optional) for distributed caching
- msgpack for efficient data serialization
- cachetools for advanced in-memory caching
Installation
- Clone the repository:Copy
- Create a virtual environment:Copy
- Install Python dependencies:Copy
- Install Node.js dependencies (for the optimized scraping system):Copy
- Install Chrome/Chromium for Pyppeteer (if not already installed)
Usage
Running Locally
- Start the server using the provided batch file (Windows):Or manually with uvicorn:CopyCopy
- Access the API at
http://localhost:8000
- Generate an MCP by making a GET request to:Copy
- Check the API documentation at
http://localhost:8000/docs
Production URL
The production server is available at:
All endpoints documented in this README are available at both the local and production URLs.
Testing the Caching System
- Make a first request to generate an MCP (this will populate the cache):Copy
- Make a second request with the same parameters (this should use the cache):The second request should be significantly faster as the result will be retrieved from the cache.Copy
Documentation
Detailed documentation is available in the docs
folder:
- API Reference - Detailed API documentation
- Endpoints Reference - Complete reference of all endpoints
- Flutter Integration - Guide for integrating with Flutter apps
- Async Tasks System - Documentation for the asynchronous task system
- Performance Improvements - Overview of performance optimizations
- Caching System - Documentation for the caching system
- Web Scraping Optimization - Details on web scraping optimizations
API Endpoints
GET /health
- Health check endpointGET /generate_mcp?topic={topic}&max_resources={max_resources}&num_nodes={num_nodes}&min_width={min_width}&max_width={max_width}&min_height={min_height}&max_height={max_height}&language={language}&category={category}
- Generate an MCP for the specified topic synchronouslytopic
(required): The topic to generate an MCP for (minimum 3 characters)max_resources
(optional): Maximum number of resources to include (default: 15, min: 5, max: 30)num_nodes
(optional): Number of nodes to include in the learning path (default: 15, min: 10, max: 30)min_width
(optional): Minimum width of the tree (nodes at first level) (default: 3, min: 2, max: 10)max_width
(optional): Maximum width at any level of the tree (default: 5, min: 3, max: 15)min_height
(optional): Minimum height of the tree (depth) (default: 3, min: 2, max: 8)max_height
(optional): Maximum height of the tree (depth) (default: 7, min: 3, max: 12)language
(optional): Language for resources (default: "pt")category
(optional): Category for the topic (e.g., "technology", "finance", "health"). If not provided, it will be detected automatically.
POST /generate_mcp_async?topic={topic}&max_resources={max_resources}&num_nodes={num_nodes}&min_width={min_width}&max_width={max_width}&min_height={min_height}&max_height={max_height}&language={language}&category={category}
- Start asynchronous generation of an MCPGET /status/{task_id}
- Check the status of an asynchronous taskGET /tasks
- List all tasksPOST /clear_cache?pattern={pattern}&clear_domain_cache={clear_domain_cache}
- Clear the cache based on a patternpattern
(optional): Pattern to match cache keys (default: "*" for all)clear_domain_cache
(optional): Whether to also clear the domain method cache (default: false)
GET /cache_stats
- Get statistics about the cache, including information about the domain method cache
Examples
Basic usage (Portuguese)
Custom number of nodes
English language
Specify category manually
Full customization
Control tree structure
Asynchronous generation
Check task status
Clear cache
Clear specific cache
Performance Improvements
The MCP Server includes several performance optimizations:
- Caching System: Results are cached to improve response times for repeated queries
- Asynchronous Task System: Long-running operations are handled asynchronously
- Resource Filtering: TF-IDF based filtering to select the most relevant resources
- Optimized Web Scraping: Efficient web scraping techniques to reduce resource usage
- Adaptive Scraping System: Automatically chooses the most efficient scraping method for each website
- Puppeteer Instance Pool: Reuses browser instances to reduce memory usage and startup time
- Domain Method Cache: Remembers which scraping method works best for each domain
- Resource Blocking: Blocks unnecessary resources (images, stylesheets, fonts) during scraping
Performance Gains
- 60-80% reduction in response time for topics already in cache
- 30-50% reduction in response time for new topics
- 40-60% reduction in memory usage during web scraping
- 3-5x increase in throughput for simultaneous requests
Deployment
The server can be deployed to various platforms:
Using Docker
Deploying to Render, Fly.io, or other platforms
Follow the platform-specific instructions for deploying a Docker container or a Python application.
License
Proprietary Software - All Rights Reserved
This software is proprietary and confidential. Unauthorized copying, distribution, modification, public display, or public performance of this software is strictly prohibited. This software is intended for use under a paid subscription model only.
© 2024 ReuneMacacada. All rights reserved.
Last commit: v1.1.2 - Correção de problemas com DuckDuckGo rate limit e Puppeteer
This server cannot be installed
hybrid server
The server is able to function both locally and remotely, depending on the configuration or use case.
A server that generates Master Content Plans (MCPs) by aggregating web resources and organizing them into structured learning paths for any topic.
Related MCP Servers
- AsecurityAlicenseAqualityA powerful MCP server for fetching and transforming web content into various formats (HTML, JSON, Markdown, Plain Text) with ease.Last updated -414612TypeScriptMIT License
- AsecurityAlicenseAqualityAn MCP server for fetching and transforming web content into various formats.Last updated -44PythonMIT License
- AsecurityFlicenseAqualityAn MCP (Model Context Protocol) server that provides Google search capabilities and webpage content analysis tools. This server enables AI models to perform Google searches and analyze webpage content programmatically.Last updated -34641TypeScript
- -securityFlicense-qualityAn MCP server that provides tools and resources for managing a coding project's todo list, allowing users to view, add, update, and delete todo items with details like priority, status, and tags.Last updated -1Python