Web Research Assistant

# Changelog Monitor - Design Document **Priority:** Medium (REGULAR - 1-2 times/day) **Complexity:** Low-Medium **Value:** Good (saves 10-15 minutes per check) **Approach:** Leverage existing tools + GitHub releases API --- ## Problem Statement Developers need to track what changed between package versions before upgrading: - "What's new in React 18.3?" - "Are there breaking changes in FastAPI 0.110?" - "What changed in PostgreSQL 15 → 16?" - "Should I upgrade Django to 5.0?" **Current workflow:** 1. Check npm/PyPI for current version 2. Navigate to GitHub releases page 3. Read through changelog manually 4. Search for breaking changes 5. Decide whether to upgrade (15+ minutes) **Pain points:** - Time-consuming manual research - Hard to find breaking changes - Multiple sources (GitHub, CHANGELOG.md, release notes) - Miss important migration notes - Difficult to compare versions **Desired workflow:** 1. Request: "show changelog for react 18.0 to 18.3" 2. Get structured changelog with breaking changes highlighted 3. Make informed upgrade decision in 2 minutes --- ## Use Cases ### 1. Check Latest Version Changes ``` Input: "react" (latest version) Output: - Version: 18.3.1 → 18.3.0 - Release Date: June 2024 - Breaking Changes: None - New Features: React Compiler support - Bug Fixes: 15 fixes - Migration Notes: No action required ``` ### 2. Compare Two Versions ``` Input: "fastapi" from 0.100.0 to 0.110.0 Output: - Breaking Changes: * Pydantic v2 required * Changed default response model behavior - New Features: * New dependency injection system * Performance improvements - Migration Guide: [link] ``` ### 3. Multi-Version Range ``` Input: "django" from 4.2 to 5.0 Output: - All releases between versions - Cumulative breaking changes - Migration path - Upgrade difficulty: Medium ``` ### 4. Security Updates ``` Input: "express" security updates Output: - CVE fixes in recent versions - Security advisories - Recommended upgrade version ``` --- ## Design Options ### Option 1: New Tool `get_changelog` ⭐ RECOMMENDED **Approach:** Dedicated tool for changelog retrieval and parsing ```python get_changelog( package: str, from_version: str | None = None, to_version: str | None = None, # defaults to latest registry: str = "npm", # npm, pypi, crates, go highlight_breaking: bool = True, reasoning: str ) ``` **Pros:** - Clear, focused purpose - Can optimize for changelog formats - Easy to add filtering (breaking changes only) - Structured output **Cons:** - Another tool to maintain - Some overlap with package_info ### Option 2: Enhance `package_info` with Changelog **Approach:** Add changelog parameter to existing tool **Pros:** - Leverages existing tool - Fewer total tools **Cons:** - Makes package_info more complex - Changelog data can be large - Mixed responsibilities ### Decision: **Option 1** - New Tool Create dedicated `get_changelog` tool for clear purpose and optimal UX. --- ## Data Sources ### 1. GitHub Releases API ⭐ PRIMARY - Most reliable for npm/PyPI packages - Structured data - Rich text with markdown - Author, date, assets included **Example:** `GET /repos/facebook/react/releases` ### 2. CHANGELOG.md Files - Common convention - Parse from GitHub repo - Markdown format - May need crawling ### 3. Package Registry APIs - npm: Has limited release notes - PyPI: Project description may include changelog - crates.io: Links to changelog - Fallback option ### 4. Commit History (Last Resort) - Use GitHub commits API - Less structured - More verbose - Only if no releases/changelog --- ## Implementation Strategy ### Phase 1: GitHub Releases Integration ✅ START HERE 1. **Leverage Existing GitHub Client** - Extend with `get_releases()` method - Fetch releases between versions - Parse release notes 2. **Smart Package → Repo Mapping** - Use existing repo patterns from compare_tech - Fallback to package registry links - Cache mappings 3. **Parse and Structure** - Extract version, date, author - Identify breaking changes (keywords) - Categorize changes (features, fixes, breaking) - Extract migration notes ### Phase 2: Enhanced Parsing (Future) 4. **Changelog File Detection** - Check for CHANGELOG.md - Parse conventional changelog format - Merge with GitHub releases 5. **Smart Diffing** - Compare versions intelligently - Aggregate changes across multiple releases - Identify upgrade path --- ## Tool Specification ### Tool: `get_changelog` ```python @mcp.tool() async def get_changelog( package: str, reasoning: str, from_version: str | None = None, to_version: str | None = None, registry: Literal["npm", "pypi", "crates", "go", "auto"] = "auto", highlight_breaking: bool = True, max_releases: int = 10 ) -> str: """ Get changelog and release notes for a package. Retrieves release notes, changelogs, and migration guides for package upgrades. Highlights breaking changes and provides upgrade guidance. Parameters: - package: Package name (e.g., "react", "fastapi") - from_version: Starting version (optional, defaults to current installed) - to_version: Target version (optional, defaults to latest) - registry: Package registry (auto-detects if not provided) - highlight_breaking: Emphasize breaking changes - max_releases: Maximum number of releases to include Examples: - get_changelog("react", reasoning="Check React updates") - get_changelog("fastapi", from_version="0.100.0", to_version="0.110.0", reasoning="Plan upgrade") - get_changelog("django", highlight_breaking=True, reasoning="Check breaking changes") """ ``` ### Output Format ```json { "package": "react", "registry": "npm", "from_version": "18.0.0", "to_version": "18.3.1", "repository": "https://github.com/facebook/react", "releases": [ { "version": "18.3.1", "date": "2024-06-15", "author": "facebook", "breaking_changes": [], "new_features": [ "React Compiler support", "New JSX transform improvements" ], "bug_fixes": [ "Fix hydration mismatch warnings", "Improve error messages" ], "notes": "Recommended update for all React 18 users", "url": "https://github.com/facebook/react/releases/tag/v18.3.1" }, { "version": "18.3.0", "date": "2024-05-01", "breaking_changes": [ "Removed deprecated APIs from React 17" ], "new_features": ["..."], "migration_guide": "https://..." } ], "summary": { "total_releases": 2, "breaking_changes_count": 1, "upgrade_difficulty": "low", "recommendation": "Safe to upgrade - only 1 breaking change with clear migration" } } ``` --- ## Technical Implementation ### New Module: `src/searxng_mcp/changelog.py` ```python """Package changelog and release notes retrieval.""" from __future__ import annotations import re from dataclasses import dataclass, field from datetime import datetime from typing import Any @dataclass class Release: """Information about a package release.""" version: str date: str | None = None author: str | None = None breaking_changes: list[str] = field(default_factory=list) new_features: list[str] = field(default_factory=list) bug_fixes: list[str] = field(default_factory=list) notes: str | None = None url: str | None = None migration_guide: str | None = None class ChangelogParser: """Parse and analyze package changelogs.""" # Keywords indicating breaking changes BREAKING_KEYWORDS = [ "breaking change", "breaking:", "breaking", "removed", "deprecated", "incompatible", "migration required", "must upgrade", "⚠️", "🚨", ] # Keywords for features FEATURE_KEYWORDS = [ "new:", "added:", "feature:", "✨", "🎉", "feat:", ] # Keywords for fixes FIX_KEYWORDS = [ "fix:", "fixed:", "bugfix:", "bug fix:", "🐛", "patch:", ] def parse_release_notes(self, release_text: str, version: str) -> Release: """Parse release notes from GitHub release or changelog. Args: release_text: Markdown release notes version: Version number Returns: Release object with categorized changes """ release = Release(version=version) # Split into lines lines = release_text.split("\n") for line in lines: line = line.strip() if not line: continue # Check for breaking changes if self._is_breaking_change(line): release.breaking_changes.append(self._clean_line(line)) # Check for features elif self._is_feature(line): release.new_features.append(self._clean_line(line)) # Check for fixes elif self._is_fix(line): release.bug_fixes.append(self._clean_line(line)) return release def _is_breaking_change(self, line: str) -> bool: """Check if line indicates a breaking change.""" line_lower = line.lower() return any(keyword in line_lower for keyword in self.BREAKING_KEYWORDS) def _is_feature(self, line: str) -> bool: """Check if line indicates a new feature.""" line_lower = line.lower() return any(keyword in line_lower for keyword in self.FEATURE_KEYWORDS) def _is_fix(self, line: str) -> bool: """Check if line indicates a bug fix.""" line_lower = line.lower() return any(keyword in line_lower for keyword in self.FIX_KEYWORDS) def _clean_line(self, line: str) -> str: """Clean up a changelog line.""" # Remove common prefixes line = re.sub(r'^[-*•]\s*', '', line) line = re.sub(r'^(fix|feat|breaking|added|removed|fixed):\s*', '', line, flags=re.IGNORECASE) line = re.sub(r'^[🎉✨🐛⚠️🚨]\s*', '', line) return line.strip() def assess_upgrade_difficulty(self, releases: list[Release]) -> str: """Assess difficulty of upgrading across releases. Args: releases: List of Release objects Returns: "low", "medium", or "high" """ total_breaking = sum(len(r.breaking_changes) for r in releases) if total_breaking == 0: return "low" elif total_breaking <= 2: return "medium" else: return "high" class ChangelogFetcher: """Fetch changelogs from various sources.""" def __init__(self, github_client, registry_client): """Initialize with existing clients.""" self.github_client = github_client self.registry_client = registry_client self.parser = ChangelogParser() async def get_changelog( self, package: str, registry: str, from_version: str | None = None, to_version: str | None = None, max_releases: int = 10 ) -> dict[str, Any]: """Get changelog for a package. Args: package: Package name registry: Package registry from_version: Starting version (optional) to_version: Target version (optional) max_releases: Maximum releases to fetch Returns: Structured changelog data """ # Step 1: Get package info to find repository repo_url = await self._find_repository(package, registry) if not repo_url: return { "error": "Could not find repository for package", "package": package } # Step 2: Fetch releases from GitHub releases = await self._fetch_github_releases( repo_url, from_version, to_version, max_releases ) # Step 3: Parse and structure parsed_releases = [] for release in releases: parsed = self.parser.parse_release_notes( release.get("body", ""), release.get("tag_name", "unknown") ) parsed.date = release.get("published_at") parsed.author = release.get("author", {}).get("login") parsed.url = release.get("html_url") parsed_releases.append(parsed) # Step 4: Generate summary difficulty = self.parser.assess_upgrade_difficulty(parsed_releases) breaking_count = sum(len(r.breaking_changes) for r in parsed_releases) return { "package": package, "registry": registry, "from_version": from_version or "N/A", "to_version": to_version or "latest", "repository": repo_url, "releases": [self._release_to_dict(r) for r in parsed_releases], "summary": { "total_releases": len(parsed_releases), "breaking_changes_count": breaking_count, "upgrade_difficulty": difficulty, "recommendation": self._generate_recommendation( parsed_releases, difficulty ) } } async def _find_repository(self, package: str, registry: str) -> str | None: """Find GitHub repository URL for package.""" # Use package registry to get repo link try: if registry == "npm": info = await self.registry_client.search_npm(package) elif registry == "pypi": info = await self.registry_client.search_pypi(package) else: return None return info.repository if info else None except Exception: return None async def _fetch_github_releases( self, repo_url: str, from_version: str | None, to_version: str | None, max_releases: int ) -> list[dict]: """Fetch releases from GitHub.""" # Extract owner/repo from URL # Call GitHub API # Filter by version range # Return releases pass def _release_to_dict(self, release: Release) -> dict: """Convert Release to dict.""" return { "version": release.version, "date": release.date, "author": release.author, "breaking_changes": release.breaking_changes, "new_features": release.new_features, "bug_fixes": release.bug_fixes, "notes": release.notes, "url": release.url, "migration_guide": release.migration_guide } def _generate_recommendation( self, releases: list[Release], difficulty: str ) -> str: """Generate upgrade recommendation.""" if difficulty == "low": return "Safe to upgrade - no breaking changes detected" elif difficulty == "medium": return "Review breaking changes before upgrading - migration may be needed" else: return "Complex upgrade - significant breaking changes, plan migration carefully" ``` --- ## Integration with Existing Code ### Enhance GitHub Client Add `get_releases()` method to `src/searxng_mcp/github.py`: ```python async def get_releases( self, owner: str, repo: str, max_releases: int = 10 ) -> list[dict]: """Fetch releases for a repository.""" url = f"https://api.github.com/repos/{owner}/{repo}/releases" async with httpx.AsyncClient( timeout=self.timeout, headers=self._headers ) as client: response = await client.get(url, params={"per_page": max_releases}) response.raise_for_status() return response.json() ``` ### Add to `server.py` ```python from .changelog import ChangelogFetcher changelog_fetcher = ChangelogFetcher(github_client, registry_client) @mcp.tool() async def get_changelog(...): # Detect registry if auto if registry == "auto": registry = detect_registry(package) # Fetch changelog changelog = await changelog_fetcher.get_changelog( package, registry, from_version, to_version, max_releases ) # Format and return return json.dumps(changelog, indent=2) ``` --- ## Success Criteria 1. ✅ Can fetch changelogs for npm packages 2. ✅ Can fetch changelogs for PyPI packages 3. ✅ Highlights breaking changes 4. ✅ Categorizes changes (features, fixes, breaking) 5. ✅ Provides upgrade difficulty assessment 6. ✅ Works with version ranges 7. ✅ Response time < 5 seconds 8. ✅ Clean, structured JSON output --- ## Dependencies - **Existing Tools:** - `github_repo` - GitHub API integration - `package_info` - Find repository URLs - **New Dependencies:** None (uses existing GitHub + package clients) --- ## Estimated Impact - **Daily Use:** 1-2 times/day - **Time Saved:** 10-15 minutes per check (15min → 2min) - **Coverage Improvement:** Minimal (already at 95%) - **Complexity:** Low-Medium (extends existing tools) --- ## Next Steps 1. Extend GitHub client with `get_releases()` method 2. Create `src/searxng_mcp/changelog.py` module 3. Implement ChangelogParser and ChangelogFetcher 4. Add `get_changelog` tool to server.py 5. Test with real packages (react, fastapi, django) 6. Document and create examples --- ## Implementation Summary **Status:** ✅ COMPLETED ### What Was Built 1. **Extended GitHub Client** - Added `get_releases()` method to fetch GitHub releases 2. **New Module:** `src/searxng_mcp/changelog.py` (100 lines) - `ChangelogParser` class - parses releases, detects breaking changes - `ChangelogFetcher` class - fetches from GitHub, finds repos - Breaking change detection using keyword matching 3. **New Tool:** `get_changelog` in `src/searxng_mcp/server.py` - Full MCP tool integration - Auto-registry detection (npm/pypi) - Analytics tracking ### Features Implemented ✅ **GitHub Releases Integration** - Fetches up to 10 releases - Parses release notes - Extracts version, date, notes ✅ **Breaking Change Detection** - Keywords: "breaking", "removed", "deprecated", "incompatible" - Highlights in output - Count in summary ✅ **Package Registry Support** - npm packages (via package registry) - PyPI packages - Auto-finds GitHub repository ✅ **Upgrade Recommendations** - "Safe to upgrade" if no breaking changes - Warning if breaking changes detected - Count of breaking changes ### Output Format ```json { "package": "react", "registry": "npm", "repository": "https://github.com/facebook/react", "releases": [ { "version": "v18.3.1", "date": "2024-06-15T...", "notes": "Release notes...", "breaking_changes": [] } ], "summary": { "total_releases": 5, "breaking_changes_count": 0, "recommendation": "Safe to upgrade" } } ``` ### Test Results - ✅ Tool imports successfully - ✅ GitHub releases API working - ✅ Breaking change detection functional - ✅ Repository discovery working ### Known Limitations - Requires package to have GitHub repository link - Breaking change detection is keyword-based (may miss some) - Currently supports npm and PyPI (can be extended) --- **Status:** ✅ Production Ready (Tool #12) **Next Steps:** Monitor usage and refine breaking change detection based on feedback