FullScope-MCP

Instructions

抓取网页内容，可选保存为txt文件

Args:
    url: 要抓取的网页URL
    save_to_file: 是否保存内容到txt文件

Returns:
    抓取结果和文件路径（如果保存）

Implementation Reference

• src/fullscope_mcp_server/server.py:304-339 (handler)

  The primary handler function for the 'scrape_webpage' tool. Decorated with @mcp.tool() for automatic registration in FastMCP. Scrapes the webpage URL using the WebScraper instance, extracts title and content with BeautifulSoup, optionally saves to a temp file, and returns formatted result.

  @mcp.tool()
  async def scrape_webpage(url: str, ctx: Context, save_to_file: bool = False) -> str:
      """
      抓取网页内容，可选保存为txt文件
      
      Args:
          url: 要抓取的网页URL
          save_to_file: 是否保存内容到txt文件
      
      Returns:
          抓取结果和文件路径（如果保存）
      """
      try:
          ctx.info(f"开始抓取网页: {url}")
          
          title, content = await scraper.scrape_url(url)
          
          result = f"网页标题: {title}\n\n网页内容:\n{content}"
          
          if save_to_file:
              # 生成文件名
              parsed_url = urlparse(url)
              filename = f"{parsed_url.netloc}_{title[:20]}.txt".replace(" ", "_")
              # 移除非法字符
              filename = "".join(c for c in filename if c.isalnum() or c in "._-")
              
              filepath = await scraper.save_content_to_file(result, filename)
              result += f"\n\n已保存到文件: {filepath}"
          
          ctx.info("网页抓取完成")
          return result
          
      except Exception as e:
          logger.error(f"网页抓取失败: {e}")
          return f"网页抓取失败: {str(e)}"

• src/fullscope_mcp_server/server.py:119-157 (helper)

  Key helper method in WebScraper class that performs HTTP GET request with httpx, parses HTML with BeautifulSoup, extracts title, cleans content by removing scripts/styles and normalizing text.

  async def scrape_url(self, url: str) -> tuple[str, str]:
      """
      抓取网页内容
      
      Args:
          url: 目标URL
          
      Returns:
          (title, content): 网页标题和清理后的文本内容
      """
      try:
          response = await self.session.get(url)
          response.raise_for_status()
          
          # 使用BeautifulSoup解析HTML
          soup = BeautifulSoup(response.text, 'html.parser')
          
          # 获取标题
          title = soup.find('title')
          title = title.get_text().strip() if title else "无标题"
          
          # 移除script和style标签
          for script in soup(["script", "style"]):
              script.decompose()
          
          # 提取主要内容
          content = soup.get_text()
          
          # 清理文本
          lines = (line.strip() for line in content.splitlines())
          chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
          content = ' '.join(chunk for chunk in chunks if chunk)
          
          return title, content
          
      except Exception as e:
          logger.error(f"网页抓取失败 {url}: {e}")
          raise Exception(f"无法抓取网页: {str(e)}")

• src/fullscope_mcp_server/server.py:158-187 (helper)

  Helper method in WebScraper class to save scraped content to a temporary .txt file, generating filename if not provided using content hash.

  async def save_content_to_file(self, content: str, filename: str = None) -> str:
      """
      保存内容到文件
      
      Args:
          content: 要保存的内容
          filename: 文件名（可选）
          
      Returns:
          保存的文件路径
      """
      if not filename:
          # 生成基于内容哈希的文件名
          content_hash = hashlib.md5(content.encode()).hexdigest()[:8]
          filename = f"scraped_content_{content_hash}.txt"
      
      # 确保文件扩展名为.txt
      if not filename.endswith('.txt'):
          filename += '.txt'
      
      filepath = Path(tempfile.gettempdir()) / filename
      
      try:
          with open(filepath, 'w', encoding='utf-8') as f:
              f.write(content)
          return str(filepath)
      except Exception as e:
          logger.error(f"保存文件失败: {e}")
          raise Exception(f"无法保存文件: {str(e)}")

• src/fullscope_mcp_server/server.py:272-273 (helper)

  Global instance of WebScraper class used by the scrape_webpage handler.

  file_processor = FileProcessor()
  rag_processor = RAGProcessor(summarizer)