mcp-server-requests

Integrations

  • Enables conversion of web pages to Markdown format, allowing for cleaner and more readable text representation of online content.

中文


mcp-server-requests

An MCP server that provides HTTP request capabilities, enabling LLMs to fetch and process web content.

Features

  • Supports converting web content to Markdown format
  • Supports filtering out content useless for LLMs
  • Supports custom User-Agent headers
  • Supports random User-Agent headers
  • Supports custom request headers in HTTP requests
  • Supports full HTTP methods (GET, POST, PUT, DELETE, PATCH)
  • LLMs can access complete HTTP response header information

Installation

git clone https://github.com/coucya/mcp-server-requests.git cd mcp-server-requests pip install .

Usage

MCP Server Configuration

{ "mcpServers": { "mcp-server-requests": { "command": "python", "args": [ "-m", "mcp_server_requests" ] } } }

Command Line

0. Start MCP Server

Start the MCP server directly:

python -m mcp_server_requests
Options
  • --user-agent TEXT: Specify custom User-Agent string
  • --random-user-agent [browser=xxx;os=xxx]: Use randomly generated User-Agent
  • --force-user-agent: Force using command line specified User-Agent, ignoring LLM provided UA
  • --list-os-and-browser: List available browsers and OS for random User-Agent generation
Option Details
  • --user-agent and --random-user-agent are mutually exclusive and cannot be used together
  • User-Agent setup methods:
    • Custom string: --user-agent "Mozilla/5.0 (...)"
    • Fully random: --random-user-agent
    • Conditional random generation:
      • Specify browser type: --random-user-agent browser=chrome
      • Specify OS: --random-user-agent os=windows
      • Both browser and OS: --random-user-agent browser=chrome;os=windows
      • Note: Browser and OS parameters are case insensitive
  • Use --list-os-and-browser to view available browsers and OS for --random-user-agent.
  • --force-user-agent controls User-Agent priority:
    • When enabled: Prioritize command line specified User-Agent (via --user-agent or --random-user-agent), ignoring LLM provided UA
    • When disabled:
      • If LLM provides User-Agent, use that
      • Otherwise use command line specified User-Agent

1. fetch - Fetch Web Content

The fetch subcommand is equivalent to the fetch tool functionality, demonstrating fetch capabilities.

python -m mcp_server_requests fetch <URL> [--return-content {raw,basic_clean,strict_clean,markdown}]

Options:

  • --return-content: Return content type (default: markdown)
    • raw: Return raw unprocessed HTML content
    • basic_clean: Basic cleanup, removing non-display tags like script, style
    • strict_clean: Strict cleanup, removing non-display tags and most HTML attributes
    • markdown: Convert HTML to clean Markdown format

Example:

python -m mcp_server_requests fetch https://example.com

2. get - Execute HTTP GET Request

The get subcommand is equivalent to the http_get tool functionality, demonstrating http_get capabilities.

python -m mcp_server_requests get <URL> [--headers HEADERS]

Options:

  • --headers: Custom request headers (format: "key1=value1;key2=value2")

3. post - Execute HTTP POST Request

The post subcommand is equivalent to the http_post tool functionality, demonstrating http_post capabilities.

python -m mcp_server_requests post <URL> [--headers HEADERS] [--data TEXT]

Options:

  • --headers: Custom request headers
  • --data: Request body data

4. put - Execute HTTP PUT Request

The put subcommand is equivalent to the http_put tool functionality, demonstrating http_put capabilities.

python -m mcp_server_requests put <URL> [--headers HEADERS] [--data TEXT]

Options: Same as POST method


5. delete - Execute HTTP DELETE Request

The delete subcommand is equivalent to the http_delete tool functionality, demonstrating http_delete capabilities.

python -m mcp_server_requests delete <URL> [--headers HEADERS] [--data TEXT]

Options: Same as POST method


Functionality

Available Tools

  1. fetch - Fetch web content
    • Parameters:
      • url (required): Target URL
      • return_content (optional): Return content type ('raw', 'basic_clean', 'strict_clean', 'markdown')
        • raw: Return raw HTML content
        • basic_clean: Return filtered HTML content, removing non-display tags like script, style
        • strict_clean: Return filtered HTML content, removing non-display tags and most useless HTML attributes
        • markdown: Return HTML converted to Markdown
  2. http_get - Execute HTTP GET request
    • Parameters:
      • url (required): Target URL
      • query (optional): Query parameter key-value pairs
      • headers (optional): Custom request headers
        • LLM may specify User-Agent in headers, whether to use it is controlled by --force-user-agent (same applies to other tools)
  3. http_post - Execute HTTP POST request
    • Parameters:
      • url (required): Target URL
      • query (optional): Query parameter key-value pairs
      • headers (optional): Custom request headers
      • data (optional): Request body data (text)
      • json (optional): Request body data (JSON)
      • data and json cannot be used together
  4. http_put - Execute HTTP PUT request
    • Parameters: Same as http_post
  5. http_patch - Execute HTTP PATCH request
    • Parameters: Same as http_post
  6. http_delete - Execute HTTP DELETE request
    • Parameters: Same as http_post

License

MIT

-
security - not tested
A
license - permissive license
-
quality - not tested

remote-capable server

The server can be hosted and run remotely because it primarily relies on remote services or has no dependency on the local environment.

Web Content Retrieval (full webpage, filtered content, or Markdown-converted), Custom User-Agent, Multi-HTTP Method Support (GET/POST/PUT/DELETE/PATCH), LLM-Controlled Request Headers, LLM-Accessible Response Headers, and more.

  1. Features
    1. Installation
      1. Usage
        1. MCP Server Configuration
        2. Command Line
        3. 0. Start MCP Server
        4. 1. fetch - Fetch Web Content
        5. 2. get - Execute HTTP GET Request
        6. 3. post - Execute HTTP POST Request
        7. 4. put - Execute HTTP PUT Request
        8. 5. delete - Execute HTTP DELETE Request
      2. Functionality
        1. Available Tools
      3. License

        Related MCP Servers

        • A
          security
          A
          license
          A
          quality
          This server enables LLMs to retrieve and process content from web pages, converting HTML to markdown for easier consumption.
          Last updated 15 hours ago
          1
          43,046
          JavaScript
          MIT License
          • Linux
          • Apple
        • A
          security
          A
          license
          A
          quality
          Extracts and transforms webpage content into clean, LLM-optimized Markdown. Returns article title, main content, excerpt, byline and site name. Uses Mozilla's Readability algorithm to remove ads, navigation, footers and non-essential elements while preserving the core content structure.
          Last updated 3 months ago
          1
          4
          11
          MIT License
        • A
          security
          F
          license
          A
          quality
          Provides functionality to fetch web content in various formats, including HTML, JSON, plain text, and Markdown.
          Last updated 3 months ago
          4
          137,083
          150
          TypeScript
        • -
          security
          A
          license
          -
          quality
          Enables retrieval and processing of web page content for LLMs by converting HTML to markdown, with support for content truncation and pagination.
          Last updated 3 months ago
          1
          1
          Python
          MIT License

        View all related MCP servers

        ID: 7cy501fadq