MCP Data Fetch Server

# 📂 MCP Data Fetch Server **MCP Data Fetch Server** is secure, sandboxed server that fetches web content and extracts data via the **Model Control Protocol (MCP)**. without executing JavaScript. --- ## Table of Contents - [Features](#features) - [Installation & Quick Start](#installation--quick-start) - [Command‑Line Options](#commandline-options) - [Integration with LM Studio](#integration-with-lm-studio) - [MCP API Overview](#mcp-api-overview) - [`initialize`](#initialize) - [`tools/list`](#toolslist) - [`tools/call`](#toolscall) - [Available Tools](#available-tools) - [`fetch_webpage`](#fetch_webpage) - [`extract_links`](#extract_links) - [`download_file`](#download_file) - [`get_page_metadata`](#get_page_metadata) - [`check_url`](#check_url) - [Security Features](#security-features) --- ## 🎯 Features - **Secure web page fetching** – strips scripts, iframes and cookie banners; no JavaScript execution. - **Rich data extraction** – retrieve links, metadata, Open Graph/Twitter cards, and downloadable resources. - **Safe file downloads** – size limits, filename sanitisation, and path‑traversal protection within a sandboxed cache. - **Built‑in caching** – optional cache directory reduces repeated network calls. - **Prompt‑injection detection** – validates URLs and fetched content for malicious instructions. --- ## 📦 Installation & Quick Start ```bash # Clone the repository (or copy the MCPDataFetchServer.1 folder) git clone https://github.com/undici77/MCPDataFetchServer.git cd MCPDataFetchServer # Make the startup script executable chmod +x run.sh # Run the server, pointing to a sandboxed working directory ./run.sh -d /path/to/working/directory ``` > 📌 **Three‑step overview** > 1️⃣ The script creates a virtual environment and installs dependencies. > 2️⃣ It prepares a cache folder (`.fetch_cache`) inside the project root. > 3️⃣ `main.py` launches the MCP server, listening on *stdin/stdout* for JSON‑RPC requests. --- ## ⚙️ Command‑Line Options | Option | Description | |--------|-------------| | `-d`, `--working-dir` | Path to the **sandboxed working directory** where all file operations are confined (default: `~/.mcp_datafetch`). | | `-c`, `--cache-dir` | Name of the **cache subdirectory** relative to the working directory (default: `cache`). | | `-h`, `--help` | Show help message and exit. | --- ## 🤝 Integration with LM Studio *(or any MCP‑compatible client)* Add an entry to your `mcp.json` configuration so that LM Studio can launch the server automatically. ```json { "mcpServers": { "datafetch": { "command": "/absolute/path/to/MCPDataFetchServer.1/run.sh", "args": [ "-d", "/absolute/path/to/working/directory" ], "env": { "WORKING_DIR": "." } } } } ``` > 📌 **Tip:** Ensure `run.sh` is executable (`chmod +x …`) and that the virtual environment can install the required Python packages on first launch. --- ## 📡 MCP API Overview All communication follows **JSON‑RPC 2.0** over *stdin/stdout*. ### `initialize` Request: ```json { "jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {} } ``` Response contains the protocol version, server capabilities and basic metadata (e.g., name = `mcp-datafetch-server`, version = `2.1.0`). ### `tools/list` Request: ```json { "jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {} } ``` Response: `{ "tools": [ …tool definitions… ] }`. Each definition includes `name`, `description` and an **input schema** (JSON Schema). ### `tools/call` Generic request shape (replace `<tool_name>` and arguments as needed): ```json { "jsonrpc": "2.0", "id": 3, "method": "tools/call", "params": { "name": "<tool_name>", "arguments": { … } } } ``` The server validates the request against the tool’s schema, executes the operation, and returns a `ToolResult` containing one or more **content blocks**. --- ## 🛠️ Available Tools ### `fetch_webpage` - **Securely fetches a web page and returns clean content in the requested format.** | Name | Type | Required | Description | |------|------|----------|-------------| | `url` | string | ✅ (no default) | URL to fetch (**http/https only**). | | `format` | string | ❌ (`markdown`) | Output format – one of `markdown`, `text`, or `html`. | | `include_links` | boolean | ❌ (`true`) | Whether to append an extracted links list. | | `include_images` | boolean | ❌ (`false`) | Whether to list image URLs in the output. | | `remove_banners` | boolean | ❌ (`true`) | Attempt to strip cookie banners & pop‑ups. | **Example** ```json { "jsonrpc": "2.0", "id": 10, "method": "tools/call", "params": { "name": "fetch_webpage", "arguments": { "url": "https://example.com/article", "format": "markdown", "include_links": true, "include_images": false, "remove_banners": true } } } ``` *Note:* The tool sanitises HTML, removes scripts/iframes, and checks for prompt‑injection patterns before returning content. --- ### `extract_links` - **Extracts and categorises all hyperlinks from a page.** | Name | Type | Required | Description | |------|------|----------|-------------| | `url` | string | ✅ (no default) | URL of the page to analyse. | | `filter` | string | ❌ (`all`) | Return only `all`, `internal`, `external`, or `resources`. | **Example** ```json { "jsonrpc": "2.0", "id": 11, "method": "tools/call", "params": { "name": "extract_links", "arguments": { "url": "https://example.com/blog", "filter": "internal" } } } ``` *Note:* Links are classified as **internal** (same domain) or **external**; resource links (images, PDFs…) can be filtered with `resources`. --- ### `download_file` - **Safely downloads a remote file into the sandboxed cache directory.** | Name | Type | Required | Description | |------|------|----------|-------------| | `url` | string | ✅ (no default) | Direct URL to the file. | | `filename` | string | ❌ (auto‑generated) | Desired filename; will be sanitised and forced into the cache directory. | **Example** ```json { "jsonrpc": "2.0", "id": 12, "method": "tools/call", "params": { "name": "download_file", "arguments": { "url": "https://example.com/files/report.pdf", "filename": "report_latest.pdf" } } } ``` *Note:* The server enforces a **100 MB** download limit, validates the URL against blocked domains/extensions, and returns the relative path inside the working directory for cross‑agent access. --- ### `get_page_metadata` - **Extracts structured metadata (title, description, Open Graph, Twitter Cards) from a web page.** | Name | Type | Required | Description | |------|------|----------|-------------| | `url` | string | ✅ (no default) | URL of the page to inspect. | **Example** ```json { "jsonrpc": "2.0", "id": 13, "method": "tools/call", "params": { "name": "get_page_metadata", "arguments": { "url": "https://example.com/product/42" } } } ``` *Note:* The tool returns a formatted text block with title, description, keywords, Open Graph properties and Twitter Card fields. --- ### `check_url` - **Performs a lightweight HEAD request to report status code, headers and size without downloading the body.** | Name | Type | Required | Description | |------|------|----------|-------------| | `url` | string | ✅ (no default) | URL to probe. | **Example** ```json { "jsonrpc": "2.0", "id": 14, "method": "tools/call", "params": { "name": "check_url", "arguments": { "url": "https://example.com/resource.zip" } } } ``` *Note:* The response includes the final URL after redirects, a concise status summary (✅ OK or ⚠️ Error), and selected HTTP headers such as `Content‑Type` and `Content‑Length`. --- ## 🔐 Security Features - **Path‑traversal protection** – all file operations are confined to the sandboxed *working directory*. - **Prompt‑injection detection** in URLs, fetched HTML and generated content. - **Blocked domains & extensions** (localhost, private IP ranges, executable/script files). - **Content‑size limits** – max 50 MB for page fetches, max 100 MB for file downloads. - **HTML sanitisation** – removes `<script>`, `<iframe>`, event handlers and other risky elements before processing. - **Cookie/banner handling** – optional removal of consent banners and pop‑ups during fetch. --- *© 2025 Undici77 – All rights reserved.*