Trafilatura MCP Server
This repository contains a Model Context Protocol (MCP) server that provides a tool-based interface to the Trafilatura library, a powerful tool for web scraping. It is designed for use with MCP-compatible clients, allowing developers and models to extract main content and metadata from web pages programmatically.
Features
Web Scraping: Utilizes Trafilatura to extract the main text content from a given URL.
Metadata Extraction: Retrieves metadata such as title, author, date, and more.
Configurable Extraction: Options to include or exclude comments and tables from the output.
Simple Tool: Exposes a single, easy-to-use
fetch_and_extract
tool.Asynchronous: Built with an asynchronous architecture for efficient I/O operations.
MCP Standard: Communicates over standard I/O, making it compatible with various MCP clients.
Prerequisites
Before running the server, you need to have Python 3.12+ and uv
installed. You will also need Node.js and npx
to run the MCP Inspector tool for testing.
Installation
Clone the repository.
git clone <repository-url> cd trafilatura_mcpCreate a virtual environment and install the required dependencies:
# Create a virtual environment uv venv # Activate the virtual environment source .venv/bin/activate # Install the dependencies uv sync
Running and Testing the Server
The MCP server is a command-line application that communicates over standard I/O. To use it, a client (like an IDE, a coding agent, or an inspector tool) must launch the server process.
Running for Diagnostics
You can run the script directly from your terminal to see if it starts without errors. This is a quick way to validate your Python environment and the script's basic syntax.
The server will start and wait for input, but you won't be able to interact with it directly from your terminal.
Testing with MCP Inspector
The recommended way to test the server interactively is with MCP Inspector. It provides an interactive shell for sending requests to your server.
Launch the Inspector: You can run the inspector without a permanent installation using
npx
. The inspector will launch your MCP server script for you. From your project directory, run:npx @modelcontextprotocol/inspector uv run -- python3 trafilatura_mcp.pyInteract with the Server: Once the inspector starts, you can connect to the server and use commands like
list_tools
andcall_tool
.Example session:
# List the available tool > list_tools # Call the 'fetch_and_extract' tool with a URL > call_tool fetch_and_extract '''{"url": "https://www.theguardian.com/us-news/2025/sep/28/mass-shootings-north-carolina-texas-new-orleans"}'''
Configuration
This server does not require any external API keys or configuration files.
Usage with an MCP Client (VS Code Example)
You can connect to this server from any standard MCP client. Here’s how to do it in a VS Code environment that supports MCP:
Configure Your MCP Client: In your IDE's MCP client settings (e.g., in
mcp.json
for VS Code), configure a new MCP server that points to the script.Example
{ "servers": { "trafilatura_scraper": { "command": "uv", "args": [ "run", "python3", "trafilatura_mcp.py" ], "cwd": "/path/to/your/project/trafilatura_mcp" } } }Note: Replace
Use the Tool: Once connected, you can use the exposed tool in your chat or agent interactions. For example, to extract content from a news article, you could send the following structured tool call:
{ "tool": "fetch_and_extract", "arguments": { "url": "https://apnews.com/article/elon-musk-x-twitter-hate-speech-antisemitism-0d35c5a69fd5c6183b729f7f3c87064a", "include_comments": false, "include_tables": true } }The server will fetch the URL, extract the main content and metadata, and return it as a JSON object.
remote-capable server
The server can be hosted and run remotely because it primarily relies on remote services or has no dependency on the local environment.
Tools
Enables web scraping and content extraction from URLs using the Trafilatura library. Extracts main text content and metadata (title, author, date) from web pages with configurable options for comments and tables.