URL Agent
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@URL Agentsummarize the website at https://example.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
URL Agent
MCP server for intelligent web crawling using a ReAct agent. Adaptively explores websites and returns structured analysis.
Features
ReAct Loop: Agent reasons about which links to follow
Multiple Providers: OpenAI, Anthropic Claude, or local Ollama
Structured Output: Returns JSON with features, requirements, APIs, edge cases, and implementation notes
MCP Server: Single
summarize_urltool for IDE integration
Related MCP server: crawl4ai-mcp-server
Quick Start
Using Docker (Recommended)
docker build -t url-agent .
docker run --rm -i -e OPENAI_API_KEY=your_key url-agentLocal Installation
# Install dependencies
uv sync
# Run the server
uv run urlagentConfiguration
Set via environment variables or .env file:
MODEL_PROVIDER=openai # openai, anthropic, or ollama
OPENAI_MODEL=gpt-4o-mini # Model name for any provider
OPENAI_API_KEY=your_key # For OpenAI
ANTHROPIC_API_KEY=your_key # For Anthropic
OLLAMA_BASE_URL=http://localhost:11434/v1 # For Ollama (optional)Using Ollama (local, free):
ollama pull llama3
export MODEL_PROVIDER=ollama
export OPENAI_MODEL=llama3
uv run urlagentUsing Anthropic Claude:
export MODEL_PROVIDER=anthropic
export OPENAI_MODEL=claude-3-5-sonnet-20241022
export ANTHROPIC_API_KEY=your_key
uv run urlagentMCP Integration
Option 1: Docker
{
"mcpServers": {
"url-agent": {
"command": "docker",
"args": ["run", "--rm", "-i", "-e", "OPENAI_API_KEY=YOUR_KEY", "url-agent"]
}
}
}Option 2: Local (after uv sync)
{
"mcpServers": {
"url-agent": {
"command": "urlagent",
"env": {
"OPENAI_API_KEY": "YOUR_KEY"
}
}
}
}Usage
Once registered with your MCP client:
Use summarize_url on https://example.comThe agent will:
Fetch the root page
Intelligently explore promising links
Stop when it has enough information or hits limits
Return structured JSON analysis
Tool Parameters
summarize_url(
url: str, # URL to analyze
max_depth: int = 2, # Maximum crawl depth
max_pages: int = 4 # Maximum pages to fetch
)Project Structure
url-agent/
src/url_agent/ # Main package
server.py # MCP server entry point
react_agent.py # ReAct loop logic
web_fetcher.py # Web scraping utilities
providers/ # LLM provider implementations
pyproject.toml
DockerfileLicense
Apache 2.0
This server cannot be deployed
Maintenance
Related MCP Connectors
Scrape, crawl and search the web for AI agents via MCP.
- RampifyOAuthdev.rampify
SEO MCP server: crawl your site, find AI-visibility gaps, and ship the fix from your coding agent.
Turn any public website into an MCP server for agents to search, read and navigate.
Free web search for AI agents. No API key required. Hosted MCP in active development.
Related MCP Servers
- FlicenseBqualityDmaintenanceAn MCP Server for Web scraping and Crawling, built using Crawl4AI224-
- AlicenseNot gradedqualityFmaintenanceA lightweight MCP server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents, enabling single-page scraping and multi-page crawling with adaptive stopping.112MIT
- -licenseNot gradedqualityCmaintenanceSelf-hosted MCP server that provides web scraping and crawling tools, integrating seamlessly with AI frameworks like OpenAI Agents SDK, Cursor, and Claude Code.4-
- AlicenseBqualityDmaintenanceMCP server for the Spider web crawling and scraping API, enabling AI agents to crawl, scrape, search, and extract web data.1347 npm2MIT