URL Agent
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@URL Agentsummarize the website at https://example.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
URL Agent
MCP server for intelligent web crawling using a ReAct agent. Adaptively explores websites and returns structured analysis.
Features
ReAct Loop: Agent reasons about which links to follow
Multiple Providers: OpenAI, Anthropic Claude, or local Ollama
Structured Output: Returns JSON with features, requirements, APIs, edge cases, and implementation notes
MCP Server: Single
summarize_urltool for IDE integration
Related MCP server: crawl4ai-mcp-server
Quick Start
Using Docker (Recommended)
docker build -t url-agent .
docker run --rm -i -e OPENAI_API_KEY=your_key url-agentLocal Installation
# Install dependencies
uv sync
# Run the server
uv run urlagentConfiguration
Set via environment variables or .env file:
MODEL_PROVIDER=openai # openai, anthropic, or ollama
OPENAI_MODEL=gpt-4o-mini # Model name for any provider
OPENAI_API_KEY=your_key # For OpenAI
ANTHROPIC_API_KEY=your_key # For Anthropic
OLLAMA_BASE_URL=http://localhost:11434/v1 # For Ollama (optional)Using Ollama (local, free):
ollama pull llama3
export MODEL_PROVIDER=ollama
export OPENAI_MODEL=llama3
uv run urlagentUsing Anthropic Claude:
export MODEL_PROVIDER=anthropic
export OPENAI_MODEL=claude-3-5-sonnet-20241022
export ANTHROPIC_API_KEY=your_key
uv run urlagentMCP Integration
Option 1: Docker
{
"mcpServers": {
"url-agent": {
"command": "docker",
"args": ["run", "--rm", "-i", "-e", "OPENAI_API_KEY=YOUR_KEY", "url-agent"]
}
}
}Option 2: Local (after uv sync)
{
"mcpServers": {
"url-agent": {
"command": "urlagent",
"env": {
"OPENAI_API_KEY": "YOUR_KEY"
}
}
}
}Usage
Once registered with your MCP client:
Use summarize_url on https://example.comThe agent will:
Fetch the root page
Intelligently explore promising links
Stop when it has enough information or hits limits
Return structured JSON analysis
Tool Parameters
summarize_url(
url: str, # URL to analyze
max_depth: int = 2, # Maximum crawl depth
max_pages: int = 4 # Maximum pages to fetch
)Project Structure
url-agent/
src/url_agent/ # Main package
server.py # MCP server entry point
react_agent.py # ReAct loop logic
web_fetcher.py # Web scraping utilities
providers/ # LLM provider implementations
pyproject.toml
DockerfileLicense
Apache 2.0
This server cannot be deployed
Maintenance
Related MCP Connectors
Scrape, crawl and search the web for AI agents via MCP.
- RampifyOAuthdev.rampify
SEO MCP server: crawl your site, find AI-visibility gaps, and ship the fix from your coding agent.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Docs: https://docs.keenable.ai/mcp-server Keenable is a free, remote MCP server that gives agents access to the web index. Search the web with ranked results and date/site filters, then fetch any indexed page as clean markdown. Works out of the box with no account or API key.
Related MCP Servers
- FlicenseBqualityDmaintenanceAn MCP Server for Web scraping and Crawling, built using Crawl4AI224-
- AlicenseNot gradedqualityDmaintenanceA lightweight MCP server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents, enabling single-page scraping and multi-page crawling with adaptive stopping.111MIT
- -licenseNot gradedqualityCmaintenanceSelf-hosted MCP server that provides web scraping and crawling tools, integrating seamlessly with AI frameworks like OpenAI Agents SDK, Cursor, and Claude Code.4-
- AlicenseBqualityDmaintenanceMCP server for the Spider web crawling and scraping API, enabling AI agents to crawl, scrape, search, and extract web data.1343 npm2MIT