Skip to main content
Glama
rossja
by rossja

URL Agent

MCP server for intelligent web crawling using a ReAct agent. Adaptively explores websites and returns structured analysis.

Features

  • ReAct Loop: Agent reasons about which links to follow

  • Multiple Providers: OpenAI, Anthropic Claude, or local Ollama

  • Structured Output: Returns JSON with features, requirements, APIs, edge cases, and implementation notes

  • MCP Server: Single summarize_url tool for IDE integration

Related MCP server: MCP WebAnalyzer

Quick Start

docker build -t url-agent .
docker run --rm -i -e OPENAI_API_KEY=your_key url-agent

Local Installation

# Install dependencies
uv sync

# Run the server
uv run urlagent

Configuration

Set via environment variables or .env file:

MODEL_PROVIDER=openai              # openai, anthropic, or ollama
OPENAI_MODEL=gpt-4o-mini          # Model name for any provider
OPENAI_API_KEY=your_key           # For OpenAI
ANTHROPIC_API_KEY=your_key        # For Anthropic
OLLAMA_BASE_URL=http://localhost:11434/v1  # For Ollama (optional)

Using Ollama (local, free):

ollama pull llama3
export MODEL_PROVIDER=ollama
export OPENAI_MODEL=llama3
uv run urlagent

Using Anthropic Claude:

export MODEL_PROVIDER=anthropic
export OPENAI_MODEL=claude-3-5-sonnet-20241022
export ANTHROPIC_API_KEY=your_key
uv run urlagent

MCP Integration

Option 1: Docker

{
  "mcpServers": {
    "url-agent": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "-e", "OPENAI_API_KEY=YOUR_KEY", "url-agent"]
    }
  }
}

Option 2: Local (after uv sync)

{
  "mcpServers": {
    "url-agent": {
      "command": "urlagent",
      "env": {
        "OPENAI_API_KEY": "YOUR_KEY"
      }
    }
  }
}

Usage

Once registered with your MCP client:

Use summarize_url on https://example.com

The agent will:

  1. Fetch the root page

  2. Intelligently explore promising links

  3. Stop when it has enough information or hits limits

  4. Return structured JSON analysis

Tool Parameters

summarize_url(
  url: str,           # URL to analyze
  max_depth: int = 2, # Maximum crawl depth
  max_pages: int = 4  # Maximum pages to fetch
)

Project Structure

url-agent/
  src/url_agent/      # Main package
    server.py         # MCP server entry point
    react_agent.py    # ReAct loop logic
    web_fetcher.py    # Web scraping utilities
    providers/        # LLM provider implementations
  pyproject.toml
  Dockerfile

License

Apache 2.0

A
license - permissive license
-
quality - not tested
D
maintenance

Maintenance

Maintainers
Response time
Release cycle
Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/rossja/url-agent'

If you have feedback or need assistance with the MCP directory API, please join our Discord server