# 1. Clone the repository
git clone https://github.com/Sonesh-2202/mcp-internet.git
cd mcp-internet

# 2. Install dependencies
uv sync

{
  "mcpServers": {
    "mcp-internet": {
      "command": "uv",
      "args": ["run", "mcp-internet"],
      "cwd": "/absolute/path/to/mcp-internet"
    }
  }
}

User: "What are the upcoming Bollywood movies in 2026?"

Tool call 1: search_web("upcoming Bollywood movies 2026")
→ Returns 10 links with snippets

Tool call 2: read_webpage("https://www.imdb.com/...")
→ Returns raw text

Tool call 3: read_webpage("https://www.wikipedia.org/...")
→ Returns raw text

AI must manually piece together the answer from raw text.

User: "What are the upcoming Bollywood movies in 2026?"

Tool call 1: smart_search("upcoming Bollywood movies 2026")
→ Returns:
  - Extracted movie list with titles, dates, genres
  - Markdown tables with ratings
  - Data from multiple sources combined
  - All in one response, ready to present

mcp-internet/
├── pyproject.toml              # Dependencies & Config
├── README.md                   # This file
├── MULTI_PLATFORM_GUIDE.md     # OS-specific setup guide
└── src/mcp_internet/
    ├── server.py               # Entry point — 25 MCP tools
    ├── tools/
    │   ├── smart_search.py     # 🌟 Intelligent search + extraction
    │   ├── search.py           # Basic DuckDuckGo search
    │   ├── webpage.py          # Page reader with structured extraction
    │   ├── news.py             # News headlines
    │   ├── youtube.py          # YouTube search & video info
    │   ├── github.py           # GitHub repos & users
    │   ├── reddit.py           # Reddit search
    │   ├── twitter.py          # Twitter/X via Nitter
    │   └── ...                 # weather, time, translate, math, etc.
    └── utils/
        ├── http_client.py      # Async HTTP with UA rotation & rate limiting
        ├── cache.py            # SQLite TTL cache
        └── extractors.py       # Domain-specific data extractors

User Query → smart_search
    ├── 1. Check cache (return if fresh)
    ├── 2. Optimize query & classify (person/movie/news/tech/general)
    ├── 3. Search DuckDuckGo (HTML scraping + ddgs fallback)
    ├── 4. Prioritize results by domain authority
    ├── 5. Scrape top 3 pages in parallel (asyncio.gather)
    ├── 6. Apply domain-specific extractors
    │       ├── Wikipedia → infobox, summary, key facts
    │       ├── IMDb → titles, ratings, cast, genres
    │       ├── LinkedIn → name, role, skills, experience
    │       └── Generic → JSON-LD, OpenGraph, tables
    ├── 7. Aggregate & format as structured markdown
    ├── 8. Cache result
    └── 9. Return comprehensive response

# Run the integration test suite
uv run python test_v3.py