How do I use whed-tools?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@whed-tools Extract WHED profile from https://harvard.edu" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

whed-tools

by Dasistaiden

Overview Schema Source Related Servers Discussions Admin

Python

Local

WHED Tools — Higher Education Intelligence Pipeline

An MCP-native pipeline for collecting structured intelligence on higher education institutions, aligned with the IAU World Higher Education Database (WHED) schema.

Scrape → Extract → Validate → Save — the Host LLM performs extraction directly using MCP tools. No external LLM required.

Built on samirsaci/mcp-webscraper.

Overview

Step	How
Scrape	MCP `crawl_website` or standalone `run_scraper.py` — schema-driven crawl, PDF extraction
Extract	Host LLM reads scraped content, uses `get_extraction_schema` + `get_db_context`
Validate	`validate_profile` — Pydantic schema + WHED DB picklist checks
Save	`save_profile` — write to `output/structured/`

Related MCP server: agent-api-gateway-mcp

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  HOST LLM  (Claude in Cursor / any MCP client)                  │
│                                                                 │
│  crawl_website(url)      →  get_extraction_schema()             │
│  scrape_url(url)             get_db_context(domain)            │
│                                     │                           │
│  Host LLM reads content and fills JSON                          │
│                                     │                           │
│  validate_profile(json)  →  save_profile(domain, json)           │
└─────────────────────────────────────────────────────────────────┘
         │                      │                       │
         ▼                      ▼                       ▼
   output/pages/           schema.py              output/structured/
   output/sites/           db_reference.py

Project Structure

mcp-webscraper/
├── MCP_server/
│   ├── server.py           # MCP entry — 9 tools (scrape + extraction)
│   ├── models/
│   └── utils/
│       └── web_scraper.py  # Scraper (static, Playwright, pdfplumber)
├── schema.py               # SchoolProfile, EXTRACTION_TEMPLATE, FIELD_URL_HINTS
├── db_reference.py         # WHED DB — picklists, reference examples, ground truth
├── run_scraper.py          # Standalone CLI — schema-driven crawl, PDF extraction
├── docs/
│   ├── USAGE_GUIDE.md      # Architecture, flow, outputs, comparison
│   ├── PROJECT_ITERATIONS.md
│   └── MCP_VS_N8N_COMPARISON.md
└── output/
    ├── pages/              # Per-page cache from crawl
    ├── sites/              # Combined site crawl
    ├── structured/         # MCP extraction output
    ├── ground_truth/       # WHED DB exports
    └── stages/             # Human review staging

Prerequisites

Python 3.10+
uv package manager
Cursor (for MCP usage)
MySQL with WHED database (optional — for DB grounding and comparison)

Installation

git clone https://github.com/your-username/mcp-webscraper.git
cd mcp-webscraper
uv sync
uv run playwright install chromium

Copy .env.example to .env and add WHED DB credentials (if available).

Connect MCP to Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "whed-tools": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/path/to/mcp-webscraper",
        "python",
        "MCP_server/server.py"
      ]
    }
  }
}

MCP Tools (whed-tools)

Tool	Description
`scrape_url`	Fetch HTML from a URL
`extract_data`	Extract by CSS selector
`extract_first`	First matching element
`batch_scrape`	Multiple URLs
`crawl_website`	Discover and crawl site (`schema_filter=True` to skip irrelevant pages)
`extract_pdf_text`	Download a PDF and extract its text content
`get_extraction_schema`	WHED field template (REQUIRED only)
`get_db_context`	Picklists + reference example for domain
`validate_profile`	Pydantic + DB picklist validation
`save_profile`	Save profile to `output/structured/`

Example prompt

"Crawl https://www.example.edu and extract a WHED profile. Use get_extraction_schema and get_db_context, then validate and save."

Standalone Scripts

Scrape (schema-driven, with PDFs)

Edit run_scraper.py (TARGET_URL, MODE, etc.), then:

uv run python run_scraper.py

Uses schema.FIELD_URL_HINTS to follow only relevant URLs
Extracts text from PDFs via pdfplumber

Schema & DB Grounding

REQUIRED fields are in EXTRACTION_TEMPLATE; DEFERRED fields are in Pydantic but not prompted.
With WHED DB: picklists, few-shot examples, and post-validation reduce hallucination.
Edit schema.py to add or reactivate fields.

Documentation

Doc	Content
USAGE_GUIDE.md	Architecture, flow, and outputs
PROJECT_ITERATIONS.md	Evolution from Ollama to MCP-native
MCP_VS_N8N_COMPARISON.md	KPI comparison with N8N + Firecrawl

License

MIT — based on samirsaci/mcp-webscraper.

This server cannot be deployed

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Maintenance

ActivityInactive

ResponsivenessNo issues

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

B2B Lead Enrichment A
io.github.Diego-Costa-Tech
Remote MCP server to enrich company profiles with structured B2B data and confidence scores.
Customer Data Platforms CRM
12
universities A
io.github.pipeworx-io
Universities MCP — Hipolabs Universities API (free, no auth)
Open Data Search Education & Learning Tools
322
luminati-io-brightdata-mcp
ai.smithery
One MCP for the Web. Easily search, crawl, navigate, and extract websites without getting blocked.…
Browser Automation Web Scraping Search
2
MESSORA A
dev.messora
Scrape, crawl and search the web for AI agents via MCP.
Web Scraping Browser Automation Search
51

Related MCP Servers

extrapify
Browser Automation Web Scraping
christ0pper
A
license
A
quality
D
maintenance
Structured web context infrastructure for AI agents. Extract reliable schema-guided JSON from websites using Claude-powered parsing, Browserless fallback rendering, and MCP-native workflows.
Updated 4 months ago (2026-05-26 05:29 UTC)
1
1
MIT
agent-api-gateway-mcp
Web Scraping Research & Data
ZachDreamZ
A
license
A
quality
B
maintenance
MCP server that extracts structured JSON from public URLs for AI agents using schemas like product, article, and company.
Updated 2 months ago (2026-07-24 01:59 UTC)
6
45 npm
MIT
Scrapy MCP
Web Scraping Browser Automation
B1unt-alt
A
license
Not graded
quality
C
maintenance
A general-purpose MCP server for crawling and extracting structured data from any website. Supports tools for crawling, single-page extraction, search-and-crawl, and schema extraction.
Updated a month ago (2026-07-29 14:03 UTC)
2
MIT
evident
Web Scraping Browser Automation Search
Kaushalendra-Marcus
A
license
A
quality
C
maintenance
An agent-agnostic web extraction and fetch layer that turns URLs into verified, typed data with confidence scores via MCP, REST, or SDK, orchestrating scraping engines behind a resilience ladder and supporting structured extraction against any schema.
Updated a month ago (2026-08-07 04:22 UTC)
5
2
Apache 2.0