Skip to main content
Glama
Dasistaiden

whed-tools

by Dasistaiden
OverviewSchemaRelated ServersScoreDiscussions
Python
Local

WHED Tools — Higher Education Intelligence Pipeline

An MCP-native pipeline for collecting structured intelligence on higher education institutions, aligned with the IAU World Higher Education Database (WHED) schema.

Scrape → Extract → Validate → Save — the Host LLM performs extraction directly using MCP tools. No external LLM required.

Built on samirsaci/mcp-webscraper.

Overview

Step

How

Scrape

MCP crawl_website or standalone run_scraper.py — schema-driven crawl, PDF extraction

Extract

Host LLM reads scraped content, uses get_extraction_schema + get_db_context

Validate

validate_profile — Pydantic schema + WHED DB picklist checks

Save

save_profile — write to output/structured/

Related MCP server: extrapify

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  HOST LLM  (Claude in Cursor / any MCP client)                  │
│                                                                 │
│  crawl_website(url)      →  get_extraction_schema()             │
│  scrape_url(url)             get_db_context(domain)            │
│                                     │                           │
│  Host LLM reads content and fills JSON                          │
│                                     │                           │
│  validate_profile(json)  →  save_profile(domain, json)           │
└─────────────────────────────────────────────────────────────────┘
         │                      │                       │
         ▼                      ▼                       ▼
   output/pages/           schema.py              output/structured/
   output/sites/           db_reference.py

Project Structure

mcp-webscraper/
├── MCP_server/
│   ├── server.py           # MCP entry — 9 tools (scrape + extraction)
│   ├── models/
│   └── utils/
│       └── web_scraper.py  # Scraper (static, Playwright, pdfplumber)
├── schema.py               # SchoolProfile, EXTRACTION_TEMPLATE, FIELD_URL_HINTS
├── db_reference.py         # WHED DB — picklists, reference examples, ground truth
├── run_scraper.py          # Standalone CLI — schema-driven crawl, PDF extraction
├── docs/
│   ├── USAGE_GUIDE.md      # Architecture, flow, outputs, comparison
│   ├── PROJECT_ITERATIONS.md
│   └── MCP_VS_N8N_COMPARISON.md
└── output/
    ├── pages/              # Per-page cache from crawl
    ├── sites/              # Combined site crawl
    ├── structured/         # MCP extraction output
    ├── ground_truth/       # WHED DB exports
    └── stages/             # Human review staging

Prerequisites

  • Python 3.10+

  • uv package manager

  • Cursor (for MCP usage)

  • MySQL with WHED database (optional — for DB grounding and comparison)

Installation

git clone https://github.com/your-username/mcp-webscraper.git
cd mcp-webscraper
uv sync
uv run playwright install chromium

Copy .env.example to .env and add WHED DB credentials (if available).

Connect MCP to Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "whed-tools": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/path/to/mcp-webscraper",
        "python",
        "MCP_server/server.py"
      ]
    }
  }
}

MCP Tools (whed-tools)

Tool

Description

scrape_url

Fetch HTML from a URL

extract_data

Extract by CSS selector

extract_first

First matching element

batch_scrape

Multiple URLs

crawl_website

Discover and crawl site (schema_filter=True to skip irrelevant pages)

extract_pdf_text

Download a PDF and extract its text content

get_extraction_schema

WHED field template (REQUIRED only)

get_db_context

Picklists + reference example for domain

validate_profile

Pydantic + DB picklist validation

save_profile

Save profile to output/structured/

Example prompt

"Crawl https://www.example.edu and extract a WHED profile. Use get_extraction_schema and get_db_context, then validate and save."

Standalone Scripts

Scrape (schema-driven, with PDFs)

Edit run_scraper.py (TARGET_URL, MODE, etc.), then:

uv run python run_scraper.py

  • Uses schema.FIELD_URL_HINTS to follow only relevant URLs

  • Extracts text from PDFs via pdfplumber

Schema & DB Grounding

  • REQUIRED fields are in EXTRACTION_TEMPLATE; DEFERRED fields are in Pydantic but not prompted.

  • With WHED DB: picklists, few-shot examples, and post-validation reduce hallucination.

  • Edit schema.py to add or reactivate fields.

Documentation

Doc

Content

USAGE_GUIDE.md

Architecture, flow, and outputs

PROJECT_ITERATIONS.md

Evolution from Ollama to MCP-native

MCP_VS_N8N_COMPARISON.md

KPI comparison with N8N + Firecrawl

License

MIT — based on samirsaci/mcp-webscraper.

This server cannot be installed

F
license - not found
-
quality - not tested
D
maintenance

How are these scores calculated?

Maintenance

–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Dasistaiden/MCP-web-scrapper-and-LLM-extractor'

If you have feedback or need assistance with the MCP directory API, please join our Discord server