Scrape multiple URLs efficiently.

Args:
    urls: List of URLs to scrape
    javascript: Set to True if the sites need JavaScript rendering
    save_path: Optional file path to save all results as a JSON array (e.g. "C:/Users/me/Desktop/batch.json").
               If a directory is given, a timestamped filename is generated automatically.

Returns:
    List of scraping results for each URL

Crawl a website to discover its structure and pages.

Args:
    start_url: Starting URL
    max_pages: Maximum pages to crawl (default 50)
    max_depth: Maximum link depth (default 3)
    same_domain_only: Stay on same domain (default True)
    schema_filter: When True, only follow URLs whose path contains keywords
                   relevant to the WHED schema (about, contact, course, program,
                   faculty, etc.). Skips login pages, media, and unrelated content.
                   Recommended for institution websites.
    save_path: Optional file path to save the site map as JSON (e.g. "C:/Users/me/Desktop/sitemap.json").
               If a directory is given, a timestamped filename is generated automatically.

Returns:
    Site map with discovered pages and statistics

Scrape a webpage and extract specific data using CSS selectors.

Args:
    url: The webpage to scrape
    css_selectors: List of CSS selectors (e.g., ["h1", "a.link", "#content"])
    attributes: List of attributes to extract for each selector (e.g., ["text", "href", "text"])
               If not provided, defaults to "text" for all selectors
    javascript: Set to True for JavaScript-rendered sites
    save_path: Optional file path to save the result as JSON (e.g. "C:/Users/me/Desktop/result.json").
               If a directory is given, a timestamped filename is generated automatically.

Returns:
    Dictionary with extracted data for each selector

Example:
    extract_data(
        url="https://example.com",
        css_selectors=["h1", "a"],
        attributes=["text", "href"]
    )

Extract the first matching element from a webpage.
Useful for getting single values like page title, main heading, etc.

Args:
    url: The webpage to scrape
    css_selector: CSS selector for the element (e.g., "h1", "title", "meta[name='description']")
    attribute: What to extract - "text" for content, or attribute name like "href", "content", "src"
    javascript: Set to True for JavaScript-rendered sites
    save_path: Optional file path to save the result as JSON (e.g. "C:/Users/me/Desktop/result.json").
               If a directory is given, a timestamped filename is generated automatically.

Returns:
    Dictionary with the extracted value

Example:
    extract_first(url="https://example.com", css_selector="title", attribute="text")

Download a PDF and extract its text content using pdfplumber.
Useful for reading course handbooks, academic calendars, prospectuses,
and other PDF documents discovered during crawling.

Args:
    url: Direct URL to a PDF file
    max_size_mb: Skip PDFs larger than this (default 5 MB)
    max_chars: Cap extracted text length (default 30000 chars)

Returns:
    Dictionary with extracted text, page count, and character count.
    Returns success=False if the PDF is too large, image-based, or unreadable.

Return WHED database reference data for a given institution domain.

Provides two types of grounding to reduce hallucination:
  1. Picklists — valid enum values (institution types, funding, divisions, etc.)
  2. Reference example — a complete record from the same country

Args:
    domain: Institution website domain (e.g. 'www.ampa.edu.au')

Returns:
    Dictionary with picklists, country code, and a reference example

Return the WHED extraction schema (REQUIRED fields only).

The host LLM should use this template to know which fields to extract
from scraped website content. Each field includes its type and priority.

Typical workflow:
  1. crawl_website / scrape_url  → get site content
  2. get_extraction_schema       → know what to extract
  3. get_db_context(domain)      → get allowed values & reference example
  4. (Host LLM extracts data)
  5. validate_profile(json)      → check the extraction
  6. save_profile(domain, json)  → persist the result

Save a validated institution profile to disk as JSON.

Args:
    domain: Institution domain (e.g. 'www.ampa.edu.au'), used as filename
    profile_json: JSON string of the profile to save
    output_dir: Directory to save into (default: output/structured)

Returns:
    Dictionary with save status and file path

Scrape a webpage and return its HTML content.

Args:
    url: The webpage URL to scrape
    javascript: Set to True for JavaScript-rendered sites (slower but handles dynamic content)
    wait_seconds: How long to wait for JavaScript to load (only used when javascript=True)
    save_path: Optional file path to save the result as JSON (e.g. "C:/Users/me/Desktop/result.json").
               If a directory is given, a timestamped filename is generated automatically.

Returns:
    Dictionary with html content, status code, and load time

Validate an extracted institution profile against the Pydantic schema
and WHED database picklists.

Args:
    profile_json: JSON string of the extracted profile
                  (must match the structure from get_extraction_schema)

Returns:
    Dictionary with validation status, cleaned data, and any warnings