Scrape a single web page. Automatically detects whether the page is static (uses httpx + BeautifulSoup) or dynamic/JS-rendered (uses Playwright headless Chromium). Returns the page title, clean content, all internal/external links, and page metadata.

BFS-crawl an entire website starting from the given root URL. Only follows internal (same-domain) links. Respects robots.txt. Avoids duplicate URLs. Limits crawl depth and total page count. Returns every scraped page with title, content, and links.

Clean raw HTML by removing scripts, styles, navigation bars, footers, cookie banners, ads, and other noise. Returns readable plain text keeping only main article content, headings, paragraphs, tables, and lists.

Split clean text into overlapping chunks (~1 000 characters each, 200-character overlap). Each chunk has a unique deterministic ID derived from the URL and position. Useful for preparing text for vector embedding or search indexing.

End-to-end pipeline: crawl every internal page of a website, clean the HTML of each page, and optionally split into chunks. Returns a structured result with every page's title, clean content, links, metadata, and (if requested) text chunks. Handles both static and dynamic pages automatically.