Launch the Chromium browser. Call this once before navigating anywhere.

Close the Chromium browser instance.

Go to a URL and return the page content, cleaned of / tags.

The response is prefixed with the HTTP status line (e.g. "HTTP 200") so a 404/500 that renders a normal-looking page doesn't get mistaken for success. timeout_ms overrides the default 30s navigation timeout for slow sites.

If the site fetches its real content (prices, listings) client-side after the initial paint, pass wait_for_selector (a CSS selector that only appears once that fetch resolves) or wait_for_idle=True (wait for network activity to quiet down) so you don't read the page before it's ready.

If the response looks like a Cloudflare/Akamai/CAPTCHA challenge page rather than real content, a "bot-challenge page" warning is prepended. This can happen even on a 200/403 that looks superficially normal.

Click an element by CSS selector (e.g. a pagination button) and return the resulting page's cleaned content.

On stateful pages (large SPA/ASP.NET markup) re-returning the whole page every click is wasteful. Pass result_selector (a CSS selector, e.g. "table#resultsGrid tbody") to get back only that part of the post-click page.

If the click triggers a client-side fetch (e.g. a filter or "load more" button) rather than a full reload, pass wait_for_selector/wait_for_idle the same way as navigate so the returned content isn't read too early.

Wait for selector to appear and/or network activity to go idle, without navigating or clicking. Use this if a previous get_page_content/get_list read caught the page before client-side content finished loading.

Return the current page's HTML with // tags stripped out.

If selector is given (a CSS selector, e.g. ".recipe-card" or "#content"), only the matching element(s) are returned instead of the whole page, which avoids truncation on large pages. If the content is longer than max_length, it's truncated and the response tells you the offset to pass on the next call to read the following chunk.

If the page looks like a Cloudflare/Akamai/CAPTCHA challenge page rather than real content, a "bot-challenge page" warning is prepended (only on the first chunk, offset=0).

Scan the current page for groups of repeated sibling elements (product cards, list items, search results, ...) and return candidate CSS selectors for each group, with a count and a text sample.

Use this on an unfamiliar site instead of eyeballing raw HTML for class names: pick a promising selector from the results and feed it to get_list (as container_selector) to actually extract the data.

Extract one record per element matching container_selector (e.g. a product card in a grid), pulling fields out of each one. This is the div/card equivalent of get_table for sites that don't use markup.

fields maps an output field name to a CSS selector relative to each container, e.g. {"title": ".product-title a", "price": "[data-a-color='base']"}. Use "" to target the container itself, or append "@attr" to read an attribute instead of text, e.g. "link": "a@href" or "image": "img@src".

Because each field is looked up independently within its own container, results never get misaligned the way separate get_page_content calls + manual zipping do when some cards are missing a field (e.g. a strikethrough "was" price only some products have). Not sure of the right selectors? Call find_repeating_elements first.

List the current page's links as {text, href}, to help decide what to visit next.

Return schema.org structured data (JSON-LD) embedded in the current page, e.g. Recipe/Product/Article objects. Many sites (WordPress recipe plugins, e-commerce) embed clean structured data this way. Check here before falling back to get_page_content.

Parse an HTML table on the current page into rows of {column_header: cell_text}, using the first matching element's first as headers.

selector picks which table (default: the first on the page, e.g. "table#programsGrid"). A cell containing a link also gets a <header>_href key. Use this instead of eyeballing get_page_content for tabular data: it does the row/cell transcription for you, so pass the results straight to add_record instead of hand-copying values.

Add one row of scraped data (column name -> value) to the in-memory dataset.

Return every row collected so far, for review before exporting.

Discard all rows collected so far. Use this when starting over, e.g. you've been exploring one source and want to switch to another without mixing rows.

Write all collected rows to a CSV file (via pandas) and return its path.