Bintu Scrape Text
bintu_scrape_textFetch one public HTTP or HTTPS URL and return its readable text, sized for an LLM context window. Only HTML/XHTML pages up to 1 MiB are supported (PDF, JSON, images and pages that need JavaScript are not). Article or main content is preferred; navigation, headers, footers, ads and cookie banners are dropped, repeated story blocks are removed (deduplicated_count), links appear as 'label (url)', code is returned in fenced blocks with indentation kept, and tables are pipe-delimited. Modes: scrape (default) returns the text trimmed to max_tokens; chunk splits it into chunks that each fit max_tokens (headings stay with their text, table header rows repeat in continuation chunks, and one indivisible unit such as a very long table row can exceed the budget); summarize returns the leading lines and whole sentences that fit max_tokens (it is not an AI-written summary). Pages are read up to about 200,000 characters (roughly 50,000 tokens): truncated=true with limiting_factor=char_cap means that ceiling was hit, token_budget means max_tokens was. tokens and approx_tokens are approximate estimates (characters divided by 4), not tokenizer counts; tokens_saved_estimate is the source-HTML estimate minus the tokens delivered in this response. low_yield=true means a large page produced very little text, usually because it needs JavaScript or a login; see yield_hint. Results may come from a cache (cache_hit, cache_layer): set cache=false to bypass cache reads and persist=false to skip the persistent write; URLs with credential-like query parameters are never stored. robots.txt is honored. JavaScript rendering (render_js) works only if the operator enables it. Failures return isError=true with a code such as invalid_or_unsafe_url, upstream_access_denied, blocked_by_robots, payload_too_large or unsupported_media_type.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "bintu_scrape_text"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute public HTTP or HTTPS URL to read. Private, loopback, link-local and metadata addresses, non-standard ports and URLs containing credentials are rejected with invalid_or_unsafe_url. | |
| mode | No | scrape: the page text trimmed to max_tokens. chunk: the whole page split into chunks that each fit max_tokens. summarize: the leading lines and whole sentences that fit max_tokens. 'chunked' is accepted as an alias for chunk. | scrape |
| cache | No | Read from the memory and persistent caches (default true). false bypasses cache reads; a fresh result may still be written to the cache. | |
| links | No | How to represent hyperlinks in extracted text. inline preserves readable link URLs; footnotes uses numbered references and a link list; false removes URLs for the lowest-token output. | inline |
| persist | No | Allow a fresh result to be written to the persistent cache (default true). false skips that write; extraction and the short-lived memory cache are unchanged. | |
| sections | No | Optional heading selectors for scrape mode. Accepts either a newline-separated string (for example "Alpha\nBeta") or an array of strings (for example ["Alpha", "Beta"]). Whitespace and case are normalized for exact heading matching. Empty or whitespace-only selectors return invalid_arguments. Each selector must be 200 characters or fewer. When provided, the persistent and memory caches are bypassed for reads; a fresh extraction may still be written when persist is true. | |
| render_js | No | Render the page in a browser before extraction. Available only if the operator enables it; otherwise render_js_disabled is returned and you should retry with false. | |
| max_tokens | No | Estimated-token budget, enforced in every mode: scrape trims to it, chunk makes each chunk fit it, summarize returns the leading text within it. Range 100 to 50000, default 6000. A single indivisible unit in chunk mode (for example one very long table row) can exceed it. Accepted alias: chunk_token_budget. | |
| chunk_token_budget | No | Accepted alias for max_tokens; max_tokens wins on conflict. Range 100 to 50000. | |
| footnote_url_policy | No | For links=footnotes only: full preserves complete URL destinations; truncated replaces over-limit destinations with non-clickable abbreviated text. Default: truncated. | truncated |
| max_footnote_url_length | No | For links=footnotes only: maximum URL length for the visible label and, with the truncated policy, the destination. Positive integer. Default: 100. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | Extraction mode: scrape returns readable text; chunk splits it into budgeted chunks; summarize selects leading text within a budget. The accepted chunked alias normalizes to chunk. | |
| split | Yes | True when content spans multiple chunks (chunk mode only; not truncation). | |
| chunks | Yes | ||
| tokens | Yes | Compatibility alias of approx_tokens. | |
| cache_hit | Yes | True when the response was served from memory or persistent cache. | |
| low_yield | No | True when a large page produced very little text (often JavaScript-rendered or login-gated). | |
| truncated | Yes | True when extracted content exceeded a limit and the returned text was shortened. | |
| source_url | Yes | The requested URL with sensitive-looking query parameters removed, safe for logs and display. | |
| yield_hint | No | Present only when low_yield is true: a short suggestion for what to try next. | |
| cache_layer | Yes | Which cache served the response: none, memory, or persistent. | |
| chunk_count | Yes | Number of chunks in the chunks array (1 for scrape and summarize modes when not split). | |
| html_tokens | Yes | Approximate token count of the fetched source HTML, computed the same way as approx_tokens. | |
| extracted_at | Yes | UTC ISO 8601 timestamp of the extraction. | |
| token_method | Yes | Method used to compute approx_tokens: ceil(characters / 4). | |
| approx_tokens | Yes | Approximate token count of the returned text: ceil(characters / 4). An estimate, not a tokenizer count. | |
| limiting_factor | Yes | Which limit bound the returned text: char_cap (extracted-text character ceiling) or token_budget (max_tokens), or none when no limit shortened the response. | |
| deduplicated_count | No | Repeated story blocks removed from the extracted text. | |
| tokens_saved_estimate | Yes | Estimated tokens saved by extraction: source-HTML approximate tokens minus returned-text approximate tokens, floored at 0. |