A fast web reader for AI assistants, served over MCP. It reads web pages and PDFs and returns only the passages that answer the question, not whole pages. It runs locally and keeps nothing.
Live benchmark on 32 questions (web pages, JSON APIs, PDFs), 2026-10-31:
Tool | Median latency | Avg tokens returned | Answer present |
web-shooter | 793 ms | 1,081 | 29/32 |
Firecrawl, cache allowed | 1,207 ms | 25,409 | 28/32 |
Firecrawl, fresh | 1,548 ms | 25,025 | 28/32 |
Jina Reader | 2,046 ms | 25,345 | 24/32 |
Faster, and about 23× fewer tokens, with the same accuracy. Caveats and the held-out check are in docs/guide.md.
Related MCP server: free-search-mcp
Tool | Use it when | Returns |
| You have URLs (or local PDF paths) | The best passages, grouped by page; whole pages without a query. Pages built by JavaScript are loaded in headless Chrome automatically |
| You want to choose which page to read next | The page's links, ranked by the query |
| You have a start page and a goal, not the URLs | The most relevant pages, as URLs with one-line snippets |
1. Install (macOS on Apple Silicon, Linux, Windows):
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/heyyviv/web-shooter/releases/latest/download/web-shooter-installer.sh | shpowershell -ExecutionPolicy Bypass -c "irm https://github.com/heyyviv/web-shooter/releases/latest/download/web-shooter-installer.ps1 | iex"Or from source (Rust 1.88+, and the only route on Intel Macs):
cargo install --git https://github.com/heyyviv/web-shooter2. Register it. In Claude Code:
claude mcp add --scope user web-shooter -- web-shooterThen run /mcp or start a new session. For Claude Desktop, Cursor and other clients, add a stdio server with the absolute path to the binary:
{ "mcpServers": { "web-shooter": { "command": "/absolute/path/to/web-shooter" } } }The first query downloads a 30 MB embedding model. Pages built by JavaScript need Chrome installed.
docs/guide.md covers usage, configuration, updating, limits and development. docs/how-it-works.md explains the internals.
MCP Registry name: mcp-name: io.github.heyyviv/web-shooter
Licensed under either of Apache License 2.0 or MIT license, at your option.
Crawl a site from a start page, following the links most relevant to query first. Returns the best pages as URLs with one-line snippets (not full text); fetch them for details. Respects robots.txt.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Page to start from. | |
| query | Yes | What you are looking for. Links are followed in order of relevance to it. | |
| keywords | No | Exact terms or synonyms that should match, to sharpen the query. | |
| max_depth | No | Maximum link hops from the start page (default 3). | |
| max_pages | No | Stop after this many pages (default 30, max 200). | |
| max_results | No | Maximum pages to return (default 10). | |
| other_sites | No | Also follow links to other sites (default false). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the load and does well: it discloses the return shape (URLs plus one-line snippets, explicitly not full text) and notes robots.txt compliance. It is silent on cost/latency and what happens on blocked or failed fetches, which are the remaining behavioral gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, each earning its place: purpose, return shape, and the follow-up action. The most decision-relevant information (what you get back and where to go next) is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter tool with no output schema, the description covers the essential unknowns: result granularity and the sibling handoff. Minor gaps remain around crawl duration, failure modes, and the distinction from `links`, but nothing that would cause a misinvocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and every parameter including defaults and the max_pages cap is documented in the schema. The description only reinforces `query` as the relevance signal; it adds no additional meaning about keywords, depth, or other_sites beyond what the schema states.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Crawl a site from a start page') plus the ordering principle (links most relevant to query first), and explicitly separates itself from the `fetch` sibling by saying results are snippets, not full text.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear two-step workflow: crawl to discover candidate pages, then `fetch` them for details. It does not address when to prefer the `links` sibling or when a plain search would be better, so it stops short of full when/when-not coverage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Read one or more web pages or PDFs (URLs, or local PDF paths) as clean markdown. With query, returns only the most relevant passages within max_tokens; PDF passages are labelled with their page number. Pages built by JavaScript (job boards, single-page apps) are loaded in headless Chrome automatically, which is slower.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | Pages or PDFs to read (URLs, or absolute paths of local PDF files). Pass several in one call rather than calling once per URL. | |
| query | No | What you are looking for. When set, only the most relevant passages are returned. | |
| keywords | No | Exact terms or synonyms that should match, to sharpen the query. | |
| max_tokens | No | Maximum tokens to return across all pages (default 2000). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden and does well: it discloses output format, query-based passage filtering, PDF page labels, and that JavaScript-heavy pages load via headless Chrome with slower performance. It omits error behavior, auth requirements, and rate limits, but provides substantial behavioral context beyond the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three tightly written sentences that front-load the core action and then add useful behavioral details. Every sentence contributes context about output, filtering, or performance, with no wasted wording.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with no annotations and no output schema, the description is mostly complete: it covers output format, query filtering, PDF handling, and JavaScript rendering behavior. It could go further on failure modes or multi-URL aggregation, but it is adequate for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds meaningful semantics for `query` and its interaction with `max_tokens`, clarifying that only relevant passages are returned within the token budget, and notes PDF page labeling.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Read') and resource ('web pages or PDFs') and clearly names the output format ('clean markdown'). It does not explicitly differentiate itself from sibling tools like 'crawl' or 'links', so sibling distinction is left implicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by the description: use this to read specific pages or PDFs, optionally with `query` to extract relevant passages. However, there is no explicit guidance on when to use this instead of alternatives like `crawl` or `links`, which is a clear gap.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List a page's links. With query, ranks them by relevance so you can pick which to fetch.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Page whose links to list. | |
| query | No | Rank links by relevance to this. Without it, links are listed in page order. | |
| keywords | No | Exact terms or synonyms that should match, to sharpen the query. | |
| max_links | No | Maximum links to return (default 30). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations supplied, the description carries the full disclosure burden. It does disclose the ordering behavior (page order vs. relevance ranking), which is genuinely useful, but says nothing about whether the page must be fetched first, how links are returned, or truncation beyond the schema's default of 30.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, zero filler, with the core action front-loaded and the query behavior immediately after. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple four-parameter read tool with a fully described schema this is largely sufficient, but with no output schema the description never indicates what a returned link looks like (URL only? anchor text?) or how many results to expect, which an agent needs to consume the output.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all four parameters are already documented, including query's ranking effect and max_links' default. The description restates the query behavior without adding format or syntax detail, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a specific verb and resource ('List a page's links') and implicitly positions itself as the discovery step before 'fetch'. It does not explicitly distinguish itself from the sibling 'crawl', so it stops short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It hints at the workflow — use query to rank links 'so you can pick which to fetch' — which implies this tool precedes fetch. There is no explicit statement of when to prefer it over crawl, nor any exclusion or prerequisite, leaving usage mostly inferred.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0crawlfetchlinksScored across 3 tools
crawl and links both surface URLs from a page's links, which could initially confuse an agent, but descriptions clearly separate site-wide discovery (crawl) from single-page link listing/ranking (links), and fetch is unambiguously for reading content. Boundaries are mostly clear.
Two tools use verb-style names (crawl, fetch) while links is a noun, a minor deviation from a single convention. All are single-word, lowercase, and immediately readable, so the inconsistency is small.
Three tools is exactly right for a web-reading server: discover (crawl), list/rank links (links), and read content (fetch). Each earns its place with no redundancy.
The core lifecycle of find-then-read is covered, including PDFs, JS-rendered pages, robots.txt compliance, and query-based relevance filtering. Minor gaps exist (no explicit sitemap or multi-URL batch crawl options), but agents can work around them by chaining crawl and fetch.
Jina AI Reader/Search MCP — turn any URL into clean LLM-ready markdown, plus web search.
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
Web search API and MCP server for developers building AI agents.