Crawl-MCP

Server Quality Checklist

Profile completionA complete profile improves this server's visibility in search results.

Latest release: v0.3.3
Disambiguation4/5
Most tools have distinct purposes, but there is potential overlap between batch_crawl and multi_url_crawl, and between single/batch versions of search and YouTube extraction. Descriptions help disambiguate, but slight confusion may occur.
Naming Consistency5/5
All tools follow a consistent verb_noun snake_case pattern, with batch_ prefix for batch operations and get_ prefix for metadata retrieval. No mixing of conventions.
Tool Count5/5
19 tools cover a broad domain (web crawling, search, YouTube, file processing, extraction) without being excessive. Each tool has a clear role and the count feels well-scoped.
Completeness5/5
The tool set covers the full lifecycle of web data extraction: crawling (single, batch, deep), search, YouTube operations, file conversion, and multiple extraction methods (entities, structured, intelligent). No obvious gaps for the stated purpose.
Average 3.9/5 across 19 of 19 tools scored. Lowest: 3/5.
See the Tool Scores section below for per-tool breakdowns.
- 7 of 7 community issues answered or closed in the last 6 months
- 2 commits in the last 12 weeks
- Last stable release on June 6, 2026
- No critical vulnerability alerts
- No high-severity vulnerability alerts
- No code scanning findings
- CI is passing
This repository is licensed under MIT License.
This repository includes a README.md file.
No tool usage detected in the last 30 days. Usage tracking helps demonstrate server value.
Tip: use the "Try in Browser" feature on the server page to seed initial usage.
Add a glama.json file to provide metadata about your server.
If you are the author, simply .
If the server belongs to an organization, first add glama.json to the root of your repository:
```
{
  "$schema": "https://glama.ai/mcp/schemas/server.json",
  "maintainers": [
    "your-github-username"
  ]
}
```
Then . Browse examples.
Add related servers to improve discoverability.

How to sync the server with GitHub?

Servers are automatically synced at least once per day, but you can also sync manually at any time to instantly update the server profile.

To manually sync the server, click the "Sync Server" button in the MCP server admin interface.

How is the quality score calculated?

The overall quality score combines two components: Tool Definition Quality (70%) and Server Coherence (30%).

Tool Definition Quality measures how well each tool describes itself to AI agents. Every tool is scored 1–5 across six dimensions: Purpose Clarity (25%), Usage Guidelines (20%), Behavioral Transparency (20%), Parameter Semantics (15%), Conciseness & Structure (10%), and Contextual Completeness (10%). The server-level definition quality score is calculated as 60% mean TDQS + 40% minimum TDQS, so a single poorly described tool pulls the score down.

Server Coherence evaluates how well the tools work together as a set, scoring four dimensions equally: Disambiguation (can agents tell tools apart?), Naming Consistency, Tool Count Appropriateness, and Completeness (are there gaps in the tool surface?).

Tiers are derived from the overall score: A (≥3.5), B (≥3.0), C (≥2.0), D (≥1.0), F (<1.0). B and above is considered passing.

Tool Scores

Behavior1/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations declare readOnlyHint=true, but the description states the tool persists data to disk, which is a write operation. This is a direct contradiction. Additionally, the description does not disclose other behavioral traits such as required permissions, rate limits, or side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, directly stating the core functionality and key usage advice. Every word earns its place with no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's 14 parameters and existence of an output schema, the description is minimal but covers the main point. However, it lacks details about return structure, error conditions, and behavioral nuances beyond the output_path feature.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema has 100% coverage for all 14 parameters, providing clear descriptions. The description adds value by explaining the effect of output_path and the slim response, which enhances understanding beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it processes large content with chunking and BM25 filtering, and mentions persisting to disk. It distinguishes itself from siblings by specifying the advanced processing features, though it doesn't explicitly differentiate from similar tools like 'process_file'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. It only implies usage via the description but lacks explicit context, exclusions, or comparisons with sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=true, which matches the extraction purpose. The description adds useful behavioral context about output_path persisting full data and slimming the response, but does not disclose other behaviors like rate limits or auth needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with two sentences, no wasted words. It front-loads the main purpose, but could be better structured by separating the core function from the output_path detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (12 parameters) and presence of an output schema, the description covers the key behavior of output_path but lacks context on when to use different extraction types or how they differ. This is adequate but not comprehensive.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents parameters thoroughly. The description adds minimal value beyond restating extraction types and output_path behavior, meeting the baseline expectation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states the tool extracts structured data using CSS selectors or LLM, which is specific and clear. It distinguishes from sibling crawling tools but does not explicitly differentiate from other extraction tools like intelligent_extract or extract_entities.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It mentions the output_path behavior but lacks prerequisites or when-not-to-use instructions, which is a gap given the large set of sibling extraction tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior1/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Description describes a side effect (writing comments to disk) that contradicts the readOnlyHint annotation, which implies no mutations. This is a serious inconsistency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences; first states core function, second explains key persistence feature. No superfluous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers main behavior, pagination, and file persistence, but the annotation contradiction undermines completeness. Output schema exists to cover return values.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema has 100% description coverage, so baseline 3. Description adds marginal value beyond schema, clarifying output_path behavior but not essential.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states verb 'Extract' and resource 'YouTube video comments'. Differentiates from sibling tools like extract_youtube_transcript.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage for extracting comments with pagination and optional file persistence, but lacks explicit when-to-use or alternative comparisons.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior1/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotation declares readOnlyHint=true, implying no state changes, but description and schema describe writing .md files and index.json to disk, which modifies the filesystem. This contradiction is severe, and per rubric, score is 1.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences front-load the purpose and state key constraints (max URLs, output_path requirement) without extraneous information. Every word is necessary and no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite the annotation contradiction, the description covers purpose, capacity, and persistence requirement. The schema fills in parameter details, and output schema exists. Missing some behavioral details like error handling for failed items (mentioned only in schema), but overall sufficient for invocation given the rich schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the parameter description in the schema already details all keys (urls, languages, output_path, etc.). The tool description adds minimal value beyond what the schema provides, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the tool extracts transcripts from multiple YouTube videos, distinguishing it from the singular counterpart extract_youtube_transcript. It specifies a max of 3 URLs per call and mentions persistence, which is specific and actionable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Description provides constraints (max 3 URLs, need output_path) but does not explicitly guide when to use this batch tool versus the singular extract_youtube_transcript or other sibling tools. Usage context is implied but exclusions and alternatives are not stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior1/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description states the tool 'converts' files, implying a write/mutation operation, but the annotation declares readOnlyHint=true. This is a direct contradiction and severely misleads the agent about the tool's side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: first states core purpose, second explains key usage pattern. Every word is necessary and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Essentials covered: conversion types and output_path. With full schema coverage and output schema present, the description is sufficiently complete for a complex tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, baseline 3. The description adds value by explaining the output_path behavior (slim response, full unsliced disk persistence) and slicing effect, going beyond schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool converts PDF, Word, Excel, PowerPoint, ZIP to markdown, and mentions the key output_path feature. It is specific and informative.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus siblings like batch_crawl or crawl_url. The description does not mention alternatives or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare readOnlyHint=true. The description adds that persisting to disk writes to a local file and alters response format. This provides behavioral context beyond the annotation, though the side effect of file writing is still safe.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with purpose and key option. No unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Description covers core purpose and one key parameter, but with 10 parameters and sibling tools, more context on summarization and differentiation would be beneficial. Output schema covers return values, so not necessary in description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so parameters are well-documented. The description adds value for output_path by explaining its effect on response, but other parameters like summarize_transcript are not elaborated beyond schema defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves video metadata and transcript availability, and mentions persisting transcript to disk. Verb 'Get' is specific. However, it does not explicitly differentiate from sibling tools like extract_youtube_transcript.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description hints at using output_path but provides no guidance on when to use this tool versus alternatives like extract_youtube_transcript. No context on exclusions or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnlyHint=true, which is consistent with crawling. Description adds that fallback strategies exist for anti-bot sites, but does not detail what those strategies are or any other behavioral traits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with core purpose, no fluff. Each sentence adds distinct value: purpose, pagination, and persistence.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 14 parameters and an output schema (not shown), the description covers the main behavioral aspects (fallback, pagination, persistence). Minor gap: no explanation of the fallback strategies themselves, but overall adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 100% schema coverage, baseline is 3. The description adds meaningful context for pagination (content_offset/content_limit) and persistence (output_path with behavior rules), going beyond the schema comments.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action ('Crawl') and target ('anti-bot sites') with fallback strategies, but does not differentiate from the sibling tool 'crawl_url' which likely has similar purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage via mention of pagination and persistence, but does not provide explicit guidance on when to choose this tool over alternatives like 'crawl_url' or 'search_and_crawl'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior1/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations set readOnlyHint=true, but the description describes writing files to disk (per-page markdown, index.json), which is a side effect. This is a direct contradiction. The description does not acknowledge or clarify the discrepancy, nor does it explain any non-destructive nature.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is succinct with only two sentences, yet covers the core purpose, output mechanism, and key configurations. Every sentence adds essential information; no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite the complexity of combining search and crawl with many options, the description covers input, persistence, response behavior, and error handling. However, it omits details on rate limiting, timeouts, or the output schema structure, which would enhance completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The parameter description inside the input schema is highly detailed, explaining each key's purpose, default behavior (e.g., 'full page bodies are written BEFORE truncation'), and special cases (failed pages). This adds significant value beyond the schema structure, achieving high clarity.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool combines Google search with crawling top results, specifying verbs 'Search' and 'crawl' on the resource 'Google results'. It distinctively differentiates from siblings like 'search_google' and 'crawl_url' by highlighting the combination.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides instructions on when to use (to combine search and crawl with persistence) but lacks explicit guidance on when not to use it or alternatives. No mention of siblings or trade-offs.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description covers key behaviors: output_path creation and validation, per-URL file writing, error handling for failures and existing files, and the effect of include_content_in_response. It does not mention authentication or rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact paragraph that front-loads the core purpose and constraints. It packs significant detail without redundancy, though it could benefit from slight restructuring for readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (8 parameters, output schema present), the description covers most behavioral aspects and parameter interactions. A minor gap is the lack of elaboration on the fallback mechanism mentioned in the title.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds meaningful context for output_path (e.g., dot-containing names allowed, index.json handling) and include_content_in_response (token efficiency). This goes beyond the schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool crawls multiple URLs with fallback and specifies a maximum of 3 URLs per call. It identifies the resource and action, but does not explicitly differentiate from siblings like multi_url_crawl or crawl_url_with_fallback.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides guidance on using output_path for persistence and notes constraints like maximum URLs. However, it lacks explicit context on when to choose this tool over alternatives or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=true, so the tool is non-destructive. The description adds the behavior of output_path enabling slim responses. No further details on rate limits, error handling, or LLM behavior are provided. Does not contradict annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: first states purpose, second explains key workflow parameter. No extraneous information. Efficient and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (12 parameters, output schema present), the description covers the main purpose and the critical output_path behavior. It could be improved by clarifying when to use this over crawling or batch tools, but overall it is sufficient for an agent to understand the tool's role.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so most parameter meanings are already clear. The description adds context for output_path, explaining its effect on response structure. This provides marginal additional value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts specific data from web pages using LLM, distinguishing it from sibling tools like crawl_url which likely return full content. The verb 'extract' and resource 'web pages' are specific.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for extracting data from web pages with optional output persistence, but does not explicitly state when to use this tool versus alternatives like batch_crawl or search_and_crawl. No exclusions or prerequisites are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations provide readOnlyHint=true, consistent with search. Description adds persistence behavior (writing to disk) and slim response, but lacks details on error handling or quota implications. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences pack the core action, limit, and optional persistence, with no redundancy. Front-loaded with purpose and constraint.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given rich schema descriptions and output schema existence, the description adequately conveys the core function. Missing behavioral details like error responses or handling of partial failures, but sufficient for typical use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with detailed parameter descriptions. The main description adds minimal extra value beyond schema, achieving baseline for a fully covered schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states it performs multiple Google searches with a max of 3 queries, clearly distinguishing it from single-query tools like search_google and crawling tools like batch_crawl.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description indicates when to use (multiple searches) but lacks explicit when-not-to-use or sibling comparisons. It implicitly suggests use for batch queries over single search but could be more direct.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description adds significant behavioral detail: how output_path slimdown works, the interaction between content_offset/limit and the on-disk file, and the include_content_in_response flag. This goes beyond basic functionality.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: three short sentences that front-load the purpose and quickly cover the most important usage scenarios. No unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 16 parameters and an output schema, the description covers the most important behaviors (JS, pagination, persistence) but omits details on less common params like css_selector or extract_media. Still, the core agent needs are addressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds value by explaining the interplay between output_path, content_offset, and content_limit, and advises when to use wait_for_js, which is not present in the schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts web page content with JavaScript support, and provides specific use cases like SPAs and pagination. However, it does not differentiate from sibling tools like 'crawl_url_with_fallback' or 'deep_crawl_site', which lowers the score slightly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit usage guidelines for key features (wait_for_js for SPAs, pagination with offsets, persisting content to disk). It lacks guidance on when not to use this tool or alternatives, but provides enough context for common scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false. The description adds no further behavioral details such as whether the genre list is static or dynamic, or any rate limits. It is minimally adequate given the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence with the verb 'Get' front-loaded, containing no unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with no parameters and an output schema, the description sufficiently conveys the tool's purpose. The agent can rely on the output schema for return structure details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has zero parameters, so the description does not need to explain them. The baseline score of 4 is appropriate as the description does not contradict or fail to add value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves available search genres for targeted searching, using a specific verb ('Get') and resource ('search genres'). It is distinct from sibling tools that perform searches themselves.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when needing genre options for a targeted search, but does not explicitly state when not to use it or provide alternatives among siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Description adds context beyond readOnlyHint annotation, detailing the output_path parameter behavior (persisting full output to disk and returning slim response). No contradictions with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with purpose, no wasted words. Efficient and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Tool has output schema covering return values, and description explains key behavioral nuance. For a complex 11-parameter tool, description is brief but adequate given schema richness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline 3 is appropriate. Description enhances understanding of the output_path parameter, explaining its effect on response format, which exceeds baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'Extract entities (emails, phones, etc.) from web pages,' providing a specific verb and resource with examples. This distinguishes it from siblings like extract_structured_data and intelligent_extract.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Description does not provide explicit when-to-use or when-not-to-use guidance. No mention of alternatives or context for selecting this tool over similar siblings like extract_structured_data.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare readOnlyHint:true. Description adds optional disk persistence via output_path and slim response behavior, providing useful context beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with purpose and genres, and a second sentence for persistence. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the existence of an output schema, the description adequately covers genre filtering and optional persistence. It omits details on result slicing but the schema covers that.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with detailed descriptions for all sub-parameters. The tool description reinforces the output_path key but adds no new meaning beyond existing schema documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Search Google with genre filtering', listing specific genres. This distinguishes it from siblings like batch_search_google or search_and_crawl.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use for single Google searches with genre filtering but does not explicitly compare to sibling tools or state when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses that writing files to output_path is a side effect, and that the response is slimmed when output_path is set. This adds value beyond the readOnlyHint annotation, but it does not cover all behavioral traits (e.g., rate limits, auth).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with purpose and key behavior. Every sentence earns its place; no redundancy with schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given tool complexity (12 parameters) and presence of output schema, the description is fairly complete. It explains the main output_path feature and behavior, but does not elaborate on other parameters or when to use depth vs alternatives.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds significant meaning to parameters like output_path, include_content_in_response, and overwrite, explaining their effects on response and file persistence. Schema coverage is 100% but the description enriches understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it crawls multiple pages from a site with configurable depth, using a verb+resource structure. It distinguishes from siblings like crawl_url by mentioning multi-page and output persistence.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for multi-page crawling with depth, but does not explicitly state when to use this tool versus alternatives like crawl_url or batch_crawl. No when-not or exclusions are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds value beyond annotations (readOnlyHint: true) by disclosing fallback crawling behavior, output_path persistence, metadata enrichment, and auto-summarization. No contradictions found. Could mention rate limits or video length constraints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Four sentences, front-loaded with the core purpose. Each sentence adds distinct information: extraction, constraints, fallback, output guidance. Slightly verbose but no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (19 params, output schema exists), the description covers key behaviors: extraction, fallback, output persistence, summarization, metadata enrichment. It omits rate limits and language constraints but the schema covers languages. Output schema reduces need to describe return values.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All 19 parameters have schema descriptions (100% coverage), and the description adds context for output_path (persist full unsliced transcript as markdown) and fallback. The description does not override schema but complements it effectively.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts YouTube transcripts with timestamps, specifying the resource (YouTube) and action (extract). It distinguishes from siblings like get_youtube_video_info (video metadata) and batch_extract_youtube_transcripts (batch variant).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides context: works only with public captioned videos, supports fallback to page crawl, and suggests using output_path for persistence. However, it does not explicitly compare to sibling tools like extract_youtube_comments or state when to avoid using this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=true. The description adds context by specifying it returns format capabilities, which goes beyond the annotation. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the action and examples. No extraneous words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given zero parameters and the presence of an output schema, the description sufficiently informs the agent about the tool's purpose and return value.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are no parameters, so the description does not need to add meaning beyond the input schema. The schema is fully covered.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves supported file formats and their capabilities, with specific examples (PDF, Office, ZIP). It is distinct from sibling tools which focus on crawling, extraction, or processing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The context is clear: this tool is used to check which formats are supported before using other file-processing tools. No explicit exclusions or alternatives are needed given the tool's simplicity.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description details output_path behavior: persisting per-URL markdown + index.json, return shape preservation, output_file key, and handling of failed items. This adds significant value beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no wasted words, front-loaded with action and key constraints. Highly efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given complexity (8 parameters, nested objects, output schema), the description covers core behavior, output path details, and return shape. Output schema exists, so return values are explained elsewhere.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds constraints like max 5 URLs and clarifies return shape, which are not in schema. This provides meaningful extra context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it performs multi-URL crawls with pattern-based config, distinguishing it from single-URL tools like crawl_url. It specifies max 5 URL patterns and mentions output persistence, which is unique.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use for multiple URLs with pattern matching and mentions max 5 patterns. It does not explicitly exclude other tools but provides enough context for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

GitHub Badge

Glama performs regular codebase and documentation scans to:

Confirm that the MCP server is working as expected.
Confirm that there are no obvious security issues.
Evaluate tool definition quality.

Our badge communicates server capabilities, safety, and installation instructions.

Card Badge

Copy to your README.md:

[![crawl-mcp MCP server](https://glama.ai/mcp/servers/walksoda/crawl-mcp/badges/card.svg)](https://glama.ai/mcp/servers/walksoda/crawl-mcp)

Score Badge

Copy to your README.md:

[![crawl-mcp MCP server](https://glama.ai/mcp/servers/walksoda/crawl-mcp/badges/score.svg)](https://glama.ai/mcp/servers/walksoda/crawl-mcp)

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/walksoda/crawl-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server