site-crawler-mcp

by coolaigit

Overview Schema Related Servers Score Discussions

Python

Local

Server Quality Checklist

Profile completionA complete profile improves this server's visibility in search results.

Latest release: v0.1.0
Disambiguation5/5
Each tool has a clearly distinct purpose: full site crawling, single page scraping, summarization, link clicking, file downloading, and querying. No overlap or ambiguity.
Naming Consistency5/5
All tools follow a consistent verb_noun pattern in snake_case (e.g., crawl_site, scrape_page, query_crawls), making the tool set predictable and easy to navigate.
Tool Count5/5
With 6 tools, the server is well-scoped. It covers the essential operations for a site crawler without unnecessary bloat or missing functionality.
Completeness4/5
The tool set covers core workflows: crawling, scraping, summarizing, interacting, downloading, and querying. Minor gaps like explicit crawl session management or update/delete operations exist, but the surface is largely complete for typical use cases.
Average 3.3/5 across 6 of 6 tools scored. Lowest: 2.4/5.
See the Tool Scores section below for per-tool breakdowns.
- No community issues in the last 6 months
- 1 commit in the last 12 weeks
- No stable releases found
- No critical vulnerability alerts
- No high-severity vulnerability alerts
- No code scanning findings
- CI is passing
This repository is licensed under MIT License.
This repository includes a README.md file.
No tool usage detected in the last 30 days. Usage tracking helps demonstrate server value.
Tip: use the "Try in Browser" feature on the server page to seed initial usage.
Add a glama.json file to provide metadata about your server.
If you are the author, simply .
If the server belongs to an organization, first add glama.json to the root of your repository:
```
{
  "$schema": "https://glama.ai/mcp/schemas/server.json",
  "maintainers": [
    "your-github-username"
  ]
}
```
Then . Browse examples.
Add related servers to improve discoverability.

How to sync the server with GitHub?

Servers are automatically synced at least once per day, but you can also sync manually at any time to instantly update the server profile.

To manually sync the server, click the "Sync Server" button in the MCP server admin interface.

How is the quality score calculated?

The overall quality score combines two components: Tool Definition Quality (70%) and Server Coherence (30%).

Tool Definition Quality measures how well each tool describes itself to AI agents. Every tool is scored 1–5 across six dimensions: Purpose Clarity (25%), Usage Guidelines (20%), Behavioral Transparency (20%), Parameter Semantics (15%), Conciseness & Structure (10%), and Contextual Completeness (10%). The server-level definition quality score is calculated as 60% mean TDQS + 40% minimum TDQS, so a single poorly described tool pulls the score down.

Server Coherence evaluates how well the tools work together as a set, scoring four dimensions equally: Disambiguation (can agents tell tools apart?), Naming Consistency, Tool Count Appropriateness, and Completeness (are there gaps in the tool surface?).

Tiers are derived from the overall score: A (≥3.5), B (≥3.0), C (≥2.0), D (≥1.0), F (<1.0). B and above is considered passing.

Tool Scores

Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It implicitly suggests a read operation ('query') but does not explicitly state that it is non-destructive, safe to call repeatedly, or what happens with empty results. No side effects, auth needs, or rate limits are disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence in Chinese, which is concise but not well-structured. It front-loads the verb but lacks any structure or additional sentences to elaborate on usage or parameters. Every sentence should earn its place, and here there is only one sentence that could be more informative.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness1/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (6 optional parameters, no output schema, no annotations), the description is severely incomplete. It does not explain return format, how filters combine, or what 'success_only' means. The agent would struggle to use this tool effectively based solely on the description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters1/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, yet the description adds no explanation for any of the 6 parameters (limit, success_only, url_contains, etc.). The description offers zero value beyond the parameter names, leaving the agent to infer their meaning without any context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'query persisted crawl results (SQLite) for reuse by other projects,' identifying the verb (query) and resource (crawl results). It distinguishes from sibling tools like crawl_site (creation) and scrape_page (web scraping). However, it could be more specific about the scope of results and the read-only nature.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., crawl_site for creating crawls, scrape_page for live scraping). There are no prerequisites, exclusions, or context for when this query is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations are absent, so description carries full burden. Only mentions destination path, but omits critical behaviors: overwrite policy, file type restrictions, error handling, or confirmation that it downloads from a given URL.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, front-loaded with verb, no fluff. However, it is underspecified, which slightly reduces effectiveness despite brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 2-parameter tool with no output schema, description fails to explain how to invoke correctly (e.g., do both parameters need to be specified? What is the output or success indication?). Lacks critical context for reliable agent use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters1/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 0% description coverage for parameters; description adds no explanation for 'url' (expected format/sources) or 'filename' (override behavior). Agent has no semantic help beyond parameter names and types.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'download' and resource 'page files' with specific destination path. This distinguishes it from sibling tools like scrape_page or click_link, which don't involve file saving.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives (e.g., when to download vs scrape vs summarize). Lacks context about prerequisites or suitable scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'click and scrape' but omits critical traits: what happens if the link is not found, whether JavaScript-triggered navigation is supported, whether it waits for page load, or if it modifies browser state. The description is too minimal to convey the tool's operational behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact sentence that front-loads the core action. It contains no filler words. However, it is slightly too terse, missing structured elements like parameter hints or usage context. It earns points for efficiency but loses one for not having any additional structure like bullet points or examples.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that there is no output schema, no annotations, and three parameters with zero schema descriptions, the description should compensate by explaining the return value (scraped content format), error cases, and prerequisites. It provides none of these. The tool's complexity (navigating and scraping) demands a richer description than what is given.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must add meaning for all three parameters. It does partially: it connects 'selector' and 'link_text' to 'CSS selector or link text'. However, the 'url' parameter is left undefined—it could be the page containing the link or the target URL—and no format or constraints are given. The description provides only marginal value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (click) and the resource (specified link via CSS selector or link text), and it adds the additional step of scraping the target page content. This differentiates it from sibling tools like scrape_page (which just scrapes without clicking) and crawl_site (which follows multiple links). The purpose is specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus alternatives like scrape_page or crawl_site. The description only states what it does, with no context about prerequisites (e.g., needing to be on a page first), nor exclusions (e.g., not for links that require authentication). The agent must infer usage from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses persistence ('并持久化') and the return types, but it does not mention mutability, side effects, or operational details like rate limits or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the action and outputs. Every word earns its place; no redundancy or fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of an output schema and annotations, the description is adequate but incomplete. It lists return types and persistence but omits details on error handling, parameter behavior, and the exact structure of the returned data.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% with no parameter descriptions. The description only implies the 'url' parameter through the tool's purpose but does not explain the 'respect_robots' parameter at all. This gap leaves the agent without guidance on an important scraping behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states 'scrape a single page' and lists the returned content (markdown, title, publish time, links). It clearly distinguishes from sibling tools like crawl_site (multiple pages) and summarize_page (summarization).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies single-page usage via '单个页面', but it does not explicitly state when to use this tool versus alternatives like crawl_site or click_link. No exclusions or prerequisites are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It discloses BFS traversal, default robots.txt respect, rate limiting (0.5s), JSON output persistence to SQLite, and fallback behavior for date filters. However, it does not mention error handling or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficient, using a few lines to convey purpose and filters. It is front-loaded with the main action and uses bullet-style listing for filters. No redundant text, though structural improvements (e.g., separating behavior from parameters) could enhance readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 11 parameters and no output schema, the description covers core crawl behavior and key filters but lacks details on crawl limits (max_depth, max_pages), output structure beyond 'JSON', and how to access persisted data (likely via query_crawls). The overall completeness is adequate but not thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so description must compensate. It explains date filters (ISO time, fallback), title filters, url_pattern (glob/regex), and toggles for robots.txt and rate limit. However, it omits max_depth, max_pages, and include_external. While start_url is obvious, the missing parameters reduce completeness.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool performs a BFS full-site crawl starting from a URL, traversing all internal links. It distinguishes itself from sibling tools like scrape_page (single page) and query_crawls (querying stored results) by specifying the crawling algorithm and scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for full-site crawling but does not explicitly contrast with siblings or provide when-not-to-use scenarios. While filters and defaults are listed, there is no direct guidance on selecting this tool over scrape_page or download_file for different tasks.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It discloses key behavioral traits: the auto mode tries LLM first and falls back to local on failure, and it specifies the llm_provider format and llm_api_key_env default. This adds useful context beyond the schema, though it stops short of explaining error scenarios or output formats.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-structured: it opens with the main purpose, then details modes and provider parameters in a clear, scannable format. Every sentence adds value with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 4 parameters, no output schema, and no annotations, the description covers the essential functional aspects: purpose, mode behaviors, and provider configuration. It doesn't mention return value or edge cases, but this is not critical for basic invocation and is adequate given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description's explanations are essential. It defines allowed mode values, provides concrete LiteLLM format examples for llm_provider, and states the default for llm_api_key_env. This compensates well for the schema's lack of descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states '概括页面内容' (summarize page content), identifying a specific verb (summarize) and resource (page). This purpose is distinct from sibling tools like scrape_page or crawl_site, making it easy for an agent to know what this tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool instead of alternatives like scrape_page or crawl_site. It does provide mode-specific guidance (auto/llm/local) and parameter details, but lacks explicit when-to-use or when-not-to-use statements relative to sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

GitHub Badge

Glama performs regular codebase and documentation scans to:

Confirm that the MCP server is working as expected.
Confirm that there are no obvious security issues.
Evaluate tool definition quality.

Our badge communicates server capabilities, safety, and installation instructions.

Card Badge

Copy to your README.md:

[![site-crawler-mcp MCP server](https://glama.ai/mcp/servers/coolaigit/site-crawler-mcp/badges/card.svg)](https://glama.ai/mcp/servers/coolaigit/site-crawler-mcp)

Score Badge

Copy to your README.md:

[![site-crawler-mcp MCP server](https://glama.ai/mcp/servers/coolaigit/site-crawler-mcp/badges/score.svg)](https://glama.ai/mcp/servers/coolaigit/site-crawler-mcp)

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/coolaigit/site-crawler-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server