Skip to main content
Glama
deenesjakoruzh
bsmi021

MCP Webscan Server

by bsmi021
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Remote

MCP 웹스캔 서버

대장간 배지

웹 콘텐츠 스캐닝 및 분석을 위한 모델 컨텍스트 프로토콜(MCP) 서버입니다. 이 서버는 웹 페이지에서 정보를 가져오고, 분석하고, 추출하는 도구를 제공합니다.

특징

  • 페이지 페칭 : 웹 페이지를 마크다운으로 변환하여 쉽게 분석할 수 있습니다.

  • 링크 추출 : 웹 페이지에서 링크를 추출하고 분석합니다.

  • 사이트 크롤링 : 콘텐츠를 발견하기 위해 웹사이트를 재귀적으로 크롤링합니다.

  • 링크 확인 : 웹 페이지에서 끊어진 링크 식별

  • 패턴 매칭 : 특정 패턴과 일치하는 URL 찾기

  • 사이트맵 생성 : 웹사이트용 XML 사이트맵 생성

Related MCP server: Open Crawler MCP Server

설치

Smithery를 통해 설치

Smithery를 통해 Claude Desktop용 Webscan을 자동으로 설치하려면:

지엑스피1

수동 설치

# Clone the repository
git clone <repository-url>
cd mcp-server-webscan

# Install dependencies
npm install

# Build the project
npm run build

용법

서버 시작

npm start

서버는 stdio 전송을 기반으로 실행되므로 Claude Desktop과 같은 MCP 클라이언트와 호환됩니다.

사용 가능한 도구

  1. fetch-page

    • 웹 페이지를 가져와서 마크다운으로 변환합니다.

    • 매개변수:

      • url (필수): 가져올 페이지의 URL입니다.

      • selector (선택 사항): 특정 콘텐츠를 타겟으로 하는 CSS 선택기입니다.

  2. extract-links

    • 웹 페이지에서 모든 링크와 텍스트를 추출합니다.

    • 매개변수:

      • url (필수): 분석할 페이지의 URL입니다.

      • baseUrl (선택 사항): 링크를 필터링할 기본 URL입니다.

      • limit (선택 사항, 기본값: 100): 반환할 링크의 최대 개수.

  3. crawl-site

    • 지정된 깊이까지 웹사이트를 재귀적으로 크롤링합니다.

    • 매개변수:

      • url (필수): 크롤링 시작 URL.

      • maxDepth (선택 사항, 기본값: 2): 최대 크롤링 깊이(0-5).

  4. check-links

    • 페이지에서 끊어진 링크를 확인합니다.

    • 매개변수:

      • url (필수): 링크를 확인할 URL입니다.

  5. find-patterns

    • 특정 패턴과 일치하는 URL을 찾습니다.

    • 매개변수:

      • url (필수): 검색할 URL입니다.

      • pattern (필수): URL을 일치시키기 위한 JavaScript 호환 정규식 패턴입니다.

  6. generate-site-map

    • 크롤링을 통해 간단한 XML 사이트맵을 생성합니다.

    • 매개변수:

      • url (필수): 사이트맵 크롤링을 위한 루트 URL입니다.

      • maxDepth (선택 사항, 기본값: 2): URL을 검색하기 위한 최대 크롤링 깊이(0-5).

      • limit (선택 사항, 기본값: 1000): 사이트맵에 포함할 URL의 최대 개수입니다.

Claude Desktop을 사용한 예시 사용

  1. Claude Desktop 설정에서 서버를 구성하세요.

{
  "mcpServers": {
    "webscan": {
      "command": "node",
      "args": ["path/to/mcp-server-webscan/build/index.js"], // Corrected path
      "env": {
        "NODE_ENV": "development",
        "LOG_LEVEL": "info" // Example: Set log level via env var
      }
    }
  }
}

  1. 대화에서 도구를 활용하세요:

Could you fetch the content from https://example.com and convert it to Markdown?

개발

필수 조건

  • 노드.js >= 18

  • 엔피엠

프로젝트 구조(리팩토링 후)

mcp-server-webscan/
├── src/
│   ├── config/
│   │   └── ConfigurationManager.ts
│   ├── services/
│   │   ├── CheckLinksService.ts
│   │   ├── CrawlSiteService.ts
│   │   ├── ExtractLinksService.ts
│   │   ├── FetchPageService.ts
│   │   ├── FindPatternsService.ts
│   │   ├── GenerateSitemapService.ts
│   │   └── index.ts
│   ├── tools/
│   │   ├── checkLinksTool.ts
│   │   ├── checkLinksToolParams.ts
│   │   ├── crawlSiteTool.ts
│   │   ├── crawlSiteToolParams.ts
│   │   ├── extractLinksTool.ts
│   │   ├── extractLinksToolParams.ts
│   │   ├── fetchPageTool.ts
│   │   ├── fetchPageToolParams.ts
│   │   ├── findPatterns.ts
│   │   ├── findPatternsToolParams.ts
│   │   ├── generateSitemapTool.ts
│   │   ├── generateSitemapToolParams.ts
│   │   └── index.ts
│   ├── types/
│   │   ├── checkLinksTypes.ts
│   │   ├── crawlSiteTypes.ts
│   │   ├── extractLinksTypes.ts
│   │   ├── fetchPageTypes.ts
│   │   ├── findPatternsTypes.ts
│   │   ├── generateSitemapTypes.ts
│   │   └── index.ts
│   ├── utils/
│   │   ├── errors.ts
│   │   ├── index.ts
│   │   ├── logger.ts
│   │   ├── markdownConverter.ts
│   │   └── webUtils.ts
│   ├── initialize.ts
│   └── index.ts    # Main server entry point
├── build/          # Compiled JavaScript (Corrected)
├── node_modules/
├── .clinerules
├── .gitignore
├── Dockerfile
├── LICENSE
├── mcp-consistant-servers-guide.md
├── package.json
├── package-lock.json
├── README.md
├── RFC-2025-001-Refactor.md
├── smithery.yaml
└── tsconfig.json

건물

npm run build

개발 모드

npm run dev

오류 처리

서버는 포괄적인 오류 처리를 구현합니다.

  • 잘못된 매개변수

  • 네트워크 오류

  • 콘텐츠 구문 분석 오류

  • URL 검증

모든 오류는 MCP 사양에 따라 올바르게 형식화되었습니다.

기여하다

  1. 저장소를 포크하세요

  2. 기능 브랜치를 생성합니다( git checkout -b feature/amazing-feature )

  3. 변경 사항을 커밋하세요( git commit -m 'Add some amazing feature' )

  4. 브랜치에 푸시( git push origin feature/amazing-feature )

  5. 풀 리퀘스트 열기

특허

MIT 라이센스 - 자세한 내용은 LICENSE 파일을 참조하세요.

Available Tools

6 tools
crawl-siteA

Recursively crawls a website starting from a given URL up to a specified maximum depth. It follows links within the same origin and returns a list of all unique URLs found during the crawl.

ParametersJSON Schema
NameRequiredDescriptionDefault
maxDepthNoThe maximum depth to crawl relative to the starting URL. 0 means only the starting URL is fetched. Max allowed depth is 5 to prevent excessive crawling. Defaults to 2.
urlYesThe starting URL for the crawl. Must be a valid HTTP or HTTPS URL.

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses key behaviors: recursion, same-origin link following, and returning unique URLs. However, it lacks details on rate limits, timeouts, authentication needs, or error handling, which are important for a crawling operation. The description does not contradict any annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the core purpose and followed by behavioral details. Every sentence earns its place by explaining the crawling process and output without redundancy or fluff, making it highly efficient and easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (crawling with recursion and depth limits), no annotations, and no output schema, the description is adequate but incomplete. It covers the basic operation and output format (list of unique URLs), but lacks information on pagination, response structure, or potential side effects, which could hinder an agent's ability to use it effectively in varied contexts.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents parameters. The description adds minimal value beyond the schema by mentioning 'starting URL' and 'maximum depth' in context, but does not provide additional semantics like examples or edge cases. With high schema coverage, the baseline is 3, but the description slightly enhances understanding, warranting a 4.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('recursively crawls'), resource ('a website'), and scope ('starting from a given URL up to a specified maximum depth'). It distinguishes from siblings by specifying it follows links within the same origin and returns unique URLs, unlike tools like 'fetch-page' (single page) or 'extract-links' (link extraction without crawling).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for discovering all URLs on a site via crawling, but does not explicitly state when to use this tool versus alternatives like 'check-links' (likely for link validation) or 'generate-site-map' (for structured sitemaps). No exclusions or prerequisites are mentioned, leaving some ambiguity about optimal use cases.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

fetch-pageA

Fetches the HTML content of a given URL. Optionally, it can select a specific part of the HTML using a CSS selector and convert only that part (or the whole body if no selector is provided) to Markdown format. Returns the resulting Markdown text.

ParametersJSON Schema
NameRequiredDescriptionDefault
selectorNoOptional CSS selector (e.g., '#main-content', '.article-body'). If provided, only the HTML content within the first matching element will be converted to Markdown. If omitted or if the selector doesn't match, the content of the 'body' tag will be used.
urlYesThe fully qualified URL of the web page to fetch. Must be a valid HTTP or HTTPS URL.

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the core behavior (fetching HTML, CSS selection, Markdown conversion) and fallback behavior (uses body tag if selector doesn't match). However, it doesn't mention error handling, rate limits, authentication needs, or performance characteristics that would be important for a web fetching tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is efficiently structured in three sentences: purpose statement, optional functionality, and return value. Every sentence earns its place by adding distinct information without redundancy. It's appropriately sized for a tool with two parameters and clear functionality.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with no annotations and no output schema, the description provides good coverage of what the tool does, how parameters interact, and what it returns. It could be more complete by mentioning potential limitations (e.g., JavaScript-rendered content, timeouts) or authentication requirements, but covers the essential functionality well given the context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 100% description coverage, providing excellent parameter documentation. The description adds value by explaining the relationship between parameters (selector is optional, affects what gets converted) and the overall workflow (fetch → optionally select → convert to Markdown). It provides context beyond individual parameter descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose with specific verbs ('fetches', 'select', 'convert') and resources ('HTML content', 'URL', 'Markdown format'). It distinguishes from siblings like 'check-links' or 'extract-links' by focusing on content retrieval and conversion rather than link analysis or crawling.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage through its functional explanation (fetch HTML, optionally select with CSS, convert to Markdown), but doesn't explicitly state when to use this tool versus alternatives like 'crawl-site' for multiple pages or 'extract-links' for link extraction. No explicit when-not or alternative guidance is provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

find-patternsA

Fetches a web page, extracts all anchor ('a') links, resolves their absolute URLs, and returns a list of links whose URLs match a given JavaScript-compatible regular expression pattern. Includes the URL and anchor text for each match.

ParametersJSON Schema
NameRequiredDescriptionDefault
patternYesA JavaScript-compatible regular expression pattern (without enclosing slashes or flags) used to test against the absolute URLs of the links found on the page. Example: 'product\/\d+' to find product links.
urlYesThe fully qualified URL of the web page to search for link patterns. Must be a valid HTTP or HTTPS URL.

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the tool's behavior well (fetching, extracting, resolving, filtering links) but lacks details on error handling, performance characteristics (e.g., timeouts, rate limits), or authentication needs. It doesn't contradict any annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the core functionality in a single, efficient sentence. Every phrase adds value: it specifies the action, the resource (anchor links), the processing (resolving URLs), the filtering mechanism (regex pattern), and the output format (list with URL and anchor text). No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity (web scraping with regex filtering), no annotations, and no output schema, the description is mostly complete. It explains what the tool does and the output format but could benefit from mentioning potential limitations (e.g., handling of JavaScript-rendered content or pagination). It adequately covers the core functionality without being exhaustive.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, providing detailed documentation for both parameters. The description adds minimal value beyond the schema by mentioning the regex is used to 'test against the absolute URLs' and that it returns 'URL and anchor text for each match,' but doesn't explain parameter interactions or edge cases. Baseline 3 is appropriate given the comprehensive schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('fetches a web page, extracts all anchor links, resolves their absolute URLs, and returns a list of links') and distinguishes it from siblings by specifying the filtering mechanism ('match a given JavaScript-compatible regular expression pattern'). It goes beyond a simple tautology by detailing the multi-step process.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use this tool (finding links matching a regex pattern on a web page) but doesn't explicitly mention when not to use it or name alternatives among the sibling tools. It implies usage for pattern-based link extraction without direct comparison to tools like 'extract-links' or 'fetch-page'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

generate-site-mapA

Crawls a website starting from a given URL up to a specified depth and generates an XML sitemap containing the discovered URLs (up to a specified limit).

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoMaximum number of URLs to include in the generated sitemap XML. Defaults to 1000. Max allowed is 5000.
maxDepthNoThe maximum depth to crawl relative to the starting URL to discover pages for the sitemap. 0 means only the starting URL. Max allowed depth is 5. Defaults to 2.
urlYesThe starting URL for the crawl to generate the sitemap. Must be a valid HTTP or HTTPS URL.

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the crawling process and output generation, but lacks details on performance (e.g., rate limits, timeouts), error handling, or authentication needs. It mentions constraints ('up to a specified limit', 'up to a specified depth'), which adds some context, but overall behavioral traits are minimally covered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, well-structured sentence that efficiently conveys the core functionality without redundancy. It front-loads key actions ('crawls', 'generates') and includes essential constraints, making every word contribute meaningfully to understanding the tool's purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity (crawling with depth/limit constraints) and no annotations or output schema, the description is adequate but incomplete. It covers the basic operation and output type, but lacks details on return values (e.g., XML structure, error responses) and behavioral aspects like performance or prerequisites, which are needed for full contextual understanding.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all parameters (url, maxDepth, limit) with details like defaults, ranges, and formats. The description adds no additional parameter semantics beyond what the schema provides, such as explaining interactions between parameters or edge cases, meeting the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('crawls a website', 'generates an XML sitemap') and the resource ('discovered URLs'), distinguishing it from siblings like 'check-links' (validation) or 'fetch-page' (single page retrieval). It explicitly mentions the scope ('starting from a given URL up to a specified depth') and output format ('XML sitemap'), making the purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like 'crawl-site' or 'extract-links'. It does not mention prerequisites, exclusions, or comparative contexts, leaving the agent to infer usage solely from the tool name and description without explicit direction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 6 tool updatesv1.0.0
    • First observedcheck-links
    • First observedcrawl-site
    • First observedextract-links
    • First observedfetch-page
    • First observedfind-patterns
    • First observedgenerate-site-map
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A3.9/5.0

Scored across 6 tools

Disambiguation3/5

There is significant overlap between tools, particularly check-links, extract-links, and find-patterns, which all involve extracting and analyzing links from web pages, leading to potential confusion. However, crawl-site and generate-site-map have more distinct purposes for recursive crawling and sitemap generation, respectively, and fetch-page stands out for content retrieval and conversion.

Naming Consistency5/5

All tool names follow a consistent kebab-case pattern with clear verb-noun structures (e.g., check-links, crawl-site, extract-links). This uniformity makes the tool set predictable and easy to navigate, with no deviations in naming style.

Tool Count5/5

With 6 tools, the count is well-scoped for a web scanning server, covering key operations like link checking, crawling, content fetching, and sitemap generation. Each tool has a clear role, avoiding bloat while providing comprehensive coverage for the domain.

Completeness4/5

The tool set covers core web scanning tasks such as link extraction, validation, crawling, and sitemap generation, with fetch-page adding content retrieval and conversion. A minor gap exists in lacking tools for security scanning (e.g., vulnerability checks) or performance analysis, but the provided tools support most common workflows effectively.

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

Appeared in Searches