olostep-mcp
Olostep MCP 서버
웹 스크래핑, 콘텐츠 추출 및 검색 기능을 위해 Olostep과 통합되는 모델 컨텍스트 프로토콜(MCP) 서버 구현체입니다. Olostep MCP 서버를 설정하려면 API 키가 필요합니다. Olostep 웹사이트에서 가입하여 API 키를 얻을 수 있습니다.
주요 기능
HTML, Markdown, JSON 또는 일반 텍스트(선택적 파서 사용)로 웹사이트 콘텐츠 스크래핑
구조화된 결과를 제공하는 파서 기반 웹 검색
인용 및 선택적 JSON 형태의 출력을 포함한 AI 답변
최대 10,000개의 URL 일괄 스크래핑
시작 URL로부터의 자율적인 사이트 크롤링
웹사이트 URL 발견 및 매핑 (포함/제외 필터 사용)
지역 타겟팅 콘텐츠를 위한 국가별 요청 라우팅
JavaScript가 많은 웹사이트를 위한 구성 가능한 대기 시간
포괄적인 오류 처리 및 보고
간단한 API 키 구성
Related MCP server: Parallel Task MCP
설치
Olostep MCP 서버에 연결하는 방법은 여러 가지가 있습니다. 워크플로우에 가장 적합한 방법을 선택하세요.
☁️ 원격 엔드포인트 (권장)
가장 간단한 방법으로, 로컬 설치가 필요하지 않습니다. 호스팅된 MCP 서버에 직접 연결하세요:
https://mcp.olostep.com/mcp인증은 Olostep API 키를 사용하여 Authorization 헤더의 Bearer 토큰을 통해 수행됩니다. 구성 예시는 아래의 클라이언트 설정 섹션을 참조하세요.
🐳 Docker Hub
공식 Docker 이미지를 가져와 실행합니다:
docker pull olostep/mcp-server
docker run -i --rm \
-e OLOSTEP_API_KEY="your-api-key" \
olostep/mcp-server🔧 로컬 Docker 빌드
소스에서 직접 이미지를 빌드하려면:
git clone https://github.com/olostep/olostep-mcp-server.git
cd olostep-mcp-server
npm install
npm run build
docker build -t olostep/mcp-server:local .
docker run -i --rm -e OLOSTEP_API_KEY="your-api-key" olostep/mcp-server:local📦 npx
npx를 사용하여 설치 없이 실행합니다:
env OLOSTEP_API_KEY=your-api-key npx -y olostep-mcpWindows (PowerShell)의 경우:
$env:OLOSTEP_API_KEY = "your-api-key"; npx -y olostep-mcpWindows (CMD)의 경우:
set OLOSTEP_API_KEY=your-api-key && npx -y olostep-mcp또는 전역으로 설치:
npm install -g olostep-mcp클라이언트 설정
Cursor
가장 쉬운 방법은 원격 엔드포인트를 사용하는 것입니다. 프로젝트 루트에 .cursor/mcp.json을 생성하거나 편집하세요:
{
"mcpServers": {
"olostep": {
"url": "https://mcp.olostep.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY_HERE"
}
}
}
}대안 (로컬): Cursor 설정 > 기능 > MCP 서버로 이동하여 "+ Add New MCP Server"를 클릭하세요:
이름:
olostep유형:
command명령어:
env OLOSTEP_API_KEY=your-api-key npx -y olostep-mcp
Claude Desktop
claude_desktop_config.json에 다음을 추가하세요:
{
"mcpServers": {
"mcp-server-olostep": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY_HERE"
}
}
}
}대안 (Docker):
{
"mcpServers": {
"olostep": {
"command": "docker",
"args": [
"run", "-i", "--rm",
"-e", "OLOSTEP_API_KEY=YOUR_API_KEY_HERE",
"olostep/mcp-server"
]
}
}
}또는 장치 터미널에서 Smithery CLI를 통해 설치:
npx -y @smithery/cli install @olostep/olostep-mcp-server --client claudeClaude Code
Claude Code MCP 구성에 원격 엔드포인트를 추가하세요:
{
"mcpServers": {
"olostep": {
"url": "https://mcp.olostep.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY_HERE"
}
}
}
}대안 (로컬):
{
"mcpServers": {
"olostep": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY_HERE"
}
}
}
}Windsurf
./codeium/windsurf/model_config.json에 다음을 추가하세요:
{
"mcpServers": {
"olostep": {
"serverUrl": "https://mcp.olostep.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY_HERE"
}
}
}
}대안 (로컬):
{
"mcpServers": {
"mcp-server-olostep": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY_HERE"
}
}
}
}VS Code
.vscode/mcp.json에 다음을 추가하세요:
{
"servers": {
"olostep": {
"type": "http",
"url": "https://mcp.olostep.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY_HERE"
}
}
}
}대안 (로컬):
{
"servers": {
"olostep": {
"type": "stdio",
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY_HERE"
}
}
}
}Metorial
옵션 1: 원클릭 설치 (권장)
Metorial 대시보드 열기
MCP 서버 디렉토리로 이동
"Olostep" 검색
"Install"을 클릭하고 API 키 입력
옵션 2: 수동 구성
Metorial MCP 서버 구성에 다음을 추가하세요:
{
"olostep": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY_HERE"
}
}
}이제 Metorial AI 채팅에서 Olostep 도구를 사용할 수 있습니다.
구성
환경 변수
OLOSTEP_API_KEY: Olostep API 키 (필수)ORBIT_KEY: 요청 라우팅을 위해 Orbit을 사용하는 경우 선택적 키.
사용 가능한 도구
1. 웹사이트 스크래핑 (scrape_website)
단일 URL에서 콘텐츠를 추출합니다. 여러 형식과 JavaScript 렌더링을 지원합니다.
{
"name": "scrape_website",
"arguments": {
"url_to_scrape": "https://example.com",
"output_format": "markdown",
"country": "US",
"wait_before_scraping": 1000,
"parser": "@olostep/amazon-product"
}
}매개변수:
url_to_scrape: 스크래핑할 웹사이트의 URL (필수)output_format: 형식 선택 (html,markdown,json또는text) - 기본값:markdowncountry: 위치별 스크래핑을 위한 선택적 국가 코드 (예: US, GB, CA)wait_before_scraping: 스크래핑 전 대기 시간(밀리초, 0-10000)parser: 특수 추출을 위한 선택적 파서 ID
응답 (예시):
{
"content": [
{
"type": "text",
"text": "{\n \"id\": \"scrp_...\",\n \"url\": \"https://example.com\",\n \"markdown_content\": \"# ...\",\n \"html_content\": null,\n \"json_content\": null,\n \"text_content\": null,\n \"status\": \"succeeded\",\n \"timestamp\": \"2025-11-14T12:34:56Z\",\n \"screenshot_hosted_url\": null,\n \"page_metadata\": { }\n}"
}
]
}2. 웹 검색 (search_web)
주어진 쿼리로 웹을 검색하고 구조화된 결과(비 AI, 파서 기반)를 얻습니다.
{
"name": "search_web",
"arguments": {
"query": "your search query",
"country": "US"
}
}매개변수:
query: 검색 쿼리 (필수)country: 지역화된 결과를 위한 선택적 국가 코드 (기본값:US)
응답:
파서 기반 결과를 나타내는 구조화된 JSON (텍스트 형태)
3. 답변 (AI) (answers)
웹을 검색하고 출처와 인용을 포함하여 원하는 JSON 구조로 AI 기반 답변을 반환합니다.
{
"name": "answers",
"arguments": {
"task": "Who are the top 5 competitors to Acme Inc. in the EU?",
"json": "Return a list of the top 5 competitors with name and homepage URL"
}
}매개변수:
task: 웹 데이터를 사용하여 답변할 질문 또는 작업 (필수)json: 선택적 JSON 스키마/객체 또는 원하는 출력 형태에 대한 짧은 설명
응답 포함 항목:
answer_id,object,task,result(제공된 경우 JSON),sources,created
4. URL 일괄 스크래핑 (batch_scrape_urls)
동시에 최대 10,000개의 URL을 스크래핑합니다. 대규모 데이터 추출에 적합합니다.
{
"name": "batch_scrape_urls",
"arguments": {
"urls_to_scrape": [
{"url": "https://example.com/a", "custom_id": "a"},
{"url": "https://example.com/b", "custom_id": "b"}
],
"output_format": "markdown",
"country": "US",
"wait_before_scraping": 500,
"parser": "@olostep/amazon-product"
}
}응답 포함 항목:
batch_id,status,total_urls,created_at,formats,country,parser,urls
5. 크롤링 생성 (create_crawl)
링크를 따라가며 웹사이트 전체를 자율적으로 발견하고 스크래핑하는 비동기 크롤링을 시작합니다. crawl_id를 반환하며, 크롤링은 백그라운드에서 실행되므로 이 응답에서 콘텐츠를 반환하지 않습니다. 이후 crawl_id와 함께 get_crawl_results를 호출하여 상태를 확인하고 스크래핑된 페이지를 가져와야 합니다 (batch_scrape_urls + get_batch_results와 동일한 2단계 패턴).
{
"name": "create_crawl",
"arguments": {
"start_url": "https://example.com/docs",
"max_pages": 25,
"follow_links": true,
"output_format": "markdown",
"country": "US",
"parser": "@olostep/doc-parser"
}
}응답 포함 항목:
crawl_id,object,status,start_url,max_pages,follow_links,created,formats,country,parser
이 호출을
get_crawl_results와 함께 사용하세요.get_batch_results에crawl_id를 전달하지 마십시오 (크롤링과 배치는 별도의 리소스입니다).
6. 맵 생성 (create_map)
웹사이트의 모든 URL을 가져옵니다. 발견 및 분석을 위해 모든 URL을 추출합니다.
{
"name": "create_map",
"arguments": {
"website_url": "https://example.com",
"search_query": "blog",
"top_n": 200,
"include_url_patterns": ["/blog/**"],
"exclude_url_patterns": ["/admin/**"]
}
}응답 포함 항목:
map_id,object,url,total_urls,urls,search_query,top_n
7. 웹페이지 콘텐츠 가져오기 (get_webpage_content)
JavaScript 렌더링을 지원하며 깔끔한 마크다운 형식으로 웹페이지 콘텐츠를 가져옵니다.
{
"name": "get_webpage_content",
"arguments": {
"url_to_scrape": "https://example.com",
"wait_before_scraping": 1000,
"country": "US"
}
}매개변수:
url_to_scrape: 스크래핑할 웹페이지의 URL (필수)wait_before_scraping: 스크래핑 시작 전 대기 시간(밀리초, 기본값: 0)country: 요청을 로드할 거주 국가 (예: US, CA, GB) (선택 사항)
응답:
{
"content": [
{
"type": "text",
"text": "# Example Website\n\nThis is the markdown content of the webpage..."
}
]
}8. 웹사이트 URL 가져오기 (get_website_urls)
웹사이트에서 관련 URL을 검색하고 쿼리 관련성 순으로 정렬하여 가져옵니다.
{
"name": "get_website_urls",
"arguments": {
"url": "https://example.com",
"search_query": "your search term"
}
}매개변수:
url: 매핑할 웹사이트의 URL (필수)search_query: URL을 정렬할 검색 쿼리 (필수)
응답:
{
"content": [
{
"type": "text",
"text": "Found 42 URLs matching your query:\n\nhttps://example.com/page1\nhttps://example.com/page2\n..."
}
]
}9. 배치 결과 가져오기 (get_batch_results)
batch_id를 사용하여 이전에 제출된 배치 스크래핑 작업의 결과를 가져옵니다.
{
"name": "get_batch_results",
"arguments": {
"batch_id": "batch_abc123"
}
}매개변수:
batch_id:batch_scrape_urls에서 반환된 배치 ID (필수)
응답 포함 항목:
batch_id,status(processing또는completed),total_urls,completed_urls,items(URL별 스크래핑 결과 배열:url,custom_id,markdown_content,html_content,json_content,text_content,status,page_metadata포함)
10. 크롤링 결과 가져오기 (get_crawl_results)
create_crawl로 시작된 비동기 크롤링의 상태와 스크래핑된 페이지를 가져옵니다. 이는 create_crawl의 필수 동반 도구입니다. create_crawl은 작업을 시작하고 crawl_id만 반환하므로, 이 도구를 통해 발견된 페이지와 콘텐츠를 실제로 가져올 수 있습니다.
{
"name": "get_crawl_results",
"arguments": {
"crawl_id": "crawl_abc123",
"formats": ["markdown"],
"items_limit": 20,
"cursor": 0
}
}매개변수:
crawl_id:create_crawl에서 반환된 크롤링 ID (필수)formats: 페이지당 가져올 형식 배열 —markdown,html,json,text(기본값:["markdown"])items_limit: 콘텐츠를 가져올 최대 페이지 수, 1–100 (기본값: 20)cursor: 발견된 페이지 목록에 대한 페이지네이션 커서 (기본값: 0)search_query: 쿼리 관련성에 따라 페이지를 순위 지정/선택하기 위한 선택적 필터
응답 포함 항목:
진행 중:
crawl_id,status(in_progress),pages_completed,pages_total, 그리고 약 10초 후에 다시 호출하라는message.완료 시:
crawl_id,status(completed),pages_returned,next_cursor,has_more, 그리고 각 항목이url,custom_id및 요청된 콘텐츠 필드(markdown_content,html_content,json_content,text_content)를 가지는pages배열.
오류 처리
서버는 강력한 오류 처리를 제공합니다:
API 문제에 대한 상세한 오류 메시지
네트워크 오류 보고
인증 실패 처리
속도 제한 정보
오류 응답 예시:
{
"isError": true,
"content": [
{
"type": "text",
"text": "Olostep API Error: 401 Unauthorized. Details: {\"error\":\"Invalid API key\"}"
}
]
}배포
Docker 이미지
MCP 서버는 Docker 이미지로 제공됩니다:
Docker Hub:
[olostep/mcp-server](https://hub.docker.com/r/olostep/mcp-server)공식 Docker MCP 레지스트리:
mcp/olostep(곧 출시 예정 - 서명 및 SBOM으로 보안 강화)GitHub 컨테이너 레지스트리:
ghcr.io/olostep/olostep-mcp-server
Docker Desktop MCP 툴킷
Olostep MCP 서버는 Docker Desktop의 공식 MCP 툴킷에 추가되고 있으며, 이를 통해 사용자는 다음을 수행할 수 있습니다:
Docker Desktop의 MCP 툴킷 UI에서 발견
원클릭 설치
시각적 구성
MCP 호환 클라이언트(Claude Desktop, Cursor 등)와 함께 사용
상태: Docker MCP 레지스트리에 제출 진행 중
지원 플랫폼
linux/amd64linux/arm64
로컬 빌드
# Clone the repository
git clone https://github.com/olostep/olostep-mcp-server.git
cd olostep-mcp-server
# Build the image
npm install
npm run build
docker build -t olostep/mcp-server .
# Run locally
docker run -i --rm -e OLOSTEP_API_KEY="your-key" olostep/mcp-server라이선스
ISC 라이선스
Available Tools
9 toolsanswersC
Search the web and return AI-powered answers in the JSON structure you want, with sources and citations.
| Name | Required | Description | Default |
|---|---|---|---|
| task | Yes | Question or task to answer using web data. | |
| json | No | Optional JSON schema/object or a short description of the desired output shape. Example object: { "book_title": "", "author": "", "release_date": "" } |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It mentions 'AI-powered answers' and 'sources and citations', which hints at synthesis and attribution, but lacks details on behavioral traits like rate limits, authentication needs, response format beyond JSON, or whether it performs web searches in real-time. For a tool with no annotations, this leaves significant gaps in understanding its operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and front-loaded, stating the core functionality in one sentence. Every phrase ('Search the web', 'return AI-powered answers', 'JSON structure you want, with sources and citations') contributes meaning without redundancy. It could be slightly more structured by separating usage hints, but it's efficient overall.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no annotations, no output schema, and a tool that performs web searches and AI synthesis, the description is incomplete. It doesn't cover critical aspects like response format details, error handling, limitations (e.g., search depth), or how 'sources and citations' are structured in the output. For a complex tool with 2 parameters, this leaves too much undefined for effective agent use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents both parameters ('task' and 'json') well. The description adds minimal value beyond the schema, mentioning 'JSON structure you want' which aligns with the 'json' parameter but doesn't provide additional semantics like examples or constraints. Baseline 3 is appropriate as the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Search the web and return AI-powered answers' with specific outputs ('JSON structure you want, with sources and citations'). It distinguishes from siblings like 'google_search' or 'scrape_website' by emphasizing AI-powered answer generation rather than raw search results or content extraction. However, it doesn't explicitly contrast with 'search_web' which might be similar.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives like 'google_search', 'search_web', or 'get_webpage_content'. The description implies usage for AI-powered answers with structured JSON output, but doesn't specify scenarios where this is preferred over simpler search tools or when not to use it (e.g., for raw data vs. synthesized answers).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
batch_scrape_urlsB
Scrape up to 10k URLs at the same time. Perfect for large-scale data extraction.
| Name | Required | Description | Default |
|---|---|---|---|
| urls_to_scrape | Yes | JSON array of objects with "url" and optional "custom_id". | |
| output_format | No | Choose format for all URLs. Default: "markdown". | markdown |
| country | No | Optional country code for location-specific scraping. | |
| wait_before_scraping | No | Wait time in milliseconds before scraping each URL. | |
| parser | No | Optional parser ID for specialized extraction. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden but offers minimal behavioral disclosure. It mentions scale ('up to 10k URLs') but doesn't cover critical aspects like rate limits, authentication needs, error handling, or what 'scrape' entails (e.g., does it extract text, metadata, full HTML?). The phrase 'at the same time' hints at concurrency but lacks specifics. For a batch operation tool with zero annotation coverage, this is inadequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise (two sentences) and front-loaded with the core functionality. Every word earns its place: first sentence defines the tool, second provides usage context. No wasted words or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity (batch scraping with 5 parameters, no annotations, no output schema), the description is incomplete. It doesn't explain what 'scrape' returns (e.g., content, status codes), how errors are handled for partial failures, or performance considerations. For a tool that could involve significant processing and network usage, more context is needed to use it effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema fully documents all 5 parameters. The description adds no parameter-specific information beyond implying the 'urls_to_scrape' parameter supports batch operations. No additional semantics, constraints, or usage examples are provided. Baseline 3 is appropriate when schema does all the work.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Scrape up to 10k URLs at the same time' specifies the verb (scrape) and resource (URLs) with a quantitative limit. It distinguishes from siblings like 'scrape_website' (singular) and 'get_webpage_content' (single page) by emphasizing batch capability. However, it doesn't explicitly differentiate from 'create_crawl' which might also handle multiple URLs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides implied usage context: 'Perfect for large-scale data extraction' suggests when to use this tool (for bulk operations). However, it lacks explicit guidance on when NOT to use it or alternatives (e.g., use 'scrape_website' for single URLs, 'get_webpage_content' for simpler extraction). No prerequisites or exclusions are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_crawlB
Autonomously discover and scrape entire websites by following links from a start URL.
| Name | Required | Description | Default |
|---|---|---|---|
| start_url | Yes | Starting URL for the crawl. | |
| max_pages | No | Maximum number of pages to crawl. | |
| follow_links | No | Whether to follow links found on pages. | |
| output_format | No | Format for scraped content. Default: "markdown". | markdown |
| country | No | Optional country code for location-specific crawling. | |
| parser | No | Optional parser ID for specialized content extraction. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. While it mentions autonomous discovery and link-following, it lacks critical behavioral details like rate limits, authentication requirements, potential for being blocked by websites, or what happens when max_pages is reached. The description doesn't explain what 'scrape' entails beyond content extraction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that communicates the core functionality without unnecessary words. It's front-loaded with the main action and resource, making it immediately understandable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex crawling tool with 6 parameters and no annotations or output schema, the description is insufficient. It doesn't address important contextual aspects like what the tool returns (scraped content format, error handling), performance characteristics, or limitations of autonomous website discovery.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds no additional parameter semantics beyond what's in the schema. The baseline score of 3 reflects adequate parameter documentation through the schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose with specific verbs ('discover and scrape entire websites') and resource ('websites'), distinguishing it from siblings like 'scrape_website' or 'get_webpage_content' by emphasizing autonomous link-following behavior.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'scrape_website' or 'batch_scrape_urls'. It mentions following links but doesn't specify scenarios where this comprehensive crawling approach is preferred over targeted scraping.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_mapC
Get all URLs on a website. Extract URLs for discovery and site analysis.
| Name | Required | Description | Default |
|---|---|---|---|
| website_url | Yes | Website URL to extract links from. | |
| search_query | No | Optional search query to filter URLs (e.g., "blog"). | |
| top_n | No | Optional limit for number of URLs returned. | |
| include_url_patterns | No | Optional glob patterns to include (e.g., "/blog/**"). | |
| exclude_url_patterns | No | Optional glob patterns to exclude (e.g., "/admin/**"). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. It states the tool extracts URLs but doesn't describe how it works (e.g., crawling depth, handling of dynamic content, rate limits, authentication needs, or error conditions). The phrase 'Get all URLs' might imply comprehensive extraction, but without behavioral details, the agent lacks transparency about what to expect from the operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise and front-loaded: the first sentence 'Get all URLs on a website' directly states the core purpose. The second sentence adds context about use cases without redundancy. Every word earns its place, making it efficient and easy to parse for an AI agent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (5 parameters, no annotations, no output schema), the description is insufficient. It doesn't explain what the tool returns (e.g., list structure, error formats), behavioral constraints, or how it differs from similar siblings. For a URL extraction tool with multiple filtering options, more context is needed to guide effective use, especially without annotations or output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% description coverage, providing clear documentation for all 5 parameters. The description adds minimal value beyond this, only implying URL extraction without detailing parameter interactions or usage examples. Since the schema does the heavy lifting, the baseline score of 3 is appropriate, though the description could have enhanced understanding with practical context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Get all URLs on a website' specifies the verb (get/extract) and resource (URLs from a website). It distinguishes from some siblings like 'get_webpage_content' (which gets content rather than URLs) and 'google_search' (which searches the web rather than extracting from a specific site). However, it doesn't explicitly differentiate from 'get_website_urls' (which might have similar functionality), keeping it from a perfect score.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It mentions 'discovery and site analysis' as general use cases, but doesn't specify when to choose this over siblings like 'get_website_urls', 'scrape_website', or 'create_crawl'. There's no mention of prerequisites, limitations, or comparative advantages, leaving the agent with minimal usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_webpage_contentC
Retrieve content of a webpage in markdown
| Name | Required | Description | Default |
|---|---|---|---|
| url_to_scrape | Yes | The URL of the webpage to scrape. | |
| wait_before_scraping | No | Time to wait in milliseconds before starting the scrape. | |
| country | No | Residential country to load the request from (e.g., US, CA, GB). Optional. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'retrieve' and 'scrape', implying a read-only operation, but doesn't address potential issues like rate limits, authentication needs, error handling, or what happens with dynamic content. This is inadequate for a scraping tool with zero annotation coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence with zero waste. It's front-loaded with the core purpose and includes the output format, making it easy to parse quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a scraping tool with no annotations and no output schema, the description is incomplete. It doesn't explain return values (e.g., structure of markdown content), error conditions, or behavioral traits like handling of JavaScript-rendered pages. This leaves significant gaps for an AI agent to use the tool effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% description coverage, so the schema fully documents all three parameters. The description adds no additional meaning beyond what's in the schema, such as explaining why 'wait_before_scraping' might be needed or how 'country' affects the scrape. Baseline 3 is appropriate when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'retrieve' and resource 'content of a webpage', specifying the output format 'in markdown'. However, it doesn't differentiate from sibling tools like 'scrape_website' or 'batch_scrape_urls', which likely have overlapping functionality.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as 'scrape_website' or 'batch_scrape_urls'. The description lacks context about use cases, prerequisites, or exclusions, leaving the agent to infer usage from the tool name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_website_urlsC
Search and retrieve relevant URLs from a website
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL of the website to map. | |
| search_query | Yes | The search query to sort URLs by. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions 'search and retrieve' but doesn't explain how the search works (e.g., depth, scope, or limitations), what 'relevant' means, potential rate limits, or authentication needs. This leaves significant gaps for a tool that interacts with external websites.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that directly states the tool's function without unnecessary words. It's front-loaded and appropriately sized for its purpose, earning full marks for conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of web interaction tools and the lack of annotations and output schema, the description is incomplete. It doesn't cover behavioral aspects like error handling, return format, or limitations, which are crucial for an agent to use this tool effectively in real-world scenarios.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents both parameters ('url' and 'search_query') adequately. The description implies these parameters are used for searching and retrieving, but doesn't add meaningful semantic context beyond what the schema provides, such as examples or constraints on the search query.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose with specific verbs ('search and retrieve') and resource ('relevant URLs from a website'). However, it doesn't explicitly distinguish this tool from sibling tools like 'scrape_website' or 'search_web', which might have overlapping functionality, preventing a perfect score.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. With multiple sibling tools like 'scrape_website', 'google_search', and 'search_web', there's no indication of context, prerequisites, or exclusions, leaving the agent to guess based on tool names alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
google_searchC
Retrieve structured data from Google search results
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | The search query to perform | |
| country | No | Country code for localized results (e.g., US, GB) | US |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool retrieves structured data, implying a read-only operation, but doesn't cover critical aspects like rate limits, authentication needs, error handling, or what 'structured data' entails (e.g., format, fields). This leaves significant gaps for a tool interacting with an external service.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that front-loads the core purpose without unnecessary words. Every part earns its place by specifying the action, data type, and source. No fluff or redundancy is present.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of a search tool with no annotations and no output schema, the description is incomplete. It doesn't explain the return values (what 'structured data' includes), error conditions, or operational constraints like rate limits. For a tool that likely involves external API calls, more context is needed to use it effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema fully documents both parameters ('query' and 'country'). The description adds no additional meaning beyond what's in the schema, such as query formatting examples or country code implications. Baseline 3 is appropriate when the schema does all the work.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('Retrieve') and resource ('structured data from Google search results'), making the purpose immediately understandable. It distinguishes this from siblings like 'search_web' by specifying Google as the source and structured data as the output. However, it doesn't explicitly contrast with all siblings (e.g., 'answers' might also retrieve information).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'search_web' or 'answers'. It doesn't mention prerequisites, constraints, or typical use cases. The agent must infer usage from the name and description alone without explicit direction.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_websiteB
Extract content from a single URL. Supports multiple formats and JavaScript rendering.
| Name | Required | Description | Default |
|---|---|---|---|
| url_to_scrape | Yes | The URL of the website you want to scrape. | |
| output_format | No | Choose format ("html", "markdown", "json", or "text"). Default: "markdown" | markdown |
| country | No | Optional country code (e.g., US, GB, CA) for location-specific scraping. | |
| wait_before_scraping | No | Wait time in milliseconds before scraping (0-10000). Useful for dynamic content. | |
| parser | No | Optional parser ID for specialized extraction (e.g., "@olostep/amazon-product"). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'JavaScript rendering' (useful context) and 'Supports multiple formats' (output behavior), but lacks critical details: whether scraping respects robots.txt, rate limits, authentication needs, error handling, or what 'extract content' specifically means. For a scraping tool with zero annotation coverage, this leaves significant behavioral gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise (two short sentences) and front-loaded with the core purpose. Every sentence adds value: first states the main action, second adds key capabilities. No wasted words, though it could be slightly more structured for clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 5 parameters, no annotations, and no output schema, the description is moderately complete. It covers the basic purpose and key features (formats, JavaScript), but lacks details on scraping behavior, error cases, or output structure. For a tool with this complexity and no structured safety/behavior annotations, it should provide more context about limitations or typical use cases.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so parameters are well-documented in the schema. The description adds minimal value beyond the schema: it implies format support ('Supports multiple formats') and JavaScript capability, but doesn't explain parameter interactions or provide additional semantic context. Baseline 3 is appropriate when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Extract content from a single URL' specifies the verb and resource. It distinguishes from sibling 'batch_scrape_urls' by emphasizing 'single URL' and from 'get_webpage_content' by mentioning format support and JavaScript rendering. However, it doesn't explicitly contrast with all siblings like 'google_search' or 'search_web'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context through 'single URL' (vs. batch) and 'JavaScript rendering' (for dynamic content), but doesn't provide explicit when-to-use guidance or alternatives. It mentions 'Supports multiple formats' which suggests format flexibility, but no clear exclusions or comparisons to siblings like 'get_webpage_content' are stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_webC
Search the web for a given query and return structured results (non-AI, parser-based).
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | Search query | |
| country | No | Optional country code for localized results (e.g., US, GB). | US |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It mentions 'parser-based' and 'non-AI' which adds some behavioral context, but fails to disclose critical traits like rate limits, authentication needs, result format, pagination, or error handling for a web search tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence with zero wasted words, clearly front-loading the core functionality. Every part earns its place by specifying the action, resource, and method.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of web search and lack of annotations or output schema, the description is incomplete. It omits details on result structure, limitations, error cases, and how it differs from siblings, leaving significant gaps for agent understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents both parameters ('query' and 'country'). The description adds no additional meaning beyond what the schema provides, such as query formatting tips or country code examples, meeting the baseline for high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('Search') and resource ('the web'), specifying it returns structured results via parser-based (non-AI) methods. It distinguishes from AI-based search tools but doesn't explicitly differentiate from sibling tools like 'google_search' or 'answers'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives like 'google_search' or 'answers', nor does it mention prerequisites or exclusions. The description implies a general web search context but lacks explicit usage instructions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
9 tool updates
v1.0.0- Added
answers - Added
batch_scrape_urls - Added
create_crawl - Added
create_map - Changed
get_webpage_content2 fields changed- added
Input schema / $schemaAdded value: +"http://json-schema.org/draft-07/schema#" - added
Input schema / additionalPropertiesAdded value: +false
- Changed
get_website_urls2 fields changed- added
Input schema / $schemaAdded value: +"http://json-schema.org/draft-07/schema#" - added
Input schema / additionalPropertiesAdded value: +false
- Changed
google_search2 fields changed- added
Input schema / $schemaAdded value: +"http://json-schema.org/draft-07/schema#" - added
Input schema / additionalPropertiesAdded value: +false
- Added
scrape_website - Added
search_web
3 tool updates
- First observed
get_webpage_content - First observed
get_website_urls - First observed
google_search
TDQS
Scored across 9 tools
There is significant overlap between tools like 'answers', 'google_search', and 'search_web' for web search functionality, and between 'scrape_website', 'get_webpage_content', and 'batch_scrape_urls' for content extraction. However, the descriptions help clarify some distinctions, such as AI-powered vs. parser-based search or single vs. batch scraping.
Most tools follow a consistent verb_noun pattern (e.g., 'create_crawl', 'get_webpage_content', 'scrape_website'), with only minor deviations like 'answers' (a noun alone) and 'google_search' (noun_verb). The naming is generally readable and predictable across the set.
With 9 tools, the count is reasonable for a web data extraction and search server. It covers multiple aspects of the domain without being overwhelming, though some overlap suggests potential consolidation could refine the scope slightly.
The toolset provides comprehensive coverage for web search, content scraping, and URL discovery, including batch operations and autonomous crawling. Minor gaps might include more advanced filtering or data processing tools, but core workflows are well-supported without dead ends.
Maintenance
Related MCP Connectors
Domain/IP, web capture, Google search/maps, YouTube, Instagram, TikTok, X and Amazon data jobs
Stealth scraping & search. Bypasses Cloudflare, DataDome & LinkedIn via Cyborg HITL approach.
Emails, phones and social links from any list of websites. Pay only for sites with a contact.
1Web search, read any page, verify or find emails, screenshots, keyword data. One key, pay per call.
Related MCP Servers
- FlicenseBqualityDmaintenanceEnables web searching and content scraping through Google Custom Search API. Provides tools to search the internet, extract webpage content, and automatically scrape search results for comprehensive information gathering.3-
- -

HasData MCP Serverofficial
AlicenseAqualityAmaintenanceDirect access to 40+ scraping and search tools. Extract structured data from Google (Search, Maps, Trends), Amazon, Airbnb, Social Media, and any web page directly into your AI agent.6320MIT
Potarix enricherofficial
AlicenseAqualityDmaintenanceFind contact/websites for any company/person912 npmMIT