code2mcp

# Bright Data SERP API Reference **Version**: 1.0 (MCP Integration) **Last Updated**: 2025-11-17 **Status**: ✅ ACTIVE **Package**: `@brightdata/mcp` **Official Docs**: https://brightdata.com/products/serp-api --- ## Overview Bright Data's SERP (Search Engine Results Page) API provides real-time search results from multiple search engines (Google, Bing, Yandex) through MCP integration. The system uses Bright Data's global proxy network to deliver accurate, unblocked search results in under 1 second. ## Authentication **Method**: Bearer Token Authentication **Server Name**: `bright-data` **Environment Variable**: `BRIGHT_DATA_API_TOKEN` **Default Token**: Configured in `mcp-servers.config.js` **Configuration**: ```javascript // mcp-servers.config.js brightData: { enabled: true, command: "npx", args: ["@brightdata/mcp"], env: { API_TOKEN: process.env.BRIGHT_DATA_API_TOKEN || "default_token" } } ``` --- ## Available Tools ### 1. searchEngine **Description**: Scrape search results from Google, Bing or Yandex. Returns SERP results in JSON (Google) or Markdown (Bing/Yandex), ideal for gathering current information, news, and detailed search results. **MCP Tool Name**: `bright-data__search_engine` **Input Interface**: ```typescript { query: string; // Search query (required) engine?: 'google' | 'bing' | 'yandex'; // Search engine (default: google) cursor?: string; // Pagination cursor for next page } ``` **Usage Examples**: ```typescript // Basic Google search const results = await searchEngine({ query: "best restaurants in New York" }); // Search on specific engine const bingResults = await searchEngine({ query: "machine learning trends 2025", engine: "bing" }); // Paginated search const page2 = await searchEngine({ query: "AI research papers", cursor: "CBQqBRiBBRAA" // Cursor from previous results }); console.log('Search results:', results); ``` **Output**: SERP results with URL, title, description for each result --- ### 2. searchEngineBatch **Description**: Run multiple search queries simultaneously (up to 10 queries). Returns JSON for Google, Markdown for Bing/Yandex. **MCP Tool Name**: `bright-data__search_engine_batch` **Input Interface**: ```typescript { queries: Array<{ query: string; engine?: 'google' | 'bing' | 'yandex'; cursor?: string; }>; } ``` **Usage Examples**: ```typescript // Batch search multiple queries const batchResults = await searchEngineBatch({ queries: [ { query: "best restaurants in NYC", engine: "google" }, { query: "top AI conferences 2025", engine: "bing" }, { query: "TypeScript best practices", engine: "google" } ] }); console.log('Batch search complete:', batchResults); // Competitive analysis const competitorAnalysis = await searchEngineBatch({ queries: [ { query: "our product vs competitors", engine: "google" }, { query: "alternative to our product", engine: "google" }, { query: "reviews for our product", engine: "google" } ] }); console.log('Competitive intelligence:', competitorAnalysis); ``` **Output**: Array of results for each query **Best Practices**: - Max 10 queries per batch - Use batch for efficiency when searching multiple related terms - Different engines return different formats (Google: JSON, Bing/Yandex: Markdown) --- ### 3. scrapeAsMarkdown **Description**: Scrape a single webpage URL with advanced options for content extraction. Returns results in Markdown format. This tool can unlock any webpage even if it uses bot detection or CAPTCHA. **MCP Tool Name**: `bright-data__scrape_as_markdown` **Input Interface**: ```typescript { url: string; // Full URL to scrape (required) } ``` **Usage Examples**: ```typescript // Scrape a single page const content = await scrapeAsMarkdown({ url: "https://example.com/article" }); console.log('Page content:', content); // After getting SERP results, extract full article const searchResults = await searchEngine({ query: "AI breakthroughs 2025" }); // Extract the first result if (searchResults.content?.[0]?.links?.[0]) { const article = await scrapeAsMarkdown({ url: searchResults.content[0].links[0].href }); console.log('Full article:', article); } ``` **Output**: Full page content in Markdown format --- ### 4. scrapeBatch **Description**: Scrape multiple webpage URLs with advanced options for content extraction (max 10 URLs). Returns results in Markdown format. Can unlock webpages with bot detection or CAPTCHA. **MCP Tool Name**: `bright-data__scrape_batch` **Input Interface**: ```typescript { urls: string[]; // Array of URLs to scrape (max 10) } ``` **Usage Examples**: ```typescript // Batch scrape multiple URLs const contents = await scrapeBatch({ urls: [ "https://site1.com", "https://site2.com", "https://site3.com" ] }); console.log('Scraped content:', contents); // Research workflow: SERP + extract multiple results const search = await searchEngine({ query: "React vs Vue comparison" }); // Extract top 3 results const urls = search.content?.[0]?.links?.slice(0, 3).map(l => l.href) || []; const articles = await scrapeBatch({ urls }); console.log('Research complete:', articles); ``` **Output**: Array of page contents in Markdown format --- ## Complete Workflow Examples ### Example 1: Market Research ```typescript // 1. Search for competitors const competitors = await searchEngine({ query: "best project management software 2025" }); // 2. Extract details from top 5 results const urls = competitors.content?.[0]?.links?.slice(0, 5).map(l => l.href) || []; const details = await scrapeBatch({ urls }); console.log('Competitor analysis:', details); ``` ### Example 2: Competitive Intelligence ```typescript // Batch search for brand mentions const mentions = await searchEngineBatch({ queries: [ { query: "ourbrand reviews", engine: "google" }, { query: "ourbrand vs competitors", engine: "google" }, { query: "problems with ourbrand", engine: "google" }, { query: "ourbrand alternatives", engine: "google" } ] }); // Extract all linked articles const allUrls = mentions.content?.flatMap(c => c.links?.slice(0, 3).map(l => l.href) || []) || []; const articles = await scrapeBatch({ urls: allUrls }); console.log('Brand intelligence:', articles); ``` ### Example 3: Content Aggregation ```typescript // Search for latest news on a topic const news = await searchEngine({ query: "AI breakthrough news today", engine: "bing" }); // Scrape each news article const urls = news.content?.[0]?.links?.map(l => l.href) || []; const articles = await scrapeBatch({ urls }); // Store in context for later use await __mcp_call('context7__store', { key: `news-${Date.now()}`, value: articles }); console.log('News aggregated:', articles.length, 'articles'); ``` --- ## Advanced Use Cases ### A/B Testing SERP Results ```typescript // Get same query from different engines const comparison = await searchEngineBatch({ queries: [ { query: "best laptops 2025", engine: "google" }, { query: "best laptops 2025", engine: "bing" }, { query: "best laptops 2025", engine: "yandex" } ] }); console.log('Cross-engine comparison:', comparison); ``` ### Automated Content Discovery ```typescript // Discover content on multiple topics const topics = ["AI", "blockchain", "cloud computing", "cybersecurity"]; const discoveries = await searchEngineBatch({ queries: topics.map(t => ({ query: `${t} trends 2025` })) }); console.log('Content discovery:', discoveries); ``` ### SERP Monitoring ```typescript // Track rankings over time (store results) const currentRankings = await searchEngine({ query: "yourbrand name" }); await __mcp_call('context7__store', { key: `rankings-${new Date().toISOString().split('T')[0]}`, value: currentRankings }); console.log('Rankings saved for monitoring'); ``` --- ## Pricing Information **Pricing Tiers** (as of 2025-11-17): - **Pay-as-you-go**: $1.50/1K results - **380K results**: $1.30/1K ($499/month) - **900K results**: $1.10/1K ($999/month) - **2M results**: $1.00/1K ($1,999/month) **Features Included**: - Under 1-second response times - 195+ country support - Real user results simulation - No charges for failed requests - 99.9% uptime SLA - Geo-location targeting included free - No limit on concurrent requests --- ## Search Engine Specifics ### Google - **Format**: JSON - **Features**: Rich snippets, knowledge panels, local results - **Best for**: Most comprehensive results, detailed data structure ### Bing - **Format**: Markdown - **Features**: Rich results, news integration - **Best for**: Alternative perspective, often faster ### Yandex - **Format**: Markdown - **Features**: Regional results, strong in Russian/Eastern European markets - **Best for**: International markets, Russian content --- ## Error Handling ### Common Issues **1. Rate Limiting**: ```typescript try { const results = await searchEngine({ query: "test" }); } catch (error) { console.log('Rate limited, waiting...'); await new Promise(resolve => setTimeout(resolve, 1000)); const results = await searchEngine({ query: "test" }); } ``` **2. No Results Found**: ```typescript const results = await searchEngine({ query: "obscure query" }); if (!results.content || results.content.length === 0) { console.log('No results, try different keywords'); } ``` **3. Invalid URL**: ```typescript try { const content = await scrapeAsMarkdown({ url: "invalid-url" }); } catch (error) { console.log('Invalid URL or page inaccessible'); } ``` --- ## Best Practices ### 1. Efficient Batch Processing - Use `searchEngineBatch` for multiple queries (max 10) - Use `scrapeBatch` for multiple URLs (max 10) - Avoid individual calls when processing many items ### 2. Error Resilience - Always wrap in try-catch for production use - Implement retry logic for transient failures - Check for empty results before processing ### 3. Query Optimization - Be specific in search queries - Use quotes for exact phrases: "machine learning" - Combine terms for better results: "AI trends 2025" ### 4. Data Storage - Store important results in Context7 for persistence - Use timestamps in keys for time-series data - Keep URLs for re-scraping if needed ### 5. Rate Management - Don't exceed reasonable query frequency - Use pagination (cursor) for large result sets - Batch related queries together --- ## Integration with Other MCP Servers ### Combine with Playwright ```typescript // Get SERP results const serp = await searchEngine({ query: "best restaurants NYC" }); // Visit top result in real browser const topUrl = serp.content?.[0]?.links?.[0]?.href; if (topUrl) { await __mcp_call('playwright__navigate', { url: topUrl }); const screenshot = await __mcp_call('playwright__screenshot', {}); } ``` ### Combine with Firecrawl ```typescript // Get results via Bright Data const serp = await searchEngine({ query: "AI research papers" }); // Use Firecrawl for deeper crawling const urls = serp.content?.[0]?.links?.slice(0, 3).map(l => l.href) || []; const crawled = await __mcp_call('firecrawl-mcp__crawl', { url: urls[0], maxDepth: 2 }); ``` ### Combine with Chrome DevTools ```typescript // Get search results const serp = await searchEngine({ query: "web scraping techniques" }); // Analyze page performance const url = serp.content?.[0]?.links?.[0]?.href; if (url) { await __mcp_call('chrome-devtools__navigate', { url }); const metrics = await __mcp_call('chrome-devtools__get_metrics', {}); } ``` --- ## Related Files - **Configuration**: `mcp-servers.config.js` - Bright Data server settings - **Generated APIs**: `generated/servers/bright-data/` - TypeScript interfaces - **Architecture**: `DOCS/Architecture/SYSTEM_MAP.md` - System overview - **Deployment**: `DEPLOYMENT.md` - PM2 setup and management --- ## Troubleshooting ### Server Connection Issues **Check if server is running**: ```bash npm run build:full # Look for "Successfully connected to bright-data" ``` **Verify API token**: ```bash echo $BRIGHT_DATA_API_TOKEN ``` **Test connection**: ```typescript const test = await searchEngine({ query: "test" }); console.log('Connection OK:', !!test); ``` ### No Results Returned **Possible causes**: - Query too obscure - try broader keywords - Rate limiting - wait and retry - Engine unavailable - try different engine **Solutions**: - Simplify query - Add time modifiers: "news", "2025" - Try different search engine - Implement retry logic ### Slow Performance **Optimization tips**: - Use batch queries instead of individual calls - Limit result extraction (use pagination) - Cache results in Context7 for reuse - Use appropriate engine for your region --- ## Change Log ### 2025-11-17 - ✅ Initial SERP integration via @brightdata/mcp - ✅ Added 4 SERP tools (searchEngine, searchEngineBatch, scrapeAsMarkdown, scrapeBatch) - ✅ Support for Google, Bing, Yandex - ✅ Batch processing capabilities - ✅ Integration with code2mcp TypeScript generation - ✅ Documentation completed --- ## Status **SERP Functionality**: ✅ FULLY ENABLED **Tools Available**: 4 **Search Engines**: Google, Bing, Yandex **Batch Support**: Yes (up to 10 queries/URLs) **Documentation**: Complete **Testing**: Ready --- **Next Steps**: 1. Review this documentation 2. Test SERP tools with your queries 3. Integrate into your workflows 4. Monitor usage and costs 5. Customize API token for production Ready to start scraping SERP data with Bright Data! 🚀