π WebScout MCP
WebScout MCP is a powerful Model Context Protocol (MCP) server designed for reverse engineering web applications, particularly chat interfaces and streaming APIs. It provides comprehensive browser automation tools to discover, analyze, and capture network traffic from complex web applications.
β¨ Key Features
π€ Automated Reverse Engineering
One-Click Analysis: Automatically navigate to web applications and capture streaming endpoints
Smart Pattern Detection: Advanced detection of SSE, WebSocket, chunked transfers, and custom streaming formats
Network Traffic Capture: Comprehensive CDP-level monitoring of all HTTP requests, responses, and WebSocket frames
Structured Data Output: Clean, parsed data with URLs, request payloads, and response patterns
π Interactive Browser Automation
Session Management: Persistent browser sessions with cookie and authentication state management
Authentication Support: Handle login forms, OAuth flows, and multi-factor authentication
Step-by-Step Navigation: Click buttons, fill forms, and navigate through complex multi-page interfaces
Visual Feedback: Take screenshots at any point to understand page state and UI elements
π― Advanced Network Monitoring
Real-Time Capture: Monitor streaming responses as they occur with configurable capture windows
Flexible Filtering: Capture all traffic or filter by POST requests, streaming responses, or URL patterns
WebSocket Support: Full capture of WebSocket frames, messages, and connection details
Memory Management: Configurable capture limits to prevent memory issues during long sessions
π οΈ Developer-Friendly Tools
14 Specialized Tools: Comprehensive toolkit for web scraping, testing, and API discovery
Headless or Visible: Run in headless mode for automation or visible mode for debugging
Error Handling: Robust error handling with detailed error messages and recovery options
Cross-Platform: Works on macOS, Linux, and Windows with consistent behavior
π Available Tools
Core Reverse Engineering
reverse_engineer_chat
- Automated analysis of chat interfaces with streaming endpoint discoverystart_network_capture
- Begin comprehensive network traffic monitoringstop_network_capture
- End capture and retrieve all collected dataget_network_capture_status
- Check capture session status and statisticsclear_network_capture
- Clear captured data without stopping the capture session
Interactive Browser Control
initialize_session
- Create a new browser session for interactive operationsclose_session
- Clean up browser resources and end sessionnavigate_to_url
- Navigate to different URLs within a sessionswitch_tab
- Switch between open browser tabs
User Interaction Simulation
click_element
- Click buttons, links, or any interactive elementsfill_form
- Fill out form fields with automatic submission optionswait_for_element
- Wait for dynamic elements to appear before continuing
Visual Inspection
take_screenshot
- Capture screenshots of viewport, full page, or specific elementsget_current_page_info
- Retrieve comprehensive page information and tab details
π Installation
Prerequisites
Node.js 18+ - Required for ES modules and modern JavaScript features
npm - Package manager for dependency installation
Quick Setup
π Usage
Method 1: MCP Server (Recommended)
Add WebScout MCP to your MCP client configuration:
Method 2: Direct CLI Usage
Method 3: Development Mode
π οΈ API Examples
Basic Chat Interface Analysis
Interactive Login Flow
Network Traffic Capture
ποΈ Architecture Overview
Workflow
Browser Launch: Opens target URL in headless Playwright browser
Network Setup: Establishes Chrome DevTools Protocol (CDP) session and route interception
Interface Detection: Automatically locates chat input elements (textarea, contenteditable, etc.)
Message Injection: Sends test message to trigger streaming responses
Traffic Capture: Monitors network requests/responses for specified time window
Pattern Analysis: Identifies streaming patterns in captured data
Data Processing: Structures captured data into clean JSON format
Streaming Detection Patterns
The system detects multiple streaming response formats:
Server-Sent Events (SSE):
data: {"content": "..."}
OpenAI-style chunks:
data: {"choices": [{"delta": {"content": "..."}}]}
Event streams:
event: message\ndata: {...}
JSON streaming: Objects with
token
,delta
,content
fieldsCustom formats:
f:{...}
,0:"..."
,e:{...}
patternsWebSocket messages: Binary/text frames with streaming data
Chunked responses: Transfer-encoding: chunked with streaming content
π Project Structure
π§ Configuration
Environment Variables
Variable | Description | Default |
| Environment mode |
|
| Enable debug logging |
|
MCP Configuration
Update your MCP client's configuration file:
Or for VS Code MCP configuration (mcp.json
):
Contributing
Fork the repository
Create a feature branch:
git checkout -b feature-name
Make your changes and add tests
Run tests:
npm test
Submit a pull request
Development Guidelines
Follow ES6+ syntax and modern JavaScript practices
Add JSDoc comments for new functions
Test your changes with multiple chat interfaces
Update documentation for new features
Ensure code passes all tests
π License
This project is licensed under the ISC License - see the LICENSE file for details.
π Acknowledgments
Built with the Model Context Protocol SDK
Powered by Playwright for browser automation
Inspired by the need for better web API discovery and testing tools
β οΈ Important Notes
Ethical Use: This tool is intended for API analysis and integration purposes only. Always respect website terms of service and robots.txt files.
Rate Limiting: Some chat interfaces may have rate limits or CAPTCHAs that could interfere with analysis.
Browser Dependencies: Playwright requires browser binaries to be installed for automation.
Network Conditions: Results may vary based on network speed and target website performance.
π Troubleshooting
Common Issues
"Browser not found" error
"Connection timeout" error
Increase
captureWindowMs
parameterCheck network connectivity
Verify target URL is accessible
"No streaming endpoints found"
Try different test messages
Increase capture window time
Verify the chat interface doesn't require authentication
MCP connection issues
Verify the absolute path in
mcp-config.json
Ensure Node.js 18+ is installed
Check MCP client logs for detailed errors
π Support
If you encounter issues or have questions:
Check the Troubleshooting section
Review existing Issues on GitHub
Create a new Issue with detailed information
WebScout MCP - Your intelligent companion for web application reverse engineering and API discovery.
Made with β€οΈ for developers, security researchers, and API enthusiasts
local-only server
The server can only run on the client's local machine because it depends on local resources.
Tools
Enables reverse engineering of web applications and chat interfaces through browser automation, network traffic capture, and streaming API discovery. Provides comprehensive tools for analyzing network patterns, capturing streaming responses, and automating complex web interactions.