README.md•5.67 kB
# LinkedIn Scraper MCP Server
A Model Context Protocol (MCP) server that provides LinkedIn profile scraping capabilities with manual credential input. This server allows you to extract comprehensive profile data including experience, education, skills, and contact information.
## Features
- **Manual Credential Input**: No need for environment variables - provide LinkedIn credentials directly in tool calls
- **Comprehensive Data Extraction**: Scrapes name, headline, location, about section, experience, education, skills, and contact info
- **Multiple Transport Methods**: Supports both HTTP (production) and STDIO (development) transports
- **Browser Automation**: Uses Selenium WebDriver with Chrome for reliable scraping
- **Anti-Detection**: Includes human-like delays and browser settings to avoid detection
## Installation
1. **Install dependencies**:
```bash
npm install
```
2. **Install Chrome browser** (if not already installed):
- The server uses Chrome WebDriver which will be automatically managed
- Ensure Chrome browser is installed on your system
3. **Build the project**:
```bash
npm run build
```
## Usage
### HTTP Transport (Recommended)
Start the server with HTTP transport:
```bash
npm start
# or
node dist/index.js
```
The server will start on `http://localhost:8080` by default.
### STDIO Transport (Development)
For local development with STDIO transport:
```bash
npm run start:stdio
# or
node dist/index.js --stdio
```
### Command Line Options
- `--port <PORT>`: Specify HTTP server port (default: 8080)
- `--stdio`: Use STDIO transport instead of HTTP
- `--help`: Show help message
## MCP Client Configuration
Add this to your MCP client configuration:
```json
{
"mcpServers": {
"linkedin-scraper": {
"url": "http://localhost:8080/mcp"
}
}
}
```
## Available Tools
### `scrape_linkedin_profile`
Scrapes a LinkedIn profile and returns comprehensive profile data.
**Parameters:**
- `url` (required): LinkedIn profile URL (e.g., "https://www.linkedin.com/in/username/")
- `email` (required): LinkedIn account email for authentication
- `password` (required): LinkedIn account password for authentication
- `headless` (optional): Run browser in headless mode (default: false)
**Example Usage:**
```json
{
"tool": "scrape_linkedin_profile",
"arguments": {
"url": "https://www.linkedin.com/in/johndoe/",
"email": "your-email@example.com",
"password": "your-password",
"headless": false
}
}
```
**Response Format:**
```json
{
"success": true,
"timestamp": "2024-01-01T12:00:00.000Z",
"profile": {
"url": "https://www.linkedin.com/in/johndoe/",
"name": "John Doe",
"headline": "Software Engineer at Tech Company",
"location": "San Francisco, CA",
"about": "Passionate software engineer...",
"experience_count": 3,
"experiences": [...],
"education_count": 2,
"education": [...],
"skills_count": 15,
"skills": [...],
"websites": [...],
"email": "john@example.com"
}
}
```
## Development
### Project Structure
```
src/
├── index.ts # Main entry point
├── cli.ts # Command-line argument parsing
├── config.ts # Configuration management
├── server.ts # Server instance creation
├── client.ts # LinkedIn scraper client
├── types.ts # TypeScript type definitions
├── tools/
│ ├── index.ts # Tool exports
│ └── linkedin.ts # LinkedIn scraping tool
└── transport/
├── index.ts # Transport exports
├── http.ts # HTTP transport
└── stdio.ts # STDIO transport
```
### Building
```bash
npm run build # Build once
npm run watch # Build and watch for changes
```
### Running in Development
```bash
npm run dev # Build and run HTTP transport
npm run dev:stdio # Build and run STDIO transport
```
## Security Considerations
- **Credential Handling**: LinkedIn credentials are passed directly in tool calls and not stored
- **Rate Limiting**: The scraper includes human-like delays to avoid being blocked
- **Browser Settings**: Uses realistic browser settings to minimize detection
- **Session Management**: Each scraping session is isolated and cleaned up properly
## Troubleshooting
### Common Issues
1. **Chrome Driver Issues**: The server automatically manages Chrome WebDriver, but ensure Chrome browser is installed
2. **LinkedIn Authentication**: If authentication fails:
- Verify your LinkedIn credentials are correct
- Check if your account has two-factor authentication enabled
- Try logging in manually first to resolve any security challenges
3. **Scraping Failures**: If scraping fails:
- Try running with `headless: false` to see what's happening
- Check if LinkedIn has updated their page structure
- Ensure stable internet connection
4. **Port Conflicts**: If port 8080 is in use:
```bash
node dist/index.js --port 3000
```
### Environment Variables
Optional environment variables:
- `PORT`: HTTP server port (default: 8080)
- `NODE_ENV`: Set to 'production' for production mode
## License
This project is for educational and research purposes. Please respect LinkedIn's Terms of Service and use responsibly.
## Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test thoroughly
5. Submit a pull request
## Disclaimer
This tool is for educational purposes only. Users are responsible for complying with LinkedIn's Terms of Service and applicable laws. The authors are not responsible for any misuse of this software.