HackerNews Job Scraper

A Python-based job scraper that extracts job postings from HackerNews "Who's Hiring" threads and exposes them via an MCP (Model Context Protocol) server for Claude Desktop integration.

Features

• HackerNews Scraper: Extracts job postings from HN threads and converts them to structured JSON

• MCP Server: Exposes scraped data through Model Context Protocol for Claude Desktop

• Caching: File-based caching system to avoid repeated requests

• Search: Search through job postings by keywords

• Job Details: Get full details of specific job postings

Quick Start

1. Setup Environment

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

2. Test the Scraper

# Run the scraper directly
python scraper.py

This will scrape the default HackerNews thread and show how many job postings were found.

3. Configure Claude Desktop

Add the following to your Claude Desktop settings (usually at ~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "hn-job-scraper": {
      "command": "python3",
      "args": ["/path/to/your/project/mcp_server.py"],
      "env": {
        "PYTHONPATH": "/path/to/your/project"
      }
    }
  }
}

Important: Replace /path/to/your/project with the actual path to this project directory.

4. Start Using with Claude Desktop

Once configured, you can use these tools in Claude Desktop:

• Search Jobs: search_jobs - Search job postings by keywords (e.g., "python", "remote", "senior")

• Get Job Details: get_job_details - Get full details of a specific job posting

• Refresh Jobs: refresh_jobs - Clear cache and fetch fresh job data

Usage Examples

Command Line Usage

# Activate virtual environment
source venv/bin/activate

# Run scraper
python scraper.py

# Start MCP server (usually called by Claude Desktop)
python mcp_server.py

Claude Desktop Integration

Once configured, you can ask Claude Desktop:

• "Search for Python jobs in the latest HackerNews hiring thread"

• "Find remote work opportunities"

• "Show me senior developer positions"

• "Get details for job posting ID 12345"

Project Structure

hackernews-jobscraper/
├── scraper.py              # Core scraping functionality
├── mcp_server.py           # MCP server implementation
├── requirements.txt        # Python dependencies
├── claude_desktop_config.json  # Example Claude Desktop config
├── cache/                  # Cached job data (created automatically)
└── README.md              # This file

How It Works

1. Scraping: The scraper fetches job postings from HackerNews "Who's Hiring" threads

2. Caching: Results are cached locally to avoid repeated requests (1-hour cache)

3. MCP Server: Exposes the data through Model Context Protocol

4. Claude Integration: Claude Desktop can search and analyze job postings

Configuration

Default HackerNews Thread

The scraper defaults to thread ID 44434574. You can change this by modifying the scrape_job_postings() call in scraper.py.

Cache Settings

Cache files are stored in the cache/ directory and expire after 1 hour. This can be modified in the HackerNewsScraper class.

Troubleshooting

Claude Desktop Not Finding Server

1. Ensure the paths in claude_desktop_config.json are absolute paths

2. Make sure the virtual environment is activated

3. Check that all dependencies are installed

4. Restart Claude Desktop after configuration changes

No Job Postings Found

1. Check your internet connection

2. Verify the HackerNews thread ID is valid

3. Try refreshing the job data using the refresh_jobs tool

Development

Running Tests

# Test scraper functionality
python scraper.py

# Test MCP server (requires Claude Desktop or MCP client)
python mcp_server.py

Adding New Features

The project is designed to be extensible. You can:

• Add new scraping sources

• Implement additional search filters

• Enhance job posting metadata

• Add data export functionality

hn_candidate_mcp