# mcp-osint

MCP server for Claude Code providing access to OSINT data sources:

- **Government** - Data.gov, LegiScan, CourtListener, Census Bureau
- **Research** - OpenAlex, Semantic Scholar, PubMed, CORE
- **Corporate** - SEC EDGAR (10-K, 10-Q), FRED (economic data)
- **Compliance** - OpenSanctions (sanctions, PEPs)
- **News/Knowledge** - GDELT, Wikidata
- **Infrastructure** - crt.sh (SSL certificates, subdomains)
- **Web** - Firecrawl (URL scraping with HTML + markdown)

## Setup

### 1. Get API Keys

Required keys:
- **LegiScan** - [legiscan.com/user/register](https://legiscan.com/user/register)
- **CourtListener** - [courtlistener.com/api](https://www.courtlistener.com/api/)
- **FRED** - [fred.stlouisfed.org/docs/api](https://fred.stlouisfed.org/docs/api/api_key.html)
- **Firecrawl** - [firecrawl.dev](https://firecrawl.dev)

Optional keys (higher rate limits):
- **Data.gov** - [api.data.gov/signup](https://api.data.gov/signup/)
- **Census** - [census.gov/data/developers](https://api.census.gov/data/key_signup.html)
- **CORE** - [core.ac.uk/services/api](https://core.ac.uk/services/api)
- **OpenSanctions** - [opensanctions.org/api](https://www.opensanctions.org/api/)

Free (no key): SEC EDGAR, GDELT, Wikidata, crt.sh, OpenAlex, Semantic Scholar, PubMed

### 2. Install & Build

```bash
cd mcp-osint
npm install
npm run build
```

### 3. Add to Claude Code

Using the CLI:

```bash
claude mcp add -s user -t stdio mcp-osint \
  -e LEGISCAN_API_KEY=your-key \
  -e COURTLISTENER_API_KEY=your-key \
  -e FRED_API_KEY=your-key \
  -e FIRECRAWL_API_KEY=your-key \
  -- node /path/to/mcp-osint/dist/index.js
```

Or manually add to your MCP settings (`~/.claude/settings.json` or VS Code settings):

```json
{
  "mcpServers": {
    "mcp-osint": {
      "command": "node",
      "args": ["/path/to/mcp-osint/dist/index.js"],
      "env": {
        "LEGISCAN_API_KEY": "your-key",
        "COURTLISTENER_API_KEY": "your-key",
        "FRED_API_KEY": "your-key",
        "FIRECRAWL_API_KEY": "your-key",
        "POLITE_EMAIL": "you@example.com"
      }
    }
  }
}
```

## Tools

### osint_search

Search across 14 OSINT data sources. Returns results with available resources.

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `query` | string | Yes | Natural language search query |
| `source` | string | | Force a specific connector (see table below) |
| `jurisdiction` | string | | State code (e.g., `"CA"`) or `"US"` |
| `year` | number | | Filter to specific year |
| `limit` | number | | Max results (default: 10) |

**Examples:**
```
"EPA air quality data California"       → Data.gov
"Michigan renewable energy bill 2024"   → LegiScan
"Brown v. Board of Education"           → CourtListener
"population by county Texas"            → Census
"machine learning medical diagnosis"    → OpenAlex/PubMed
"Apple 10-K filing 2024"                → SEC EDGAR
"GDP quarterly growth rate"             → FRED
"Russian sanctions oligarchs"           → OpenSanctions
"Ukraine conflict news"                 → GDELT
"microsoft.com subdomains"              → crt.sh
```

### osint_preview

Preview a resource's schema and sample data before fetching.

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `resource_id` | string | Yes | Resource ID from osint_search |
| `row_limit` | number | | Sample rows for tabular data (default: 5) |
| `max_bytes` | number | | Max bytes for text preview (default: 4000) |

### osint_get

Fetch data from a resource ID or URL. Automatically handles web pages, PDFs, and structured data.

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `target` | string | Yes | URL (http/https) or resource_id from osint_search |
| `output_path` | string | | Path to save binary files (required for PDFs) |
| `question` | string | | What to extract (e.g., "all data", "key findings") |
| `summarize` | boolean | | If true with question, returns only relevant content (default: false) |
| `columns` | string[] | | Specific columns to return (resource_id only) |
| `filters` | object[] | | Filter conditions (resource_id tabular data only) |
| `limit` | number | | Max rows for tabular data (default: 100) |

**Behavior by target type:**

| Target | Behavior |
|--------|----------|
| Web URL | Returns markdown + raw HTML + SHA256 hash via Firecrawl |
| PDF URL | Downloads to `output_path`, returns file path + SHA256 |
| Binary URL | Downloads to `output_path`, returns file path + SHA256 |
| Resource ID | Extracts data via connector with optional filtering |

**Examples:**
```
osint_get target="https://example.com/article"
osint_get target="https://example.com/paper.pdf" output_path="./downloads/paper.pdf"
osint_get target="pubmed:paper:12345:abstract" question="key findings"
```

### osint_list_sources

List all data sources and their configuration status.

## Environment Variables

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `LEGISCAN_API_KEY` | Yes | - | LegiScan legislative data |
| `COURTLISTENER_API_KEY` | Yes | - | CourtListener judicial data |
| `FRED_API_KEY` | Yes | - | FRED economic data |
| `FIRECRAWL_API_KEY` | Yes | - | Firecrawl web scraping |
| `DATAGOV_API_KEY` | | - | Data.gov (higher limits) |
| `CENSUS_API_KEY` | | - | Census Bureau (higher limits) |
| `CORE_API_KEY` | | - | CORE open access papers |
| `OPENSANCTIONS_API_KEY` | | - | OpenSanctions compliance data |
| `POLITE_EMAIL` | | - | Email for polite API usage (OpenAlex, PubMed, SEC) |
| `MCP_DEBUG` | | `true` | Debug logging; set to `"false"` to disable |
| `MCP_LOG_DIR` | | `./logs` | Log directory; set to `"none"` to disable |

## Connectors

| Connector | Source | Data Types | Key Required |
|-----------|--------|------------|--------------|
| `data_gov` | Data.gov | Datasets, resources | Recommended |
| `legiscan` | LegiScan | Bills, votes, sponsors | Yes |
| `courtlistener` | CourtListener | Cases, opinions, dockets | Yes |
| `census` | Census Bureau | Demographics, statistics | Recommended |
| `openalex` | OpenAlex | Papers, authors, citations | No |
| `semantic_scholar` | Semantic Scholar | Papers, authors, citations | No |
| `pubmed` | PubMed/NCBI | Medical papers, abstracts | No |
| `core` | CORE | Open access papers | Recommended |
| `sec_edgar` | SEC EDGAR | 10-K, 10-Q, company filings | No |
| `fred` | FRED | Time series, economic data | Yes |
| `opensanctions` | OpenSanctions | Sanctions, PEPs | Yes |
| `gdelt` | GDELT | News, global events | No |
| `wikidata` | Wikidata | Entities, knowledge graph | No |
| `crt_sh` | crt.sh | SSL certificates, subdomains | No |

## Development

```bash
npm install       # Install dependencies
npm run build     # Compile TypeScript
npm run dev       # Watch mode
npm start         # Run server
```

## Testing

```bash
# Test connector metadata and identifiers
npx tsx test/test-all-connectors.ts

# Test data retrieval (downloads files)
npx tsx test/test-data-retrieval.ts

# Test Firecrawl HTML+Markdown
npx tsx test/test-firecrawl.ts

# Run comprehensive MCP tool tests
npx tsx test/test-mcp-scenarios.ts
```

## Architecture

```
src/
├── index.ts              # MCP server entry point
├── types.ts              # Shared types
├── intent.ts             # Query parsing and routing
├── router.ts             # Connector selection
├── logger.ts             # Logging utility
├── cache.ts              # SQLite + file caching
├── retry.ts              # Retry with backoff
└── connectors/
    ├── base.ts           # Base connector class
    ├── data-gov.ts       # Data.gov/CKAN
    ├── legiscan.ts       # LegiScan
    ├── courtlistener.ts  # CourtListener
    ├── census.ts         # Census Bureau
    ├── openalex.ts       # OpenAlex
    ├── semantic-scholar.ts
    ├── pubmed.ts         # PubMed/NCBI
    ├── core.ts           # CORE
    ├── sec-edgar.ts      # SEC EDGAR
    ├── fred.ts           # FRED
    ├── opensanctions.ts  # OpenSanctions
    ├── gdelt.ts          # GDELT
    ├── wikidata.ts       # Wikidata
    ├── crt-sh.ts         # crt.sh
    └── firecrawl.ts      # Firecrawl
```

## License

MIT