# DataSF MCP Server

A Model Context Protocol (MCP) server that provides LLMs with seamless access to San Francisco's open data portal (DataSF), powered by the Socrata platform.

## Overview

This MCP server enables AI assistants like Claude to search, explore, and query San Francisco's public datasets through a simple, standardized interface. It handles the complexity of the Socrata API, provides intelligent column name correction, and includes schema caching for optimal performance.

### Key Features

- 🔍 **Dataset Search & Discovery** - Find datasets by keywords or browse by category
- 📊 **Schema Retrieval** - Get column names and data types before querying
- 💬 **SoQL Query Execution** - Run SQL-like queries against any dataset
- 🎯 **Fuzzy Column Matching** - Auto-corrects typos in column names
- ⚡ **Schema Caching** - Reduces API calls with intelligent caching
- 🔐 **Optional Authentication** - Supports Socrata App Tokens for higher rate limits
- ✅ **Property-Based Testing** - Comprehensive correctness guarantees

## Available Tools

### 1. `search_datasf`
Search for datasets by keywords.

**Parameters:**
- `query` (string, required): Search keywords (1-500 characters)
- `limit` (number, optional): Max results (default: 5, max: 20)

**Example:**
```
Search for police incident datasets
```

### 2. `list_datasf`
Browse available datasets, optionally filtered by category.

**Parameters:**
- `category` (string, optional): Filter by category
- `limit` (number, optional): Max results (default: 5, max: 20)

**Example:**
```
List recent public safety datasets
```

### 3. `get_schema`
Get the schema (columns and data types) for a specific dataset.

**Parameters:**
- `dataset_id` (string, required): Dataset 4x4 ID (format: `xxxx-xxxx`)

**Example:**
```
Get the schema for dataset wg3w-h783
```

### 4. `query_datasf`
Execute a SoQL (Socrata Query Language) query against a dataset.

**Parameters:**
- `dataset_id` (string, required): Dataset 4x4 ID
- `soql` (string, required): SoQL query (1-4000 characters)
- `auto_correct` (boolean, optional): Enable column name correction (default: true)

**Example:**
```
Query dataset wg3w-h783: SELECT incident_category, COUNT(*) GROUP BY incident_category LIMIT 10
```

## Installation

### Prerequisites
- Node.js 18 or higher
- npm or yarn

### Local Setup (Optional)

If you want to run or modify the server locally:

1. Clone the repository:
```bash
git clone https://github.com/fwextensions/datasf-mcp.git
cd datasf-mcp
```

2. Install dependencies:
```bash
npm install
```

3. Run the server:
```bash
npm start
```

The server uses `tsx` to run TypeScript directly without a build step.

## Usage

### Testing with MCP Inspector

For the MCP Inspector, you'll need to use the local installation:

```bash
# First, clone and install locally
git clone https://github.com/fwextensions/datasf-mcp.git
cd datasf-mcp
npm install

# Then run the inspector
npx -y @modelcontextprotocol/inspector tsx src/index.ts
```

In the inspector UI, use:
- **Command:** `tsx`
- **Arguments:** `src/index.ts` (or absolute path if running from outside the directory)

### Quick Start with npx (Recommended)

The easiest way to use the server is directly from GitHub using npx:

```json
{
  "mcpServers": {
    "datasf": {
      "command": "npx",
      "args": ["-y", "github:fwextensions/datasf-mcp"],
      "env": {
        "SOCRATA_APP_TOKEN": "your-optional-token"
      }
    }
  }
}
```

This will automatically download and run the latest version from GitHub without any manual installation.

### Local Installation

Alternatively, clone and install locally:

```bash
git clone https://github.com/fwextensions/datasf-mcp.git
cd datasf-mcp
npm install
```

Then use the absolute path in your MCP configuration (see below).

### Configuration for Claude Desktop

Add to your Claude Desktop config file:

**Windows:** `%APPDATA%\Claude\claude_desktop_config.json`  
**macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`  
**Linux:** `~/.config/Claude/claude_desktop_config.json`

**Option 1: Using npx (recommended)**
```json
{
  "mcpServers": {
    "datasf": {
      "command": "npx",
      "args": ["-y", "github:fwextensions/datasf-mcp"],
      "env": {
        "SOCRATA_APP_TOKEN": "your-optional-token"
      }
    }
  }
}
```

**Option 2: Using local installation**
```json
{
  "mcpServers": {
    "datasf": {
      "command": "npx",
      "args": ["tsx", "/absolute/path/to/datasf-mcp/src/index.ts"],
      "env": {
        "SOCRATA_APP_TOKEN": "your-optional-token"
      }
    }
  }
}
```

**Important:** Replace `/absolute/path/to/datasf-mcp` with the actual full path to where you cloned this project.

### Configuration for Kiro IDE

Create or edit `.kiro/settings/mcp.json`:

**Option 1: Using npx from GitHub (recommended)**
```json
{
  "mcpServers": {
    "datasf": {
      "command": "npx",
      "args": ["-y", "github:fwextensions/datasf-mcp"],
      "env": {
        "SOCRATA_APP_TOKEN": "your-optional-token"
      },
      "disabled": false,
      "autoApprove": []
    }
  }
}
```

**Option 2: Using local installation**
```json
{
  "mcpServers": {
    "datasf": {
      "command": "npx",
      "args": ["tsx", "src/index.ts"],
      "env": {
        "SOCRATA_APP_TOKEN": "your-optional-token"
      },
      "disabled": false,
      "autoApprove": []
    }
  }
}
```

## Getting a Socrata App Token

The server works without authentication for public data, but an App Token increases rate limits:

1. Visit https://data.sfgov.org/
2. Sign up for a free account
3. Navigate to Developer Settings
4. Create a new App Token
5. Add it to your MCP configuration

## Development

### Project Structure

```
datasf-mcp-server/
├── src/
│   ├── index.ts              # MCP server entry point
│   ├── socrataClient.ts      # Socrata API client
│   ├── validator.ts          # Input validation with Zod
│   ├── fuzzyMatcher.ts       # Column name auto-correction
│   ├── cache.ts              # Schema caching
│   ├── errorHandler.ts       # Error handling utilities
│   └── __tests__/
│       └── property/         # Property-based tests
├── dist/                     # Compiled JavaScript output
├── package.json
└── tsconfig.json
```

### Available Scripts

- `npm run build` - Compile TypeScript to JavaScript
- `npm start` - Run the compiled server
- `npm test` - Run all tests
- `npm run test:watch` - Run tests in watch mode

### Running Tests

```bash
npm test
```

The project uses property-based testing with `fast-check` to ensure correctness across a wide range of inputs.

## Architecture

The server follows a modular architecture:

1. **MCP Server** - Handles protocol communication via stdio
2. **Socrata Client** - Manages HTTP requests to Socrata APIs
3. **Validator** - Validates all inputs using Zod schemas
4. **Fuzzy Matcher** - Corrects column name typos using Fuse.js
5. **Schema Cache** - Caches dataset schemas in memory (5-minute TTL)
6. **Error Handler** - Classifies and formats errors for LLM consumption


## Example Queries

Once configured in your LLM, you can ask questions like:

- "Search for datasets about housing in San Francisco"
- "What's the schema for the police incidents dataset (wg3w-h783)?"
- "Show me the top 10 incident categories from the police incidents dataset"
- "Find all building permits issued in 2024"
- "What datasets are available about transportation?"

## API Endpoints Used

The server interacts with three Socrata APIs:

- **Discovery API**: `https://api.us.socrata.com/api/catalog/v1` - Dataset search and browsing
- **Views API**: `https://data.sfgov.org/api/views/{id}.json` - Schema retrieval
- **Resource API**: `https://data.sfgov.org/resource/{id}.json` - Data querying

## Error Handling

The server provides descriptive error messages for:

- **Validation errors** - Invalid input format or length
- **Not found** - Dataset doesn't exist
- **Rate limiting** - Too many requests (add App Token to resolve)
- **Timeouts** - Request exceeded 30 seconds
- **API errors** - Socrata-specific errors (e.g., SoQL syntax errors)

## Contributing

Contributions are welcome! The project uses:

- TypeScript for type safety
- Zod for runtime validation
- fast-check for property-based testing
- Vitest as the test runner

## License

MIT

## Resources

- [DataSF Portal](https://data.sfgov.org/)
- [Socrata API Documentation](https://dev.socrata.com/)
- [SoQL Query Language](https://dev.socrata.com/docs/queries/)
- [Model Context Protocol](https://modelcontextprotocol.io/)

## Troubleshooting

**Server not starting**
- Ensure you ran `npm run build` first
- Check that Node.js 18+ is installed

**Tools not showing up in LLM**
- Verify the path in your config is absolute
- Restart your LLM application after adding the config
- Check the LLM's logs for connection errors

**Rate limiting errors**
- Add a Socrata App Token to your configuration
- Reduce the frequency of requests

**Column name errors in queries**
- Use `get_schema` first to see valid column names
- Enable `auto_correct: true` (default) for automatic typo correction