# redpanda-console-mcp

[![CI](https://github.com/KietDev-JS/redpanda-console-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/KietDev-JS/redpanda-console-mcp/actions/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)](https://www.python.org)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

An [MCP](https://modelcontextprotocol.io) server that lets an AI assistant read
and search Kafka messages through the **Redpanda Console HTTP API**.

It talks to the Console you already run, over HTTPS — so it works from a laptop
or CI runner that has no route to the Kafka broker port, inherits whatever
authentication sits in front of the Console, and needs no Kafka client library.

```
"Find the order-service messages mentioning trace 0f9573db from yesterday"
   └── search_messages(topic=…, text="0f9573db", start_timestamp_ms=…)
```

## Contents

- [Why this exists](#why-this-exists)
- [Requirements](#requirements)
- [Install](#install)
- [Configuration](#configuration)
- [Client setup](#client-setup)
- [Tools](#tools)
- [Message shape](#message-shape)
- [How it works](#how-it-works)
- [Security](#security)
- [Troubleshooting](#troubleshooting)
- [Development](#development)

## Why this exists

Most Kafka MCP servers speak the native Kafka protocol on port 9092. That is a
poor fit when the broker sits behind a VPC boundary and the only thing exposed
is the Console UI. This server uses the Console's own API, which means:

| | Native Kafka client | This server |
|---|---|---|
| Network requirement | Broker port (9092) | Console HTTPS port |
| Auth | SASL/mTLS config | Console's existing auth |
| Deserialisation | Client-side | Console (Avro/Protobuf/JSON registry-aware) |
| Content search | Fetch everything, filter locally | Server-side, only matches transferred |

Because deserialisation happens in the Console, messages come back as the same
text a human sees in the web UI, including schema-registry-decoded payloads.

## Requirements

- Python 3.10+
- Network access to a Redpanda Console instance (tested against Console v3.x
  fronting Apache Kafka 3.4)

## Install

```bash
git clone https://github.com/KietDev-JS/redpanda-console-mcp.git
cd redpanda-console-mcp
pip install .
```

For an isolated install that still exposes the command globally:

```bash
pipx install .
```

Verify the entry point resolves. The server communicates over stdio and takes
no command-line flags, so check that the executable exists rather than running
it bare:

```bash
which redpanda-console-mcp      # macOS / Linux
where.exe redpanda-console-mcp  # Windows
```

## Configuration

All configuration is environment-driven. **No hostname or credential is baked
into the code** — `CONSOLE_BASE_URL` is required and the server refuses to start
without it.

| Variable | Required | Default | Description |
|---|---|---|---|
| `CONSOLE_BASE_URL` | yes | — | Console base URL, e.g. `https://console.example.com` |
| `CONSOLE_API_KEY` | no | empty | Sent as `Authorization: Bearer <key>` |
| `CONSOLE_TIMEOUT` | no | `60` | HTTP timeout in seconds |
| `CONSOLE_VERIFY_TLS` | no | `true` | Set `false` only for trusted self-signed hosts |

Copy [`.env.example`](.env.example) to `.env` as a starting point. `.env` is
gitignored.

## Client setup

### Claude Desktop

`%APPDATA%\Claude\claude_desktop_config.json` (Windows) or
`~/Library/Application Support/Claude/claude_desktop_config.json` (macOS):

```json
{
  "mcpServers": {
    "redpanda-console": {
      "command": "redpanda-console-mcp",
      "env": {
        "CONSOLE_BASE_URL": "https://console.example.com"
      }
    }
  }
}
```

### Claude Code

```bash
claude mcp add redpanda-console \
  --env CONSOLE_BASE_URL=https://console.example.com \
  -- redpanda-console-mcp
```

### opencode

`~/.config/opencode/opencode.json`:

```json
{
  "mcp": {
    "redpanda-console": {
      "type": "local",
      "command": ["redpanda-console-mcp"],
      "enabled": true,
      "environment": {
        "CONSOLE_BASE_URL": "https://console.example.com"
      }
    }
  }
}
```

### Cursor / Windsurf / other MCP clients

Any stdio-capable client works. If the client cannot resolve the installed
script, invoke the module directly:

```json
{
  "command": "python",
  "args": ["-m", "redpanda_console_mcp"],
  "env": { "CONSOLE_BASE_URL": "https://console.example.com" }
}
```

See [docs/SETUP.md](docs/SETUP.md) for a step-by-step walkthrough, including
verification and a connectivity checklist.

## Tools

| Tool | Purpose |
|---|---|
| `list_topics` | List topics with partition count and replication factor |
| `cluster_info` | Cluster status, version, broker and partition counts |
| `describe_topic` | Effective configuration for one topic |
| `fetch_latest` | Most recent N messages |
| `fetch_by_offset` | N messages starting at an offset, reading forwards |
| `fetch_by_time` | N messages from the first offset at/after a timestamp |
| `search_messages` | Messages whose key or value contains a substring |

Common parameters:

- `partition_id` — `-1` (default) reads every partition; pass a number to
  target one. Paging with `fetch_by_offset` is only deterministic against a
  single partition, because an offset means different things per partition.
- `max_results` — capped at **500**. The Console aborts the response stream
  above that, so the limit is enforced client-side with a clear error rather
  than surfacing as a truncated read.
- `start_offset` (search only) — `-1` recent, `-2` oldest (**default**),
  `-3` newest/live, or an explicit offset.

`search_messages` scans from the **oldest** message by default so matches are
not silently missed. On a high-volume topic, narrow the range with
`start_timestamp_ms` or `start_offset` to keep the scan cheap.

## Message shape

Every fetch tool returns a list of objects with a stable schema:

```json
{
  "partition": 0,
  "offset": 107260,
  "timestamp": "2026-09-21T08:56:27.720000+00:00",
  "timestamp_ms": 1789980987720,
  "key": null,
  "value": "{\"ActionName\":\"HideFlightStaff\"}",
  "headers": [{ "key": "trace-id", "value": "0f9573db", "binary": false }],
  "key_encoding": "PAYLOAD_ENCODING_NULL",
  "value_encoding": "PAYLOAD_ENCODING_JSON",
  "value_size_bytes": 513,
  "compression": "COMPRESSION_TYPE_ZSTD",
  "value_is_binary": false
}
```

Notes:

- `value` is the Console's **deserialised** payload, so Avro and Protobuf
  messages arrive as readable JSON.
- Payloads that are not valid UTF-8 are returned base64-encoded with
  `value_is_binary: true`, rather than being corrupted by lossy decoding.
- `timestamp_ms` is kept alongside the ISO string so it can be fed straight
  back into `fetch_by_time`.

## How it works

The Console exposes two HTTP surfaces, and this server speaks both:

1. **Dataplane REST** (`/v1/...`) — ordinary JSON, used for topics and configs.
2. **Connect RPC** (`/<package>.<Service>/<Method>`) — Buf's
   [Connect protocol](https://connectrpc.com/docs/protocol). Unary methods use
   plain JSON; `ListMessages` is *server-streaming* and frames each JSON payload
   behind a 5-byte prefix (1 flag byte + 4-byte big-endian length).

Two details are easy to get wrong and are handled explicitly:

- **Unary calls must not use the streaming content type.** Sending
  `application/connect+json` to a unary method returns HTTP 415.
- **proto3 omits zero values.** `partitionId: 0` and `offset: 0` are absent
  from the JSON entirely, so they are defaulted to `0`, not `null`. Treating a
  missing field as unknown mislabels every message on partition 0.

Message search is executed by the Console's sandboxed JavaScript interpreter
(goja), so filtering happens next to the data and only matches cross the wire.

## Security

- **No credentials in the repository.** Configuration is environment-only, and
  `ConsoleConfig` marks the API key `repr=False` so an accidental log or
  traceback will not print it.
- **JavaScript injection is blocked.** The search term is embedded in the
  server-side filter via `json.dumps`, which emits a properly escaped string
  literal. Quotes, backslashes, newlines and the JS-specific U+2028/U+2029
  separators cannot break out into executable code — this is covered by
  dedicated tests.
- **Path injection is blocked.** Topic names are percent-encoded before being
  placed in a URL path.
- **TLS verification is on by default** and must be disabled explicitly.
- **The server is read-only.** It exposes no produce, delete, or config-write
  operation; it cannot modify your cluster.

Access control is inherited from the Console: this server can read exactly what
the supplied credential can read. Scope the token accordingly, and remember
that topic contents will be sent to whichever model backs your MCP client.

## Troubleshooting

| Symptom | Cause and fix |
|---|---|
| `Configuration error: CONSOLE_BASE_URL is not set` | Set it in the MCP client's `env` block, not just your shell — MCP servers do not inherit an interactive shell profile. |
| `Expected JSON … check that CONSOLE_BASE_URL points at a Redpanda Console API root` | The URL resolves to an SSO login page or proxy error. Confirm `curl $CONSOLE_BASE_URL/v1/topics` returns JSON. |
| `HTTP 415` | A proxy is rewriting the `Content-Type` header. |
| `UNKNOWN_TOPIC_OR_PARTITION` | The topic does not exist, or the credential cannot see it. Check `list_topics`. |
| `max_results must not exceed 500` | A Console-side limit. Page with `fetch_by_offset` against a single partition. |
| Search returns nothing on a busy topic | The scan reached `max_results` of *consumed* messages before finding matches. Narrow with `start_timestamp_ms`. |
| Timeouts on large fetches | Raise `CONSOLE_TIMEOUT`. |

## Development

```bash
pip install -e ".[dev]"

pytest                 # test suite (runs on asyncio and trio)
pytest --cov=redpanda_console_mcp --cov-report=term-missing
ruff check . && ruff format --check .
mypy                   # strict mode
```

The test suite runs entirely against an in-memory `httpx` transport, so it
needs no Kafka cluster, no Console, and no network. CI covers Python 3.10–3.13
on Linux, macOS and Windows.

Layout:

```
src/redpanda_console_mcp/
  console.py   # HTTP + Connect RPC transport
  models.py    # proto3 JSON normalisation
  filters.py   # server-side JS filter construction
  service.py   # Console semantics and validation
  server.py    # MCP tool definitions
```

Contributions are welcome — please keep `ruff`, `mypy` and the test suite green.

## License

[MIT](LICENSE)