search_messages
Find Kafka messages by JavaScript predicate over key, value, headers, offset, timestamp, or schema fields; group matches into buckets, count them, or export results to a file.
Instructions
Search message key, value, headers or metadata with a JavaScript predicate. The script returns true for a match and receives value (parsed JSON, a decoded Avro/Protobuf/JSON Schema record, or text), key, headers, partition, offset, timestamp, value_bytes, key_bytes, format, schema_id and decode_error. Omit it to match all messages. Schema-encoded values are searched by field exactly like JSON.
group_by summarises instead of listing: an expression over the same variables returns a bucket name, and every match in the range is counted per bucket with one example address each. Use it to break a DLQ down by error header, find hot keys on a partition, or count messages per schema_id.
Every partition is read together, one chunk deep at a time, so a limited newest-first search returns the newest matches in the topic rather than the newest in whichever partition was read first. Kafka orders records only within a partition, so matches are merged by timestamp; producers set timestamps unless the topic uses LogAppendTime.
Kafka has no server-side search, so scans are bounded. Check complete, stopped_reason and scanned_ranges before treating no matches as conclusive. Use count_only for large result sets, and download to save them to a file on the user's machine without storing anything on the server: over stdio the file is saved at once, and over HTTP the response is a link and a command to run there, the search streaming every match when it is fetched. A script that runs past timeout_seconds is interrupted and counted in script_errors; scripts are not memory-sandboxed, so keep predicates simple.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| topic | Yes | Topic to search. Matched exactly and case-sensitively. | |
| script | No | Optional JavaScript that decides whether a message matches. Return true to keep it. In scope: value (the parsed document for JSON, and the decoded record for Avro, Protobuf, JSON Schema or a configured format; the raw text otherwise), key (string, decoded document when the key has a schema, or null), headers (object of header name to string), partition, offset, timestamp (a Date), value_bytes and key_bytes (raw sizes, before decoding), format (json, avro, protobuf, json_schema, msgpack, text, binary or null), schema_id (number or null) and decode_error (string or null). Examples: return key === 'order-123'; return value.eventType === 'NEW' && value.payload.amount >= 500; return value.payload.cancelledAt === null. Omit to match every message. | |
| download | No | Optional. Instead of returning matches, save every match to a file on the user's own machine. Over stdio the file is saved right away and download.path names it. Over HTTP the response is a link and a command, e.g. curl, to run on the user's machine; the search runs when the link is fetched. Nothing is stored on the server. Use it instead of returning thousands of messages, or whenever the user wants the messages in a file. | |
| group_by | No | Optional JavaScript expression, with the same variables as script, that returns a bucket name for each matching message, e.g. return headers['error-reason'] or return key or return schema_id. Every match in the range is counted per bucket, like count_only, and no message bodies are returned. null and undefined form the bucket null. Cannot be combined with download. | |
| direction | No | Optional scan direction: newest_first (default) or oldest_first. Decides which matches are kept when max_matches cuts the search short. Every partition is read together, so newest_first means newest in the topic, ordered by timestamp, rather than newest in one partition. | |
| to_offset | No | Optional exclusive offset to stop scanning at, applied to every searched partition. | |
| count_only | No | Optional. When true, scan the whole range and return only how many messages matched, with a per-partition breakdown and no message bodies. Use this first when a query may match a great many messages, then ask the user how they want them before fetching any. | |
| max_groups | No | Optional number of largest buckets to return with group_by. Defaults to 20, at most 1000. groups_truncated says whether more existed. | |
| partitions | No | Optional partitions to restrict the search to. Defaults to every partition. Do not guess a partition from a message key: producers may set the partition explicitly, so the key does not determine it. | |
| from_offset | No | Optional inclusive offset to start scanning from, applied to every searched partition. | |
| max_matches | No | Optional maximum number of matches to return. Defaults to 10, at most 1000. Use download to export more. | |
| parallelism | No | Optional number of concurrent readers, from 1 to 16. Defaults to 1. It splits a single-partition topic's offsets between readers, which makes a full scan of one large partition faster. A multi-partition topic is already read across its partitions together, so this does not apply there. Worth using for count_only, download or a full scan of one partition. | |
| to_timestamp | No | Optional exclusive end time (RFC3339). Resolved to the first offset at or after this time; messages whose own timestamp is at or after it are skipped and not counted as scanned. | |
| from_timestamp | No | Optional inclusive start time (RFC3339). Resolved to the first offset at or after this time; messages whose own timestamp is earlier are skipped and not counted as scanned. | |
| max_value_bytes | No | Optional maximum value bytes to return per match. Defaults to 512, at most 1048576. Longer values are cut and flagged with truncated=true. | |
| timeout_seconds | No | Optional wall-clock limit for the scan in seconds. Defaults to 30, at most 3600. | |
| max_messages_scanned | No | Optional maximum number of messages to read before giving up. Defaults to 10000, at most 100000000. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| topic | Yes | ||
| groups | No | ||
| matches | Yes | ||
| complete | Yes | ||
| download | No | ||
| match_count | Yes | ||
| script_errors | No | ||
| scanned_ranges | Yes | ||
| stopped_reason | Yes | ||
| groups_truncated | No | ||
| scanned_messages | Yes | ||
| written_messages | No | ||
| matches_by_partition | No |