de en es ja ko ru zh

llmprobe

Name: llmprobe
Author: Jwrede

by Jwrede

Overview Schema Source Related Servers Discussions Admin

Local

llmprobe

Зондирование конечных точек API LLM. Измерение TTFT, задержки, пропускной способности. Один бинарный файл, ноль SDK.

License: MIT llmprobe MCP server

llmprobe — это CLI-инструмент, который зондирует конечные точки API LLM и измеряет метрики, важные для надежности в продакшене: время до первого токена (TTFT), общую задержку, пропускную способность генерации (токенов/сек) и частоту ошибок.

Используйте его для разовой проверки работоспособности, непрерывного мониторинга или в качестве CI-шлюза, который блокирует развертывание, если ваш провайдер LLM работает нестабильно.

demo

Быстрый старт

Скачайте предварительно собранный бинарный файл из последнего релиза (Linux, macOS, Windows; amd64 и arm64).

Или установите из исходного кода:

go install github.com/Jwrede/llmprobe@latest

Создайте probes.yml (или скопируйте прилагаемый пример):

providers:
  - name: openai
    api_key: ${OPENAI_API_KEY}
    models:
      - name: gpt-4o
        thresholds:
          max_ttft: 2s
      - name: gpt-4o-mini
        thresholds:
          max_ttft: 500ms

  - name: anthropic
    api_key: ${ANTHROPIC_API_KEY}
    models:
      - name: claude-sonnet-4-20250514
        thresholds:
          max_ttft: 1s

Запустите зонд:

$ llmprobe probe

Provider   Model                    Status    TTFT    Latency  Tok/s  Tokens  Error
--------   -----                    ------    ----    -------  -----  ------  -----
openai     gpt-4o                   healthy   312ms   2100ms   68.4   42
openai     gpt-4o-mini              healthy   98ms    814ms    112.3  56
anthropic  claude-sonnet-4-20250514 healthy   420ms   2831ms   52.1   38
azure      gpt-4o                   healthy   289ms   1950ms   71.2   44
bedrock    anthropic.claude-3-5...  degraded  1820ms  4510ms   28.1   38

4 healthy, 1 degraded, 0 errors

Related MCP server: LLM API Benchmark MCP Server

Что он измеряет

Метрика	Что это значит
TTFT	Время от отправки запроса до первого токена контента. Это то, что пользователи ощущают как «задержку» перед началом потоковой передачи ответа.
Задержка	Общее время от запроса до закрытия потока.
Ток/с	Пропускная способность генерации: количество токенов, создаваемых в секунду после первого токена. Рассчитывается как `token_count / (latency - ttft)`.
Токены	Общее количество выходных токенов. Предпочтение отдается метаданным использования провайдера, если они доступны, в противном случае используется подсчет событий SSE.
Статус	`healthy` (здоров), если все пороги пройдены, `degraded` (деградировал), если превышен любой порог, `error` (ошибка), если запрос не удался.

Команды

`llmprobe probe`

Разовая проверка работоспособности. Зондирует все настроенные конечные точки и выводит результаты.

llmprobe probe                        # table output
llmprobe probe -f json                # JSON output
llmprobe probe --fail-on degraded     # exit 1 if any endpoint is degraded
llmprobe probe -c custom-config.yml   # custom config path

Коды выхода для CI:

`--fail-on`	Выход 0	Выход 1
`error` (по умолчанию)	healthy или degraded	любая ошибка
`degraded`	только healthy	degraded или error
`none`	всегда	никогда

`llmprobe watch`

Непрерывный мониторинг. Зондирует все конечные точки с заданным интервалом и выводит строку сводки для каждой итерации.

llmprobe watch                          # default 60s interval
llmprobe watch --interval 30s           # custom interval
llmprobe watch --tui                    # live terminal dashboard with TTFT chart
llmprobe watch --tui --load data.jsonl  # load historical data into the dashboard
llmprobe watch -f json                  # JSONL output (one line per result)

Флаг --tui запускает интерактивную панель мониторинга в терминале с графиком TTFT, цветовой легендой и таблицей статистики. Используйте --load для импорта исторических данных JSONL (из llmprobe watch -f json > data.jsonl).

llmprobe

$ llmprobe watch --interval 30s

Watching 4 endpoints every 30s (Ctrl+C to stop)

[14:01:02] All 4 endpoints healthy.
[14:01:32] All 4 endpoints healthy.
[14:02:02] 3 healthy, 1 degraded, 0 errors. DEGRADED: openai/gpt-4o (TTFT 1820ms)
[14:02:32] All 4 endpoints healthy.

Интеграция с CI

Используйте llmprobe probe в качестве шлюза перед развертыванием:

# .github/workflows/deploy.yml
- name: Check LLM providers
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: |
    go install github.com/Jwrede/llmprobe@latest
    llmprobe probe --fail-on degraded

Это заблокирует развертывание, если какой-либо провайдер LLM в данный момент демонстрирует сниженную производительность.

MCP-сервер

llmprobe MCP server

llmprobe включает встроенный сервер Model Context Protocol, позволяющий Claude Code и другим хостам MCP проверять работоспособность API LLM непосредственно из рабочего процесса агента.

Запуск сервера

llmprobe mcp

Это запускает MCP-сервер через stdio.

Регистрация в Claude Code

claude mcp add --transport stdio llmprobe -- llmprobe mcp

После регистрации Claude Code может вызывать инструменты llmprobe во время любого разговора.

Доступные инструменты

Инструмент	Описание
`probe_all`	Зондирует все настроенные конечные точки из `probes.yml`. Возвращает TTFT, задержку, пропускную способность и статус работоспособности для каждой модели. Принимает необязательный параметр `config` для указания пути к пользовательскому конфигу.
`probe_model`	Зондирует одну модель без файла конфигурации. Требует `provider` (openai, anthropic, google, azure, bedrock), `model` (идентификатор модели) и `api_key_env` (переменная окружения, содержащая API-ключ).
`list_providers`	Выводит список всех провайдеров и моделей в файле конфигурации с их порогами. Используйте это для обнаружения доступных моделей перед зондированием.
`get_config`	Возвращает полную разобранную конфигурацию, включая значения по умолчанию, провайдеров, модели и пороги.

Пример использования: Агент вызывает list_providers, чтобы увидеть, какие модели настроены, а затем probe_all, чтобы убедиться в их работоспособности перед развертыванием изменений.

Конфигурация

defaults:
  prompt: "Hello"                                # probe prompt
  max_tokens: 20                                 # max output tokens
  timeout: 30s                                   # per-probe timeout
  concurrency: 5                                 # max parallel probes

providers:
  - name: openai                    # openai, anthropic, google, azure, bedrock
    api_key: ${OPENAI_API_KEY}      # env var expansion
    base_url: https://custom.api    # optional, override endpoint
    models:
      - name: gpt-4o
        prompt: "Say hello."        # override default prompt
        max_tokens: 10              # override default max_tokens
        thresholds:
          max_ttft: 2s              # alert if TTFT exceeds this
          max_latency: 10s          # alert if total latency exceeds this
          min_tokens_per_sec: 20    # alert if throughput drops below this

  - name: azure
    api_key: ${AZURE_OPENAI_API_KEY}
    base_url: https://your-resource.openai.azure.com
    api_version: "2024-10-21"       # optional, defaults to 2024-10-21
    models:
      - name: gpt-4o               # deployment name

  - name: bedrock
    access_key: ${AWS_ACCESS_KEY_ID}
    secret_key: ${AWS_SECRET_ACCESS_KEY}
    region: us-east-1
    models:
      - name: anthropic.claude-3-5-sonnet-20241022-v2:0

API-ключи и учетные данные AWS поддерживают синтаксис ${ENV_VAR}. Разворачиваются только поля учетных данных, поэтому ссылки на переменные окружения в промптах или именах моделей остаются без изменений.

Провайдеры, совместимые с OpenAI

Многие провайдеры (Groq, Together AI, Fireworks, DeepSeek, Mistral, OpenRouter, Ollama, vLLM) предоставляют API, совместимый с OpenAI. Они работают «из коробки» при установке base_url:

providers:
  # Groq
  - name: openai
    api_key: ${GROQ_API_KEY}
    base_url: https://api.groq.com/openai
    models:
      - name: llama-3.3-70b-versatile

  # DeepSeek
  - name: openai
    api_key: ${DEEPSEEK_API_KEY}
    base_url: https://api.deepseek.com
    models:
      - name: deepseek-chat

  # Together AI
  - name: openai
    api_key: ${TOGETHER_API_KEY}
    base_url: https://api.together.xyz
    models:
      - name: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo

  # Local Ollama
  - name: openai
    api_key: unused
    base_url: http://localhost:11434/v1
    models:
      - name: llama3.2

Архитектура

probes.yml
  -> Config loader (YAML + env var expansion)
    -> Probe engine (concurrent goroutines per provider/model)
      -> Provider clients (raw HTTP + SSE parsing, no SDKs)
        -> Results (TTFT, latency, tokens/sec, status)
          -> Output (table, JSON, JSONL)

Каждый клиент провайдера — это тонкая HTTP-обертка, которая отправляет потоковый запрос и анализирует ответ. SDK LLM не импортируются. Парсер SSE обрабатывает как события только с данными (OpenAI, Google), так и именованные события (Anthropic). Клиент Bedrock реализует подпись SigV4 и парсинг бинарного потока событий AWS с нуля.

TTFT измеряется с момента отправки HTTP-запроса до первого события, содержащего фактический текстовый контент (не назначения ролей или метаданные).

Провайдеры

Провайдер	Конечная точка	Авторизация	Формат потока
OpenAI	`/v1/chat/completions`	`Authorization: Bearer`	SSE, `[DONE]` sentinel
Anthropic	`/v1/messages`	`x-api-key` header	named-event SSE
Google	`/v1beta/models/{model}:streamGenerateContent?alt=sse`	`key` query param	SSE
Azure OpenAI	`/openai/deployments/{model}/chat/completions`	`api-key` header	SSE, `[DONE]` sentinel
AWS Bedrock	`/model/{model}/converse-stream`	SigV4	AWS binary event stream
OpenAI-compat	`/v1/chat/completions` (custom base_url)	`Authorization: Bearer`	SSE

Совместимость с OpenAI охватывает: Groq, Together AI, Fireworks, DeepSeek, Mistral, OpenRouter, Ollama, vLLM и любую конечную точку, поддерживающую API чат-комплитов OpenAI.

Живой бенчмарк

llm-bench использует llmprobe для проведения непрерывного публичного бенчмарка основных API LLM. Результаты публикуются в виде открытого набора данных JSONL и интерактивной панели мониторинга в терминале на bench.jonathanwrede.de.

Дорожная карта

Отслеживание базовых показателей: хранение скользящих перцентилей, оповещение, когда текущий зонд превышает базовый уровень Nx
Экспорт метрик OpenTelemetry для интеграции с Grafana/Datadog
Конечная точка Prometheus /metrics
Валидация структурированного вывода: проверка корректности парсинга ответов в режиме JSON

Лицензия

MIT

Available Tools

4 tools

get_configA

Return the full parsed configuration including defaults, providers, models, and thresholds. Useful for understanding the current probe setup or debugging configuration issues.

ParametersJSON Schema

Name	Required	Description	Default
`config`	No	path to probes.yml config file

TDQS

A4.1/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Describes return contents but does not mention side effects, auth requirements, or rate limits. No annotations exist to supplement.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two efficient sentences, no redundancy, front-loaded with purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Adequately covers purpose, return content, and common use cases for a simple tool with one optional parameter and no output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema covers the single parameter with description. Description adds no extra meaning beyond 'full parsed configuration'; baseline 3 due to high coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states it returns the full parsed configuration, listing included elements (defaults, providers, models, thresholds). Distinct from siblings like list_providers or probe_all.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Indicates usefulness for understanding setup or debugging, implying context. Lacks explicit when-not-to-use or comparison to siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_providersA

List all providers and models defined in the config file. Returns provider names, model identifiers, and any configured thresholds. Use this to discover what models are available before probing.

ParametersJSON Schema

Name	Required	Description	Default
`config`	No	path to probes.yml config file

TDQS

A4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Implicitly a read operation, but with no annotations, the description should explicitly state it is read-only or disclose any side effects. It lacks explicit non-destructive guarantee.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences: first states what it does, second tells when to use it. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple single-parameter tool with no output schema, the description is sufficiently complete, covering purpose, returns, and usage context. Minor gap in behavioral transparency.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a single parameter described. The description adds no additional meaning beyond what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it lists providers and models from a config file, specifying the exact information returned (names, identifiers, thresholds). This distinguishes it from sibling tools like 'probe_all' or 'get_config'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly advises using it 'before probing', providing clear usage context. However, it does not contrast with siblings or specify when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

probe_allA

Probe all configured LLM API endpoints. Returns TTFT (ms), total latency (ms), throughput (tokens/sec), and health status for every model in the config file.

ParametersJSON Schema

Name	Required	Description	Default
`config`	No	path to probes.yml config file

TDQS

A3.9/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. It discloses return values and that it uses a config file, but does not mention side effects, error handling, or whether it's read-only. Adequate but not comprehensive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence that is concise, front-loaded, and contains no filler. Every word adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description adequately explains the return values. The single optional parameter is well-described. Sibling tool context implies complementarity with 'probe_model'. Complete for a probing tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a clear description for the 'config' parameter. The description adds context by explaining that the config file determines which endpoints are probed, enhancing the schema's meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Probe all configured LLM API endpoints') and specifies the exact metrics returned (TTFT, latency, throughput, health status). It distinguishes from sibling 'probe_model' which likely targets a single model.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives like 'probe_model' or 'get_config'. The description does not specify prerequisites or scenarios where this tool is appropriate.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

probe_modelA

Probe a single LLM model by provider and model name. Use this for ad-hoc checks without a config file. Returns TTFT (ms), total latency (ms), throughput (tokens/sec), and health status.

ParametersJSON Schema

Name	Required	Description
`provider`	Yes	provider name (openai, anthropic, google, azure, bedrock)
`model`	Yes	model identifier (e.g. gpt-4o, claude-sonnet-4-20250514)
`api_key_env`	Yes	environment variable name containing the API key
`base_url`	No	optional base URL for OpenAI-compatible endpoints (e.g. http://localhost:8000)
`label`	No	optional display name for the endpoint (e.g. vllm-local)

TDQS

A4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It discloses return values (TTFT, latency, throughput, health status) and states it probes a model, but fails to mention side effects (e.g., real API call) or safety properties (read-only vs. destructive). This is adequate but not comprehensive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, each carrying essential information: first sentence states purpose and parameters, second sentence clarifies usage context and return values. No redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 5-parameter tool with no output schema, the description adequately covers return values and usage context. It could mention that the tool makes a live API call, but overall completeness is high for a simple diagnostic tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description does not add parameter-level details beyond what the schema already provides; it only mentions 'provider and model name' generically.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Probe') and explicitly states the resource ('single LLM model by provider and model name'). It also distinguishes from siblings (probe_all, list_providers) by noting ad-hoc single-model use without a config file.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly recommends use for 'ad-hoc checks without a config file', implying when to use this tool. Sibling names (probe_all, list_providers, get_config) provide contrast, but no explicit exclusions or when-not-to-use guidance are given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

6 tool updatesv1.4.0May 29, 2026
- Removedcheck_model
- Addedget_config
- Addedlist_providers
- Removedprobe
- Addedprobe_all
- Addedprobe_model
2 tool updatesv0.1.0May 7, 2026
- First observedcheck_model
- First observedprobe

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.3/5.0

Scored 2026-05-29 21:15 across 4 tools

Disambiguation5/5

Each tool has a distinct purpose: configuration retrieval, provider listing, bulk probing, and single model probing. No overlap.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern (get_config, list_providers, probe_all, probe_model) with appropriate verbs.

Tool Count5/5

4 tools is well-scoped for the LLM probing domain, covering necessary operations without bloat.

Completeness5/5

The tool set covers all essential probe operations: viewing config, listing providers, probing all endpoints, and probing a single endpoint, with no obvious gaps.

Maintenance

ActivityInactive

ResponsivenessUnresponsive

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

LLM Latency Tracker A
dev.llmlatency
Measured latency, time to first token and uptime for ~45 AI inference APIs, by region.
Monitoring Observability
222
AI API Radar A
com.dropwatchhq
Live LLM API price + status radar across 11 providers, with public per-model price HISTORY.
AI & Machine Learning Monitoring
42
LLM Red-Team Scanner A
dev.workers.manhliemcn4euwlu.llmrt-companion
35-probe LLM/agent security red-team scan (injection, jailbreak, MCP abuse) with report.
Penetration Testing Security
41
traced-multi-llm-proxy
ai.getvda
Traced Multi-LLM Proxy
AI & Machine Learning Observability
1

Related MCP Servers

LLM MCP Bridge
Developer Tools Testing & QA Tools Code Execution
ramgeart
F
license
B
quality
D
maintenance
Provides a universal bridge to interact with any OpenAI-compatible LLM API (local or cloud), enabling model testing, benchmarking, quality evaluation, and chat operations with performance metrics.
Updated 10 months ago (2025-12-06 01:13 UTC)
8
-
LLM API Benchmark MCP Server
Testing & QA Tools Monitoring Developer Tools
Yoosu-L
A
license
Not graded
quality
Not graded
maintenance
Enables benchmarking of Large Language Model APIs by measuring performance metrics such as generation throughput, prompt throughput, and Time To First Token (TTFT) with configurable concurrency levels and parameters.
Updated a year ago (2025-06-26 05:01 UTC)
1
MIT
ppb-mcp
AI & Machine Learning Research & Data
paulplee
A
license
Not graded
quality
D
maintenance
Exposes queryable GPU inference benchmark data (quantization, throughput, VRAM, concurrent users) as tools for LLM clients.
Updated 4 months ago (2026-05-27 16:47 UTC)
MIT
qa-probeofficial
Testing & QA Tools API Testing
LS-SIEM-LLP
A
license
A
quality
C
maintenance
Probes your live API and classifies why each endpoint failed (root cause, evidence, and a calibrated confidence level), exposed over MCP so your AI assistant debugs from evidence instead of guessing. Works with FastAPI, Express, Next.js, tRPC, and GraphQL.
Updated 3 months ago (2026-06-25 08:01 UTC)
8
3 npm
2
Apache 2.0

llmprobe

Быстрый старт

Что он измеряет

Команды

llmprobe probe

llmprobe watch

Интеграция с CI

MCP-сервер

Запуск сервера

Регистрация в Claude Code

Доступные инструменты

Конфигурация

Провайдеры, совместимые с OpenAI

Архитектура

Провайдеры

Живой бенчмарк

Дорожная карта

Лицензия

Available Tools

Tool Schema Changelog

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

qa-probeofficial

`llmprobe probe`

`llmprobe watch`