Skip to main content
Glama
nerdit-ai

Nerdit

Official
by nerdit-ai
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

serve_model

Launch a local AI model as an OpenAI-compatible API endpoint so apps can connect through the OpenAI SDK; setup runs asynchronously—poll status until running.

Instructions

Serve a local AI model as an OpenAI-compatible endpoint.

    Serving is asynchronous: the call returns immediately with the
    model row in ``building`` while the daemon pulls the image and weights
    off-tick; poll ``list_models`` (or ``get_service``) until it is
    ``running``. Lifecycle (stop / restart / delete) goes through the
    existing service tools (``stop_service`` / ``restart_service`` /
    ``remove_service``).

    An app connects to this model by declaring an ``[ai.<name>]`` binding
    (provider ``ollama``, model = this model); at each launch it receives
    ``NERDIT_AI_<NAME>_URL/_KEY/_MODEL``, and the ``default`` binding also
    sets ``OPENAI_BASE_URL/OPENAI_API_KEY/OPENAI_MODEL`` so an unmodified
    OpenAI SDK reaches this endpoint with no code change.

Input Schema

NameRequiredDescriptionDefault
gpusNoGPUs to reserve for the model server; 0 (the default) is CPU inference, and allocated GPUs are shared with other workloads. The ``vllm`` backend needs at least 1 and refuses 0 with ``model.gpu_required``.
nameNoService name for the model (lowercase DNS label). Omit and the name is the backend-prefixed, sanitized model reference (``llama3.1:8b`` on ollama → ``ollama-llama3-1-8b``).
modelYesModel reference to pull and serve, e.g. ``llama3.1:8b`` (ollama) or a Hugging Face repo id (vLLM): letters, digits, ``. - _ /`` with an optional ``:tag``.
backendNoServing backend: ``ollama`` (CPU or GPU) or ``vllm`` (GPU-only, one model per server, weights from Hugging Face). Omit to use the daemon's configured default; an unknown name is refused with ``model.unknown_backend``.
max_model_lenNovLLM only: context-window cap in tokens (``--max-model-len``), 256 to 262144. Omit and the daemon decides: an operator flag in ``[models].vllm_extra_args``, else 4096 on a GPU under 16 GB VRAM, else vLLM's own default. Any other backend refuses it with ``model.backend_param``.
idempotency_keyNoCaller-chosen key: a retry with the same key replays the first result instead of acting twice. Omit and a fresh key is minted per call, so each call acts.
gpu_memory_utilizationNovLLM only: fraction of VRAM the engine may claim (``--gpu-memory-utilization``), 0.1 to 0.95. Omit and the daemon decides: an operator flag in ``[models].vllm_extra_args``, else 0.85 on a GPU under 16 GB VRAM, else vLLM's own default. Any other backend refuses it with ``model.backend_param``.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden and does so well: it discloses asynchronous immediate-return semantics, the 'building' intermediate state, background image/weight pulling, and where lifecycle actions live. It also explains how apps bind to the endpoint and what env vars result.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core purpose, then async behavior, then app binding. Efficient overall, though the [ai.<name>] binding paragraph is somewhat tangential to invoking the tool and could be trimmed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-param mutation tool with no annotations and no output schema, the description covers the async lifecycle, polling target, and error behavior well enough to call correctly. It does not describe the shape of the returned 'model row', though the schema and error codes compensate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter (gpus, backend, max_model_len, idempotency_key, etc.) is already documented with constraints and error codes. The description adds no parameter-level meaning beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Serve a local AI model as an OpenAI-compatible endpoint.' It is distinguishable from siblings like list_models and get_service, and it explicitly routes lifecycle operations to stop_service/restart_service/remove_service.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear operational context: serving is asynchronous, poll list_models or get_service until running, and use the existing service tools for lifecycle. It does not, however, distinguish this tool from the sibling 'serve' tool, leaving one alternative unaddressed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools