Skip to main content
Glama

Server Details

LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.

If you are the author of this connector, you can claim ownership by verifying the domain or GitHub account it belongs to. Claimed connector authors can inspect health checks, view analytics, and manage their listing.
Status
Healthy
Uptime
100.0% over the last 24 hours
Last Tested
Transport
Streamable HTTP · MCP 2025-11-25
URL
Repository
ia-qa-com/mcp
GitHub Stars
0
Server Listing
IA-QA MCP server
ChatGPT
–
Claude
–

TDQS

A3.6/5.0

Scored across 39 tools

Disambiguation3/5

Many tools cluster around near-identical text-comparison work — similarity_score, embedding_similarity, run_semantic_tests, compare_responses, analyze_responses, consistency_check, response_quality_score — and the token trio (count_tokens, context_window_check, token_budget_calculator) plus the validation trio (llm_format_check, llm_output_validator, llm_json_schema_check) overlap heavily. The descriptions do a lot of cross-referencing to draw boundaries (e.g. 'A 2-way head-to-head with structural JSON diff is compare_responses'), which rescues the set from worse, but an agent must read carefully to pick correctly.

Naming Consistency4/5

Every name is snake_case with clear domain vocabulary, no camelCase or casing drift. The verb ordering is mixed (verb_noun like run_semantic_tests, compare_models vs noun_verb like model_info, ab_test_report, skill_interview), but names remain readable and predictable within their cluster.

Tool Count2/5

39 tools is well past the comfortable range and the surface spans eval contracts, RAG retrieval, model catalogs, token budgeting, CI generation, skill testing, tool discovery, and self-rating. The breadth is real, but several near-duplicate capabilities (three similarity scorers, three token/cost tools) mean not every tool clearly earns its place.

Completeness4/5

For an LLM/RAG QA domain the lifecycle is largely covered: test generation (prompt_test_suite, needle_haystack_generate, generate_eval_yaml), execution (run_eval_contract, run_vlm_test_suite, llm_generate), scoring (similarity, hallucination, metamorphic, consistency), RAG retrieval evaluation (bm25_score, rag_relevance_rank, rerank_evaluate), cost/context (estimate_llm_cost, model_info), and CI gating (generate_ci_workflow). Minor gaps: no golden-dataset/dataset management or result persistence, and no real embedding-generation tool despite referring to one.

Available Tools

39 tools
ab_test_reportA
Read-onlyIdempotent
Inspect

Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: winner names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.

Inputs: variant_a (required), variant_b (required). Returns: max, min, mean, count, +7 more

analyze_responsesA
Read-onlyIdempotent
Inspect

Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a reference (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). A 2-way head-to-head with structural JSON diff is compare_responses.

Inputs: responses (required), reference. Returns: count, summary, consensus, reference_ranking

bm25_scoreA
Read-onlyIdempotent
Inspect

Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.

Inputs: query (required), documents (required), b, k1, +1 more. Returns: b, k1, index, query, +6 more

build_rag_promptA
Read-onlyIdempotent
Inspect

Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.

Inputs: query (required), chunks (required), language, cite_sources, +2 more. Returns: prompt, system_prompt, chunks_included, included_chunks, +3 more

compare_modelsA
Read-onlyIdempotent
Inspect

Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a "best model" — run your cases on each to choose.

Inputs: models (required). Returns: rows, cheapest, prices_as_of, largest_context, +1 more

compare_responsesA
Read-onlyIdempotent
Inspect

Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a reference (ground truth) is given, scores each output against it and picks the closer one. If model + api_key are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.

Inputs: response_a (required), response_b (required), task, model, +5 more. Returns: judge, labelA, labelB, metrics, +2 more

consistency_checkA
Read-onlyIdempotent
Inspect

Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. True semantic consistency: run_semantic_tests in embedding mode. Essential for determinism testing.

Inputs: responses (required), check_facts. Returns: verdict, fact_drift, avg_similarity, response_count, +3 more

context_window_checkA
Read-onlyIdempotent
Inspect

Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.

Inputs: model (required), messages (required), max_output_tokens. Returns: fits, role, chars, index, +11 more

conversation_analyzeA
Read-onlyIdempotent
Inspect

Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.

Inputs: messages (required). Returns: turn_count, repetitions, topic_drift, user_messages, +5 more

cot_analyzerA
Read-onlyIdempotent
Inspect

Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.

Inputs: reasoning (required), expected_conclusion. Returns: markers, step_count, total_chars, total_lines, +5 more

count_tokensA
Read-onlyIdempotent
Inspect

Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Shows whether a text fits the context window of a model and what it costs. Returns token estimate, character count, and word count.

Inputs: input (required). Returns: chars, words, tokens_estimate

embedding_similarityA
Read-onlyIdempotent
Inspect

Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: true vector embeddings are run_semantic_tests with mode="embeddings" and an OpenAI key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.

Inputs: batch, text_a, text_b, methods. Returns: mode, count, scores, text_a, +2 more

estimate_llm_costA
Read-onlyIdempotent
Inspect

Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models.

Inputs: model (required), input_tokens (required), output_tokens. Returns: model, rates, input_tokens, output_tokens, +3 more

extract_json_from_textA
Read-onlyIdempotent
Inspect

Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON.

Inputs: input (required). Returns: json, source

find_toolA
Read-onlyIdempotent
Inspect

Search available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, dev, encoding, eval, integrations, llm, meta, qa, rag, security, text, web.

Inputs: query (required), category, max_results, with_schema. Returns: hint, tool, count, query, +5 more

generate_ci_workflowA
Read-onlyIdempotent
Inspect

Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails the build on their verdict, "selector_drift" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM. Returns the YAML, the secrets to create and the next steps. Pair with generate_eval_yaml for the contract.

Inputs: cron, gate, provider, triggers, +8 more. Returns: gate, path, yaml, notes, +2 more

generate_eval_yamlA
Read-only
Inspect

Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.

Inputs: description (required), task_type, system_prompt, scenario_count. Returns: yaml, task_type, model_used, scenario_count

get_testing_guidelinesA
Read-onlyIdempotent
Inspect

Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.

Inputs: topic. Returns: tip, topic, usage, keywords, +2 more

hallucination_checkA
Read-onlyIdempotent
Inspect

Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". Entailment: run_semantic_tests (NLI/embedding) or a calibrated judge.

Inputs: cases, answer, strict, context. Returns: detail, message, numbers, overlap, +14 more

list_llm_modelsA
Read-onlyIdempotent
Inspect

List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.

Inputs: provider. Returns: total, filter, models, providers

llm_format_checkA
Read-onlyIdempotent
Inspect

Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.

Inputs: output (required), expected_format (required), regex_pattern. Returns: valid, checks, failed, passed, +2 more

llm_generateA
Read-only
Inspect

Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited.

Inputs: prompt (required), model, system, max_tokens, +1 more. Returns: model, usage, content, provider, +1 more

llm_json_schema_checkA
Read-onlyIdempotent
Inspect

Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.

Inputs: output (required), schema (required). Returns: valid, errors, error_count, parse_error, +1 more

llm_output_validatorA
Read-onlyIdempotent
Inspect

Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.

Inputs: output (required), max_length, min_length, check_safety, +5 more. Returns: total, checks, failed, passed, +1 more

metamorphic_checkA
Read-onlyIdempotent
Inspect

Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.

Inputs: base (required), variants (required), mode, api_key, +3 more. Returns: mode, summary, verdict, weakest, +4 more

model_infoA
Read-onlyIdempotent
Inspect

Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.

Inputs: model (required). Returns: model, pricing_per_1k

needle_haystack_generateA
Read-only
Inspect

Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.

Inputs: needle (required), question (required), tokens, position. Returns: needle, haystack, position, question, +3 more

prompt_test_suiteA
Read-onlyIdempotent
Inspect

Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.

Inputs: user_prompt (required), system_prompt (required), max_tokens, temperature, +6 more. Returns: rubric, categories, total_tests, instructions, +1 more

rag_relevance_rankA
Read-onlyIdempotent
Inspect

Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.

Inputs: query (required), chunks (required), top_k. Returns: rank, index, query, score, +5 more

rate_toolA
Destructive
Inspect

Give honest usage feedback on an IA-QA MCP tool. Give a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Maintainers read ratings, one vote per agent and tool a month: comments and per-tool scores are not published, only the overall average and vote count. Comments are screened: one carrying a credential is not stored, personal data becomes placeholders — describe what happened, not what a model should do. Skip routine successes: signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })

Inputs: score (required), tool_name (required), comment. Returns: ok, score, comment, message, +4 more

rerank_evaluateA
Read-only
Inspect

Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels (INCONCLUSIVE without them). Modes: (1) give every passage the score YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).

Inputs: query (required), passages (required), top_k, api_key, +1 more. Returns: k, mode, model, ranked, +7 more

response_quality_scoreA
Read-onlyIdempotent
Inspect

Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). Meaning against a reference answer: run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.

Inputs: question (required), response (required), max_length, expected_keywords. Returns: grade, stats, reason, signals, +4 more

run_eval_contractA
Read-only
Inspect

Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model.

Inputs: api_keys, overrides, contract_path, validate_only, +1 more. Returns: valid, issues, summary, metadata, +3 more

run_semantic_testsA
Read-onlyIdempotent
Inspect

Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.

Inputs: cases (required), mode, api_key, thresholds, +1 more. Returns: mode, cases, summary, verdict, +1 more

run_vlm_test_suiteA
Read-only
Inspect

Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, overall PASS/WARNING/FAIL against threshold (N/A when no assertion could run), and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.

Inputs: model (required), api_key (required), test_cases (required), image_url, +4 more. Returns: model, total, passed, overall, +5 more

similarity_scoreA
Read-onlyIdempotent
Inspect

Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.

Inputs: batch, metrics, reference, threshold, +1 more. Returns: f1, mode, count, recall, +2 more

skill_interviewB
Read-onlyIdempotent
Inspect

Help a person write a SKILL.md they can test. No answers: the questions (why + example), written for the person; invented answers make the result meaningless. With answers: remarks per answer (no model), the SKILL.md, scenarios for test_skill, a starter .ia-eval.yaml.

Inputs: name, answers. Returns: object

test_skillB
Read-only
Inspect

Test a SKILL.md. No model, every call: lint (gate on lint.valid) and review (description lacks what/when, words shared with other_skills, leftovers, no done-check, missing files; why + fix each). Unless lint_only, a model picks a skill per request from names+descriptions: trigger_accuracy PASS ≥80, FIX ≥50, else BLOCK; your scenarios = gate, generated = signal; unanswered = INCONCLUSIVE. Server key 10 runs/day.

Inputs: skill_md (required), files, model, api_key, +5 more. Returns: lint, review, verdict

token_budget_calculatorA
Read-onlyIdempotent
Inspect

Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds the model's context window (any model from list_llm_models).

Inputs: model (required), context, user_input, system_prompt, +4 more. Returns: model, warnings, breakdown, context_window, +3 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Provides advanced evaluation tools for assessing AI safety, alignment, and performance of LLM outputs. Enables programmatic evaluation of quality, safety metrics like toxicity and PII detection, and operational metrics including carbon footprint and cost estimation.
      4
      Apache 2.0
    • A
      license
      A
      quality
      D
      maintenance
      Enables version-controlled golden dataset management and semantic evaluation for RAG/LLM pipelines using TF-IDF cosine similarity, without requiring an LLM API key.
      9
      40 PyPI
      MIT
    Try in Browser

    Glama MCP Gateway

    Add one secure layer between your agents and this server.