IA-QA Eval & RAG — 38 tools
Server Details
LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.
- Status
- Healthy
- Uptime
- 100.0% over the last 24 hours
- Last Tested
- Transport
- Streamable HTTP · MCP 2025-11-25
- URL
- Repository
- ia-qa-com/mcp
- GitHub Stars
- 0
- Server Listing
- IA-QA MCP server
- ChatGPT
- –
- Claude
- –
TDQS
Scored across 39 tools
Many tools cluster around near-identical text-comparison work — similarity_score, embedding_similarity, run_semantic_tests, compare_responses, analyze_responses, consistency_check, response_quality_score — and the token trio (count_tokens, context_window_check, token_budget_calculator) plus the validation trio (llm_format_check, llm_output_validator, llm_json_schema_check) overlap heavily. The descriptions do a lot of cross-referencing to draw boundaries (e.g. 'A 2-way head-to-head with structural JSON diff is compare_responses'), which rescues the set from worse, but an agent must read carefully to pick correctly.
Every name is snake_case with clear domain vocabulary, no camelCase or casing drift. The verb ordering is mixed (verb_noun like run_semantic_tests, compare_models vs noun_verb like model_info, ab_test_report, skill_interview), but names remain readable and predictable within their cluster.
39 tools is well past the comfortable range and the surface spans eval contracts, RAG retrieval, model catalogs, token budgeting, CI generation, skill testing, tool discovery, and self-rating. The breadth is real, but several near-duplicate capabilities (three similarity scorers, three token/cost tools) mean not every tool clearly earns its place.
For an LLM/RAG QA domain the lifecycle is largely covered: test generation (prompt_test_suite, needle_haystack_generate, generate_eval_yaml), execution (run_eval_contract, run_vlm_test_suite, llm_generate), scoring (similarity, hallucination, metamorphic, consistency), RAG retrieval evaluation (bm25_score, rag_relevance_rank, rerank_evaluate), cost/context (estimate_llm_cost, model_info), and CI gating (generate_ci_workflow). Minor gaps: no golden-dataset/dataset management or result persistence, and no real embedding-generation tool despite referring to one.
Available Tools
39 toolsab_test_reportARead-onlyIdempotentInspect
Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: winner names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.
Inputs: variant_a (required), variant_b (required). Returns: max, min, mean, count, +7 more
analyze_responsesARead-onlyIdempotentInspect
Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a reference (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). A 2-way head-to-head with structural JSON diff is compare_responses.
Inputs: responses (required), reference. Returns: count, summary, consensus, reference_ranking
bm25_scoreARead-onlyIdempotentInspect
Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.
Inputs: query (required), documents (required), b, k1, +1 more. Returns: b, k1, index, query, +6 more
build_rag_promptARead-onlyIdempotentInspect
Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.
Inputs: query (required), chunks (required), language, cite_sources, +2 more. Returns: prompt, system_prompt, chunks_included, included_chunks, +3 more
compare_modelsARead-onlyIdempotentInspect
Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a "best model" — run your cases on each to choose.
Inputs: models (required). Returns: rows, cheapest, prices_as_of, largest_context, +1 more
compare_responsesARead-onlyIdempotentInspect
Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a reference (ground truth) is given, scores each output against it and picks the closer one. If model + api_key are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.
Inputs: response_a (required), response_b (required), task, model, +5 more. Returns: judge, labelA, labelB, metrics, +2 more
consistency_checkARead-onlyIdempotentInspect
Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. True semantic consistency: run_semantic_tests in embedding mode. Essential for determinism testing.
Inputs: responses (required), check_facts. Returns: verdict, fact_drift, avg_similarity, response_count, +3 more
context_window_checkARead-onlyIdempotentInspect
Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.
Inputs: model (required), messages (required), max_output_tokens. Returns: fits, role, chars, index, +11 more
conversation_analyzeARead-onlyIdempotentInspect
Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.
Inputs: messages (required). Returns: turn_count, repetitions, topic_drift, user_messages, +5 more
cot_analyzerARead-onlyIdempotentInspect
Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.
Inputs: reasoning (required), expected_conclusion. Returns: markers, step_count, total_chars, total_lines, +5 more
count_tokensARead-onlyIdempotentInspect
Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Shows whether a text fits the context window of a model and what it costs. Returns token estimate, character count, and word count.
Inputs: input (required). Returns: chars, words, tokens_estimate
embedding_similarityARead-onlyIdempotentInspect
Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: true vector embeddings are run_semantic_tests with mode="embeddings" and an OpenAI key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.
Inputs: batch, text_a, text_b, methods. Returns: mode, count, scores, text_a, +2 more
estimate_llm_costARead-onlyIdempotentInspect
Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models.
Inputs: model (required), input_tokens (required), output_tokens. Returns: model, rates, input_tokens, output_tokens, +3 more
extract_json_from_textARead-onlyIdempotentInspect
Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON.
Inputs: input (required). Returns: json, source
find_toolARead-onlyIdempotentInspect
Search available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, dev, encoding, eval, integrations, llm, meta, qa, rag, security, text, web.
Inputs: query (required), category, max_results, with_schema. Returns: hint, tool, count, query, +5 more
generate_ci_workflowARead-onlyIdempotentInspect
Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails the build on their verdict, "selector_drift" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM. Returns the YAML, the secrets to create and the next steps. Pair with generate_eval_yaml for the contract.
Inputs: cron, gate, provider, triggers, +8 more. Returns: gate, path, yaml, notes, +2 more
generate_eval_yamlARead-onlyInspect
Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.
Inputs: description (required), task_type, system_prompt, scenario_count. Returns: yaml, task_type, model_used, scenario_count
get_testing_guidelinesARead-onlyIdempotentInspect
Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.
Inputs: topic. Returns: tip, topic, usage, keywords, +2 more
hallucination_checkARead-onlyIdempotentInspect
Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". Entailment: run_semantic_tests (NLI/embedding) or a calibrated judge.
Inputs: cases, answer, strict, context. Returns: detail, message, numbers, overlap, +14 more
list_llm_modelsARead-onlyIdempotentInspect
List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.
Inputs: provider. Returns: total, filter, models, providers
llm_format_checkARead-onlyIdempotentInspect
Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.
Inputs: output (required), expected_format (required), regex_pattern. Returns: valid, checks, failed, passed, +2 more
llm_generateARead-onlyInspect
Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited.
Inputs: prompt (required), model, system, max_tokens, +1 more. Returns: model, usage, content, provider, +1 more
llm_json_schema_checkARead-onlyIdempotentInspect
Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.
Inputs: output (required), schema (required). Returns: valid, errors, error_count, parse_error, +1 more
llm_output_validatorARead-onlyIdempotentInspect
Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.
Inputs: output (required), max_length, min_length, check_safety, +5 more. Returns: total, checks, failed, passed, +1 more
metamorphic_checkARead-onlyIdempotentInspect
Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.
Inputs: base (required), variants (required), mode, api_key, +3 more. Returns: mode, summary, verdict, weakest, +4 more
model_infoARead-onlyIdempotentInspect
Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.
Inputs: model (required). Returns: model, pricing_per_1k
needle_haystack_generateARead-onlyInspect
Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.
Inputs: needle (required), question (required), tokens, position. Returns: needle, haystack, position, question, +3 more
prompt_test_suiteARead-onlyIdempotentInspect
Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.
Inputs: user_prompt (required), system_prompt (required), max_tokens, temperature, +6 more. Returns: rubric, categories, total_tests, instructions, +1 more
rag_relevance_rankARead-onlyIdempotentInspect
Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.
Inputs: query (required), chunks (required), top_k. Returns: rank, index, query, score, +5 more
rate_toolADestructiveInspect
Give honest usage feedback on an IA-QA MCP tool. Give a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Maintainers read ratings, one vote per agent and tool a month: comments and per-tool scores are not published, only the overall average and vote count. Comments are screened: one carrying a credential is not stored, personal data becomes placeholders — describe what happened, not what a model should do. Skip routine successes: signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })
Inputs: score (required), tool_name (required), comment. Returns: ok, score, comment, message, +4 more
rerank_evaluateARead-onlyInspect
Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels (INCONCLUSIVE without them). Modes: (1) give every passage the score YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).
Inputs: query (required), passages (required), top_k, api_key, +1 more. Returns: k, mode, model, ranked, +7 more
response_quality_scoreARead-onlyIdempotentInspect
Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). Meaning against a reference answer: run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.
Inputs: question (required), response (required), max_length, expected_keywords. Returns: grade, stats, reason, signals, +4 more
run_eval_contractARead-onlyInspect
Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model.
Inputs: api_keys, overrides, contract_path, validate_only, +1 more. Returns: valid, issues, summary, metadata, +3 more
run_semantic_testsARead-onlyIdempotentInspect
Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.
Inputs: cases (required), mode, api_key, thresholds, +1 more. Returns: mode, cases, summary, verdict, +1 more
run_vlm_test_suiteARead-onlyInspect
Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, overall PASS/WARNING/FAIL against threshold (N/A when no assertion could run), and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.
Inputs: model (required), api_key (required), test_cases (required), image_url, +4 more. Returns: model, total, passed, overall, +5 more
similarity_scoreARead-onlyIdempotentInspect
Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.
Inputs: batch, metrics, reference, threshold, +1 more. Returns: f1, mode, count, recall, +2 more
skill_interviewBRead-onlyIdempotentInspect
Help a person write a SKILL.md they can test. No answers: the questions (why + example), written for the person; invented answers make the result meaningless. With answers: remarks per answer (no model), the SKILL.md, scenarios for test_skill, a starter .ia-eval.yaml.
Inputs: name, answers. Returns: object
test_skillBRead-onlyInspect
Test a SKILL.md. No model, every call: lint (gate on lint.valid) and review (description lacks what/when, words shared with other_skills, leftovers, no done-check, missing files; why + fix each). Unless lint_only, a model picks a skill per request from names+descriptions: trigger_accuracy PASS ≥80, FIX ≥50, else BLOCK; your scenarios = gate, generated = signal; unanswered = INCONCLUSIVE. Server key 10 runs/day.
Inputs: skill_md (required), files, model, api_key, +5 more. Returns: lint, review, verdict
token_budget_calculatorARead-onlyIdempotentInspect
Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds the model's context window (any model from list_llm_models).
Inputs: model (required), context, user_input, system_prompt, +4 more. Returns: model, warnings, breakdown, context_window, +3 more
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
LLM evaluation: compare models on cost and quality, score outputs, run benchmarks, analyse agents.
Free, no signup: AI crawler check, llms.txt validator, ChatGPT/Perplexity/Gemini citation check.
Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.
AI visibility across 8 LLM platforms. 30 tools, most need no API key. EU-hosted, free.
Related MCP Servers
AlicenseNot gradedqualityCmaintenanceDetect fabrication and hallucination in any LLM output. Score responses from GPT-4o, Claude, Gemini, Llama and 30+ models. Free tier included.MIT
Trustwise MCP Serverofficial
AlicenseNot gradedqualityDmaintenanceProvides advanced evaluation tools for assessing AI safety, alignment, and performance of LLM outputs. Enables programmatic evaluation of quality, safety metrics like toxicity and PII detection, and operational metrics including carbon footprint and cost estimation.4Apache 2.0
io.github.HZYAI/ragscoreofficial
AlicenseNot gradedqualityDmaintenanceEnables generating QA datasets from documents and evaluating RAG systems with detailed metrics, failure diagnosis, and visual reports through natural language.22 PyPI15Apache 2.0- AlicenseAqualityDmaintenanceEnables version-controlled golden dataset management and semantic evaluation for RAG/LLM pipelines using TF-IDF cosine similarity, without requiring an LLM API key.940 PyPIMIT
Glama MCP Gateway
Add one secure layer between your agents and this server.