Evaluation dataset check
wever_evaluation-dataset-checkAgent Evaluation: submit one evaluation dataset description: its name, size, source (production logs, written by experts, generated, a public benchmark or a mix), version and when it was published, the categories the agent must handle beside the categories covered with their example counts and whether hard and edge cases are included on purpose, how the reference labels were made (experts, trained annotators, crowd workers, a model, the agent’s own builders or no one) with the number of labelers, their agreement, a written guide and review of model labels, and the leakage controls: whether the set is held out from training and tuning, whether overlap was checked and found, whether prompt examples come from the set, whether anything was tuned against it, and the model’s training cutoff. The signed report says category by category whether the set covers what the agent must handle, whether the labels can be trusted and whether anything the agent learned from is in the test, and names every required category with no examples, no labels, examples shared with training or tuning data, prompt examples drawn from the set and tuning against the set, then every category under 30 examples, no hard cases, category counts above the dataset size, unreviewed model labels, labels by the agent’s builders, one labeler, agreement under 0.6, no guide, no split, no overlap check and a public benchmark published before the model’s training cutoff, with the fix for each. This reads the set an agent is scored on: whether a benchmark claim’s figures match the source it cites is wever_benchmark-evidence-check and whether a claim is supported by offered evidence is wever_claim-evidence-check. Verdict dataset_unfit, dataset_gaps, dataset_unverified or dataset_sound. POST /api/evaluation-dataset-check. Reads only the material in the call: no network request, fetch, money movement, message, sharing, configuration change or external write. Paid URL: 0.10 USDC per call on Base via x402 exact, with the shared 10 signed-wallet free calls per rolling 30 days. Bind X-Wever-Wallet-Proof to the exact JSON arguments and the tool route. Returns a report with an Ed25519 signed receipt. Contract: /api/agent-evaluation-contract. Human page: https://weverlabs.com/agent-evaluation/.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"dataset": "<dataset>"
},
"name": "wever_evaluation-dataset-check"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| label | No | Optional name recorded in the report. | |
| dataset | Yes | ||
| leakage | No | ||
| coverage | No | ||
| labeling | No |