noisefloor
noisefloor
Is this number real, or is it noise?
Zero dependencies. Standard library only. Works as a Python package or as an MCP server.
The problem
You ship a change and check the A/B test. Then you check again an hour later. Then tomorrow. You stop when it looks like a win.
That procedure lies to you, and not by a little:
400 A/A tests, both arms identical, peeked every 20 observations | False winners |
Two-proportion z-test | 40.2% |
noisefloor | 0.0% |
Both arms had the same 10% conversion rate, so every "winner" was false. A test tuned to be wrong 5% of the time was wrong 40% of the time, purely because someone looked more than once.
This isn't a discipline problem. Peeking is the rational thing to do when a bad variant is costing money. It's a maths problem, and it has a solution.
Reproduce the table with pytest tests/test_calibration.py.
Related MCP server: backtest-reality-check
Install
pip install noisefloorUse
from noisefloor import experiment
experiment.compare(a_successes=500, a_total=5000,
b_successes=750, b_total=5000)
# {'decided': True, 'winner': 'B',
# 'reading': "B wins: its interval sits entirely above A's",
# 'A': {'rate': 0.1, 'interval': [0.08366, 0.11809]},
# 'B': {'rate': 0.15, 'interval': [0.13049, 0.17101]}, ...}Run it after every single observation if you like. The guarantee holds at every sample size simultaneously, so stopping early, stopping late, or stopping because your manager walked past all cost you nothing.
The other four checks
from noisefloor import change, coverage, forecast, multiple
# Did this metric actually change? Catches collapses as well as spikes.
change.scan(daily_signups)
# {'state': 'changed', 'direction': 'down',
# 'reading': 'CHANGED (down), 8.5x its usual spread', ...}
# Did the metric move, or did my sample size move?
coverage.check(conversion_rate, sessions_per_day)
# {'verdict': 'SAMPLING_ARTIFACT',
# 'reading': 'your sample size changed by -41% at the same time and it tracks
# this metric historically (correlation 0.70)...'}
# What should the next reading be?
forecast.next_value(latency_p95)
# How good have these forecasts actually been?
forecast.score(latency_p95)
# {'empirical_coverage': 0.803, 'calibrated': True, 'n_misses': 197,
# 'worst_misses': [...]} # misses are always published
# I watch 40 metrics. Which genuinely stand out?
multiple.select({'signups': 3.2, 'latency': 812.0, 'errors': 1.1, ...})
# {'selected': ['latency'],
# 'reading': '1 of 40 metrics worth looking at: latency'}As an MCP server
Assistants read metrics constantly and assert significance the way people do — by eyeballing a change and calling it. This gives them a way to check.
{
"mcpServers": {
"noisefloor": { "command": "uvx", "args": ["--from", "noisefloor", "noisefloor-mcp"] }
}
}mcp-name: io.github.beepboop2025/noisefloorSix tools: ab_test, did_it_change, real_or_sampling, forecast_next, score_forecasts, which_metrics_matter.
Why these methods
Every check is anytime-valid or distribution-free — the two properties that survive contact with how dashboards are really used: looked at whenever someone feels like it, and stopped when they see what they want.
experiment— beta-binomial mixture test martingale. A mixture of likelihood ratios is a non-negative martingale under the null, so Ville's inequality bounds the chance it ever crosses1/alpha. Exact for Bernoulli outcomes rather than a worst-case bound, which is where the power comes from. (theory, safe testing)change— conformal Shiryaev-Roberts detector. Rank-based p-values, no distributional assumption, with a stated average time between false alarms. Two-sided by default, because half of what goes wrong is a number going to zero. (nonexchangeable conformal)coverage— conditions the metric on its own denominator. Almost every metric worth watching is a rate over a sample you don't control.forecast— adaptive conformal intervals, valid under arbitrary distribution shift, graded by the Weighted Interval Score, a proper rule so the scoreboard can't be gamed by hedging. (ACI, decaying steps, WIS)multiple— e-Benjamini-Hochberg. Controls false discoveries across all your metrics at once, under arbitrary dependence — which matters, because real metrics move together. (e-BH, merging)
What it costs
Anytime validity isn't free. At any fixed sample size the interval is wider than a one-look interval, so calling the same effect takes roughly twice the data — 10% vs 13% resolves at a median of about 14,300 observations here, against roughly 7,000 for a correctly run one-look test.
You're buying the right to stop whenever you want. For most teams that's a bargain, because the realistic alternative isn't a clean one-look test. It's a one-look test being peeked at, which is the 40% column above.
Design
Nothing here returns a number it can't stand behind. Not enough history returns NOT_ENOUGH_HISTORY, not a confident zero. A metric that moved with its own sample size returns SAMPLING_ARTIFACT, not a finding. Forecast misses are published in full and there's no flag to hide them.
Every result carries its method and, where one exists, its guarantee. Quote them — the point is that the claim can be backed rather than asserted.
Zero dependencies means it installs in Lambda, edge runtimes and locked-down build images where adding scipy is a procurement conversation. Every result is deterministic: same input, same answer, forever, with no RNG anywhere.
Provenance
These engines were built for Palimpsest, a public-good censorship observatory, where publishing a number you can't defend is the whole failure mode. The sampling-artifact check exists because a censorship index there fell 60.3 → 55.6 while the measurements underneath it fell 353,676 → 208,933. The index hadn't moved. The instrument had.
Licence
MIT.
Available Tools
6 toolsab_testA
Can you call a winner on an A/B test yet? Uses anytime-valid confidence sequences, so it is SAFE TO RUN AFTER EVERY OBSERVATION — peeking does not inflate the false-positive rate the way a t-test or z-test does.
| Name | Required | Description | Default |
|---|---|---|---|
| alpha | No | error budget, default 0.05 | |
| a_label | No | ||
| a_total | Yes | total observations in arm A | |
| b_label | No | ||
| b_total | Yes | total observations in arm B | |
| a_successes | Yes | conversions in arm A | |
| b_successes | Yes | conversions in arm B |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully carries the behavioral burden. It discloses the key behavior: peeking is safe and does not inflate false-positive rates. This is critical for correct usage. However, it does not mention other behaviors like mutability or side effects, which are not relevant here.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the core question, and immediately providing the essential statistical property. Every sentence adds value without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool lacks an output schema and the description does not explain what the tool returns (e.g., recommendation, p-value, winner). Given 7 parameters and no output schema, the description is incomplete for an AI agent to fully understand the tool's behavior.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers 71% of parameters with descriptions. The description adds no additional meaning to parameters beyond what the schema provides. For the 29% not covered (a_label, b_label), the description does not compensate. Baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly asks 'Can you call a winner on an A/B test yet?' which directly states the tool's purpose. It specifies the methodology (confidence sequences) and distinguishes from sibling tools like forecast_next or did_it_change.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states that the tool uses anytime-valid confidence sequences and that it is safe to run after every observation without inflating false-positive rates. It contrasts with t-test/z-test, providing clear usage guidance, though no direct comparison to siblings is given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
did_it_changeA
Did a metric actually change, or is the move noise? Detects both rises AND collapses against the metric's own history, with a stated false-alarm rate and no assumption about the distribution.
| Name | Required | Description | Default |
|---|---|---|---|
| values | Yes | the metric's history, oldest first | |
| half_life | No | optional: readings after which old history counts half, for drifting metrics | |
| two_sided | No | detect drops as well as rises (default true) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses key behavioral traits: detection of both rises and drops, use of historical data, a false-alarm rate, and no distribution assumption. This is fairly transparent, though it doesn't clarify whether the tool is read-only or how the false-alarm rate is set.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, well-structured sentence that front-loads the core question. It is concise and efficient, though could be slightly more scannable with a brief list. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has no output schema, and the description fails to specify what the tool returns (e.g., boolean, p-value, signal strength). This is a significant gap, especially for a statistical tool where output interpretation is critical. The description also doesn't explain how the false-alarm rate is configured or communicated.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds algorithm context (false-alarm rate, no distribution assumption) but does not elaborate on parameter syntax beyond what the schema provides. No parameter-specific details are added, so it meets but does not exceed the baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: determining whether a metric change is real or noise. It specifies 'detects both rises AND collapses' and includes unique algorithm properties (false-alarm rate, no distribution assumption), distinguishing it from siblings like ab_test (hypothesis test) or which_metrics_matter (metric selection).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool should be used when you need to know if a change is noise, providing context like 'against the metric's own history'. However, it does not explicitly state when not to use it or mention alternatives among siblings, slightly limiting guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
forecast_nextC
What should the next reading be, and within what range? Range adapts to the metric's recent volatility and stays valid even when the metric shifts.
| Name | Required | Description | Default |
|---|---|---|---|
| values | Yes | the metric's history, oldest first | |
| nominal | No | range coverage, default 0.8 |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It states that the range adapts to recent volatility and stays valid with shifts, but does not disclose how many values are needed, what model is used, or what happens with insufficient data. No mention of errors or limitations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very concise, consisting of two sentences. It is front-loaded with the key question. However, the use of a question format might be less direct for an AI agent. No redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has a simple input schema but no output schema. The description is incomplete because it does not specify the structure of the output (e.g., object with 'prediction' and 'range' fields). It also lacks guidance on edge cases like very short history or extreme volatility. For a forecasting tool, more detail is needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with clear descriptions for both 'values' and 'nominal'. The tool description adds no additional meaning beyond the schema. Given high coverage, baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a question to state the tool's purpose: predicting next reading and range. It mentions adapting to volatility, which hints at a unique feature. However, it does not explicitly state 'forecast' or 'predict' as a verb, and could be more direct. It distinguishes well from sibling tools which have different focuses.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool vs alternatives. There is no mention of prerequisites, data requirements, or scenarios where other tools like 'score_forecasts' or 'did_it_change' would be better suited. The description only implies use for forecasting, but lacks comparative context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
real_or_samplingA
Did the metric move, or did the sample size underneath it move? Run this before reporting any RATE as a change — conversion rates, error rates and click-through all shift when the denominator shifts, for reasons that have nothing to do with the thing being measured.
| Name | Required | Description | Default |
|---|---|---|---|
| values | Yes | the metric's history, oldest first | |
| sample_sizes | Yes | the denominator behind each reading, same order and length |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full burden of behavioral disclosure. It describes the conceptual behavior (checking denominator shifts) but does not specify the output format, assumptions, or edge cases. The agent knows the goal but not the result type.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise, consisting of two short sentences. It front-loads the core question and immediately provides actionable context. Every word earns its place, and there is no waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of an output schema and annotations, the description should ideally specify the return value. It adequately explains the use case and parameter relationships but leaves a notable gap regarding what the tool returns, which is essential for an agent to use it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, providing basic descriptions for both parameters. The tool description adds meaningful context by linking parameters to real-world rate metrics and denominator shifts, enhancing understanding beyond the raw schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: to determine if a change in a rate metric is due to actual movement or a shift in the denominator. It uses a specific verb 'run' and resource 'rate metric change'. The description effectively distinguishes from sibling tools like 'ab_test' and 'did_it_change' by focusing on the artifact of denominator shifts.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit guidance on when to use the tool: 'Run this before reporting any RATE as a change', with examples like conversion rates and error rates. It implies the context for use but does not explicitly mention when not to use it or list alternatives; however, the context is clear enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
score_forecastsA
How good would these forecasts actually have been? Grades every prediction the tool would have made over the history, using only what was known at the time, and reports calibration plus the worst misses.
| Name | Required | Description | Default |
|---|---|---|---|
| values | Yes | the metric's history, oldest first | |
| nominal | No | range coverage, default 0.8 |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It discloses that the tool uses only information known at the time (avoiding look-ahead bias) and reports calibration and worst misses. No destructive side effects mentioned, which is appropriate for a read-only evaluation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, front-loaded with purpose, no redundant information. Every word contributes meaning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description adequately hints at return values ('calibration plus worst misses'). Parameter descriptions in schema complement well. No significant gaps for a tool of this complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with parameter descriptions for 'values' and 'nominal'. The description adds no new meaning beyond what the schema already provides, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool 'grades every prediction the tool would have made over the history' and 'reports calibration plus the worst misses', using specific verbs 'grade' and 'reports'. It distinguishes from siblings like 'forecast_next' by focusing on historical evaluation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives like 'forecast_next' or 'ab_test'. The description implies usage for historical forecast evaluation but does not provide explicit context or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
which_metrics_matterB
You watch many metrics; which genuinely stand out? Controls the false discovery rate across all of them at once, which per-metric thresholds do not: forty metrics each alerting wrongly 5% of the time means two false alarms every round.
| Name | Required | Description | Default |
|---|---|---|---|
| alpha | No | false-discovery rate, default 0.1 | |
| evidence | Yes | metric name -> evidence value, e.g. the 'evidence' field from did_it_change |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description bears full burden. It explains the statistical concept (FDR control) and provides an example, but does not describe output format, assumptions, or edge cases (e.g., required dependencies like did_it_change evidence values).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences with an engaging tone. The first sentence is a question but quickly leads to explanation. No filler, every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and sibling tools, the description misses key context: the evidence parameter expects output from a sibling (did_it_change), and the output is not described. The tool's niche is clear but integration details are lacking.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds no extra detail about parameters beyond what the schema provides (e.g., alpha default, evidence structure).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool controls false discovery rate across many metrics, contrasting with per-metric thresholds. It includes a concrete example illustrating the problem. However, it does not explicitly differentiate from sibling tools beyond the conceptual niche.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use when multiple metrics are tested and false discovery control is needed, but lacks explicit guidance on when to use versus alternatives (e.g., when to use ab_test or score_forecasts).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
6 tool updates
v0.1.2- First observed
ab_test - First observed
did_it_change - First observed
forecast_next - First observed
real_or_sampling - First observed
score_forecasts - First observed
which_metrics_matter
TDQS
Each tool targets a distinct statistical question: A/B testing, change detection, forecasting, denominator shift, forecast evaluation, and multiple testing correction. No overlap in purposes.
All tool names follow a consistent verb_noun pattern in lowercase snake_case, e.g., did_it_change, forecast_next, which_metrics_matter. No deviations.
6 tools is an ideal number for a focused statistics toolkit, covering essential operations without being overwhelming or sparse.
The set covers key statistical tasks (A/B testing, change detection, forecasting, multiple testing), but lacks a sample size/power analysis tool, which is a minor gap.
Maintenance
Related MCP Connectors
Build, inspect and read adaptive A/B tests. Config travels in the URL; no account or API key needed.
Exact statistics & probability: distributions, hypothesis tests, CIs, Bayesian updates, regression.
Find novel, statistically validated patterns in tabular data — hypothesis-free.
The statistical analyst in your AI chat — validated, citable, re-runnable analysis of your data.
Related MCP Servers
- AlicenseAqualityAmaintenanceEval-integrity statistics for AI benchmark claims — multiple-testing correction, power/MDE for model gaps, judge-bias and leaderboard-rank checks. Catches a benchmark number that won't survive a second look.9MIT
- AlicenseNot gradedqualityBmaintenanceChecks whether a trading backtest survives its own statistics: deflated Sharpe, multiple-testing correction against a best-of-N-noise benchmark, minimum track record length, and fill realism. Takes no market data and no API keys, and cannot recommend a trade — it only reports that a result is weaker than claimed or not yet provable.MIT
- AlicenseNot gradedqualityBmaintenanceProvides honest A/B test verdicts using interactive cards, with statistical calculations for conversion rates, sample sizes, and Bayesian probability-to-beat.MIT
- AlicenseAqualityBmaintenanceAnswers "sales dropped since last week — where?" by comparing a target period against a weekday-adjusted baseline and localizing which attribute combinations (e.g. channel=web, or Tuesday nights) explain the shift. Runs fully offline on your own CSV — no API key, no ML training, read-only.2MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/beepboop2025/noisefloor'
If you have feedback or need assistance with the MCP directory API, please join our Discord server