Reference
Statistics vocabulary
The exact meaning of every statistical term Kno reports, confidence intervals, holdouts, controls, Bonferroni correction, underpowered.
The authoritative treatment: What the numbers mean.
| Term | Meaning in Kno |
|---|---|
| Delta | Paired per-case difference: the same case scored with and without the asset |
| 95% confidence interval | Every reported delta carries one, or the delta is not reported at all |
| Holdout | A slice of cases sealed at baseline; nothing, not valuation, not selection, reads it until validate |
| Control reserve | A third of dev cases reserved before routing, so controls never run dry |
| Fresh controls | Routed cases re-measured without the asset; regression is measured separately from improvement |
| Bonferroni correction | Every keep/reject decision at 1 − (1 − 0.95) / n_screened; the error budget is the family of decisions, not any one of them |
| Underpowered | Fewer than 20 holdout cases, Kno reports no delta rather than a bad number |
| Winner’s curse | Selection inflates the best candidates; Kno corrects the multiplicity and names the remaining inflation instead of hiding it |
min_detectable_harm |
The smallest regression the control sample could see, the limit of what “no harm found” can claim |
| Gaps | GAP / UNKNOWN verdicts: cases the current data cannot move |
| Error rate | A baseline with a 5%+ error rate is unusable, not just noisy |
The holdout number is the one you may put in a slide.