feat: add onod3000-calibrated devlog post

This commit is contained in:
randogoth 2026-10-25 19:07:45 +02:00
parent 77e19d25d4
commit c5f7d24914

View file

@ -0,0 +1,37 @@
Title: onod3000 Was Failing Perfectly Good Random Data
Date: 2026-10-25 19:07
Category: RANDOM
Slug: onod3000-calibrated
Summary: onod3000 0.2.0 stops trusting textbook p-values that are wrong for small samples, calibrates every test against 20,000 random samples per size, and rates a whole sample with one score and a survival depth.
Tags: rust, onod3000, statistics
[onod3000]({filename}20250118_onod3000.md) is my Rust port of *Paul Uszak*'s *ent3000*, a randomness test suite for people building hardware random number generators and testing them on samples under a megabyte. I ported it faithfully, test by test, and for almost two years I assumed that a faithful port of a careful suite gave trustworthy answers.
It didn't. I found out while testing STRIX, a ring-oscillator TRNG I have been building, when the Kolmogorov-Smirnov test failed half of its runs. It also failed 3 of 6 runs on `/dev/urandom`, which is about as good a source of random bytes as a Linux machine has, and it gave a different p-value for the same file each time. The Java original compares the data against a pseudo-random sample, so the reference itself changed on every run. The KS test in 0.2.0 compares byte values against the exact uniform distribution over 0 to 255 instead, and on `/dev/urandom` it now fails about 1% of the time at alpha 0.01, which is what it should do.
## The other tests were wrong too
Once I started feeding known-good data through every test, KS wasn't the only one. Several p-values rely on approximations that only hold for large samples, or on constants that don't match the real spread:
- *Shannon* compares against exactly 8 bits per byte, but measured entropy always comes out a little lower on a finite sample.
- *Gaps* ignores repeats past the end of the sample.
- *Compression* uses a standard deviation that is simply made up.
- *Avalanche* divides by the spread of a single window instead of the spread of the mean.
So on truly random data Shannon failed every sample up to 4 KB, Gaps failed every sample up to 2 KB, and Avalanche never failed anything at all. A test that rejects everything small and a test that accepts everything are both useless, and I had been reading their output for almost two years.
## Calibration
Rather than patch each formula, onod3000 now measures. `examples/calibrate.rs` ran every test on 20,000 random samples at each power-of-two size from 256 bytes to 1 MB and stored where each z-score landed, which takes about 40 minutes on 14 cores. The *calibrated* p-value says how unusual a z-score is compared with those runs. The old z-score and p-value columns stay, and still match the Java original, so nothing that compared against ent3000 breaks.
The 15 calibrated p-values are then combined into one with the Cauchy combination test ([Liu & Xie 2020](https://doi.org/10.1080/01621459.2018.1554485)), whose null distribution was measured by simulation too, because Shannon, ChiByte, KS and MeanByte all look at the same byte counts and can't be treated as independent. The *score* is `-log10` of that combined p-value: under 2 passes, 2 to 4 is suspect, 4 or more fails. A perfect source doesn't score zero. One good sample in a hundred lands in suspect, and a source that keeps landing there is the real warning sign.
There is also a *survival depth*, ported from *Comscire*'s QNGmeter: the tests run on the first 256 bytes, then 512, then 1 KB and so on, and the depth is how far the sample gets before the combined p-value drops below 10⁻⁴. It shows roughly how much data a flaw needs before it shows up.
## 0.2.0
The release also moves onod3000 to the Apache-2.0 license, with a NOTICE file and third-party notices. Paul Uszak's original is still credited, and still the reason any of this exists. I just don't trust a p-value anymore until I have watched it on data I know is random.
## Repository
[onod3000](https://code.randogoth.com/randogoth/onod3000)