3.8 KiB
Title: onod3000 Was Failing Perfectly Good Random Data Date: 2026-10-25 19:07 Category: RANDOM Slug: onod3000-calibrated Summary: onod3000 0.2.0 stops trusting textbook p-values that are wrong for small samples, calibrates every test against 20,000 random samples per size, and rates a whole sample with one score and a survival depth. Tags: rust, onod3000, statistics
onod3000 is my Rust port of Paul Uszak's ent3000, a randomness test suite for people building hardware random number generators and testing them on samples under a megabyte. I ported it faithfully, test by test, and for almost two years I assumed that a faithful port of a careful suite gave trustworthy answers.
It didn't. I found out while testing STRIX, a ring-oscillator TRNG I have been building, when the Kolmogorov-Smirnov test failed half of its runs. It also failed 3 of 6 runs on /dev/urandom, which is about as good a source of random bytes as a Linux machine has, and it gave a different p-value for the same file each time. The Java original compares the data against a pseudo-random sample, so the reference itself changed on every run. The KS test in 0.2.0 compares byte values against the exact uniform distribution over 0 to 255 instead, and on /dev/urandom it now fails about 1% of the time at alpha 0.01, which is what it should do.
The other tests were wrong too
Once I started feeding known-good data through every test, KS wasn't the only one. Several p-values rely on approximations that only hold for large samples, or on constants that don't match the real spread:
- Shannon compares against exactly 8 bits per byte, but measured entropy always comes out a little lower on a finite sample.
- Gaps ignores repeats past the end of the sample.
- Compression uses a standard deviation that is simply made up.
- Avalanche divides by the spread of a single window instead of the spread of the mean.
So on truly random data Shannon failed every sample up to 4 KB, Gaps failed every sample up to 2 KB, and Avalanche never failed anything at all. A test that rejects everything small and a test that accepts everything are both useless, and I had been reading their output for almost two years.
Calibration
Rather than patch each formula, onod3000 now measures. examples/calibrate.rs ran every test on 20,000 random samples at each power-of-two size from 256 bytes to 1 MB and stored where each z-score landed, which takes about 40 minutes on 14 cores. The calibrated p-value says how unusual a z-score is compared with those runs. The old z-score and p-value columns stay, and still match the Java original, so nothing that compared against ent3000 breaks.
The 15 calibrated p-values are then combined into one with the Cauchy combination test (Liu & Xie 2020), whose null distribution was measured by simulation too, because Shannon, ChiByte, KS and MeanByte all look at the same byte counts and can't be treated as independent. The score is -log10 of that combined p-value: under 2 passes, 2 to 4 is suspect, 4 or more fails. A perfect source doesn't score zero. One good sample in a hundred lands in suspect, and a source that keeps landing there is the real warning sign.
There is also a survival depth, ported from Comscire's QNGmeter: the tests run on the first 256 bytes, then 512, then 1 KB and so on, and the depth is how far the sample gets before the combined p-value drops below 10⁻⁴. It shows roughly how much data a flaw needs before it shows up.
0.2.0
The release also moves onod3000 to the Apache-2.0 license, with a NOTICE file and third-party notices. Paul Uszak's original is still credited, and still the reason any of this exists. I just don't trust a p-value anymore until I have watched it on data I know is random.