feat: add onod3000-calibrated devlog post
This commit is contained in:
parent
77e19d25d4
commit
c5f7d24914
1 changed files with 37 additions and 0 deletions
37
randogoth.com/content/20261025_onod3000-calibrated.md
Normal file
37
randogoth.com/content/20261025_onod3000-calibrated.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
Title: onod3000 Was Failing Perfectly Good Random Data
|
||||
Date: 2026-10-25 19:07
|
||||
Category: RANDOM
|
||||
Slug: onod3000-calibrated
|
||||
Summary: onod3000 0.2.0 stops trusting textbook p-values that are wrong for small samples, calibrates every test against 20,000 random samples per size, and rates a whole sample with one score and a survival depth.
|
||||
Tags: rust, onod3000, statistics
|
||||
|
||||
[onod3000]({filename}20250118_onod3000.md) is my Rust port of *Paul Uszak*'s *ent3000*, a randomness test suite for people building hardware random number generators and testing them on samples under a megabyte. I ported it faithfully, test by test, and for almost two years I assumed that a faithful port of a careful suite gave trustworthy answers.
|
||||
|
||||
It didn't. I found out while testing STRIX, a ring-oscillator TRNG I have been building, when the Kolmogorov-Smirnov test failed half of its runs. It also failed 3 of 6 runs on `/dev/urandom`, which is about as good a source of random bytes as a Linux machine has, and it gave a different p-value for the same file each time. The Java original compares the data against a pseudo-random sample, so the reference itself changed on every run. The KS test in 0.2.0 compares byte values against the exact uniform distribution over 0 to 255 instead, and on `/dev/urandom` it now fails about 1% of the time at alpha 0.01, which is what it should do.
|
||||
|
||||
## The other tests were wrong too
|
||||
|
||||
Once I started feeding known-good data through every test, KS wasn't the only one. Several p-values rely on approximations that only hold for large samples, or on constants that don't match the real spread:
|
||||
|
||||
- *Shannon* compares against exactly 8 bits per byte, but measured entropy always comes out a little lower on a finite sample.
|
||||
- *Gaps* ignores repeats past the end of the sample.
|
||||
- *Compression* uses a standard deviation that is simply made up.
|
||||
- *Avalanche* divides by the spread of a single window instead of the spread of the mean.
|
||||
|
||||
So on truly random data Shannon failed every sample up to 4 KB, Gaps failed every sample up to 2 KB, and Avalanche never failed anything at all. A test that rejects everything small and a test that accepts everything are both useless, and I had been reading their output for almost two years.
|
||||
|
||||
## Calibration
|
||||
|
||||
Rather than patch each formula, onod3000 now measures. `examples/calibrate.rs` ran every test on 20,000 random samples at each power-of-two size from 256 bytes to 1 MB and stored where each z-score landed, which takes about 40 minutes on 14 cores. The *calibrated* p-value says how unusual a z-score is compared with those runs. The old z-score and p-value columns stay, and still match the Java original, so nothing that compared against ent3000 breaks.
|
||||
|
||||
The 15 calibrated p-values are then combined into one with the Cauchy combination test ([Liu & Xie 2020](https://doi.org/10.1080/01621459.2018.1554485)), whose null distribution was measured by simulation too, because Shannon, ChiByte, KS and MeanByte all look at the same byte counts and can't be treated as independent. The *score* is `-log10` of that combined p-value: under 2 passes, 2 to 4 is suspect, 4 or more fails. A perfect source doesn't score zero. One good sample in a hundred lands in suspect, and a source that keeps landing there is the real warning sign.
|
||||
|
||||
There is also a *survival depth*, ported from *Comscire*'s QNGmeter: the tests run on the first 256 bytes, then 512, then 1 KB and so on, and the depth is how far the sample gets before the combined p-value drops below 10⁻⁴. It shows roughly how much data a flaw needs before it shows up.
|
||||
|
||||
## 0.2.0
|
||||
|
||||
The release also moves onod3000 to the Apache-2.0 license, with a NOTICE file and third-party notices. Paul Uszak's original is still credited, and still the reason any of this exists. I just don't trust a p-value anymore until I have watched it on data I know is random.
|
||||
|
||||
## Repository
|
||||
|
||||
[onod3000](https://code.randogoth.com/randogoth/onod3000)
|
||||
Loading…
Add table
Add a link
Reference in a new issue