diff --git a/readme.md b/readme.md index ba83174..f9787d9 100644 --- a/readme.md +++ b/readme.md @@ -12,16 +12,20 @@ generating in the sub 1 MB space. Successor to John Walker’s venerable `ent`. Pipe binary entropy data into the `onod3000` command line tool or call it - with `-f ` to load entropy from a file. + with `-f ` to load entropy from a file. Add `--depth-off` to + skip the survival depth. ``` -`Onod3000` provides multiple randomness tests, each outputting a statistic, a z-score andd a p-value to assess the conformity of the data to random behavior. +`Onod3000` provides multiple randomness tests, each outputting a statistic, a z-score and a p-value to assess the conformity of the data to random behavior. On top of that it rates a whole sample with one calibrated score, see [Scoring](#scoring). ## Testing Methodology - **Null Hypothesis**: The data conforms to random behavior. - **P-Value Interpretation**: Low p-values (< 0.01) indicate a significant deviation from randomness. +- **Pass**: a test passes when its calibrated p-value (`Cal-P`, see [Scoring](#scoring)) is at least 0.01. Samples under 256 bytes have no calibration, so there the test's own p-value is used. + +Even a perfect source fails each test 1% of the time, so with 15 tests about 1 in 7 good samples shows at least one ❌. Use the score for an overall verdict. All tests have been carefully ported from [Paul Uszak's Java code](http://www.reallyreallyrandom.com/gitbucketlabhub/) (ent3000-0.6.0-beta). Java libraries were substituted with Rust crate equivalents when possible. The suite includes the following tests: @@ -44,27 +48,100 @@ More details on some individual tests [available here](http://www.reallyreallyra cat /dev/random | head -c 259200 | onod3000 Testing 259200 bytes from stdin. --------------------------------------------------------- -Randomness Test Value Z-Score P-Value Pass --------------------------------------------------------- -Shannon 7.999 -0.3669 0.7137 ✅ -Monobit 0.500 0.9000 0.3681 ✅ -ChiBit 4.555 -0.8612 0.8039 ✅ -ChiByte 236.288 -0.8286 0.7939 ✅ -MeanByte 127.312 -1.2970 0.1946 ✅ -Compression 1.000 0.0868 0.9308 ✅ -Kolm.-Smirnov 0.004 2.0722 0.0272 ✅ -Pi 3.139 -1.3171 0.1878 ✅ -Shells 33.593 -0.0493 0.4874 ✅ -Gaps 7.402 -0.3767 0.5954 ✅ -Avalanche 80.006 0.0009 0.9993 ✅ -Runs 129288.000 0.8208 0.4118 ✅ -RunUps 2.647 1.1645 0.1038 ✅ -Prediction 0.500 -0.0255 0.9796 ✅ -UnCorrelation -0.001 -0.3725 0.7096 ✅ --------------------------------------------------------- +---------------------------------------------------------------- +Randomness Test Value Z-Score P-Value Cal-P Pass +---------------------------------------------------------------- +Shannon 7.999 -0.3799 0.7040 0.6693 ✅ +Monobit 0.500 0.6722 0.5014 0.5007 ✅ +ChiBit 12.457 1.1141 0.1320 0.1320 ✅ +ChiByte 244.810 -0.4512 0.6656 0.6630 ✅ +MeanByte 127.442 -0.3986 0.6902 0.6934 ✅ +Compression 1.000 0.0868 0.9308 1.0000 ✅ +Kolm.-Smirnov 0.001 0.4724 0.9789 0.9604 ✅ +Pi 3.138 -1.0970 0.2726 0.2602 ✅ +Shells 24.512 -1.1506 0.9068 0.8923 ✅ +Gaps 5.994 -0.7085 0.7405 0.6891 ✅ +Avalanche 80.135 0.0214 0.9829 0.0848 ✅ +Runs 129129.000 0.1080 0.9140 0.9160 ✅ +RunUps 2.604 1.1345 0.1066 0.1073 ✅ +Prediction 259198.000 0.3321 0.2254 0.2172 ✅ +UnCorrelation 0.000 0.0144 0.9885 0.9842 ✅ +---------------------------------------------------------------- 15/15 tests passed. --------------------------------------------------------- +---------------------------------------------------------------- +Score 0.06 (pass) · weakest: Avalanche p=0.085 · Depth 21.0+ bits +---------------------------------------------------------------- +``` + +## Scoring + +Besides the individual tests, `onod3000` rates a whole sample with one **score** and a **survival depth**. Both need at least 256 bytes. + +### Calibrated p-values + +Some of the tests' own p-values are wrong on random data. They rely on approximations that only hold for large samples, or on constants that don't match the real spread: + +- **Shannon**: compares against exactly 8 bits, but measured entropy comes out lower on finite samples. +- **Gaps**: ignores repeats past the end of the sample. +- **Compression**: its standard deviation is made up. +- **Avalanche**: divides by the spread of a single window instead of the spread of the mean. + +So on truly random data Shannon fails every sample up to 4 KB, Gaps fails every sample up to 2 KB, and Avalanche never fails at all. + +The score therefore doesn't use those p-values. `examples/calibrate.rs` ran every test on 20,000 random samples at each size from 256 B to 1 MB (powers of two) and stored where each test's z-score landed. The **calibrated p-value** (`Cal-P`) says how unusual a z-score is compared with those random runs: + +- Sizes in between are interpolated, and sizes above 1 MB are extrapolated, both linearly in 1/√n. That is how Shannon's bias and Avalanche's spread change with size, and it leaves the other tests unchanged. +- Compression's z-score on random data is a fixed value for each length (deflate's overhead), so it is compared against a reference random sample of the same length instead. + +The `Z-Score` and `P-Value` columns are unchanged and still match the Java original. + +### Score + +The 15 calibrated p-values are combined with the Cauchy combination test ([Liu & Xie 2020](https://doi.org/10.1080/01621459.2018.1554485)). Its null distribution was also measured by simulation, so the combined p-value stays correct even though several tests look at the same thing (Shannon, ChiByte, KS and MeanByte all come from the byte counts). + +The score is `-log10(combined p)`, capped at 12: + +| Score | Verdict | Meaning | +| ----- | ------- | ------- | +| < 2 | pass | nothing unusual | +| 2 – 4 | suspect | as unusual as 1 in 100 to 1 in 10,000 random samples | +| ≥ 4 | fail | rarer than 1 in 10,000 random samples | + +A perfect source does not score 0. On random data the combined p-value is uniform, so 1% of good samples score 2 or more and 0.01% score 4 or more. A source that keeps landing in suspect is the real warning sign. The weakest test (lowest calibrated p-value) is reported with the score. + +### Accuracy + +How often fresh random data (not used to build the tables) gets a combined p-value below each threshold. For a correct calibration that is exactly the threshold: 5%, 1% and 0.01%. Values a little under target mean the score errs toward passing good data. + +| Sample size | Random samples checked | p < 0.05 | p < 0.01 (score ≥ 2, suspect) | p < 0.0001 (score ≥ 4, fail) | +| ----------- | ---------------------- | -------- | ----------------------------- | ---------------------------- | +| 256 B | 2,000,000 | 4.98% | 0.88% | 0.0046% | +| 1,000 B | 20,000 | 4.77% | 0.95% | – | +| 1 KB | 1,000,000 | 4.84% | 0.88% | 0.0025% | +| 4 KB | 1,000,000 | 4.62% | 0.84% | 0.0022% | +| 10,000 B | 20,000 | 4.88% | 0.90% | – | +| 100,000 B | 20,000 | 5.23% | 1.06% | – | +| 259,200 B | 2,000 | 4.60% | 0.85% | – | +| 1 MB | 2,000 | 5.25% | 1.10% | – | +| 1,500,000 B | 2,000 | 4.95% | 1.05% | – | +| 4,000,000 B | 2,000 | 4.50% | 0.65% | – | + +Sizes that are not a power of two use interpolation, and sizes above 1 MB use extrapolation. "–" means too few samples to measure a 0.01% rate. Reproduce with `cargo run --release --example calibrate -- --check N --sizes A,B,...`. + +### Survival depth + +Ported from Comscire's QNGmeter: the tests are run on the first 256 B, 512 B, 1 KB and so on up to the whole sample. The depth is log2 of the number of bits in the longest prefix that passed, where all shorter prefixes passed too. A prefix fails when its combined p-value drops below 10⁻⁴, QNGmeter's freeze level. + +- `+` means the whole sample survived. +- A depth below the sample size shows roughly how much data it takes for a flaw to show up. +- Computing the depth takes up to twice as long. `--depth-off` skips it. + +### Rebuilding the calibration + +```bash +cargo run --release --example calibrate # simulates and writes src/calibration/table.rs, ~40 min on 14 cores +cargo run --release --example calibrate -- --from-dump target/calibration-runs.bin # rebuilds the table from saved runs in seconds +cargo run --release --example calibrate -- --check 2000 # checks the compiled-in table on fresh data ``` ## Rust Library @@ -74,8 +151,16 @@ use onod3000::Onod; fn main() { let data = vec![...]; // Your binary data - let p_value = Onod::monobit(&data); + let (_, _, p_value) = Onod::monobit(&data); println!("Monobit Test P-Value: {:.4}", p_value); + + // One test with its calibrated p-value: (statistic, z, p, calibrated p) + let (_, _, _, p_cal) = Onod::calibrated("gaps", &data); + println!("Gaps calibrated P-Value: {:?}", p_cal); + + // The whole sample; `true` also computes the survival depth + let score = Onod::score(&data, true); + println!("Score {:.2} ({}), depth {:.1} bits", score.score, score.verdict.as_str(), score.depth_bits); } ``` @@ -83,77 +168,114 @@ fn main() { The library ships with FFI and optional Python bindings so it can be used as C or Python library. +### C + +Link against the static or dynamic library and include [include/onod3000.h](include/onod3000.h): + +```c +double result[3]; // statistic, z-score, p-value +onod_run("monobit", data, len, result); + +OnodScore score; +if (onod_score(data, len, true, &score)) // true: also compute the survival depth + printf("score %.2f, weakest %s\n", score.score, + score.weakest >= 0 ? onod_test_name(score.weakest) : "-"); +``` + +`OnodScore.verdict` is 0 pass, 1 suspect, 2 fail or 3 insufficient data. `p_cal` holds the calibrated p-value per test, in `onod_test_name` order. + ### Python Install using `maturin`: ```bash maturin build --release --features python ``` +```python +from onod import Onod + +stat, z, p = Onod.run("monobit", data) +score = Onod.score(data) # Onod.score(data, depth=False) skips the survival depth +print(score["score"], score["verdict"], score["weakest"]) +``` + ## Comparison To test the implementation three files of binary random data have been analyzed using the original Java program (version 0.6.0-beta) and this Rust port, expecting the same p-values. The test files can be found in the test folder. +Each table has two parts: + +- **Java ent3000 / Rust onod3000**: each test's own p-value in both programs. These should match. **Java vs Rust** notes where they don't. +- **Rust calibrated**: the p-value this port uses for the `Pass` mark and the score (see [Calibrated p-values](#calibrated-p-values)). It is the same z-score, compared with 20,000 random samples of the same size instead of a textbook distribution. The Java program has no equivalent. It differs from the raw value most where the raw value is wrong for the sample size, like Shannon and Gaps on the 1 KB file. + +Below each table is the `Score` line that `onod3000` prints for that file. + ### Testing 1024 bytes from test1.bin (/dev/random) -Test | Java ent3000 | Rust onod3000 | Note | -------------- | ------------ | ------------- | ------------- | -Shannon | N/A | 0.0000 | | -Monobit | 0.6746 | 0.6746 | | -ChiBit | 0.9299 | 0.9299 | | -ChiByte | 0.9819 | 0.9819 | | -MeanByte | 0.7964 | 0.7964 | | -Compression | N/A | 0.0146 | | -KS | 0.5313 | 0.9998 | Different method | -Pi | 0.3784 | 0.3784 | | -Shells | 0.7559 | 0.7559 | | -Gaps | 0.0000 | 0.0000 | | -Avalanche | 0.8794 | 0.8794 | | -Runs | 0.0850 | 0.0850 | | -RunUps | 0.6537 | 0.6537 | | -Prediction | 0.6164 | 0.6164 | | -UnCorrelation | 0.0921 | 0.0918 | Close enough | +Test | Java ent3000 | Rust onod3000 | Java vs Rust | Rust calibrated | +------------- | ------------ | ------------- | ------------------- | --------------- | +Shannon | N/A | 0.0000 | Not in Java | 0.0379 | +Monobit | 0.6746 | 0.6746 | | 0.6676 | +ChiBit | 0.9299 | 0.9299 | | 0.9313 | +ChiByte | 0.9819 | 0.9819 | | 0.9812 | +MeanByte | 0.7964 | 0.7964 | | 0.8037 | +Compression | N/A | 0.0146 | Not in Java | 1.0000 | +KS | 0.5313 | 0.9998 | Different KS method | 0.9986 | +Pi | 0.3784 | 0.3784 | | 0.3760 | +Shells | 0.7559 | 0.7559 | | 0.8171 | +Gaps | 0.0000 | 0.0000 | | 0.5107 | +Avalanche | 0.8794 | 0.8794 | | 0.4529 | +Runs | 0.0850 | 0.0850 | | 0.0886 | +RunUps | 0.6537 | 0.6537 | | 0.6972 | +Prediction | 0.6164 | 0.6164 | | 0.6305 | +UnCorrelation | 0.0921 | 0.0918 | Close enough | 0.1029 | + +`Score 0.02 (pass) · weakest: Shannon p=0.038 · Depth 13.0+ bits` Feel free to provide more results, and I'll continue updating the table! ### Testing 259,200 bytes from test2.bin (/dev/random) -Test | Java ent3000 | Rust onod3000 | Note | -------------- | ------------ | ------------- | ------------- | -Shannon | N/A | 0.6853 | | -Monobit | 0.2103 | 0.2103 | | -ChiBit | 0.5590 | 0.5590 | | -ChiByte | 0.3748 | 0.3748 | | -MeanByte | 0.0183 | 0.0183 | | -Compression | N/A | 0.9308 | | -KS | 0.2900 | 0.1158 | Different method | -Pi | 0.9765 | 0.9765 | | -Shells | 0.7235 | 0.7235 | | -Gaps | 0.6425 | 0.6425 | | -Avalanche | 0.9995 | 0.9995 | | -Runs | 0.3310 | 0.3310 | | -RunUps | 0.9393 | 0.9393 | | -Prediction | 0.1519 | 0.1519 | | -UnCorrelation | 0.4497 | 0.4497 | | +Test | Java ent3000 | Rust onod3000 | Java vs Rust | Rust calibrated | +------------- | ------------ | ------------- | ------------------- | --------------- | +Shannon | N/A | 0.6853 | Not in Java | 0.7644 | +Monobit | 0.2103 | 0.2103 | | 0.2096 | +ChiBit | 0.5590 | 0.5590 | | 0.5617 | +ChiByte | 0.3748 | 0.3748 | | 0.3722 | +MeanByte | 0.0183 | 0.0183 | | 0.0195 | +Compression | N/A | 0.9308 | Not in Java | 1.0000 | +KS | 0.2900 | 0.1158 | Different KS method | 0.0975 | +Pi | 0.9765 | 0.9765 | | 0.9700 | +Shells | 0.7235 | 0.7235 | | 0.7126 | +Gaps | 0.6425 | 0.6425 | | 0.5852 | +Avalanche | 0.9995 | 0.9995 | | 0.9539 | +Runs | 0.3310 | 0.3310 | | 0.3271 | +RunUps | 0.9393 | 0.9393 | | 0.9450 | +Prediction | 0.1519 | 0.1519 | | 0.1486 | +UnCorrelation | 0.4497 | 0.4497 | | 0.4519 | + +`Score 0.29 (pass) · weakest: MeanByte p=0.020 · Depth 21.0+ bits` ### Testing 259,200 bytes from test3.bin (Hardware QRNG) -Test | Java ent3000 | Rust onod3000 | Note | -------------- | ------------ | ------------- | ---- | -Shannon | N/A | 0.6978 | | -Monobit | 0.4926 | 0.4926 | | -ChiBit | 0.8428 | 0.8428 | | -ChiByte | 0.5785 | 0.5785 | | -MeanByte | 0.9601 | 0.9601 | | -Compression | N/A | 0.9308 | | -KS | 0.7395 | 0.7777 | Different method | -Pi | 0.2806 | 0.2806 | | -Shells | 0.7711 | 0.7711 | | -Gaps | 0.1937 | 0.1937 | | -Avalanche | 0.9932 | 0.9932 | | -Runs | 0.2500 | 0.2500 | | -RunUps | 0.9710 | 0.9710 | | -Prediction | 0.7173 | 0.7173 | | -UnCorrelation | 0.4674 | 0.4674 | | +Test | Java ent3000 | Rust onod3000 | Java vs Rust | Rust calibrated | +------------- | ------------ | ------------- | ------------------- | --------------- | +Shannon | N/A | 0.6978 | Not in Java | 0.8522 | +Monobit | 0.4926 | 0.4926 | | 0.4922 | +ChiBit | 0.8428 | 0.8428 | | 0.8436 | +ChiByte | 0.5785 | 0.5785 | | 0.5766 | +MeanByte | 0.9601 | 0.9601 | | 0.9583 | +Compression | N/A | 0.9308 | Not in Java | 1.0000 | +KS | 0.7395 | 0.7777 | Different KS method | 0.7233 | +Pi | 0.2806 | 0.2806 | | 0.2718 | +Shells | 0.7711 | 0.7711 | | 0.7504 | +Gaps | 0.1937 | 0.1937 | | 0.1532 | +Avalanche | 0.9932 | 0.9932 | | 0.4964 | +Runs | 0.2500 | 0.2500 | | 0.2491 | +RunUps | 0.9710 | 0.9710 | | 0.9775 | +Prediction | 0.7173 | 0.7173 | | 0.7221 | +UnCorrelation | 0.4674 | 0.4674 | | 0.4686 | + +`Score 0.07 (pass) · weakest: Gaps p=0.153 · Depth 21.0+ bits` ### Conclusions @@ -161,6 +283,8 @@ Although the implementation of the randomness tests closely follows the original All other differences are generally negligible for practical purposes and do not affect the overall functionality or statistical significance of the tests. +The raw Shannon and Gaps p-values of 0.0000 on the 1 KB file are not a flaw in the data. Both tests reject almost every random sample that small (see [Calibrated p-values](#calibrated-p-values)), in the Java original too for Gaps. Calibrated, they come out at 0.0379 and 0.5107, and all three files pass. + ## License This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.