In papers, we can see people use NIST STS test suite to evaluate RNG.
However, I think that many people just evaluate short sequences, less samples, or only for several times (most are just a single time).
Perhaps, it is because people have tendency to report "SUCCESS", not facts.
Well, it seems that the test suite is not designed/developed for very long sequence (>> 1-million bit sequence).
Thus, I ran the test suite over 499 files, while a single file has 125MB (= 1,000,000,000 bits, that is for 1,000 samples of 1,000,000-bit sequence). All parameters are defaults.
The Random Numbers
The files are output of FST-01 running NeuG (it took about two weeks for 499 files).
The MCU on FST-01 is STM32F103. It has two built-in A/D converters. NeuG uses A/D converters' outputs as entropy sources. The noise is considered as noise of quantization error, plus noise from power supply, etc.
NeuG uses SHA-256 conditioning component to remove bias of noise.
Results
- Tested on 499 files.
| No errors | 349 files |
| Some errors | 150 files |
- Error of propotion of passing sequences
| 1 failure | 112 files |
| 2 failures | 28 files |
| 3 failures | 2 files |
| 4 failures | 2 files |
- Error of uniformity of p-values
| 1 failure | 11 files |
Errors
Here is the detail of errors:
000: 63 65 62 69 46 60 57 72 64 49 0.343781 592/607 * RandomExcursions 002: 68 58 81 52 61 71 67 57 50 70 0.152806 620/635 * RandomExcursions 005: 125 108 79 94 95 102 112 109 93 83 0.040371 978/1000 * NonOverlappingTemplate 011: 98 115 95 106 89 112 79 93 104 109 0.248014 977/1000 * NonOverlappingTemplate 018: 127 114 92 92 101 107 81 93 102 91 0.067300 977/1000 * NonOverlappingTemplate 018: 52 55 83 60 67 82 58 74 31 65 0.000068 * 624/627 RandomExcursions 020: 105 104 110 101 105 93 80 108 97 97 0.639202 979/1000 * NonOverlappingTemplate 020: 105 105 108 102 108 88 82 109 96 97 0.579021 979/1000 * NonOverlappingTemplate 021: 103 102 97 97 98 109 98 99 94 103 0.995777 979/1000 * NonOverlappingTemplate 021: 103 103 96 98 98 107 99 99 95 102 0.998740 979/1000 * NonOverlappingTemplate 022: 98 118 111 94 88 96 106 104 89 96 0.500279 979/1000 * NonOverlappingTemplate 022: 120 103 103 86 93 97 101 92 97 108 0.524101 979/1000 * NonOverlappingTemplate 025: 100 98 101 88 107 111 96 93 95 111 0.807412 977/1000 * NonOverlappingTemplate 027: 104 82 95 106 110 104 98 103 102 96 0.788728 979/1000 * NonOverlappingTemplate 029: 99 88 105 103 96 103 105 87 107 107 0.837781 978/1000 * NonOverlappingTemplate 040: 81 59 66 66 60 66 58 63 57 63 0.643730 624/639 * RandomExcursions 040: 59 60 48 60 65 75 63 71 66 72 0.446112 624/639 * RandomExcursionsVariant 042: 109 104 104 119 88 102 95 99 87 93 0.469232 979/1000 * NonOverlappingTemplate 042: 115 117 105 89 90 91 97 99 106 91 0.377007 979/1000 * NonOverlappingTemplate 043: 113 103 74 110 75 117 108 90 106 104 0.013380 978/1000 * Serial 050: 120 97 97 107 99 100 90 98 94 98 0.727851 979/1000 * FFT 052: 104 91 104 98 112 100 106 102 91 92 0.878618 979/1000 * NonOverlappingTemplate 053: 110 99 116 107 92 81 107 85 103 100 0.266235 976/1000 * NonOverlappingTemplate 055: 117 108 100 95 105 99 91 92 103 90 0.680755 974/1000 * ApproximateEntropy 056: 72 61 61 70 59 55 69 54 61 67 0.754354 614/629 * RandomExcursionsVariant 060: 104 109 102 118 96 96 98 87 97 93 0.649612 979/1000 * NonOverlappingTemplate 064: 88 98 98 111 98 112 95 94 88 118 0.392456 979/1000 * NonOverlappingTemplate 064: 123 107 100 93 95 97 111 93 95 86 0.310049 979/1000 * NonOverlappingTemplate 064: 111 97 99 103 97 101 98 107 92 95 0.967382 976/1000 * Universal 065: 123 115 97 93 95 108 93 88 98 90 0.214439 979/1000 * OverlappingTemplate 067: 104 103 105 107 88 96 102 97 108 90 0.886162 979/1000 * NonOverlappingTemplate 068: 85 88 111 90 98 108 109 114 96 101 0.390721 1000/1000 * NonOverlappingTemplate 073: 106 110 106 109 103 89 86 102 103 86 0.546283 978/1000 * NonOverlappingTemplate 073: 115 103 86 98 102 102 94 108 90 102 0.693142 978/1000 * NonOverlappingTemplate 075: 123 106 98 107 94 94 90 94 104 90 0.399442 978/1000 * OverlappingTemplate 085: 111 89 71 95 120 104 90 94 113 113 0.018036 979/1000 * NonOverlappingTemplate 090: 99 101 98 102 78 103 104 98 102 115 0.583145 978/1000 * FFT 090: 140 96 105 113 106 88 90 100 81 81 0.000911 974/1000 * Universal 098: 110 82 106 118 101 92 95 91 104 101 0.373625 977/1000 * NonOverlappingTemplate 101: 97 96 113 106 86 91 92 104 111 104 0.612147 979/1000 * NonOverlappingTemplate 103: 125 85 113 94 103 93 88 97 108 94 0.134944 979/1000 * NonOverlappingTemplate 105: 104 105 100 85 107 119 105 82 80 113 0.068145 979/1000 * NonOverlappingTemplate 114: 108 96 88 108 101 107 96 96 104 96 0.910091 978/1000 * NonOverlappingTemplate 115: 95 98 113 89 90 98 105 107 96 109 0.745908 978/1000 * NonOverlappingTemplate 115: 61 72 49 51 61 81 57 97 46 58 0.000042 * 629/633 RandomExcursionsVariant 116: 94 113 83 96 110 100 79 109 111 105 0.172816 978/1000 * NonOverlappingTemplate 118: 132 96 90 88 100 105 111 102 79 97 0.026588 979/1000 * NonOverlappingTemplate 120: 102 106 108 117 103 97 88 84 97 98 0.510153 979/1000 * NonOverlappingTemplate 127: 114 103 93 92 114 95 98 88 98 105 0.620465 979/1000 * NonOverlappingTemplate 129: 109 95 98 98 101 96 111 103 85 104 0.832561 977/1000 * NonOverlappingTemplate 134: 108 101 90 88 98 146 101 113 78 77 0.000038 * 991/1000 NonOverlappingTemplate 134: 125 106 92 112 89 90 106 81 106 93 0.072964 975/1000 * NonOverlappingTemplate 135: 120 94 82 104 100 94 99 107 101 99 0.471146 978/1000 * FFT 136: 111 97 80 94 107 94 107 107 97 106 0.540204 977/1000 * NonOverlappingTemplate 137: 116 117 87 100 91 89 87 101 113 99 0.183547 979/1000 * NonOverlappingTemplate 138: 68 54 56 62 74 58 62 50 55 79 0.169397 603/618 * RandomExcursions 139: 125 105 101 106 100 101 79 96 94 93 0.196920 975/1000 * NonOverlappingTemplate 139: 100 94 110 90 84 112 117 101 92 100 0.358641 977/1000 * NonOverlappingTemplate 141: 104 86 83 90 91 110 106 123 99 108 0.118120 979/1000 * NonOverlappingTemplate 141: 128 95 103 88 87 102 98 92 96 111 0.145326 979/1000 * NonOverlappingTemplate 143: 102 117 83 100 115 87 114 94 92 96 0.168112 979/1000 * NonOverlappingTemplate 145: 115 92 86 105 98 115 100 94 97 98 0.546283 977/1000 * NonOverlappingTemplate 150: 70 69 79 58 50 57 66 59 65 61 0.384916 619/634 * RandomExcursions 150: 80 62 64 68 54 52 62 62 75 55 0.239540 617/634 * RandomExcursionsVariant 151: 104 90 108 90 106 89 93 117 107 96 0.494392 979/1000 * NonOverlappingTemplate 151: 103 111 90 96 112 97 93 97 99 102 0.866097 979/1000 * NonOverlappingTemplate 152: 93 109 102 97 87 102 97 101 100 112 0.859637 979/1000 * NonOverlappingTemplate 154: 100 84 119 86 82 111 104 102 101 111 0.122325 979/1000 * NonOverlappingTemplate 161: 105 123 92 92 79 100 105 108 99 97 0.201189 979/1000 * CumulativeSums 161: 121 94 97 85 95 100 93 103 109 103 0.452173 978/1000 * NonOverlappingTemplate 164: 105 105 92 104 107 96 98 89 108 96 0.911413 979/1000 * NonOverlappingTemplate 166: 110 108 91 101 103 93 102 95 88 109 0.781106 979/1000 * NonOverlappingTemplate 167: 111 83 107 85 108 95 96 103 104 108 0.457825 978/1000 * NonOverlappingTemplate 171: 133 91 88 101 100 118 101 82 112 74 0.001014 976/1000 * NonOverlappingTemplate 181: 125 82 92 95 110 115 76 88 112 105 0.007369 979/1000 * NonOverlappingTemplate 184: 94 93 78 94 87 85 140 101 100 128 0.000095 * 990/1000 NonOverlappingTemplate 185: 122 113 95 84 101 79 102 103 112 89 0.056426 978/1000 * NonOverlappingTemplate 189: 119 97 89 90 103 95 96 103 115 93 0.415422 979/1000 * BlockFrequency 191: 111 91 111 93 95 83 99 121 99 97 0.250558 979/1000 * NonOverlappingTemplate 192: 74 39 53 64 63 70 45 68 97 57 0.000018 * 623/630 RandomExcursions 193: 96 91 87 125 90 84 113 103 102 109 0.088226 978/1000 * NonOverlappingTemplate 193: 110 108 98 94 109 104 96 90 99 92 0.849708 977/1000 * NonOverlappingTemplate 193: 102 90 88 96 106 112 100 109 91 106 0.697257 979/1000 * NonOverlappingTemplate 193: 51 56 56 63 68 54 75 53 61 71 0.326749 592/608 * RandomExcursions 209: 107 97 113 107 113 82 89 119 81 92 0.052610 978/1000 * NonOverlappingTemplate 217: 111 111 104 108 101 80 104 100 90 91 0.419021 979/1000 * NonOverlappingTemplate 217: 131 109 93 98 95 102 86 96 96 94 0.126658 979/1000 * NonOverlappingTemplate 220: 111 99 113 95 91 87 87 114 90 113 0.213309 979/1000 * NonOverlappingTemplate 224: 107 108 103 78 95 108 103 112 101 85 0.293952 978/1000 * NonOverlappingTemplate 224: 108 104 104 87 90 106 104 115 81 101 0.347257 979/1000 * NonOverlappingTemplate 227: 97 87 112 88 99 90 122 107 101 97 0.268917 979/1000 * NonOverlappingTemplate 229: 110 123 102 95 111 86 96 100 96 81 0.134172 978/1000 * NonOverlappingTemplate 233: 72 72 66 46 64 66 64 56 67 55 0.369073 612/628 * RandomExcursions 238: 116 89 88 106 105 96 98 107 96 99 0.670396 979/1000 * NonOverlappingTemplate 253: 105 99 111 101 104 98 106 90 91 95 0.904708 976/1000 * Rank 260: 71 57 59 60 60 55 60 61 57 61 0.972269 585/601 * RandomExcursions 262: 132 100 100 84 111 92 90 81 117 93 0.007057 976/1000 * Universal 274: 84 104 110 93 87 115 93 107 116 91 0.186566 1000/1000 * NonOverlappingTemplate 274: 124 93 103 98 110 105 98 90 83 96 0.229559 977/1000 * NonOverlappingTemplate 275: 116 108 98 102 106 98 80 107 91 94 0.406499 979/1000 * NonOverlappingTemplate 275: 63 61 79 59 59 62 70 48 57 53 0.280230 596/611 * RandomExcursions 293: 83 96 92 100 118 116 109 95 94 97 0.275709 979/1000 * LinearComplexity 296: 114 70 107 81 114 87 76 100 117 134 0.000012 * 983/1000 NonOverlappingTemplate 298: 70 64 62 59 48 66 56 67 66 61 0.725066 604/619 * RandomExcursionsVariant 298: 60 67 68 51 59 54 58 69 60 73 0.586055 604/619 * RandomExcursionsVariant 300: 106 108 119 101 100 100 79 105 90 92 0.281232 979/1000 * NonOverlappingTemplate 302: 106 95 107 91 109 99 99 107 88 99 0.861264 978/1000 * NonOverlappingTemplate 304: 118 103 91 116 98 98 103 88 97 88 0.363593 978/1000 * NonOverlappingTemplate 304: 122 104 91 111 93 85 101 102 106 85 0.190654 977/1000 * NonOverlappingTemplate 306: 109 92 105 107 80 107 114 105 100 81 0.186566 978/1000 * NonOverlappingTemplate 306: 87 85 72 119 119 90 130 81 111 106 0.000083 * 991/1000 Serial 308: 110 107 80 91 115 103 111 93 83 107 0.132640 978/1000 * NonOverlappingTemplate 308: 105 109 99 98 105 97 90 101 99 97 0.979226 979/1000 * NonOverlappingTemplate 315: 102 110 90 114 88 94 95 92 106 109 0.548314 979/1000 * NonOverlappingTemplate 317: 116 80 85 97 151 99 95 93 100 84 0.000016 * 986/1000 NonOverlappingTemplate 319: 104 113 101 127 82 85 81 98 103 106 0.027497 979/1000 * NonOverlappingTemplate 320: 113 94 100 99 99 103 100 103 75 114 0.314544 979/1000 * NonOverlappingTemplate 320: 69 62 62 53 67 65 64 66 52 55 0.792601 598/615 * RandomExcursions 324: 105 95 105 104 89 101 102 96 113 90 0.832561 979/1000 * NonOverlappingTemplate 325: 112 107 91 90 94 105 101 105 100 95 0.846338 979/1000 * NonOverlappingTemplate 326: 113 103 106 105 91 101 103 101 88 89 0.743915 978/1000 * NonOverlappingTemplate 326: 129 90 102 98 119 86 89 104 102 81 0.017425 979/1000 * NonOverlappingTemplate 327: 121 105 108 107 107 88 84 95 103 82 0.127393 979/1000 * OverlappingTemplate 328: 115 94 86 95 109 107 90 91 120 93 0.190654 979/1000 * NonOverlappingTemplate 329: 119 88 86 90 102 120 80 96 112 107 0.033584 979/1000 * NonOverlappingTemplate 336: 118 98 101 102 101 94 106 98 87 95 0.735908 974/1000 * NonOverlappingTemplate 336: 123 95 94 97 113 79 102 96 100 101 0.196920 978/1000 * NonOverlappingTemplate 347: 80 76 115 81 139 98 117 89 107 98 0.000049 * 994/1000 NonOverlappingTemplate 349: 65 57 70 67 57 63 69 56 65 61 0.929192 614/630 * RandomExcursions 350: 121 114 96 90 100 101 96 97 95 90 0.433590 976/1000 * NonOverlappingTemplate 355: 67 67 57 56 84 66 62 62 54 58 0.295803 616/633 * RandomExcursions 357: 135 107 77 87 100 111 97 84 113 89 0.001704 978/1000 * NonOverlappingTemplate 358: 119 90 102 99 93 104 93 99 104 97 0.733899 979/1000 * LinearComplexity 360: 115 106 105 102 89 100 96 86 103 98 0.703417 979/1000 * NonOverlappingTemplate 363: 112 110 97 107 97 79 97 96 106 99 0.520102 979/1000 * NonOverlappingTemplate 366: 129 104 114 82 78 92 102 92 110 97 0.012562 975/1000 * FFT 369: 63 64 72 59 60 61 63 58 57 56 0.957439 598/613 * RandomExcursions 371: 118 98 100 89 103 104 81 96 107 104 0.422638 979/1000 * NonOverlappingTemplate 373: 118 102 120 101 99 98 77 103 83 99 0.075254 979/1000 * NonOverlappingTemplate 374: 119 96 99 103 101 109 101 87 109 76 0.164425 978/1000 * NonOverlappingTemplate 376: 65 62 59 56 65 58 54 50 57 59 0.942955 570/585 * RandomExcursions 378: 93 100 111 108 99 99 99 99 93 99 0.968863 979/1000 * NonOverlappingTemplate 379: 104 107 89 104 95 120 104 91 86 100 0.419021 978/1000 * NonOverlappingTemplate 389: 75 58 56 69 61 74 65 65 62 56 0.674478 626/641 * RandomExcursions 390: 99 93 100 94 90 114 115 96 98 101 0.711601 979/1000 * NonOverlappingTemplate 393: 103 81 90 100 113 98 103 91 111 110 0.388990 977/1000 * NonOverlappingTemplate 398: 71 65 66 64 65 65 53 56 57 66 0.866664 612/628 * RandomExcursions 400: 94 107 95 111 108 91 107 111 93 83 0.452173 979/1000 * NonOverlappingTemplate 405: 113 96 113 93 104 88 101 102 96 94 0.719747 978/1000 * NonOverlappingTemplate 406: 115 98 99 103 111 95 116 78 84 101 0.128874 976/1000 * NonOverlappingTemplate 406: 118 105 92 89 110 103 104 86 103 90 0.380407 979/1000 * Universal 408: 102 109 102 109 107 82 102 97 95 95 0.733899 979/1000 * Serial 413: 101 97 98 80 104 110 105 99 108 98 0.715679 979/1000 * NonOverlappingTemplate 416: 73 54 68 53 61 70 59 64 59 70 0.637119 612/631 * RandomExcursionsVariant 416: 67 67 63 47 70 60 67 51 80 59 0.168437 616/631 * RandomExcursionsVariant 421: 93 104 112 97 84 89 98 116 96 111 0.357000 979/1000 * NonOverlappingTemplate 422: 99 105 97 93 84 101 100 112 102 107 0.800005 1000/1000 * CumulativeSums 424: 66 69 48 69 67 54 57 81 52 65 0.108170 613/628 * RandomExcursions 427: 110 89 88 101 97 92 98 110 99 116 0.534146 979/1000 * NonOverlappingTemplate 427: 56 72 44 67 67 56 62 57 70 61 0.324600 597/612 * RandomExcursionsVariant 432: 126 95 88 91 97 106 113 110 90 84 0.067722 979/1000 * NonOverlappingTemplate 434: 98 100 98 97 100 92 97 99 104 115 0.950247 979/1000 * NonOverlappingTemplate 438: 104 102 89 92 105 101 94 104 97 112 0.886162 979/1000 * NonOverlappingTemplate 449: 138 101 98 88 101 100 105 69 88 112 0.000734 977/1000 * FFT 451: 115 99 83 88 84 119 97 108 101 106 0.127393 979/1000 * NonOverlappingTemplate 451: 115 100 82 88 84 117 100 106 103 105 0.159020 979/1000 * NonOverlappingTemplate 455: 109 91 104 89 106 93 86 101 117 104 0.450297 978/1000 * NonOverlappingTemplate 455: 133 102 77 97 100 89 102 97 103 100 0.038313 977/1000 * NonOverlappingTemplate 459: 113 97 91 123 94 112 80 88 108 94 0.064418 979/1000 * Frequency 460: 120 102 103 87 104 88 105 92 89 110 0.310049 978/1000 * NonOverlappingTemplate 461: 100 98 103 109 110 81 104 79 110 106 0.244236 977/1000 * NonOverlappingTemplate 464: 126 82 99 81 87 109 104 109 102 101 0.046568 972/1000 * NonOverlappingTemplate 465: 76 63 63 56 56 65 68 54 46 65 0.314618 597/612 * RandomExcursionsVariant 466: 62 61 63 54 66 51 65 62 85 63 0.239540 613/632 * RandomExcursions 472: 78 117 76 89 104 105 84 146 97 104 0.000011 * 994/1000 NonOverlappingTemplate 477: 107 88 82 109 99 98 116 97 111 93 0.320607 978/1000 * NonOverlappingTemplate 477: 101 114 79 102 93 119 74 89 139 90 0.000067 * 987/1000 NonOverlappingTemplate 479: 103 105 111 94 67 107 119 85 111 98 0.015598 979/1000 * NonOverlappingTemplate 479: 115 95 100 109 81 104 101 100 83 112 0.248014 978/1000 * NonOverlappingTemplate 479: 122 101 103 106 89 87 92 116 90 94 0.173770 979/1000 * NonOverlappingTemplate 482: 117 105 96 86 98 97 102 93 102 104 0.727851 979/1000 * NonOverlappingTemplate 483: 120 106 99 95 77 104 102 118 90 89 0.076658 978/1000 * NonOverlappingTemplate 486: 123 114 101 84 93 103 98 85 90 109 0.105618 979/1000 * NonOverlappingTemplate 487: 105 89 114 93 82 114 112 87 105 99 0.186566 978/1000 * NonOverlappingTemplate 488: 114 105 97 99 106 87 119 85 86 102 0.201189 977/1000 * NonOverlappingTemplate 494: 95 93 91 108 94 116 99 123 83 98 0.147815 979/1000 * NonOverlappingTemplate 495: 105 109 105 82 113 84 100 90 117 95 0.165340 979/1000 * NonOverlappingTemplate 495: 127 117 89 100 94 104 96 105 91 77 0.030602 978/1000 * NonOverlappingTemplate 496: 115 99 83 82 111 115 113 105 86 91 0.056069 979/1000 * NonOverlappingTemplate 496: 96 94 103 112 101 98 110 87 109 90 0.678686 979/1000 * NonOverlappingTemplate 496: 108 106 102 105 88 83 116 92 102 98 0.446556 978/1000 * NonOverlappingTemplate 496: 112 108 109 114 86 93 88 91 109 90 0.251837 979/1000 * Universal 497: 111 87 87 96 92 109 95 102 119 102 0.339271 979/1000 * NonOverlappingTemplate