-
Notifications
You must be signed in to change notification settings - Fork 78
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement XORoshiro-256 PRNG for Enhanced Random Number Generation Efficiency #555
Implement XORoshiro-256 PRNG for Enhanced Random Number Generation Efficiency #555
Conversation
In evaluating this algorithm, I observe significant advantages, especially for legacy computing environments. Notably, CPUs from the Pentium 4 era and earlier stand to gain from the algorithm's simplicity and efficiency. The algorithm not only produces statistically high-quality numbers but also impressively passes the NIST test in 186 out of 188 instances. This level of performance suggests it is more than adequate for tasks like HDD wiping. In terms of speed, it is estimated to operate approximately 4-5 times faster than the MT19937 algorithm, presenting a potential area for further exploration on older hardware platforms. The suggestion here is that in the context of pseudorandom number generation for these applications, the bottleneck might more likely be the computational capacity of the CPU rather than the HDD's capabilities. It would be interesting to see this algorithm's performance tested on such older systems to validate these observations. Please verify as well for thread safety. |
Another thing we might take into consideration. It is possible to speed up this code by around 100% again when using AVX2 instructions. But it will make the code much less portable. Just wanted to mention this. My current implementation uses simple operations, so it will work regardless of the platform. |
@Knogle xoroshiro256 passes, no verification errors on a 18 drive wipe. Looking good! 👍 Only problem regarding merging this, is that the AES-CTR code is present in the xoro branch. I should have really got you to put the AES-CTR code into it's own branch as well. That's why you should always create a branch, not make changes directly to the master (except when you are working alone, but even then I think it's a good idea to always create a branch). Personally, what I would do is make a backup of the xoro and AES code somewhere. Delete your fork, refork nwipe then create a AES-CTR branch Your master will be zero commits ahead of the upstream master. Once I merge them you can then switch to your local master, delete your branches and There are no doubt cleverer ways of doing this like stashing & popping stashes but I tend to only do that from within the Kdevelop IDE rather than on the command line. However separating AES and xoro in separate branches makes it so much easier to merge and keep different features separate just in case something needs to be reverted due to some unforeseen problem. |
The problem you have is that your master contains the non merged AES-CTR code and your xoro branch contains both the non merged AES-CTR and xoro code. Your master should match the upstream master, it can be x commits behind but not x commits ahead of upstream and then the two branches. That's why I suggest reforking. You could probably revert your master back but it might be cleaner and quicker to delete you github fork and refork it. |
Hey, i am back from my vacation, i will try to take a look today or tomorrow in order to fix that :) |
Did an interactive rebase now, can you check if it works? |
What i've found out. I'm currently printing all random numbers into a text file. ROUND 1 + ROUND 1 VERIFICATION: Both files match, everything OK! |
Sorry for creating confusion. This was related to AES. |
I think xoro is looking good, I ran simultaneously on 18 discs without any verification issues so should be able to merge that soon. What are your plans for AES? Are you going to use openssl's thread locking functions or write your own AES prng? |
Sounds great!
Means: Let's say the key is
result will be 12345678 Now nwipe want's to verify the stream, but after the function was called,
result will be 45678910 The AES key which is derived form the seed by nwipe.c is still the same, but the other values differ. So i have to make sure somehow, that the complete state is the same and deterministic. |
As requested by others, performance of ISAAC-64 regarding the NIST suite.
|
Checking a 16 drive wipe on Xoro tonight, then I'll work on merging the sha-hmac tomorrow. nwipe_xoro_test-2024-03-21_23.04.18.mp4 |
Sounds great! Is the data throughput also acceptable? |
Yes, throughput is very good. Thanks for adding those two prngs. Much appreciated. |
This pull request introduces the implementation of the XORoshiro-256 pseudorandom number generator (PRNG), a significant enhancement to our project's capabilities in generating high-quality random numbers. XORoshiro-256, standing for "XOR/rotate/shift/rotate", is part of the well-regarded XORoshiro family of PRNGs. This PR aims to leverage the algorithm's advantages, including speed, efficiency, and the quality of randomness, to benefit various non-cryptographic applications within our project.
Key Benefits of XORoshiro-256:
1. High Performance:
XORoshiro-256 is designed for speed, making it an excellent choice for applications requiring a high volume of random numbers with minimal computational overhead. Its efficient use of XOR, shift, and rotate operations ensures fast execution on modern hardware. Around 1.8GB/s when writing to a ramdisk.
2. Excellent Statistical Properties:
The algorithm has been rigorously tested and exhibits excellent statistical properties for a wide range of applications. Its randomness quality meets the standards of most non-cryptographic use cases, including simulations, gaming, and randomized algorithms.
3. Low Memory Footprint:
With a state size of only 256 bits, XORoshiro-256 maintains a minimal memory footprint, making it suitable for applications with limited memory resources or those requiring multiple independent PRNG instances.
4. Easy to Implement and Use:
The simplicity of the XORoshiro family's design philosophy is preserved in XORoshiro-256, ensuring an easy implementation and integration process. This PR includes both the core algorithm implementation and utility functions for initialization and random number generation, providing a ready-to-use solution for immediate application benefits.
Integration Details:
The integration of XORoshiro-256 into our project is straightforward, with the PR including all necessary source and header files. Developers can initialize the PRNG with a seed and then generate random numbers as needed, with functions provided for both individual number generation and filling buffers with random data.
This addition represents a strategic enhancement of our project's foundational components, supporting a broad spectrum of future development opportunities and performance optimizations.
![dd15dfc8-e977-4711-ba53-8d799289213a](https://private-user-images.githubusercontent.com/26367057/313049530-4d29538e-716d-4613-8544-585f2354698d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU1OTMyNDMsIm5iZiI6MTcyNTU5Mjk0MywicGF0aCI6Ii8yNjM2NzA1Ny8zMTMwNDk1MzAtNGQyOTUzOGUtNzE2ZC00NjEzLTg1NDQtNTg1ZjIzNTQ2OThkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA2VDAzMjIyM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRhNjcwMGVhOTk5YjAwZGNlODY2YTJiYjliOWEwMTk4NTZlZDI2ZWIyMDUwY2ZjN2JhNmViMDhhYjMwZjE4YzMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.hbt-nRd1c546GODveYRqgT9tH9hRIfgElFk07Y7vCCY)