Conversation
I've focused on the slowest parts, and been unable to speed them up. Pushing this up just for discussion right now.
n1zyy
left a comment
n1zyy
left a comment
There was a problem hiding this comment.
Note that all of this is experimentation. Lots and lots of it has room for improvement, but I'm just trying to show some of what I've poked at. Really my conclusion here is "I'm not sure this is a worthwhile endeavor," so please don't waste a lot of cycles trying to improve my timekeeping code.
| puts "#{elapsed} sec. from '#{@previous_timer}' to '#{name}'" | ||
| end | ||
| @previous_timer = name | ||
| end |
There was a problem hiding this comment.
Ignore this mediocre method.
There was a problem hiding this comment.
Have you seen the JobHelpers::Timer class we have? example usage in
identity-idp/app/jobs/resolution_proofing_job.rb
Lines 122 to 124 in 10e3f8b
There was a problem hiding this comment.
I did, but not until I'd already written this. 😿
| f.write(data) | ||
| `gzip #{f.path}` | ||
| timestamp(:gzip_shell) | ||
| end |
There was a problem hiding this comment.
This is dumb and not using dir at all, and I wouldn't interpolate unescaped arguments inside backticks if this code were going anywhere. But, the outcome of this is that shelling out to gzip here is not any faster than when we call Zlib.gzip above. It takes the same amount of time.
|
|
||
| key = cipher.random_key | ||
| iv = cipher.random_iv | ||
| timestamp(:get_and_iv_assignment) |
There was a problem hiding this comment.
| timestamp(:get_and_iv_assignment) | |
| timestamp(:key_and_iv_assignment) |
But this is just insignificant instrumentation that shouldn't be merged.
| file.close | ||
| timestamp(:large_file_write) | ||
| `xxd -p -c 0 base16_me.txt > base16_done.txt` | ||
| timestamp(:xxd) |
There was a problem hiding this comment.
Again, this is sloppy experimental code, but the outcome is that the call to xxd is taking about 10x longer than the Base16.encode16 call. I'm curious to dig into why, but on the surface, this does not look like a promising speed boost.
There was a problem hiding this comment.
What sizes of data did you use for these tests? Previously it took until like 100k batches for the shell to win
There was a problem hiding this comment.
I did 260k events. (10k, then I added 250k because it was too small.)
| ) | ||
| end | ||
| puts 'Done!' | ||
| end |
There was a problem hiding this comment.
This is shockingly slow; about 3 minutes and CPU-bound. I should move the event generation outside the loop, including the .to_jwe call, so the inside of the loop is just writing static strings to Redis.
Initially this was 10k instead of 250k and the couple seconds it took was immaterial. I wanted the data to be quasi-realistic.
|
Closing out as most of this failed to give any performance increase. Replaced by #7706 |
2 participants
I've focused on the slowest parts, and been unable to speed them up. Pushing this up just for discussion right now.