Speed up the commons functions #2
Conversation
My 10M (sqlite3_opt_batched) run goes from 42s to 26s. Switching to https://docs.python.org/3/library/os.html#os.urandom would make this even faster, but I don't think that's the point of the exercise so leaving it at this. I think it's still nice to have because it reduces the time by a lot and makes it _faster for me to benchmark_
huh, I wonder why is this happening. This is for 100M records or were you trying with bigger num? The way it works currently is, it builds the entire DB in memory and flushes to disk at the end. However, it should not take lot of memory (I am assuming you have 8GB+ memory) |
|
16GB total memory, but I do run earlyoom so that might have had an impact. I was also trying mostly against a RAM disk ( |
|
Thank you! I am merging this now. I will post the new numbers and how much faster it has gotten on my machine |
|
I ran the script today, for CPython, the running time reduced to almost half. Thats crazy! Before: After: |
My 10M (sqlite3_opt_batched) run goes from 42s to 26s.
Switching to https://docs.python.org/3/library/os.html#os.urandom would
make this even faster, but I don't think that's the point of the
exercise so leaving it at this. I think it's still nice to have because
it reduces the time by a lot and makes it faster for me to benchmark
result for
threaded_batched.pyagainstpypy3on my i7 for all 100M records:real 111.251 user 104.187 sys 4.483 pcpu 97.68I seem to be running out of memory though (process keeps getting OOM-killed, wondering if that can be fixed somehow).