Adding semaphore to lower memory usage #1#4

Closed

thomasw21 wants to merge 7 commits intobigscience-workshop:mainfrom

thomasw21:semaphore

Commits on Jul 15, 2021

Adding semaphore to lower memory usage
Thomas
committed
Make semaphore sharable
Thomas
committed
Add back \n
Thomas
committed
Turns out there's a possiblity to deadlock, use imap_unordered instead
Thomas
committed

Commits on Jul 16, 2021

Convert to tensor in multiprocessing
Thomas
committed
This should be faster
Thomas
committed
Acquire lock via chunks instead
Thomas
committed