Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
opt(Restore): Make restore map phase faster (#8038)
With this change, we can get ~450 MBps output throughput for map phase sustained. This is what we got on a 48 core AWS machine. ``` alpha1 | I0917 08:03:27.921912 17 restore_map.go:547] Restore MAP 01h04m30s len(reqCh): 0 len(writeCh): 0 read: 472 GiB. output: 1.6 TiB. rate: 437 MiB/sec. nextFileId: 2474 writers: 0 jemalloc: 0 B. alpha1 | I0917 08:03:27.921934 17 restore_map.go:559] Restore MAP Done in 01h04m30s. ``` Changes: * Make numGo equal to number of cores * Don't throttle. * Create a bigger buffer before calling merge * Rewrite mapper to remove sendForWriting out of the critical path. * Fix up a deadlock * Fix up the mapper * add write ch * Use many goroutines to create map files * Use fewer goroutines * Use 75% of the cores for mapping * Stagger the writes * Half writers can write at a time * Print num writers * Range for file would be 1/4 to 1 * Reduce number of pending requests Co-authored-by: Ahsan Barkati <[email protected]>
- Loading branch information