Implement non-mapped async IO for CUDA on Windows. #7896

Merged

slaren merged 7 commits into

ggml-org:masterfrom

mtavenrath:direct_io

Jun 17, 2024

Commits on Jun 12, 2024

Implement non-mapped async IO for CUDA on Windows. On a fast Gen5 NVMe drive this change improves model load time by >3x while it should be the same (or slightly faster) on any other drive.
mtavenrath
committed
Free resources except for backend.
mtavenrath
committed