[FRONTEND] let CacheManager write to temp dir instead of temp file

[yundai424]

Summary

there've been multiple issues discussing around the FileNotFoundError on compilation when CompiledKernel is trying to read from the listed ASM files. #2688 #4002 vllm-project/vllm#6103 etc. and there have been some attempts to address it such as #3544 . This PR attempts to explain the root cause and suggest a fix.

Why

When a kernel is being compiled, triton first writes IRs to triton cache dir (ref). Inside of the write operation, the process first writes it to a temp file unique to the current process (plus a uuid to distinguish between multiple processes with same PID on different hosts sharing the same underlying FS) (ref) and then atomically os.replace it to the final file name. Afterwards the CompiledKernel lists all the IRs and reads them (ref).

On multiprocess set up this may however result in a race condition. Let's focus on a case where there's one host with 2 processes on it.

At the time when pid 1 lists ASMs, the dir may contain temp files generated from another process pid 2. However at the time when pid 1 proceeds to read bytes from the listed files, pid2 may have already os.replaceed its temp files, so pid 1 will encounter FileNotFoundError when trying to read the temp file generated by pid 2. IBM/vllm#35 (comment) also believes this is the root cause.

How

There're multiple potential solutions towards this, as mentioned in IBM/vllm#35 (comment) as well:

• let each process write to a private temp dir instead so glob won't bother taking the temp stuff into consideration
• or, exclude tmp.pid_* from glob

This PR tries to go with the 1st approach to avoid adding an assumption on the tmp file pattern (which is only used in runtime/cache.py) in compiler/compiler.py but is open to any suggestion. Thanks!

Complete the following tasks before sending your PR, and replace [ ] with
[x] to indicate you have done them.

• I am not making a trivial change, such as fixing a typo in a comment.

• I have written a PR description following these
  rules.

• I have run pre-commit run --from-ref origin/main --to-ref HEAD.

• Select one of the following.

  • I have added tests.
    • /test for lit tests
    • /unittest for C++ tests
    • /python/test for end-to-end tests
  • This PR does not need a test because not applicable.

• Select one of the following.

  • I have not added any lit tests.
  • The lit tests I have added follow these best practices,
    including the "tests should be minimal" section. (Usually running Python code
    and using the instructions it generates is not minimal.)

[ThomasRaoux]

Looks reasonable to me

[tdoublep]

awesome thanks @yundai424 - this looks like it should fix the issue we have in vLLM.

is there a rough ETA for when this would be in a Triton release? trying to plan whether we need to merge a work-around in the meantime.