[DRIVER][VLLM] Auto-retry kernel compilation with large GRF mode on build failure

[Egor-Krivov]

Problem

When a Triton kernel requires too many registers on Intel XPU, the IGC backend
compiler fails with "total scratch space exceeds HW supported limit" (PTSS).
The existing retry logic in load_binary() only handles the case where
compilation succeeds but has a high spill count (>1000). When compilation
fails entirely, the error propagates immediately without attempting large
GRF mode.

Users must manually add grf_mode='256' to work around this, which is not
discoverable and differs from NVIDIA where such issues don't occur.

Fixes #3777

Solution

Extend the GRF retry logic in load_binary() (driver.c) to also cover
complete build failures:

• When zeModuleCreate fails with ZE_RESULT_ERROR_MODULE_BUILD_FAILURE
  and no GRF mode was explicitly set, clear the error and retry compilation
  with -cl-intel-256-GRF-per-thread
• If the retry succeeds, continue with the large-GRF kernel
• If the retry also fails, propagate the original error
• Print a recovery message to stderr so the user knows the initial error
  was handled
• Apply the same retry-on-failure pattern to the ocloc offline compilation
  path in make_zebin() (compiler.py)

The existing spill-count-based retry (successful compilation but >1000 spills)
is unchanged and still applies when the build-failure retry is not triggered.

Testing

Added parametrized test test_auto_grf_on_build_failure covering:

• grf_mode='default': build fails → auto-retries with large GRF → succeeds
• grf_mode='256': explicit large GRF → compiles directly, no retry
• grf_mode='128': explicit small GRF → fails, no retry (respects user choice)

When I run the original reproducer from #3777 I now get a pass:

(triton) (312) jovyan@jupyter-ekrivov:~/triton/intel-xpu-backend-for-triton/issues/3777/ut$ python test_old.py 
L0 build module failed. Log: 
warning: [RetryManager] Start recompilation of the kernel
in kernel: 'sample_recovered_tokens_kernel'

error: total scratch space exceeds HW supported limit for kernel sample_recovered_tokens_kernel: 270848 bytes (max permitted PTSS 262144 bytes)
error: backend compiler failed build.

(I): Build failure recovered by retrying with large GRF mode for "sample_recovered_tokens_kernel"

[Egor-Krivov]

I write to stderr because stderr at this point already contains error message from IGC and we need to tell the user that issue is probably fixed

[anmyachev]

Make sense for me. Since it's a change in the driver - let's update your test to cover more cases.

[anmyachev]

Let's test both: make_zebin and load_binary. I guess TRITON_XPU_GEN_NATIVE_CODE should help for this.

[anmyachev]

Should we raise the initial exception here to be align with make_zebin?

[Egor-Krivov]

@anmyachev I updated the PR based on your feedback, but GH doesn't show changes for some reason
main...egor/issue_3777

[Egor-Krivov]

Ok, after 36ab67d PR got updated.

[anmyachev]

One small comment, everything else is LGTM!

[jikunshang]

may I know which triton-xpu release will contain this fix?