[ROCm] ROCm triton pin update

[jataylo]

Bold attempt to see how the CI likes a triton commit bump :)

AMD specific updates:

• [AMD] Disable block merging to avoid block argument explosion triton-lang/triton#4176
  ... More to add

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @hongxiayang

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/130625

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures, 1 Cancelled Job, 2 Unrelated Failures

As of commit dae6a2f with merge base df0494b ():

NEW FAILURES - The following jobs have failed:

• inductor / cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
  AllenaiLongformerBase
• inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
  AllenaiLongformerBase
• inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
  cm3leon_generate
• inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
  test_ops 2/4 failed!
• inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
  test_ops_gradients 2/3 failed!
• inductor / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
  test_ops 2/4 failed!
• inductor / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
  test_ops_gradients 2/3 failed!
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (inductor_avx2, 1, 2, linux.10xlarge.avx2) (gh)
  test_ops 3/4 failed!
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (inductor_avx2, 2, 2, linux.10xlarge.avx2) (gh)
  test_ops_gradients 2/2 failed!
• inductor-periodic / cuda12.1-py3.10-gcc9-sm80 / test (inductor_torchbench_smoketest_perf, 1, 1, linux.gcp.a100) (gh)
  AllenaiLongformerBase
• linux-binary-manywheel / manywheel-py3_8-cuda11_8-split-test / test (gh)
  Process completed with exit code 1.
• linux-binary-manywheel / manywheel-py3_8-cuda11_8-test / test (gh)
  Process completed with exit code 1.
• linux-binary-manywheel / manywheel-py3_8-cuda12_1-split-test / test (gh)
  Process completed with exit code 1.
• linux-binary-manywheel / manywheel-py3_8-cuda12_1-test / test (gh)
  Process completed with exit code 1.
• linux-binary-manywheel / manywheel-py3_8-cuda12_4-split-test / test (gh)
  Process completed with exit code 1.
• linux-binary-manywheel / manywheel-py3_8-cuda12_4-test / test (gh)
  Process completed with exit code 1.
• rocm / linux-focal-rocm6.1-py3.8 / test (default, 6, 6, linux.rocm.gpu.2) (gh)
  test_transformers.py::TestSDPACudaOnlyCUDA::test_flash_attention_vs_math_ref_grads_batch_size_8_seq_len_q_512_seq_len_k_2048_head_dim_16_is_causal_False_dropout_p_0_0_float16_scale0_cuda_float16
• trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_AVX512, 1, 1, amz2023.linux.2xlarge) (gh)
  inductor/test_efficient_conv_bn_eval 1/1 failed!
• trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, amz2023.linux.2xlarge) (gh)
  inductor/test_efficient_conv_bn_eval 1/1 failed!
• trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 3, 5, amz2023.linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_torchinductor.py::GPUTests::test_any_cuda

CANCELLED JOB - The following job was cancelled. Please retry:

• Build Triton wheels / Build Triton Conda (3.11) (gh)
  ##[error]The operation was canceled.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, amz2023.linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
  'Test'
• rocm / linux-focal-rocm6.1-py3.8 / test (default, 3, 6, linux.rocm.gpu.2) (gh) (similar failure)
  inductor/test_kernel_benchmark.py::TestKernelBenchmark::test_split_scan

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[jataylo]

@pytorchbot rebase

[pytorchmergebot]

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

[pytorchmergebot]

Successfully rebased new-triton-rocm-pin-1207 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout new-triton-rocm-pin-1207 && git pull --rebase)

[jataylo]

Just testing moving upstream pin also so we can find ROCm exclusive failures

[jataylo]

I'm seeing many sympy related errors, hopefully these are fixed with a rebase. But we do see 1 or 2 real failures from the commit bump exclusively on ROCm -

TestSelectAlgorithm.test_addmm_fp16 - AssertionError: Incorrect result from choice TritonTemplateCaller(/tmp/tmp0qhe2wis/nl/cnlgjadb7fxqmphrkrgahould4qfakwpppbcamn3y7bd7v6rpu3o.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=16, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, matrix_instr_nonkdim=0, num_stages=0, num_warps=4)

Leads to Tensor-likes issue

TestCompiledAutograd.test_free_activation_memory - AssertionError: False is not true

But this was previously disabled so may be flakey/resolved issue

[jataylo]

@pytorchbot rebase

[pytorchmergebot]

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

[pytorchmergebot]

Successfully rebased new-triton-rocm-pin-1207 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout new-triton-rocm-pin-1207 && git pull --rebase)

[jataylo]

Not planned.