[Triton/Gluon] Add Triton-based Conv3D kernels

[saeid-rostami]

Motivation

This PR adds Triton Conv3D implementation with optimized
routing for common video-model and VAE workloads.

The public API supports FP16 and BF16 inputs, NCDHW and channels-last-3D
execution, optional bias, asymmetric stride/padding/dilation, and activations.

Technical Details

The Conv3D router selects between the following kernel families:

• Specialized 1x1x1 Conv3D
• NCDHWc 3x3x3 Conv3D with fused activation packing
• Native NDHWC 3x3x3 Conv3D
• General im2col-free Conv3D
• Winograd F(4x4, 3x3) over the H/W axes
• Winograd with an NCDHWc input path

The implementation also adds:

• Conv3D weight-prepack and Winograd-filter caches
• Layout-preserving NCDHW and channels-last-3D behavior
• Shape-driven production routing
• Complete launch keys containing all three spatial axes
• Tuned gfx1201 launch configurations
• Configuration-table coverage for gfx1200, gfx942, and gfx950, initially
  seeded from the gfx1201 values
• Public export through aiter.ops.triton
• Real-model-derived benchmark shape coverage for:
  • Wan2.2 A14B VAE
  • Wan2.2 TI2V-5B VAE
  • LTX-Video 0.9.5 VAE
  • InternVideo2 and Qwen-family video patch embeddings

The benchmark reports per-layer correctness, selected Triton kernels,
kernel-only and repack-inclusive performance, aggregate throughput, and
call-count-weighted encoder/decoder totals. It exits nonzero when a requested
shape errors or fails correctness.

Test Plan

The Conv3D test suite covers:

• FP16 and BF16
• NCDHW and channels-last-3D layouts
• Every direct Conv3D route
• Optional bias
• none, relu, relu6, and gelu activations
• Asymmetric dimensions, stride, padding, and dilation
• Batched, odd-channel, and tail-masking shapes
• Winograd transform and output correctness
• Output dtype preservation
• Weight-cache reuse, invalidation, and LRU eviction
• Actual production-router dispatch
• Benchmark database, reporting, and CLI behavior
• Invalid argument and unsupported configuration handling

Test Result

Tests were run on an AMD Radeon AI PRO R9700 (gfx1201) after clearing the
Triton compilation cache.

Conv3D: 174 passed in 223.69s

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
multigpu	Aiter multi-GPU tests on the 8-GPU runner
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5640 --add-label <label>

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to opt this PR out.

[brunomazzottiamd]

@saeid-rostami, as far as I can see your PRing from your fork. Try to publish your feature branch to ROCm repo and then PR the feature branch to main branch in the same ROCm repo.

Lots of Triton CI test shards are failing with the following error:

docker: Error response from daemon: toomanyrequests: You have reached your unauthenticated pull rate limit.

This happens because CI isn't logging to Docker Hub, with CI team credentials. This step is skipped when PRing from a fork:

https://github.com/ROCm/aiter/blob/main/.github/workflows/triton-test.yaml#L183

      - name: Docker login
        if: ${{ github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork }}
        env:
          DOCKER_USERNAME: ${{ secrets[needs.ci_config.outputs.docker_username_secret] }}
          DOCKER_PASSWORD: ${{ secrets[needs.ci_config.outputs.docker_password_secret] }}

[brunomazzottiamd]

@saeid-rostami, as far as I can see your PRing from your fork. Try to publish your feature branch to ROCm repo and then PR the feature branch to main branch in the same ROCm repo.

Lots of Triton CI test shards are failing with the following error:

docker: Error response from daemon: toomanyrequests: You have reached your unauthenticated pull rate limit.

This happens because CI isn't logging to Docker Hub, with CI team credentials. This step is skipped when PRing from a fork:

https://github.com/ROCm/aiter/blob/main/.github/workflows/triton-test.yaml#L183

      - name: Docker login
        if: ${{ github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork }}
        env:
          DOCKER_USERNAME: ${{ secrets[needs.ci_config.outputs.docker_username_secret] }}
          DOCKER_PASSWORD: ${{ secrets[needs.ci_config.outputs.docker_password_secret] }}

Postponing this since it requires opening a new PR. We can move on to a new branch / PR at ROCm fork if Docker rate limiting becomes a real issue in the future.

[brunomazzottiamd]

More comments on the Conv3D wrapper function and Conv3D unit tests.

[brunomazzottiamd]

Please make sure we can run these unit tests in gfx942 and gfx950 AITER CI runners. They will be a guarantee that future commits won't break Conv3D correctness! Remember that we don't have RDNA runners!

You don't need to tune the kernels for CDNA, just pick any reasonable block size and related meta parameters - be conservative in this regard. I'll be happy if everything is green in CDNA CI (no perf promises at all).

[saeid-rostami]

I think this is because this PR coming from my fork and PR cannot access the Rocm CI, I'll fix this part when all the reviews done.

[brunomazzottiamd]

PLMK when you're able to push to ROCm fork, then I can give it a quick try locally in some CDNA hosts.

[brunomazzottiamd]

LGTM! Let's wait for @nidal567 last feedback!

[nidal567]

I think Rule 4 is not fully satisfied. The Conv3D config files are in the expected architecture folders, and they are placed in their correct respective directories, but the exact-shape pins under gfx942 and gfx950 are copied from the gfx1201 tuning data. The loader resolves exact-shape entries before M_LEQ_* buckets and the generic fallback, so production Conv3D workloads can select launch parameters that were not validated for these target CDNA architectures.

I would say to ensure this is validated across the CDNA archs. The safe fix is to remove the conv3d_* directories from gfx942 and gfx950, or replace them with independently measured CDNA configs. Otherwise the fallback safety path is bypassed and Rule 4 is not met.

Other than that concern, LGTM!

[saeid-rostami]

I think Rule 4 is not fully satisfied. The Conv3D config files are in the expected architecture folders, and they are placed in their correct respective directories, but the exact-shape pins under gfx942 and gfx950 are copied from the gfx1201 tuning data. The loader resolves exact-shape entries before M_LEQ_* buckets and the generic fallback, so production Conv3D workloads can select launch parameters that were not validated for these target CDNA architectures.

I would say to ensure this is validated across the CDNA archs. The safe fix is to remove the conv3d_* directories from gfx942 and gfx950, or replace them with independently measured CDNA configs. Otherwise the fallback safety path is bypassed and Rule 4 is not met.

Other than that concern, LGTM!

Thanks for pointing this out. I only tuned the Conv3D kernels on RDNA hardware. The gfx942 and gfx950 configurations were initialized using the gfx1201 values so the Conv3D paths could run in AITER CI, same approach that we did on Conv2d PR,. Those variables have not been performance-tuned or fully validated for CDNA. I’m happy to help with that work, but I’m on the RDNA team and do not have access to CDNA machines. If someone with access to those systems help run the configurations and share the results, I can make any necessary adjustments.

[brunomazzottiamd]

@nidal567 and @saeid-rostami, for the Conv2D PR I added untuned gfx942 and gfx950 configs with just "any" entry, using safe defaults (small block sizes, num_stages=1 to avoid LDS exhaustion errors).

Take the following two files as an example.

aiter/ops/triton/configs/gfx942/triton/conv/conv_general/DEFAULT.json:

{
    "any": {
        "BLOCK_M": 64,
        "BLOCK_N": 64,
        "BLOCK_K": 64,
        "GROUP_SIZE_M": 4,
        "num_warps": 4,
        "num_stages": 1
    }
}

aiter/ops/triton/configs/gfx950/triton/conv/conv_general/DEFAULT.json:

{
    "any": {
        "BLOCK_M": 64,
        "BLOCK_N": 64,
        "BLOCK_K": 64,
        "GROUP_SIZE_M": 4,
        "num_warps": 4,
        "num_stages": 1
    }
}

I think this is the best approach because:

1. we don't pollinate CDNA configs with RDNA configs - there's no assumption of similarity
2. it's crystal clear that there's no tuning at all for CDNA
3. safe configs, as described before, will guarantee UTs passing in CDNA

We don't need to tune for CDNA, it's out of scope - we won't do this unless we have a clear demand.

Does it work IYO?

[saeid-rostami]

@brunomazzottiamd
Based on new rule for pushing to Rocm repos we have to create Fine Grained Token; the ordinary token are no longer accepted. I created one last week and still waiting for approval. After getting approved, I'll create new PR and close this one.

[saeid-rostami]

Closing this PR in favor of #5952, which contains the same Triton Conv3D work
and all follow-up review fixes from this discussion. The replacement PR uses an in-
repository ROCm/aiter branch so the full CI pipeline can access the organization’s
credentials.

Thank you, @brunomazzottiamd and @nidal567, for the reviews and guidance. Please continue
any remaining review and CI validation on #5952. This PR is being closed only
because the source branch moved; the work itself is continuing in the replacement PR.