Skip to content

Port the PaddlePaddle deltas from paddle/v1.27.0 onto 1.28.0 - #12

Merged
sneaxiy merged 9 commits into
PFCCLab:paddle/v1.28.0from
ForFishes:paddle/v1.28.0-port
Sep 3, 2026
Merged

Port the PaddlePaddle deltas from paddle/v1.27.0 onto 1.28.0#12
sneaxiy merged 9 commits into
PFCCLab:paddle/v1.28.0from
ForFishes:paddle/v1.28.0-port

Conversation

@ForFishes

@ForFishes ForFishes commented Sep 3, 2026

Copy link
Copy Markdown

Carry the PaddlePaddle deltas from paddle/v1.27.0 over to the new
paddle/v1.28.0 branch. Without them Paddle cannot drive these kernels at all
(the dtype tables resolve against PyTorch, TensorDesc rejects a Paddle place,
torch.memory_format does not exist, numel() becomes a blocking D2H, ...).

Base is paddle/v1.28.0 = upstream 1.28.0-rc @ 601b3f35f, which has the same
tree (b4b9b06cc) as the v1.28.0 release tag, so the branch is pristine
upstream 1.28.0 and this PR is exactly the Paddle delta on top of it.

Produced with git rebase --onto <1.28.0 base> 0018f8dff paddle/v1.27.0, so every
original commit keeps its author, message and granularity.

Commits

# commit ported from upstream 1.28.0 status
1 Compat with PaddlePaddle 7f42e084 fork-private
2 add compat for nvtx.range and ExternalStream (#2) ab717cb2 fork-private
3 fix device=cuda move (#3) 38ede449 fork-private
4 [Fix] Protect prob <=1 when cudnn backward #4 6ed7850f not fixed — exp2 argument still unclamped
5 Fix v1.27.0 error (#8) 2aef928b fork-private
6 DSA: fix dK SMEM handoff race and two latent NaN sources #9 b4389638 not fixed — no dk_reduce_barrier, both denom_out buffers still uninitialised
7 DSA: keep the indexer backward grad_loss guard host-side #11 1afa2ff3 not fixed — predicate still built from numel()
8 DSA: apply the same exp2 clamp to the H16 specialisation new see below
9 Apply black (line-length 160) to the ported files new see below

Two of the nine paddle/v1.27.0 commits are not here, because on this base
they carry no content:

The v1.27.0 indexer_backward_sm100.py performance regression (range_constexpr
range(unroll_full=True), upstream NVIDIA#376) is already fixed in 1.28.0 by
upstream NVIDIA#549, so nothing is carried for it.

Conflict resolutions worth reviewing

Upstream reworked the code four of the fork hunks sat on:

  1. datatypes.py _torch_to_cutlass_data_type — 1.28.0 resolves the
    framework via sys.modules.get("torch") instead of importing it. A real
    PyTorch is commonly installed alongside Paddle, so that probe hands back
    PyTorch and every Paddle dtype misses the mapping. Changed to probe
    "paddle", which keeps upstream's "do not import on behalf of another
    framework" property. This one is load-bearing: without it
    _torch_to_cutlass_data_type(paddle.bfloat16) returns None and every
    downstream dtype check silently degrades.
  2. api_base.py TensorDesc.__post_init__ — gained the same _torch()
    probe plus a framework-neutral Device. Probes Paddle directly and keeps
    Device in the accepted set.
  3. api_base.py is_contiguous — the v1.27.0 workaround is resolved
    in favour of upstream. The old signature was memory_format: torch.memory_format = torch.contiguous_format, which fails at def time
    because Paddle has no memory_format; upstream changed it to Any = None
    and guards every torch reference, so the raise NotImplementedError stopgap
    is no longer needed and the contiguity check works again under Paddle.
  4. dsa_bwd_sm100.py — rewritten upstream by Improve performance of SM100 DSA backward kernel NVIDIA/cudnn-frontend#684, so [Fix] Protect prob <=1 when cudnn backward #4's clamp landed at
    the new exp2 site by hand rather than as a context-matched patch.

Commit 8 is the one change that is not a port: dsa_bwd_sm100_h16.py is new
in 1.28.0 (NVIDIA#664, added after v1.27.0 was cut), is a copy of the same epilogue and
carries the identical unclamped exp2. Drop that commit if you would rather keep
the branch a strict port.

Commit 9 exists because the ported Paddle code predates the current
.pre-commit-config.yaml (black, --line-length 160) and six files were not
format-clean; the 1.28.0 base already was.

Verification

black --line-length 160 --fast clean over all 18 touched files. No C/C++/CUDA
files touched, so clang-format is a no-op. All 18 files byte-compile.

Smoke-tested on B30Z (sm_103) with Paddle 3.4.0 under
paddle.enable_compat(scope={"cudnn"}), which is how paddlefleet_ops drives
this package:

  • import cudnn succeeds and the eager api_base / DSA imports resolve (23 DSA symbols)
  • _torch_to_cutlass_data_type: paddle.bfloat16BFloat16, paddle.float32Float32
  • _is_torch_tensor(paddle_tensor) → True; to_cute_tensor builds a cute tensor
  • _validate_grad_loss_tensor accepts (), (1,), (1,1) → shape (1,); rejects numel 2 and fp16; and with Tensor.numel monkeypatched to raise, still passes — i.e. the guard is provably host-side
  • TensorDesc accepts a Paddle place and is_contiguous() returns True
  • resolve_stream / torch_stream_context round-trip

Not verified: no kernel was actually launched, so the #9 dK barrier and the #4
clamp are carried on code review plus their original v1.27.0 validation, not on a
fresh numerical run.

Merge requirements

merge-requirements currently fails for two reasons, and only one of them is
mine to fix:

  • this PR has no Milestone — please set it from the sidebar;
  • the repository secret PROJECT_READ_TOKEN is not configured, so the workflow
    cannot read the Projects field and fails closed. That affects every PR on this
    repo, not just this one, and needs a repo admin (or the
    cat-routine-update label as an exemption).

SigureMo and others added 9 commits September 3, 2026 21:54
Co-authored-by: root <root@tjdm-9d8tvhbv.bcc-tjdm.baidu.com>
Not a port: dsa_bwd_sm100_h16.py is new in upstream 1.28.0 (NVIDIA#664, added
after v1.27.0 was cut).  It is a copy of the dsa_bwd_sm100.py epilogue and
carries the identical unclamped exp2 that PFCCLab#4 fixes in the main kernel, so
the same NaN is reachable there.

Drop this commit if you would rather keep the branch a strict port of
paddle/v1.27.0.
The ported PaddlePaddle code predates the current .pre-commit-config.yaml
(black, --line-length 160), so six files were not format-clean.  The
1.28.0 base already was, and the pre-commit workflow runs
`pre-commit run --all-files` on the PR head.
@sneaxiy
sneaxiy merged commit 572ff23 into PFCCLab:paddle/v1.28.0 Sep 3, 2026
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants