[Refactor | Feature] Introducing NDSL backend

[FlorianDeconinck]

🚀 BREAKING CHANGE - Scheduled for 2026.02.00

Description

Historically backend argument has always directly related to gt4py's backend. But with time and more and more usage of dace, we have added orchestration, leading to a split in the concept.

Also, we are more and more specializing code/optimizations by backend which calls for centralization.

Lastly, the backend naming scheme makes sense for the framework developer (of gt4py or dace) but not for the ndsl user.

This PR aims at introducing a NDSL backend concept solving all the above:

• unified gt4py/dace entry point
• simplified naming scheme
• shortcuts to common used backend both by naming scheme and via API
• extensible class to embed future optimization pipeline and/or key specific code

Also:

• deprecate gt4py_utils.is_gpu_backend in favor of inlined Backend.is_gpu_backend
• deprecate gt4py_utils.is_fortran_aligned in favor of inlined Backend.is_fortran_aligned

How has this been tested?

Unit tests will run and decide it's fate.

pyFV3 update PR | pySHiELD update PR | pace update PR

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (e.g. add new modules to docs/docstrings/)
• My changes generate no new warnings
• Any dependent changes have been merged and published in downstream modules
• New check tests, if applicable, are included

[FlorianDeconinck]

This is ready for a first review @twicki / @romanc

It's MASSIVE change, focus on config/backend.py which is the true change that gets propagated everywhere.

ToDo:

• A unit tests (?)
• Branch out pyFV3, pySHiELD, Pace with a "2026.02.00" update and tests further with backend properly swapped

[romanc]

This refactor is so overdue - thanks digging in!

Two general comments w.r.t. the planned API refactor #243

1. Where does ndsl/config/backend.py fit into the newly thought directory structure?
2. Is it possible to avoid from ndsl import Backend inside NDSL? Might not be possible, but if we can avoid it now, we have less work later. I've only seen two of these imports: in dace_config.py and refine_transients.py.

For docs, the entry in mkdocs.yaml and the "corresponding md file" are missing. We need to automate / tool this up in the future ... it's too much manual work and easily forgotten.

[romanc]

Thanks for all the changes. I was starting to think how we are going to merge this PR with as little CI disruption as possible. I'd propose the following process

• with this PR, we skip the CI hooks to pyFV3, pySHiELD, and pace
• we prepare PRs in pyFV3, pySHiELD, and pace to update to NDSL 2026.02.00 (those could be used¹ to check that this PR doesn't break anything after updating)
• we merge this PR as the last one before the February release
• we release NDSL 2026.02.00
• we update pyFV3, pySHiELD, and pace to 2026.02.00
• we re-enable the CI hooks in NDSL as the first PR after the release

What do you think? Too much over-engineering and -thinking?

Footnotes

1. by temporarily pointing the CI hooks of NDSL towards these branches instead of skipping them. ↩

[FlorianDeconinck]

pyFV3 PR testing this PR

[FlorianDeconinck]

pySHiELD testing branch

[FlorianDeconinck]

PR ready for show.

⚠️ BEFORE MERGE TURN THE CI BACK ONTO develop, THEN FORCE MERGE ⚠️

[romanc]

Looking good 🥳