feat[next]: SDFGConvertible Program for dace_fieldview backend #1742

DropD · 2024-11-19T11:00:16Z

Description

Add a decrator.Program subclass, which implements SDFGConvertible to dace_fieldview backend, analogous to the one in dace_iterator. Conditionally shadow decorator.Program with it and reactivate the orchestration tests by using dace_fieldview instead of dace_iterator.

One caveat: The toolchain is not ready for pure CompileTimeConnectivities in all cases yet, so the test_sdfgConvertible_connectivities had to be adjusted for the moment.

Requirements

All fixes and/or new features come with corresponding tests.
Important design decisions have been documented in the approriate ADR inside the docs/development/ADRs/ folder.

If this PR contains code authored by new contributors please make sure:

The PR contains an updated version of the AUTHORS.md file adding the names of all the new contributors.

src/gt4py/next/program_processors/runners/dace_fieldview/program.py

src/gt4py/next/ffront/decorator.py

edopao · 2024-11-19T11:36:46Z

src/gt4py/next/program_processors/runners/dace_fieldview/program.py

+                if conn_id not in self.connectivity_tables_data_descriptors:
+                    conn = self.connectivities[name]
+                    self.connectivity_tables_data_descriptors[conn_id] = dace.data.Array(
+                        dtype=dace.int64 if conn.index_type == np.int64 else dace.int32,


I think that dace dtypes has a utility to parse numpy types:
dace.dtypes.typeclass(conn.index_type)

Trying that lead to KeyError: dtype('int64') in typeclass().

and trying dace.dtypes.typeclass(conn.index_type.type) leads to wrong stencil results. Edit: this might have instead been caused by auto_optimize=False.

edit: after merging main, it passes. However according to type hints of NeighborTableOffsetProvider, your version should work and mine should fail. Yet, in the tests, the connectivities are set up in a way so that my version works and yours fails. The current version on the other hand is safe in both cases.

If I can get the test to produce the correct types so the official version works, I will change it. Otherwise I would prefer to leave this one until the neighbor table construction is anyway changed by what @havogt is working on.

Could not get it to work, it seems like the type hints are wrong, but there is no point investing time in this when the API will change heavily anyway.

src/gt4py/next/program_processors/runners/dace_fieldview/program.py

src/gt4py/next/program_processors/runners/dace_iterator/__init__.py

kotsaloscv · 2024-11-20T13:12:55Z

src/gt4py/next/program_processors/runners/dace_iterator/__init__.py

-                    continue
-                if param.id not in sdfg.gt4py_program_input_fields:
-                    continue
-                sdfg.offset_providers_per_input_field.setdefault(param.id, []).extend(list(shifts))


Same here.

All these properties are needed for the automatic halo exchange placement.

Results from discussion in a separate channel for "offset_providers_per_input_field":

it is correct, as only used in icon4py which uses "unstructured" fields. There is max one horizontal dimension per field.

trace_shifts should work™️ partially on GTIR (but not accross as_fieldop).

currently not actively used in icon4py, so defer implementation and put a TODO instead.

src/gt4py/next/program_processors/runners/dace_iterator/__init__.py

src/gt4py/next/program_processors/runners/dace_fieldview/program.py

edopao · 2024-11-25T09:03:43Z

src/gt4py/next/program_processors/runners/dace_fieldview/program.py

+        # Add them as dynamic properties to the SDFG
+        program = typing.cast(
+            itir.Program, gtir_stage.data
+        )  # we already checked that our backend uses GTIR


we already checked that our backend uses GTIR

where? otherwise we could make an assert isinstance().

We checked that self.backend is a DaCe backend. Probably should check that it is specifically the dace_fieldview backend.

I see, thank you for the explanation. No, not needed to check that it is the fieldview backend because the iterator backend will be removed soon (next week).

src/gt4py/next/program_processors/runners/dace_fieldview/program.py

src/gt4py/next/program_processors/runners/dace_iterator/__init__.py

DropD · 2024-11-26T13:15:23Z

src/gt4py/next/ffront/decorator.py

+    connectivities: Optional[common.OffsetProvider] = (
+        None  # TODO(ricoh): replace with common.OffsetProviderType once the temporary pass doesn't require the runtime information
+    )


@havogt I had to change this to satisfy mypy. Now it is also consistent with CompileTimeArgs

makes sense

havogt · 2024-11-26T13:22:47Z

src/gt4py/next/program_processors/runners/dace_fieldview/program.py

+        return {name for name in cls().visit(program) if name in field_param_names}
+
+
+class InputNamesExtractor(SymbolNameSetExtractor):


Could you move these utilities to some module in iterator/transforms. I would like to use them in other places.

DropD added 2 commits November 18, 2024 13:38

[wip] start adding sdfg convertible Program for dace-fieldview

4c41f02

add SDFGConvertible Program replacement to dace_fieldview

481af34

edopao reviewed Nov 19, 2024

View reviewed changes

DropD added 4 commits November 19, 2024 15:09

improve generate_sdfg args, remove old Program replacement

ad0a7b2

Merge branch 'main' into gtir-sdfg-convertible

307b537

turn auto_optimize back on in __sdfg__

7c4d197

disable auto_opt once again in __sdfg__

3c56151

edopao reviewed Nov 20, 2024

View reviewed changes

src/gt4py/next/program_processors/runners/dace_fieldview/program.py Outdated Show resolved Hide resolved

support only CUDA device type in dace_fieldview Program

c5b4c43

kotsaloscv self-requested a review November 20, 2024 09:34

kotsaloscv requested changes Nov 20, 2024

View reviewed changes

edopao reviewed Nov 20, 2024

View reviewed changes

src/gt4py/next/program_processors/runners/dace_fieldview/program.py Outdated Show resolved Hide resolved

[wip] bring back extra sdfg attributes for halo placement

835c115

edopao reviewed Nov 25, 2024

View reviewed changes

DropD commented Nov 25, 2024

View reviewed changes

src/gt4py/next/program_processors/runners/dace_iterator/__init__.py Show resolved Hide resolved

DropD added 4 commits November 26, 2024 11:16

partially add halo exchange helper attrs with tests

c126cb2

add dace/gt4py type parsing crosscheck

0b90583

Merge branch 'main' into gtir-sdfg-convertible

8c1f9f4

fix dace_fieldview.program tests

77c250b

DropD commented Nov 26, 2024

View reviewed changes

havogt reviewed Nov 26, 2024

View reviewed changes

edopao mentioned this pull request Nov 28, 2024

feat[next]: Remove dace_iterator backend and pass_manager_legacy #1753

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat[next]: SDFGConvertible Program for dace_fieldview backend #1742

feat[next]: SDFGConvertible Program for dace_fieldview backend #1742

DropD commented Nov 19, 2024 •

edited

Loading

edopao Nov 19, 2024

DropD Nov 19, 2024 •

edited

Loading

DropD Nov 19, 2024 •

edited

Loading

DropD Nov 19, 2024

DropD Nov 19, 2024

kotsaloscv Nov 20, 2024

DropD Nov 25, 2024

edopao Nov 25, 2024

DropD Nov 26, 2024

edopao Nov 26, 2024

DropD Nov 26, 2024

havogt Nov 26, 2024

havogt Nov 26, 2024

		return {name for name in cls().visit(program) if name in field_param_names}


		class InputNamesExtractor(SymbolNameSetExtractor):

feat[next]: SDFGConvertible Program for dace_fieldview backend #1742

Are you sure you want to change the base?

feat[next]: SDFGConvertible Program for dace_fieldview backend #1742

Conversation

DropD commented Nov 19, 2024 • edited Loading

Description

Requirements

Choose a reason for hiding this comment

DropD Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

DropD Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DropD commented Nov 19, 2024 •

edited

Loading

DropD Nov 19, 2024 •

edited

Loading

DropD Nov 19, 2024 •

edited

Loading