[python] CI broken on main (mitigated: needs follow-up)

[johnkerl]

Python full CI manual run:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/12164118211/job/33924984198

FAILED apis/python/tests/test_basic_xarray_io.py::TestDenseNDDataArray1D::test_getitem_with_steps[key0] - Failed: DID NOT RAISE <class 'NotImplementedError'>
FAILED apis/python/tests/test_basic_xarray_io.py::TestDenseNDDataArray3D::test_getitem_with_steps[key0] - Failed: DID NOT RAISE <class 'NotImplementedError'>
= 2 failed, 6338 passed, 23 skipped, 2 xfailed, 2450 warnings in 433.07s (0:07:13) =

TileDB-SOMA/apis/python/tests/test_basic_xarray_io.py

Lines 80 to 82 in ee8e967

	def test_getitem_with_steps(self, xr_soma_data_array, key):
	with pytest.raises(NotImplementedError):
	xr_soma_data_array[key].data.compute()

This appears to be new as of today.

[XanthosXanthopoulos]

This is caused by the latest Dask version 2024.11.2 probably this specific PR

[johnkerl]

Finding by @XanthosXanthopoulos:

I looked it a bit further and the new version create some extra potential issues down the road. The passed SOMAArray is copied entirely that's why we are not raising a exception. The slicing operation is acting upon the copied array which support the stepped indexing.

A potential issue is now that everything is copied we are going to require more memory.

Adding @jp-dark as assignee to triage after (well-deserved!) PTO.

[XanthosXanthopoulos]

After some more digging the previously mentioned PR is not causing the issue. The task graph differs between the two versions. The newer version has a read op that reads a chunk of data and then the next node reads back the requested slice compared to the previous version which reads the requested slice directly.

With that in mind the memory issue I mentioned earlier doesn't seem to be actually an issue.