Use correct shape for obsm/varm layers in anndata export

[aaronwolen]

Overview

This addresses a bug in the ExperimentAxisQuery anndata exporter that is specific to obsm/varm arrays.

Details

Currently, the number of columns needed for the resulting numpy array is taken from the SparseNDArray's shape property, which provides the maximum capacity of the underlying TileDB array, rather than the occupied capacity. This results in the exporter allocating a far larger numpy ndarray than necessary when obsm/varm layers are included in the output.

This issue is specific to obsm/varm arrays because, unlike X/obsp/varp, and as described here, their second dimension doesn't map to the obs or var axis and can include an arbitrary number of columns.

Solution

With this PR the correct number of columns needed for the resulting numpy array is computed using the number of unique values present in the COO-formatted arrow table's "soma_dim_1" column, which is already in-memory and doesn't require any additional TileDB queries.

This solution is a temporary workaround, as SparseNDArray.shape's behavior will change in an upcoming release of TileDB-SOMA (see here for status).

Note: The TileDB-SOMA SparseNDArray class does provide a used_shape property that returns the occupied capacity of the underlying TileDB array that could be used, however, this is specific to the TileDB implementation and is not part of the SOMA API specification.

[sc-51048].

[johnkerl]

🚢