Stop PPDataProxy accessing the file when no data is needed.

[pp-mo]

We just found that PP and Fieldsfile loading has been slow and heavy on memory, apparently since Dask 2.0.

In these cases, the PPDataProxy does not actually use the file data, so this fix allows it to return a suitable empty array object of the expected type and shape, without reading the file.

---

The reason appears to be that dask.array.from_array(arraylike) now extracts an empty slice of the argument, like arraylike[0:0, 0:0...].

Note, this is even worse than it sounds :

1. the PP field will often need to be unpacked, which is time consuming
2. since the 0-size slice is stored within the dask object, and it is a view referencing the full data array, hence all field data is loaded, and remains in memory !

See this line in meta_from_array, called from here in Array.__new__, as used in from_array.

Speculation : I think dask is doing this to capture the metadata of a slice from the argument, which it presumably believes can be different from that of the argument itself. I'm guessing this is needed for generalised array types e.g. dask, CUDA

[trexfeathers]

Thanks @pp-mo, I follow the logic of all this. There are a couple of changes to address 🙂

[pp-mo]

Hi @trexfeathers sorry for the delay on this -- I forgot to push it up !

Following offline discussion, I've now reduced the scope of the behaviour, in favour of simpler code that only supports the exact current need (e.g. it no longer catches "[1:1]" as an empty slice case)

However, I've also rather increased the complexity of the testing.
Again sorry, but I did spot some existing problems!
e.g. the previous code could not handle a single index key, because Python passes that as a bare object + not a 1-tuple.

As usual, it's tough to be comprehensive because numpy has so many indexing styles.
So, I'm taking it for granted that Dask will only use slices and integer indices
This excludes (e.g.) :

• keys which are arrays or lists (fancy indexing)
• ellipsis
• None / newaxis
• boolean arrays

I'm just not considering those as possibilities.
Does this seem sound ??

[pp-mo]

Note: when we have agreed on an implementation here, we should also adopt it in iris-grib.

[trexfeathers]

I've added some comments @pp-mo, although as we discussed you will be abstracting this code to be outside the PPDataProxy specifically, so that it can also serve for GRIB files too

[pp-mo]

Thanks @trexfeathers
I think I have it all done now. Review away !

[trexfeathers]

All looks great, great idea with the abstraction into util 👏