Dask notes

[SimonHeybrock]

• If loading many chunks naively, which are then processed and gather, deserialization of duplicate meta data (attributes) becomes quite costly.
  • Is this a performance issue with our h5py-based serialization approach?
  • Note that just loading this meta data from the Nexus file on every chunk has probably even more overhead.

Applications

• Process large number of files, either in parallel (on cluster) or serially (locally) where not all fit memory.
• Imaging.
• Large files, such as long runs, load time ranges.
• Workflow inherent parallelism such as processing sample and vanadium independently.
• Load and plot slices of large chunked volume (may require dask collection?)

To do

• Support lazy loading like h5py using something like scn.open_nexus(filename)['pulse_time', start:stop]