Provide default options for chunking of datasets (issue #635)

[ehennestad]

Fix #635

Motivation

Provide better chunking options for files in cloud storage

How to test the behavior?

To be determined

Checklist

• Have you ensured the PR description clearly describes the problem and solutions?
• Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
• If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

[ehennestad]

Some open questions:

• Should each possible property / dataset name be present in the configuration (chunk_params.json)? Or should there be one default that applies to all datasets, and overwrites would happen on specific datasets?
• Should each dataset be a candidate for chunking, or only datasets like "data", "timestamps". Any others?
• Should chunking be specified for datasets that are smaller than the chunk size? I.e chunkSize == maxSize?
• Should chunk_dimensions be specified for each set of dimension options, similar to how it is done in the nwb schema?

I.e. https://github.com/NeurodataWithoutBorders/nwb-schema/blob/473fcc41e871288767cfb37d83315cca7469b9d1/core/nwb.base.yaml#L100-L110

dims:
    - - num_times
    - - num_times
      - num_DIM2
    - - num_times
      - num_DIM2
      - num_DIM3
    - - num_times
      - num_DIM2
      - num_DIM3
      - num_DIM4

@bendichter

[ehennestad]

Current implementation for schema/definition:

    "default_dataset_configuration": {
        "layout": "chunked",
        "target_chunk_size": {
            "value": 10000000,
            "unit": "bytes"
        },
        "chunk_dimensions": [null],
        "compression": {
            "algorithm": "gzip",
            "level": 6,
            "parameters": {},
            "prefilters": ["shuffle"]
        }
    },
    specific_dataset_overrides...

Open questions:
@bendichter
Should chunk_dimensions be specified for each set of dimension options, similar to how it is done in the nwb schema?

dims:
    - - num_times
    - - num_times
      - num_DIM2
    - - num_times
      - num_DIM2
      - num_DIM3
    - - num_times
      - num_DIM2
      - num_DIM3
      - num_DIM4

What would be the syntax?
Example (json):

"chunk_dimensions": [ [null], [null, 32], [null, 32, max] ]

[ehennestad]

Need to determine what to do with nested data types. For example: A RoiResponseSeries can be part of a Fluorescence or a DfOverF group. Should the spec support defining configuration for nested neurodata types dependent on were they are located, i.e:

"Fluorescence": {
        "RoiResponseSeries": {
            "data": {
                "chunk_dimensions": [null, 16]}
        }
    },
    "DfOverF": {
        "RoiResponseSeries": {
            "data": {
                "chunk_dimensions": [null, 32]}
        }
    }

[codecov]

Codecov Report

Attention: Patch coverage is 79.62963% with 33 lines in your changes missing coverage. Please review.

Project coverage is 94.79%. Comparing base (1a2f696) to head (b47f153).

Files with missing lines	Patch %	Lines
...config/+internal/resolveDatasetConfigForDataType.m	56.00%	11 Missing ⚠️
+io/+config/+internal/configureDataPipeFromData.m	60.86%	9 Missing ⚠️
+io/+config/readDatasetConfiguration.m	71.42%	4 Missing ⚠️
+schemes/listDatasetsOfNeurodataType.m	81.81%	4 Missing ⚠️
+io/+config/+internal/computeChunkSizeFromConfig.m	90.32%	3 Missing ⚠️
+io/+config/applyDatasetConfiguration.m	95.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #636      +/-   ##
==========================================
- Coverage   95.27%   94.79%   -0.48%     
==========================================
  Files         124      132       +8     
  Lines        5139     5301     +162     
==========================================
+ Hits         4896     5025     +129     
- Misses        243      276      +33     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.