Rework DynamicTable/DynamicTableRegion.get when the table contains a DTR

[rly]

Motivation

Rework of #556. Fix #552.

When a DynamicTable dt contains a DynamicTableRegion dtr and dt.get(), dt.__getitem__(), dtr.get(), or dtr.__getitem__() are called, then:

• If df=True, index=True (default), then a pandas DataFrame will be returned where the entries for the DTR column are the indices of the DTR.
• If df=True and index=False, then a pandas DataFrame will be returned where the entries for the DTR column are nested DataFrames (this works for multiple levels of nesting).
• If df=False and index=True, a list of lists will be returned where the list corresponding to the DTR column contains the indices of the DTR.
• If df=False and index=False, an error will be raised. Returning a list of lists where the DTR column contains lists of lists gets very messy in the ragged DTR case, and the nesting removes all column names and structure, so the returned value is difficult to interpret/parse. It is recommended to set df=True if you want to return a row with DynamicTableRegions resolved into nested DataFrames.
• Adds lots of tests to make sure this works for the various combinations of: [__getitem__, get(df=True, index=True), get(df=True, index=False), get(df=False, index=True)] with the data stored as a [np.ndarray, list, h5py.Dataset] with tables with scalar, 1D arrays, ragged 1D arrays, 2D arrays, DTRs, ragged DTRs at multiple levels of nesting.

Removed name mangling when a DynamicTable dt contains a DynamicTableRegion dtr and dt.get(), dt.__getitem__(), dtr.get(), or dtr.__getitem__() are called. This is technically a breaking change, though it did not work correctly in many cases.

When a DynamicTable dt contains a DynamicTableRegion dtr, and dt.to_dataframe() is called, then:

• If index=True, then a pandas DataFrame will be returned where the entries for the DTR column are the indices of the DTR. This is equivalent to table.get(slice(None, None, None)).
• If index=False (default), then a pandas DataFrame will be returned where the entries for the DTR column are nested DataFrames (this works for multiple levels of nesting). This is equivalent to table.get(slice(None, None, None), index=False).

Also increase h5py minimum version from 2.9 to 2.10 because 2.9 does not support __getitem__ on an h5py.Dataset using a list of indices.

Checklist

• Did you update CHANGELOG.md with your changes?
• Have you checked our Contributing document?
• Have you ensured the PR clearly describes the problem and the solution?
• Is your contribution compliant with our coding style? This can be checked running flake8 from the source directory.
• Have you checked to ensure that there aren't other open Pull Requests for the same change?
• Have you included the relevant issue number using "Fix #XXX" notation where XXX is the issue number? By including "Fix #XXX" you allow GitHub to close issue #XXX when the PR is merged.

[codecov]

Codecov Report

Merging #579 (5e59d97) into rc/3.0.0 (b93cde4) will decrease coverage by 0.06%.
The diff coverage is 88.76%.

@@             Coverage Diff              @@
##           rc/3.0.0     #579      +/-   ##
============================================
- Coverage     86.30%   86.24%   -0.07%     
============================================
  Files            41       41              
  Lines          8289     8310      +21     
  Branches       1786     1790       +4     
============================================
+ Hits           7154     7167      +13     
- Misses          799      807       +8     
  Partials        336      336              

Impacted Files	Coverage Δ
src/hdmf/common/table.py	83.33% <87.50%> (-0.15%)	⬇️
src/hdmf/common/resources.py	89.77% <94.11%> (-2.78%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b93cde4...5e59d97. Read the comment docs.

[rly]

This PR is nearly ready for review. There are two outstanding issues:

1. string handling for h5py 3 in tests
2. ragged compound types (like in the NWB epochs table) are not being handled correctly

[rly]

This is now ready for review, tested with h5py 2.10 and 3.3.

[oruebel]

It would be useful to add the description of the behavior of the get functions that is included here in the description of the PR also in the tutorial for DynamicTable. https://hdmf.readthedocs.io/en/latest/tutorials/dynamictable.html#sphx-glr-tutorials-dynamictable-py