RAD-120: Read Pattern

[PaulHuwe]

Resolves RAD-120

Closes #

This PR adds read pattern to the exposure group.

Checklist

• Schema changes discussed at RAD Review Board meeting
• Added entry in CHANGES.rst under the corresponding subsection
• Updated relevant roman_datamodels utilities and tests
• PLWishMaster Jenkins link: https://plwishmaster.stsci.edu:8081/job/RT/job/Roman-Developers-Pull-Requests/127/

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (3d5b215) 93.54% compared to head (5b43f62) 93.54%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #233   +/-   ##
=======================================
  Coverage   93.54%   93.54%           
=======================================
  Files           4        4           
  Lines         124      124           
=======================================
  Hits          116      116           
  Misses          8        8           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[kdupriestsci]

Looks good to me. Filed related RDADS ticket RDADS-587

[ddavis-stsci]

I'm a bit confused here.
Do we not need to preserve the grouping? For instance if the read pattern is
[1, 2], [3, 4, 5], [7, 8, 9], where 6 is skipped, turning this into an array yields,
array([1, 2, 3, 4, 5, 7, 8, 9])
but the read pattern [1, 2 ,3, 4, 5], [7, 8, 9] also gives the above array.
So how does the user find the grouping? Go back to the ma table names? or does this simply not matter here?

[schlafly]

I agree that the grouping is key here. But the type is an array of arrays, which looks fine to me?

[PaulHuwe]

I agree that the grouping is key here. But the type is an array of arrays, which looks fine to me?

It should be, but I can add an explicit test for this structural preservation in RCAL-543.

[ddavis-stsci]

I'm not sure that works for a list of list with an unequal number of elements, so if I try to convert the above list into an array I get

np.array([[1], [2, 3], [4], [5, 6, 7, 8]])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (4,) + inhomogeneous part.

Am I missing something?
I can turn that list into a 2-d array. Is that what we're proposing?
For the archive catalog that will be stored as the original list? e.g. '[1]', '[2,3]', '[4]', ...

[schlafly]

Is the RAD schema going to try to push that into a numpy ndarray as you're showing? Can we mark the type as list rather than array? We were imagining the archive storing str([[1], [2, 3], [4], [5, 6, 7, 8]]) = '[[1], [2, 3], [4], [5, 6, 7, 8]]'

[WilliamJamieson]

This cannot be an "array of arrays", that would by definition require all of the member arrays to be the same shape.

In order for this to work it must remain a "list of lists" as each constituent list can have different shape.

If we must have an array compatible format then it would be better to build a 1D array whose index represents the reads and values represent the grouping. A null flag such as -1 can be used to for skipped reads. E.G.

[1, 2], [3, 4, 5], [7, 8, 9]

Would be instead represented as

[0, 0, 1, 1, 1, -1, 2, 2, 2]

[PaulHuwe]

This implementation appears to work fine?
https://github.com/spacetelescope/roman_datamodels/pull/154/files#diff-0adb667cd397bd56edce10eec11e1b10821740a10d72bd148b09645fc9108968R211

[WilliamJamieson]

Note that we can do this as unsigned integers by defining 0 as the skipped positions. Then the example becomes

[1, 1, 2, 2, 2, 0, 3, 3, 3]

[WilliamJamieson]

The confusion may also be coming from this term. Recall, the schemas are written using "jsonschema", which is based on "json" originating in JavaScript. So the "type" refers to the type you would see in JavaScript, which happens to term "array" to mean a resizable indexed collection of items which in Python we would call a "list". Hence, the schema here is describing a "list of lists" in Python.

[ddavis-stsci]

Looks good.