You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have run into several cases where a ListArray or FixedSizedListArray ends up being nullable while its inner-types are non-nullable. Some of the codegen still ends up placing a validity-mask on these inner types, which can lead to discrependencies in the validation of our round-trip tests.
These are sort of unreachable nulls in a schema that says they shouldn't be nullable and so they fall into a grey area. On one hand you could argue they are inconsistent with the schema definition. On the other hand, the data that's in those inner arrays is undefined and so if something were to unwrap those inner arrays and look at them without the outer context, having nulls in there is possible the more correct thing to do.
In order to get the round-trip tests passing as part of #2991 we added this validity mask to the rust code.
Two pieces of work are associated with this issue:
Review and make sure we are happy with the current state of affairs. As an alternative to the current approach we might instead consider making the C++ / Python code avoid adding these nulls or make the round-trip validation checks more lenient.
We should add some more architectural primer/documentation on how Rerun interprets the intention and interactions between the schema definition and the validity masks.
The text was updated successfully, but these errors were encountered:
We have run into several cases where a ListArray or FixedSizedListArray ends up being nullable while its inner-types are non-nullable. Some of the codegen still ends up placing a validity-mask on these inner types, which can lead to discrependencies in the validation of our round-trip tests.
These are sort of unreachable nulls in a schema that says they shouldn't be nullable and so they fall into a grey area. On one hand you could argue they are inconsistent with the schema definition. On the other hand, the data that's in those inner arrays is undefined and so if something were to unwrap those inner arrays and look at them without the outer context, having nulls in there is possible the more correct thing to do.
In order to get the round-trip tests passing as part of #2991 we added this validity mask to the rust code.
Two pieces of work are associated with this issue:
The text was updated successfully, but these errors were encountered: