DRIVERS-3031 Clarify BSON Binary Vector design and encoding

[ghost]

This is a change to the text of the BSON Binary Vector spec which intends to clarify the design without altering any of the intended technical content.

This PR addresses DRIVERS-3031. The spec claimed "All values use the little-endian format", but this was unclear and incorrect.

This PR makes no changes to tests. Test changes will be needed both for improved cross-language generality and for DRIVERS-3095. That will be a separate PR, possibly concurrent with CDRIVER-5641.

The new text avoids using "endian" terms except in one place where the inconsistency is specifically pointed out. The new version avoids making generalizations about all supported formats, and it instead describes the ordering for individual data formats.

Two main changes in overall writing style were included in the update:

• The spec now has a neutral point of view with regard to low-level and high-level languages. Data type conversion is a specified but optional feature. Conversions and unpacking are positioned secondary to direct element access.
• The spec now describes both the extensible interpretation of the header (field by field) and the tightly-specified interpretation (a list of allowed bit patterns) in order to offer organizing principles while making them explicitly optional. The "dtype" concept is no longer given special mention.

The "Padding" field is given a general interpretation whereas previously this was unclear. This general interpretation (items not bits, always from what would have been the vector's end) matches the one in the earlier BSON Binary Vector design document.

Additional reserved size codes are documented according to the table in the earlier design document.

Additional "SHOULD" validity criteria were added beyond what drivers currently implement or test:

• Unused bits (indicated by Padding) SHOULD be zero
• Vectors with multi-byte elements SHOULD NOT have any extra trailing bytes.

Before and after renderings:

https://specifications.readthedocs.io/en/latest/bson-binary-vector/bson-binary-vector/
https://specifications--1752.org.readthedocs.build/en/1752/bson-binary-vector/bson-binary-vector/

Please complete the following before merging:

• Update changelog.
• [ ] Test changes in at least one language driver. No implementation changes.
• [ ] Test these changes against all server versions and topologies (including standalone, replica set, sharded
  clusters, and serverless). No implementation changes.

[BorisDog]

We had a discussion about the reserved bytes with @caseyclements , but I am not sure I remember its outcome.
Do we know why should drivers be limiting the reserved bytes to 0?
If the server starts sending additional data, do we necessarily want to break the users on older driver versions?

[ghost]

This requirement follows from the design goal for comparison stability. If reserved bit contents were unspecified, we would fail to meet the goal that BSON Binary Vectors can be compared for equality by comparing the BSON Binary for equality.

The combination of MUST and SHOULD here is an implementation of the "robustness principle" for compatibility with existing implementations. If we were doing this from scratch, I'd encourage MUST both for production and validation.

[BorisDog]

Is this required by the server?
From user's perspective this might be seen as inconvenience.
For example if the vector comes from another system (as it probably will), user would be required to zero the bits to avoid runtime exceptions. As opposed to just limiting the buffer length by "padding" parameter.

Also is this MUST consistent with SHOULD in:
"Drivers SHOULD reject Vectors with any unused bits in the final byte set to 1"

[ghost]

If packed bit data is coming from an external system that may leave unused bits set, it's a beneficial task to explicitly zero them. It's the comparison stability goal again.

[BorisDog]

In .NET we implemented deserializing the vector data to both

1. Native types, like float[]
2. New dedicated vector types like BinaryVectorFloat32

In the case of PACKED_BITS and the native byte[] representation (as opposed to BinaryVectorPackedBit) we are loosing the Padding value when reading. Which causes data corruption in a round trip. So we disallowed Padding != 0 with native types.
Do you think it's worth mentioning in spec?

[ghost]

This seems like a problem with the .NET implementation? I noticed this in a few places, drivers failing to preserve the 'Padding' value when representing PACKED_BITS as a byte array. This doesn't look correct to me.

This leads into a question about intended compatibility. Right now the spec assumes all drivers will provide a full implementation of the three data types. If drivers wish to provide fewer types, or an incomplete implementation of PACKED_BITS, that may motivate creating named levels of support.

[caseyclements]

If such a big change to a spec is proposed, please allow time for everyone to review.

[caseyclements]

This is a complete rewrite of the description of this binary subtype. It had been vetted by quite a few senior engineers. What is the intention here?

[ghost]

Hi Casey! This PR follows directly from multiple conversations on Slack, which then turned into DRIVERS-3031 which I'm now scheduled to work on. That ticket documents the motivations for the change.

[caseyclements]

If you would like to clarify the sentence, "All values use the little-endian format", I would hope that you could do it by a change of no more than a couple lines.

[ghost]

Please take a look at the clarifications here and let me know if there are specific problems with my proposed changes.

[ghost]

I just noticed #1750, which overlaps with this. @qingyang-hu and I both added the requirement not to have any extra data beyond complete multi-byte elements. (No partial FLOAT32)

In #1750 this was added as a hard requirement, but I wrote it as SHOULD here to avoid breaking existing drivers.

I do think it would be beneficial to describe this as a hard requirement. I'd like to add a similar hard requirement that unused bits in the final byte are zero'ed. (Upgrading the SHOULD here to a MUST.)

[ghost]

I've been asked to replace this PR with a suite of smaller changes.