Experimental Avro/Protobuf/Arrow reference encoders + PubSub adapters

[marcschier]

Summary

Adds two experimental class libraries to the stack that implement the draft OPC UA DataEncoding extensions (Avro binary, Protobuf, Apache Arrow) as hand-rolled reference encoder/decoders, plus PubSub NetworkMessage adapters, with NUnit round-trip tests. These are the C# reference implementation of the spec drafts in the marcschier/opcua-drafts repo.

New projects

• Libraries/Opc.Ua.Core.Experimental — IEncoder/IDecoder implementations:
  • Avro — streaming Avro binary (zig-zag varint, little-endian floats, nullable unions, NodeId-by-parts), full Variant/Matrix support, plus Action invoke/response + Discovery. SchemaId = CRC-64-AVRO (Rabin) fingerprint with Avro single-object prefix.
  • Protobuf — hand-rolled protobuf wire format; scoped to gRPC service messages only.
  • Arrow — genuinely typed Apache Arrow (Apache.Arrow 18.1.0): typed arrays, StructArray, ListArray, dense UnionArray, FixedSizeBinaryArray. No JSON/base64/blob carrier (asserted by tests).
• Libraries/Opc.Ua.PubSub.Experimental — INetworkMessageEncoder/Decoder:
  • Avro adapter drives the real AvroEncoder per field (self-describing Avro-in-Avro bytes).
  • Arrow adapter emits a real columnar RecordBatch (rows = DataSetMessages, typed per-field + leading header columns; NetworkMessage headers in schema metadata) per the Part 14 Arrow batch mapping (ADBC/historian oriented).
• Test projects for both, Apache.Arrow 18.1.0 added to central package management, four projects added to UA.slnx.

Target frameworks / CI

• The experimental libraries target net8.0;net9.0;net10.0 only (they rely on Apache.Arrow and net5+ BCL APIs such as BitConverter.SingleToUInt32Bits, Convert.ToHexString, Stream.ReadExactly). They opt into RestrictForLegacyTfm so the per-TFM CI matrix entries pinned to legacy TFMs (net472/net48/netstandard2.x) no-op them (verified locally).
• Verified: both libraries build 0 warnings / 0 errors on net8.0/net9.0/net10.0; Opc.Ua.Core.Experimental.Tests 17/17 and Opc.Ua.PubSub.Experimental.Tests 3/3 pass on net8.0 and net10.0; legacy CustomTestTarget builds no-op cleanly.

Conformance / scope

• Round-trip-only conformance for this first reference (encode → decode reproduces the input), including null vs empty vs present-zero, nullable scalars, arrays, matrices (dimensions + values), all four NodeId kinds, Variant/ExtensionObject/DataValue/DiagnosticInfo, and float NaN/±0/±Inf bit preservation.
• Out-of-scope / unsupported decode paths fail loud (NotSupportedException) rather than corrupt.

Code-review fixes included

• Avro (critical): a present-empty array previously emitted a spurious Avro block terminator that desynced the following field — and, because every 1-D array Variant carries a present-empty dimensions array, such Variants decoded as Null. Fixed in AvroEncoder.WriteArray + a dedicated multi-field regression test.
• Arrow: dense-union declared type-ids now include the written code so external Arrow readers (pyarrow/ADBC) can map Variant/DataValue union slots; XmlElement now round-trips instead of being silently dropped.
• Removed a dead/misleading tautology guard in the Arrow PubSub adapter.

Known limitation

• Protobuf optional presence: the reference decoder reconstructs optional-field presence by probing field names against the wire; combined with positional field numbering this is unreliable for structures with two or more optional fields where an earlier optional is absent. The in-scope gRPC service messages do not use such structures. A general fix requires an explicit on-wire encoding mask (documented in ProtobufDecoder).

Companion to the encoding spec drafts in marcschier/opcua-drafts (PR #4).

---

Follow-up: closed encoder gaps, Avro perf, and performance numbers

• Protobuf Variant arrays/matrices now encode/decode across all built-in body types (previously threw InvalidCastException for matrix/array Variants). Round-trip tests added.
• Arrow Variant dense-union completed to cover the full OPC UA type model — every built-in scalar/array/matrix body (previously only a handful; e.g. Variant<Boolean> was unsupported). Round-trip tests added.
• Avro codec performance: buffered, span-based varint IO and pooled UTF-8; IO buffers rented from ArrayPool so small-message allocation is not penalised (Avro scalar encode alloc ~9.1 KB/op → ~0.9 KB/op, below Binary) while retaining the large-payload batching win.
• The Part 6 + Part 14 comparison harness now runs with no n/a cells. Build 0/0 across net8/9/10; Core 118 + PubSub 8 tests + all round-trips green.
• Measured results and value-add analysis are captured in the companion spec PR (core-specs/extras/performance/OPC-UA-Encoding-Performance-Comparison.md).

Follow-up: Arrow bare-RecordBatch (batch) framing

Adds ArrowIpcFraming {Stream, Batch} to the Arrow PubSub encoder. Batch emits a bare Arrow RecordBatch without the embedded Schema message (a fixed ~1.2 KB/message); the schema is announced once out of band and resolved by SchemaId (the Arrow Flight model). The decoder caches schemaId -> schema (from a full stream or CacheSchema) and reconstructs the stream to decode bare batches. Measured savings: ~39% for a single sample, ~37% at 10 samples, shrinking to ~1.8% at 1000 (schema amortised). Bare-batch round-trip tests + an IPC framing-breakdown measurement added; build 0/0 all TFMs, PubSub tests 10/10.

Update: ArrowIpcFraming.Batch is now the encoder default (aligns with the spec default and the Avro schema-exchange model); the Part 14 harness measures both framings.

[marcschier]

Superseded by three focused PRs that port each reference encoder into the production stack (next to Binary/Json/Xml/Uadp), removing the need for the *.Experimental libraries:

• Add Avro DataEncoding to the stack (1/3, supersedes part of #7) #10 — Avro DataEncoding -> stack (owns shared SchemaId + PubSub schema-exchange/cache infra)
• Add Protobuf (gRPC) DataEncoding to the stack (2/3, supersedes part of #7) #11 — Protobuf/gRPC encoder -> stack (independent)
• Port Arrow reference encoder into the stack (Types + PubSub) #12 — Arrow DataEncoding -> stack (stacked on Add Avro DataEncoding to the stack (1/3, supersedes part of #7) #10)

Recommended merge order: #11 (independent), then #10, then #12 (rebases onto main automatically once #10 merges). Closing this PR in favor of the three above.