Introduce an efficient binary codec for model events (at least transactions and spans), for persisting to local storage. This would be used in tail-based sampling, and perhaps in other processing such as #5936.
For the most efficient codec, we should use a statically defined encoding. Barring arguments made for alternatives, let's use protobuf.
In order to have a statically defined encoding we'll need our model to be stable. The events we produce are now substantially ECS-compliant (which is stable), with a few exceptions. For example, we emit transaction.duration.us or span.duration.us; ideally we would record only the ECS event.duration field (#5999).
I suggest we split this up into two phases:
Phase 1
- Continue to maintain package model by hand, and continue to use this to emitting JSON documents
- Introduce protobuf definitions, generate code in a separate package with protoc
- Generate code to translate between handwritten model types and protoc-generated Go types
Phase 2
For phase 2 we'll need some more changes to remove any remaining logic in the model package, e.g. setting <event>.duration.us. This can be done in an ingest pipeline.
Introduce an efficient binary codec for model events (at least transactions and spans), for persisting to local storage. This would be used in tail-based sampling, and perhaps in other processing such as #5936.
For the most efficient codec, we should use a statically defined encoding. Barring arguments made for alternatives, let's use protobuf.
In order to have a statically defined encoding we'll need our model to be stable. The events we produce are now substantially ECS-compliant (which is stable), with a few exceptions. For example, we emit
transaction.duration.usorspan.duration.us; ideally we would record only the ECSevent.durationfield (#5999).I suggest we split this up into two phases:
Phase 1
Phase 2
For phase 2 we'll need some more changes to remove any remaining logic in the model package, e.g. setting
<event>.duration.us. This can be done in an ingest pipeline.