Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ffi: Add WIP Serializer class for the next IR stream format. #446

Merged
merged 4 commits into from
Jun 22, 2024

Conversation

LinZhihao-723
Copy link
Member

Description

When migrating to the next IR stream format, there are two important differences:

  1. The serialization is no longer stateless in the next IR format. The stream has to maintain a schema tree across the entire IR stream.
  2. In the current IR format, bytes can be written into the IR buffer in the same order as encoded: first encoded variables, then logtype, and finally timestamp. This is not true in the next IR format. For structured key-value pairs, the overall packet of a new record would be: [new_schema_tree_nodes, keys, values]. Each sub-packet has to be encoded in a separate buffer and merged into one after the entire record is processed. This means we need more than one internal buffer during the serialization, even though only the final merged buffer is exposed to the upper layer APIs.

To adapt to these changes, we wrap all the necessary data structures into a new class named Serializer to keep track of the states, as well as maintain necessary data structures. In details:

  • It maintains an underlying schema tree for the stream.
  • It maintains all the necessary internal buffers for serialization. These internal buffers are private members. In the future PRs, serialization methods will be added as member functions and they will have direct access to these buffers.
  • It provides an API to access the view of the serialized IR buffer (containing serialized bytes).

Note: this class is designed to provide serialization functionalities only. The upper-level caller should be responsible for writing the serialized bytes into IO streams properly. In addition, it doesn't provide a call to terminate the IR stream. The upper-level caller should decide when to terminate the stream by append clp::ffi::ir_stream::cProtocol::Eof at the end of the stream.

Validation performed

  • Added a unit test case to ensure the serializer can be successfully created with the preamble section serialized properly; also tested to ensure it can properly serialize a UTC offset change.
  • Passsed clang-tidy checks for the Serializer implementation.
  • Didn't execute clang-tidy against the unit test file since it is not synchronized with the latest coding guideline.

@kirkrodrigues kirkrodrigues requested a review from gibber9809 June 17, 2024 18:31
gibber9809
gibber9809 previously approved these changes Jun 21, 2024
Copy link
Contributor

@gibber9809 gibber9809 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. How about "ffi: Add Serializer class for new IR stream format." for the title?

components/core/src/clp/ffi/ir_stream/Serializer.cpp Outdated Show resolved Hide resolved
components/core/tests/test-ir_encoding_methods.cpp Outdated Show resolved Hide resolved
components/core/tests/test-ir_encoding_methods.cpp Outdated Show resolved Hide resolved
components/core/tests/test-ir_encoding_methods.cpp Outdated Show resolved Hide resolved
components/core/src/clp/ffi/ir_stream/Serializer.hpp Outdated Show resolved Hide resolved
components/core/src/clp/ffi/ir_stream/Serializer.hpp Outdated Show resolved Hide resolved
components/core/src/clp/ffi/ir_stream/Serializer.hpp Outdated Show resolved Hide resolved
components/core/src/clp/ffi/ir_stream/Serializer.hpp Outdated Show resolved Hide resolved
components/core/src/clp/ffi/ir_stream/Serializer.hpp Outdated Show resolved Hide resolved
Copy link
Member

@kirkrodrigues kirkrodrigues left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. How about "ffi: Add Serializer class for new IR stream format." for the title?

How about a slight update:

ffi: Add WIP Serializer class for the next IR stream format.

@LinZhihao-723 LinZhihao-723 changed the title ffi: Add support for IR serializer of the next IR stream format. ffi: Add WIP Serializer class for the next IR stream format. Jun 22, 2024
@LinZhihao-723
Copy link
Member Author

LGTM. How about "ffi: Add Serializer class for new IR stream format." for the title?

How about a slight update:

ffi: Add WIP Serializer class for the next IR stream format.

lgtm

@LinZhihao-723 LinZhihao-723 merged commit 628ba3d into y-scope:main Jun 22, 2024
11 checks passed
jackluo923 pushed a commit to jackluo923/clp that referenced this pull request Dec 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants