ffi: Add Deserializer class to deserialize log events from key-value pair IR format.
#511
Conversation
LinZhihao-723
changed the title
Deserializer class to deserialize log events from key-value pair IR format.Deserializer class to deserialize log events from key-value pair IR format.
| * This class: | ||
| * - maintains all the necessary internal data structure to track deserialization state; | ||
| * - provide APIs to deserialize log events from an IR stream. |
There was a problem hiding this comment.
I think this is obvious, right?
There was a problem hiding this comment.
Right. I've added more details to explain the APIs should work as a transaction
Co-authored-by: kirkrodrigues <[email protected]>
| [[nodiscard]] auto deserialize_schema( | ||
| ReaderInterface& reader, | ||
| encoded_tag_t& tag, | ||
| std::vector<SchemaTreeNode::id_t>& schema |
There was a problem hiding this comment.
I think for someone new to this code, schema might be a confusing/overloaded term (and it's not something that we've documented well in SchemaTree). So perhaps we should create a type alias and add a docstring for it explaining that it's a collection of schema tree leaf node IDs that represents the schema of a KV-pair log event.
| } | ||
|
|
||
| auto schema_tree_node_tag_to_type(encoded_tag_t tag) -> std::optional<SchemaTreeNode::Type> { | ||
| std::optional<SchemaTreeNode::Type> type; |
There was a problem hiding this comment.
I think SchemaTreeNode::Type is a trivial type, so could we return it directly from the method without triggering a copy?
| encoded_tag_t str_packet_tag{}; | ||
| if (auto const err{deserialize_tag(reader, str_packet_tag)}; | ||
| IRErrorCode::IRErrorCode_Success != err) | ||
| { | ||
| return err; | ||
| } | ||
| std::string key_name; | ||
| if (auto const err{deserialize_string(reader, str_packet_tag, key_name)}; | ||
| IRErrorCode::IRErrorCode_Success != err) | ||
| { | ||
| return err; | ||
| } |
There was a problem hiding this comment.
Can we extract this into a method? Then you also wouldn't need the comment.
Co-authored-by: kirkrodrigues <[email protected]>
Co-authored-by: kirkrodrigues <[email protected]>
|
The centOS build failed due to #521 and it is irrelevant to this PR. |
…e pair IR format. (y-scope#511) Co-authored-by: kirkrodrigues <[email protected]>
…e pair IR format. (y-scope#511) Co-authored-by: kirkrodrigues <[email protected]>
Description
This PR implements
Deserializerto deserialize log events from a key-value pair IR format.Deserializeris a class that stores all the necessary states of a key-value pair IR stream (currently including the schema tree and the UTC offset). It also provides API to read until the next key-value pair log event. The read operation is designed to be a transaction: if a failure happens during the deserialization, all the states will be reverted.Notice that the class is designed to take any arbitrary
ReadInterfaceto read bytes from a stream. The caller needs to maintain the stream to get the expected deserialization results. Due to the current limitations on our ffi layer design, we can't bind this class to one particular reader until all ffi libraries have implemented their reader wrapper.Validation performed