-
Notifications
You must be signed in to change notification settings - Fork 2.2k
fix!(codecs): Use '\0' delimiter as default stream decode framer #20778
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
441a19e
66837aa
7b419d4
1255c19
2de28c8
d7c5ecf
04e19c5
28ef000
7c49d9d
126f526
77edb33
a98c619
fd97584
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| Now the GELF codec with stream-based sources uses null byte (`\0`) by default as messages delimiter instead of newline (`\n`) character. This better matches GELF server behavior. | ||
|
|
||
| ### Configuration changes | ||
|
|
||
| In order to maintain the previous behavior, you must set the `framing.method` option to the `character_delimited` method and the `framing.character_delimited.delimiter` option to `\n` when using GELF codec with stream-based sources. | ||
|
|
||
| ### Example configuration change for socket source | ||
|
|
||
| #### Previous | ||
|
|
||
| ```yaml | ||
| sources: | ||
| my_source_id: | ||
| type: "socket" | ||
| address: "0.0.0.0:9000" | ||
| mode: "tcp" | ||
| decoding: | ||
| codec: "gelf" | ||
| ``` | ||
|
|
||
| #### Current | ||
|
|
||
| ```yaml | ||
| sources: | ||
| my_source_id: | ||
| type: "socket" | ||
| address: "0.0.0.0:9000" | ||
| mode: "tcp" | ||
| decoding: | ||
| codec: "gelf" | ||
| framing: | ||
| method: "character_delimited" | ||
| character_delimited: | ||
| delimiter: "\n" | ||
| ``` | ||
|
|
||
| authors: jorgehermo9 | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,6 +15,12 @@ pub struct CharacterDelimitedDecoderConfig { | |
| } | ||
|
|
||
| impl CharacterDelimitedDecoderConfig { | ||
| /// Creates a `CharacterDelimitedDecoderConfig` with the specified delimiter and default max length. | ||
| pub const fn new(delimiter: u8) -> Self { | ||
| Self { | ||
| character_delimited: CharacterDelimitedDecoderOptions::new(delimiter, None), | ||
| } | ||
| } | ||
| /// Build the `CharacterDelimitedDecoder` from this configuration. | ||
| pub const fn build(&self) -> CharacterDelimitedDecoder { | ||
| if let Some(max_length) = self.character_delimited.max_length { | ||
|
|
@@ -53,7 +59,7 @@ pub struct CharacterDelimitedDecoderOptions { | |
|
|
||
| impl CharacterDelimitedDecoderOptions { | ||
| /// Create a `CharacterDelimitedDecoderOptions` with a delimiter and optional max_length. | ||
| pub fn new(delimiter: u8, max_length: Option<usize>) -> Self { | ||
| pub const fn new(delimiter: u8, max_length: Option<usize>) -> Self { | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Noticed this and didn't hurt |
||
| Self { | ||
| delimiter, | ||
| max_length, | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -334,14 +334,16 @@ impl DeserializerConfig { | |
| DeserializerConfig::Native => FramingConfig::LengthDelimited(Default::default()), | ||
| DeserializerConfig::Bytes | ||
| | DeserializerConfig::Json(_) | ||
| | DeserializerConfig::Gelf(_) | ||
| | DeserializerConfig::NativeJson(_) => { | ||
| FramingConfig::NewlineDelimited(Default::default()) | ||
| } | ||
| DeserializerConfig::Protobuf(_) => FramingConfig::Bytes, | ||
| #[cfg(feature = "syslog")] | ||
| DeserializerConfig::Syslog(_) => FramingConfig::NewlineDelimited(Default::default()), | ||
| DeserializerConfig::Vrl(_) => FramingConfig::Bytes, | ||
| DeserializerConfig::Gelf(_) => { | ||
| FramingConfig::CharacterDelimited(CharacterDelimitedDecoderConfig::new(0)) | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was thinking about this these days... I think that GELF Http maybe uses newline delimiter? I can't find anything related to streaming gelf in Http here https://go2docs.graylog.org/5-0/getting_in_log_data/gelf.html, but maybe someone will rely on that for http sources also and this will be a breaking change in those cases too, not only with TCP. If you are ok with that, I will include that breaking changes and the "fix" (manually setting the newline delimiter in
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Interesting, I haven't heard of anyone using GELF over HTTP, but it seems plausible. TCP seems to be the dominant transport so I think it makes sense to configure the defaults to work well with that. |
||
| } | ||
| } | ||
| } | ||
|
|
||
|
|
@@ -467,3 +469,23 @@ impl format::Deserializer for Deserializer { | |
| } | ||
| } | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use super::*; | ||
|
|
||
| #[test] | ||
| fn gelf_stream_default_framing_is_null_delimited() { | ||
| let deserializer_config = DeserializerConfig::from(GelfDeserializerConfig::default()); | ||
| let framing_config = deserializer_config.default_stream_framing(); | ||
| matches!( | ||
| framing_config, | ||
| FramingConfig::CharacterDelimited(CharacterDelimitedDecoderConfig { | ||
| character_delimited: CharacterDelimitedDecoderOptions { | ||
| delimiter: 0, | ||
| max_length: None, | ||
| } | ||
| }) | ||
| ); | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -362,12 +362,14 @@ impl SerializerConfig { | |||||
| FramingConfig::LengthDelimited(LengthDelimitedEncoderConfig::default()) | ||||||
| } | ||||||
| SerializerConfig::Csv(_) | ||||||
| | SerializerConfig::Gelf | ||||||
| | SerializerConfig::Json(_) | ||||||
| | SerializerConfig::Logfmt | ||||||
| | SerializerConfig::NativeJson | ||||||
| | SerializerConfig::RawMessage | ||||||
| | SerializerConfig::Text(_) => FramingConfig::NewlineDelimited, | ||||||
| SerializerConfig::Gelf => { | ||||||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Huh, yeah, this does seem like an oversight. I'm trying to understand the impact of it though since the original PR did seem to resolve the issue for people 🤔
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It seems that the changed struct was It seems that a few places relies on
Where we can see its relying on SerializerConfig::default_stream_framing in order to get the default framing for a given encoder.
There are a lot of sinks that relies on vector/src/codecs/encoding/config.rs Line 110 in 67a5e46
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Aha, gotcha, thanks for looking into that. |
||||||
| FramingConfig::CharacterDelimited(CharacterDelimitedEncoderConfig::new(0)) | ||||||
| } | ||||||
| } | ||||||
| } | ||||||
|
|
||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome, thanks for including the example for using the previous default.
Small nit that we try to use YAML rather than TOML as the default these days.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done!