Conversation
ruflin
added
in progress
|
@tsg I'm not sure if we should enable this by default or only have it for example in all our modules enabled. |
ruflin
force-pushed
the
raw-message
branch
2 times, most recently
from
e5e67b8 to
9eedc54
Compare
Current default for ignore_above is 1024. This is too short for some command line entries on Windows. This increases it to 2048. Closes elastic#8076
CHANGELOG.asciidoc
Outdated
There was a problem hiding this comment.
This entry does not belong here.
CHANGELOG.asciidoc
Outdated
There was a problem hiding this comment.
Please, add the number of the PR.
kvch
left a comment
kvch
left a comment
There was a problem hiding this comment.
We agreed that this feature is going to be disabled by default, because it has a big impact on users. If it is enabled silently on update for users, their whole installation could break. The size of the events are doubled, so they run out of storage. At this point this must be a conscious decision of users to enable it. Thus, we ensure they can prepare for/acknowledge the increased storage requirements.
If we decide later to enable it by default, we can do it in 7.x.
|
I've just looked at the issue which references it. I still believe that on input level, raw should not be added to the event. But I am fine with adding this to the modules. I am keeping my review as request changes, because currently it enables keeping the raw message for log input, not just for modules. |
filebeat/harvester/reader/limit.go
Outdated
There was a problem hiding this comment.
I am on the fence with this. Raw message should not be truncated, because it is a form of processing. But this could protect users from injecting too long messages to their pipelines.
There was a problem hiding this comment.
I would go with truncating the raw message.
There was a problem hiding this comment.
I would actually go with your first proposal and not truncate it. At the same time I would not index log.original.
Can you elaborate on what you mean with too long message to the pipeline?
There was a problem hiding this comment.
I mean such a big message which could disrupt the output of Filebeat.
By default filebeat adds log.message to each event which contains the raw unprocessed message. This can be disabled by setting `raw_message` to false to save disk space. The message in `log.message` contains a full multiline message, is encoded and the new line at the end is stripped.
ruflin
left a comment
ruflin
left a comment
There was a problem hiding this comment.
It would be interesting to see what the actual impact on enabling it would mean on the storage level. My hope is that it has a smaller impact as the same data exists twice it "could" compress well.
@kvch @ph This change currently only applies to the log input. Should we to something similar for the other inputs?
filebeat/harvester/reader/limit.go
Outdated
There was a problem hiding this comment.
I would actually go with your first proposal and not truncate it. At the same time I would not index log.original.
Can you elaborate on what you mean with too long message to the pipeline?
|
I think we should. In case of |
|
Closing in favor of #8448 |
2 participants
By default filebeat adds log.message to each event which contains the raw unprocessed message. This can be disabled by setting
raw_messageto false to save disk space.The message in
log.messagecontains a full multiline message, is encoded and the new line at the end is stripped.