fluentd is not able to start inside EKS pod #583
Comments
Follow up for #534 Fix #583 Signed-off-by: Takuro Ashie <[email protected]>
Follow up for #534 Fix #583 Signed-off-by: Takuro Ashie <[email protected]>
|
Sorry, I linked to this wrongly from #585. |
|
yeah, just checked latest tag v1-debian-elasticsearch and it keeps failing... |
|
hmm, does it occur even though the default daemonset image? (I mean apply -f without additional configMap ) |
|
hello, checked w/o ConfigMap and it was able to proceed with startup but failed on some K8S API calls which is expected as pod does not have any SA assigned. regarding the second question - it doesn't, i can get only the error I pasted into issue |
121 if k.start_with?('@')
122 if root_element || ELEM_SYMBOLS.include?(elem_name)
123 parse_error! "'@' is the system reserved prefix. Don't use '@' prefix parameter in the configuration: #{k}"
124 else
125 # TODO: This is for backward compatibility. It will throw an error in the future.
126 $log.warn "'@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: #{k}"
127 end
128 endHmm, there are 2 problems.
I used the following config to confirm it: <source>
@type tail
path /home/aho/Projects/Fluentd/fluentd/issues/3269/*.log
tag foo.bar
<parse>
@type json
@time_key nil # <<<<==== This line shouldn't include "@" =====>>>>>>
</parse>
@label @OUTPUT
</source>
<label @OUTPUT>
<match **>
@type stdout
</match>
</label> |
It's caused by this commit: fluent/fluentd@b89c9f6 |
By #3352, an early stage config parser is introduced to enable setting log rotation by system config. While this stage, the global logger doesn't exist yet, so that it causes crash when the config parser try to output log. This commit avoid the crash by checking $log. Same logs will be output at the later stage so that it's not needed in the early stage. See also: fluent/fluentd-kubernetes-daemonset#583 Signed-off-by: Takuro Ashie <[email protected]>
|
It will be fixed by fluent/fluentd#3451 |
|
I reviewed config and found that I have only one suspicious statement: Full block: I cannot change it right now as it's based on application logs which provide |
No problem |
|
Fixed in fluentd but gem & fluentd-kubernetes-daemonset aren't released yet. |
|
Found one more suspicious thing: All other fields seem to be valid config parameters such as |
|
Ah, I got it. It's not also replaceable for you, isn't it? |
|
yeah, it comes from app logging and we need to use this transformation... anyway we're good to go with version 1.12 and wait for fixes :) thank you for investigation and response :) |
|
I have the same issue with fluent/fluentd-kubernetes-daemonset:v1.13.1-debian-elasticsearch7-3.0 here is my config pods-fluent.conf: |-
<source>
@type tail
read_from_head true
tag kubernetes.*
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
exclude_path ["/var/log/containers/fluent*"]
<parse>
@type kubernetes
@type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json'}"
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source> |
It's caused by using if !prev_match.start_with?('@')
$log.warn "'include' is deprecated. Use '@include' instead"
end |
|
v1.13.2 images have been released. |
Today's morning couple of our EKS workers have been updated to the latest EKS Worker AMI and suddenly fluentd pods failed to start on those new nodes with the following error:
Image:
fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearchUPD: it's not related to new EKS AMI, it came from new version on image... Let me try to pin image tag to some older versions
UPD 2: it works as expected on previous image version (
fluent/fluentd-kubernetes-daemonset:v1.12-debian-elasticsearch7-1)The text was updated successfully, but these errors were encountered: