Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log format compatibility #1793

Merged
merged 3 commits into from
Aug 13, 2024
Merged

Log format compatibility #1793

merged 3 commits into from
Aug 13, 2024

Conversation

pepov
Copy link
Member

@pepov pepov commented Aug 12, 2024

Add support for the automatic configuration of docker compatible log parsing with the CRI log format.

This has the following benefits:

  • automatic json log parsing can happen using the Merge_Log Fluent Bit kubernetes filter
  • downstream parsers can use the log field instead of message as they did with the docker runtime
  • the concat and parser filters are automatically set back to use the log field instead of message

The example in the PR demonstrates a working configuration:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: containerd
spec:
  enableDockerParserCompatibilityForCRI: true

After this, fluentbit can be configured through either the Logging or the FluentbitAgent resource. (Fluentbit configured through the nodeAgents field of the Logging resource is not supported, since nodeAgents are deprecated)

Example log line produced:

{
  "log": "2024-08-12T14:19:29.672991171Z stderr F [2024/08/12 14:19:29] [ info] [input:tail:tail.0] inotify_fs_add(): inode=2939617 watch_fd=17 name=/var/log/containers/containerd-fluentd-0_default_fluentd-e46f1fcb1b63e7458fc43c079b7455f8e1305e551939ca128361a9574a194ed7.log",
  "kubernetes": {
    "pod_name": "containerd-fluentbit-mchwz",
    "namespace_name": "default",
    "pod_id": "22f86078-26f1-4202-bbc7-1dd5ddce20ec",
    "labels": {
      "app.kubernetes.io/instance": "containerd",
      "app.kubernetes.io/managed-by": "containerd",
      "app.kubernetes.io/name": "fluentbit",
      "controller-revision-hash": "6bb6f7f5f4",
      "pod-template-generation": "2"
    },
    "annotations": {
      "checksum/cri-log-parser.conf": "d902a0ee964e9398e637b581be851cdf50ab2846e82003d2f5e2feef82bef95d",
      "checksum/fluent-bit.conf": "dc9727d915c447b414dc05df2d9a6f23246cdca345309eb3107cc16ae8369b53"
    },
    "host": "logging",
    "container_name": "fluent-bit",
    "docker_id": "5cf032406344fdf41d76ce4489ee6f3ca092e9207ec49ab6209cc2bcf950e593",
    "container_image": "fluent/fluent-bit:3.0.4"
  }
}

@pepov pepov added the enhancement New feature or request label Aug 12, 2024
Copy link
Contributor

@tarokkk tarokkk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pepov pepov merged commit 8c11c30 into master Aug 13, 2024
19 checks passed
@pepov pepov deleted the containerd-compatibility branch August 13, 2024 10:35
@pepov pepov added this to the 4.9 milestone Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants