Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kafkareceiver] improve metrics and logs to include more information to debug consumers #28631

Closed
ben-childs-docusign opened this issue Oct 25, 2023 · 5 comments

Comments

@ben-childs-docusign
Copy link
Contributor

ben-childs-docusign commented Oct 25, 2023

Component(s)

receiver/kafka

Is your feature request related to a problem? Please describe.

We were seeing issues with our consumers not consuming all the data from our kafka broker (skipping messages) but we didn't have enough logs / metrics data to effectively debug until we added it manually.

Describe the solution you'd like

  1. Log topic name, and initial offset when logging start-consumer-group this will help to identify consumer groups where the offset is not being committed successfully.
  2. Update the name tag on consumer metrics to include the partition / topic name or add a new tag with the partition / topic. Currently each partition consumer ends up emitting the metrics with the same tags and they overwrite eachother.

Describe alternatives you've considered

No response

Additional context

Here is an example update that we found helpful:
image

@ben-childs-docusign ben-childs-docusign added enhancement New feature or request needs triage New item requiring triage labels Oct 25, 2023
@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions
Copy link
Contributor

Pinging code owners for receiver/kafka: @pavolloffay @MovieStoreGuy. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@crobert-1
Copy link
Member

Sounds like a good idea to me, feel free to open a PR if you're interested. 👍

Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Jan 10, 2024
Copy link
Contributor

This issue has been closed as inactive because it has been stale for 120 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants