Skip to content

Conversation

@faec
Copy link
Contributor

@faec faec commented Jan 27, 2025

Add queue/output metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: added.{events, bytes}, consumed.{events, bytes}, removed.{events, bytes}, and filled.{events, bytes, pct}, all within libbeat.pipeline.queue, and libbeat.output.write.latency.{count, max, median, p99}.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@faec faec added bug Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-8.x Automated backport to the 8.x branch with mergify labels Jan 27, 2025
@faec faec self-assigned this Jan 27, 2025
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 27, 2025
@faec faec marked this pull request as ready for review January 28, 2025 21:20
@faec faec requested review from a team as code owners January 28, 2025 21:20
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@faec faec enabled auto-merge (squash) January 29, 2025 18:06
Copy link
Member

@pickypg pickypg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some questions before approving for Stack Monitoring.

@faec
Copy link
Contributor Author

faec commented Jan 29, 2025

Note that I just added a few more fields under output.latency. These are by request of @strawgate for elastic/integrations#12488 (which is essentially a super-issue of #42093) since he is updating the relevant dashboards.

Copy link
Member

@pickypg pickypg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@faec faec merged commit 7eb2bdd into elastic:main Jan 29, 2025
142 checks passed
mergify bot pushed a commit that referenced this pull request Jan 29, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)
@mauri870
Copy link
Member

mauri870 commented Jan 30, 2025

This change seems to have broken some of the metricbeat tests, because of the new metrics:

E           Exception: Keys:
E           
E           ['beat.stats.libbeat.pipeline.queue.added.events', 'beat.stats.libbeat.pipeline.queue.added.bytes', 
'beat.stats.libbeat.pipeline.queue.consumed.events', 'beat.stats.libbeat.pipeline.queue.consumed.bytes', 
'beat.stats.libbeat.pipeline.queue.removed.events', 'beat.stats.libbeat.pipeline.queue.removed.bytes', 
'beat.stats.libbeat.pipeline.queue.filled.events', 'beat.stats.libbeat.pipeline.queue.filled.bytes', 
'beat.stats.libbeat.pipeline.queue.filled.pct']
E           
E           not documented in event

https://buildkite.com/elastic/beats-metricbeat/builds/13523#0194b889-3630-4849-a023-e9e352e1afd6/192-456

@strawgate
Copy link
Contributor

@mauri870 i'll make sure the integration tests are passing with my follow-on pr #42514

@jlind23 jlind23 added the backport-8.18 Automated backport to the 8.18 branch label Jan 31, 2025
@jlind23
Copy link
Contributor

jlind23 commented Jan 31, 2025

@faec adding backport 8.18 label as the branch has been created right before this merge

@jlind23 jlind23 added the backport-8.17 Automated backport with mergify label Jan 31, 2025
mergify bot pushed a commit that referenced this pull request Jan 31, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)
mergify bot pushed a commit that referenced this pull request Jan 31, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)
jlind23 pushed a commit that referenced this pull request Jan 31, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)

Co-authored-by: Fae Charlton <[email protected]>
jlind23 pushed a commit that referenced this pull request Jan 31, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)

Co-authored-by: Fae Charlton <[email protected]>
faec added a commit that referenced this pull request Jan 31, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)

Co-authored-by: Fae Charlton <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-8.x Automated backport to the 8.x branch with mergify backport-8.17 Automated backport with mergify backport-8.18 Automated backport to the 8.18 branch bug Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants