Fix Node and container resource limit metrics missing intermittently

[swiatekm]

Proposed commit message

Fix Node and container resource limit metrics missing intermittently.

This is a bug very recently introduced by the refactor in #41216. Metadata watchers are not just responsible for updating metadata, but also Node and container metrics. Only updating the latter eagerly when metadata is requested leads to races, where the values may be missing depending on the order in which metrics are fetched.

This fix decouples metrics calculation from metadata calculation. Metrics now have their own handlers attached to the watcher, and are completely detached from metadata enrichers. I don't like the resulting architecture that much, as it concentrates a lot of logic in the watcher. But it is an improvement over the status quo, and I'd like to fix this bug promptly before we release it to users.

The bug was quite difficult to catch in E2E tests, as it could take some time to appear. I've tested this change much more carefully, and haven't seen any issues after hours of running it in my test cluster.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

Simplest way is to install elastic-agent standalone and look at the default Kubernetes dashboard.

Related issues

• Closes Pod and container resource limit metrics missing intermittently #41432

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @swiatekm? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[swiatekm]

I've no idea what the problem is with linting on darwin. Looks like a build error, but we don't build metricbeat on MacOS, so it's difficult to diagnose. I can reproduce it locally by running golangci-lint with GOOS=darwin CGO_ENABLED=1, but the error is plain incorrect.

EDIT: Just added an exception, similar to #33649 .

[mauri870]

The code looks good overall, but I don't have deep knowledge of the integration. It would be helpful to get a review from another developer.

[MichaelKatsoulis]

@swiatekm The changes look good to me. I tested them also and everything looks to be working as expected.
Maybe you could also update the enrichers.md file to also describe the updated process.

[swiatekm]

@MichaelKatsoulis I'll update the documentation in a follow-up, I don't want to hold this PR up.