[8.18](backport #7936) Use upstream prometheus receiver again

[mergify]

What does this PR do?

Switches from our fork of the Otel prometheus receiver to the upstream version. This fork was driven by a dependency conflict with the prometheus library, which was resolved in elastic/beats#43540.

To achieve this, we bump the beats dependency version. I was also forced to downgrade github.com/elastic/cloud-on-k8s, as it depended on a version of github.com/prometheus/common which was incompatible with the new prometheus library. @pkoutsovasilis I hope that's ok.

Why is it important?

We want to use upstream components where possible.

Checklist

• I have read and understood the pull request guidelines of this project.
  - [ ] My code follows the style guidelines of this project
  - [ ] I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
  - [] I have added tests that prove my fix is effective or that my feature works
  - [ ] I have added an entry in ./changelog/fragments using the changelog tool
  - [ ] I have added an integration test or an E2E test

---

This is an automatic backport of pull request #7936 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 0c78a0f has failed:

On branch mergify/bp/8.18/pr-7936
Your branch is up to date with 'origin/8.18'.

You are currently cherry-picking commit 0c78a0fb2.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   NOTICE.txt
	both modified:   go.mod
	both modified:   go.sum
	both modified:   internal/pkg/otel/README.md
	deleted by us:   internal/pkg/otel/configtranslate/otelconfig_test.go

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 7d14063

Failed CI Steps

• check-ci
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Windows 2022
• Unit tests - Windows 2022
• Unit tests - Windows 2016
• Unit tests - Windows 2016
• Unit tests - macOS 15 ARM
• Unit tests - macOS 15 ARM
• Unit tests - Windows 10
• Unit tests - Windows 10
• Unit tests - Windows 11
• Unit tests - Windows 11
• Win2022:non-sudo:default
• Win2022:non-sudo:default
• Win2025:non-sudo:default
• Win2025:non-sudo:default
• x86_64:non-sudo: default
• x86_64:non-sudo: default
• x86_64:non-sudo: default
• x86_64:non-sudo: default
• v1.27.16:amd64:basic,wolfi,complete,complete-wolfi,service,cloud
• v1.27.16:amd64:basic,wolfi,complete,complete-wolfi,service,cloud
• v1.28.9:amd64:basic,wolfi,complete,complete-wolfi,service,cloud
• v1.28.9:amd64:basic,wolfi,complete,complete-wolfi,service,cloud
• v1.29.8:amd64:basic,wolfi,complete,complete-wolfi,service,cloud
• v1.29.8:amd64:basic,wolfi,complete,complete-wolfi,service,cloud
• v1.30.8:amd64:basic,wolfi,complete,complete-wolfi,service,cloud
• v1.30.8:amd64:basic,wolfi,complete,complete-wolfi,service,cloud
• v1.31.0:amd64:basic,wolfi,complete,complete-wolfi,service,cloud
• v1.31.0:amd64:basic,wolfi,complete,complete-wolfi,service,cloud

History

• 💔 Build #20450 failed 5d8afe8
• 💔 Build #20415 failed 3e30009

cc @swiatekm

[swiatekm]

We've decided not to do this in 8.18, as we don't want to bump the Otel libraries.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube