[beatreceiver] - Add status reporting

[VihasMakwana]

Proposed commit message

This PR adds status reporting for beatreceivers. The status reporting is added while creating the runners. The first PR (#44528) was quite "hacky" and it had go deep down to inject status reporters.

This PR adds a runner factory wrapper that will:

1. Call the parent factory to create the runner
2. Inject status reporter

The code responsible for doing the above tasks will live in libbeat and we will only enable it for beatreceivers. From an the beat receiver high level, it will do following:

1. The beater will be created in createReceiver
2. We will add the factory wrapper

   beats/x-pack/libbeat/cmd/instance/receiver.go

   Lines 80 to 83 in 344bbce

   	if w, ok := br.beater.(cfgfile.WithFactoryWrapper); ok {
   	groupReporter := status.NewGroupStatusReporter(host)
   	w.WithFactoryWrapper(status.StatusReporterFactory(groupReporter))
   	}

3. The receiver will kick off the beater

   beats/x-pack/libbeat/cmd/instance/receiver.go

   Lines 76 to 81 in 6286492

   	func (br *BeatReceiver) Start() error {
   	if err := br.beater.Run(&br.beat.Beat); err != nil {
   	return fmt.Errorf("beat receiver run error: %w", err)
   	}
   	return nil
   	}

Note:

To accomplish the above steps, it is essential that we create the runners in beater.Run(...). Currently, metricbeat creates runners during the beater creation phase and starts them in beater.Run(...). This PR moves the runner creation code in beater.Run(...) to closely align with filebeat's implementation.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Related Beat receivers do not correctly report status back to the Elastic Agent elastic-agent#8210

Screenshots

Output

Here's output of running two streams (degraded) together:

┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   ├─ status: (DEGRADED) 1 or more components/units in a degraded state
   ├─ pipeline:logs/_agent-component/filestream-default
   │  ├─ status: StatusRecoverableError [error while running harvester: cannot read from file source: /var/log/elasticAgent-install-20240625_133733.log]
   │  ├─ exporter:elasticsearch/_agent-component/default
   │  │  └─ status: StatusOK
   │  └─ receiver:filebeatreceiver/_agent-component/filestream-default
   │     └─ status: StatusRecoverableError [error while running harvester: cannot read from file source: /var/log/elasticAgent-install-20240625_133733.log]
   └─ pipeline:logs/_agent-component/system/metrics-default
      ├─ status: StatusRecoverableError [Error fetching data for metricset system.process: error fetching process list: non fatal error; reporting partial metrics: error fetching PID metrics for 607 processes, most likely a "permission denied" error. Enable debug logging to determine the exact cause.]
      ├─ exporter:elasticsearch/_agent-component/default
      │  └─ status: StatusOK
      └─ receiver:metricbeatreceiver/_agent-component/system/metrics-default
         └─ status: StatusRecoverableError [Error fetching data for metricset system.process: error fetching process list: non fatal error; reporting partial metrics: error fetching PID metrics for 607 processes, most likely a "permission denied" error. Enable debug logging to determine the exact cause.]

Testing

1. Checkout this PR locally
2. Go to elastic-agent and follow this guide to test local beats changes
3. Package agent with mage package
4. Follow steps on Beat receivers do not correctly report status back to the Elastic Agent elastic-agent#8210 to install agent and verify the status

Closes elastic/elastic-agent#8210

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @VihasMakwana? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mauri870]

Quite an elegant solution for this problem!

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b wrap-runner-factory upstream/wrap-runner-factory
git merge upstream/main
git push upstream wrap-runner-factory

[VihasMakwana]

@mauri870 @khushijain21 I've added new test cases and have made changes to benchmark modules for testing. We can now make benchmark module return error if we want, to test status reporting.
Please take a look!

[cmacknz]

This LGTM, my only remaining comment is that seeing the FactoryWrapper methods outside of the context of this PR makes it quite hard to figure out what they do.

Could we rename them to have OtelStatus (or Otel or similar) in the name so we know what they are injecting when we read them?

Alternatively, can you add comments to all of them if you want to leave the names alone?