[Meta] Cloud Foundry Support

[blakerouse]

Overview

Cloud Foundry loggregator provides a gRPC stream to receive a combined logs and metrics about the applications running inside of Cloud Foundry. Loggregator provides selectors on connection to gRPC to split up the logs vs metrics.

Filebeat and metricbeat can use the gRPC stream with selectors to split up the events to process the appropriate events respectively as inputs to send to elasticsearch.

Libbeat work to complete

• Add common loggregator connection to share between metricbeat and filebeat.
• Add common add_cloudfoundry_metadata to add application metadata to events shared between metricbeat and filebeat.
• Add support for V1 Add support for cloudfoundry logs collection using consumer v1 API #19125 Add support for cloudfoundry metrics collection using consumer v1 API #19268
• Add integration tests https://github.com/elastic/beats/issues/17539
• Throttling per organization Add event rate quota per Cloud Foundry organization #21020

Filebeat work to complete

• Add cloudfoundry input to filebeat.
• Add documentation for cloudfoundry input.
• Test log files exist for the grok patterns (Logs come from gRPC protocol buffer events)
• Generated output for at least 1 log file exists
• Add a cloudfoundry module

Metricbeat work to complete

• Add cloudfoundry module.
• Add documentation for cloudfoundry module.
• Example data.json exists and an automated way to generate it exists (go test -data) (Currently no way to autogenerate)
• Test environment in Docker exist for integration tests (Cannot run Cloud Foundry in Docker)
• Test environment based on kind and kubecf
• Add a metricset with metrics grouped https://github.com/elastic/beats/issues/21532
• Some metrics are received twice in the same timestamp, but with different values, confirm if they have different tags

Usage work to complete

• Fields follow ECS and naming conventions
• Add documentation on how to run filebeat and metricbeat for Cloud Foundry.
  • Extend documentation to cover use cases with persistent cache. [Cloudfoundry] Document local persistent cache #21560
• Add dashboard for Kibana. When CF metricbeat module is installed, relevant dashboards are auto-installed #20798

[exekias]

cc @andresrc @mukeshelastic