Add cloudfoundry module to metricbeat by blakerouse · Pull Request #16671 · elastic/beats

blakerouse · 2020-02-27T19:08:02Z

What does this PR do?

Adds a new cloudfoundry module to x-pack/metricbeat. Adds 3 metric sets container, counter, and value.

This module/metricsets are different from standard metricsets as the Cloud Foundry loggregator pushes to metrics to the client. Each metricset shares the same connection from the module so it does not create a connection per metricset.

Why is it important?

Adds support for metrics from cloudfoundry.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~I have made corresponding change to the default configuration files~~
~~I have added tests that prove my fix is effective or that my feature works~~

Author's Checklist

Receives metrics from Cloud Foundry loggregator.
Selecting individual metricsets works.

How to test this PR locally

With a running Cloud Foundry run metricbeat with the following configuration:

metricbeat.modules:
  - &cloudfoundry
    module: cloudfoundry
    api_address: https://api.dev.cfdev.sh
    client_id: firehose-to-filebeat
    client_secret: verysecret
    ssl:
      verification_mode: none

processors:
  - add_cloudfoundry_metadata:
      <<: *cloudfoundry

output.console:
  pretty: true

Related issues

Relates [Meta] Cloud Foundry Support #16520

Use cases

Screenshots

Logs

* Document Kibana Space ID setting * Update reference YML files * Update libbeat/docs/shared-kibana-config.asciidoc Co-Authored-By: DeDe Morton <dede.morton@elastic.co> * Updating reference YML files * Adding link to doc Co-authored-by: DeDe Morton <dede.morton@elastic.co>

We have changed the type from a special custom type to a slice of string. This reduce the number of exposed types and make it easier in general to work with in the tests.

…nges (elastic#16584)

* Add vpn into vpc metricset * Add NATGateway and TrasitGateway

* Update cisco.asciidoc Making Elasticsearch explicit in the instructions * Update docs source in the module dir Co-authored-by: Loek van Gool <loek.vangool@elastic.co>

Fields generator keeps all open files while generating the global fields.yml file. This is not needed and may lead to "too many open files" errors in some environments.

Add metricsets: node, proxy

sayden

I have taken an overall look and it seems fine.

However it will be nice to have some comments for maintainers to help in the logic flow 😉

blakerouse · 2020-03-02T18:09:27Z

@kaiyan-sheng Added the config fields to the reference yaml if you could give it another look. Thanks!

kaiyan-sheng

LGTM

* Split up parts into different metricsets. * Add value and container metricsets. * Add documentation. * Revert metricbeat.yml. * Run make update. * Fix documentation add changelog. * Fix bad change in azure module. * Run make update. * Improve reference config. * Document Kibana Space ID setting (elastic#16622) * Document Kibana Space ID setting * Update reference YML files * Update libbeat/docs/shared-kibana-config.asciidoc Co-Authored-By: DeDe Morton <dede.morton@elastic.co> * Updating reference YML files * Adding link to doc Co-authored-by: DeDe Morton <dede.morton@elastic.co> * Remove the special pins type for `ca_sha256` (elastic#16615) We have changed the type from a special custom type to a slice of string. This reduce the number of exposed types and make it easier in general to work with in the tests. * Remove feature.Describer and enhance Details (elastic#16658) * Make memqueue.Broker internal (now memqueue.broker) (elastic#16667) * [docs] Add skeleton files for 7.7 release highlights and breaking changes (elastic#16584) * [Metricbeat] Add vpc metricset for aws module (elastic#16111) * Add vpn into vpc metricset * Add NATGateway and TrasitGateway * Update cisco.asciidoc (elastic#16434) * Update cisco.asciidoc Making Elasticsearch explicit in the instructions * Update docs source in the module dir Co-authored-by: Loek van Gool <loek.vangool@elastic.co> * Fix wording in dev guide (elastic#16497) * Added NetBeat (elastic#12612) * Close files earlier when generating fields files (elastic#16681) Fields generator keeps all open files while generating the global fields.yml file. This is not needed and may lead to "too many open files" errors in some environments. * [Metricbeat] Add new module for Redis Enterprise (elastic#16482) Add metricsets: node, proxy * Add doppler_address, uaa_address, rlp_address to config.reference.yml. Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Steffen Siering <steffen.siering@elastic.co> Co-authored-by: Fae Charlton <fae.charlton@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Loek van Gool <loek.vangool@elastic.co> Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com> (cherry picked from commit 07b03a7)

) * Add cloudfoundry module to metricbeat (#16671) * Split up parts into different metricsets. * Add value and container metricsets. * Add documentation. * Revert metricbeat.yml. * Run make update. * Fix documentation add changelog. * Fix bad change in azure module. * Run make update. * Improve reference config. * Document Kibana Space ID setting (#16622) * Document Kibana Space ID setting * Update reference YML files * Update libbeat/docs/shared-kibana-config.asciidoc Co-Authored-By: DeDe Morton <dede.morton@elastic.co> * Updating reference YML files * Adding link to doc Co-authored-by: DeDe Morton <dede.morton@elastic.co> * Remove the special pins type for `ca_sha256` (#16615) We have changed the type from a special custom type to a slice of string. This reduce the number of exposed types and make it easier in general to work with in the tests. * Remove feature.Describer and enhance Details (#16658) * Make memqueue.Broker internal (now memqueue.broker) (#16667) * [docs] Add skeleton files for 7.7 release highlights and breaking changes (#16584) * [Metricbeat] Add vpc metricset for aws module (#16111) * Add vpn into vpc metricset * Add NATGateway and TrasitGateway * Update cisco.asciidoc (#16434) * Update cisco.asciidoc Making Elasticsearch explicit in the instructions * Update docs source in the module dir Co-authored-by: Loek van Gool <loek.vangool@elastic.co> * Fix wording in dev guide (#16497) * Added NetBeat (#12612) * Close files earlier when generating fields files (#16681) Fields generator keeps all open files while generating the global fields.yml file. This is not needed and may lead to "too many open files" errors in some environments. * [Metricbeat] Add new module for Redis Enterprise (#16482) Add metricsets: node, proxy * Add doppler_address, uaa_address, rlp_address to config.reference.yml. Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Steffen Siering <steffen.siering@elastic.co> Co-authored-by: Fae Charlton <fae.charlton@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Loek van Gool <loek.vangool@elastic.co> Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com> (cherry picked from commit 07b03a7) * Fix changelog. * Run mage fmt. * Fix imports to have v7. * Run mage fmt update. * Fix changelog.

blakerouse added 4 commits February 27, 2020 11:25

Split up parts into different metricsets.

7284be3

Add value and container metricsets.

3d2db66

Add documentation.

ce7e84d

Revert metricbeat.yml.

66ac24e

blakerouse requested a review from a team as a code owner February 27, 2020 19:08

blakerouse added enhancement review Team:Platforms Label for the Integrations - Platforms team labels Feb 27, 2020

blakerouse and others added 16 commits February 27, 2020 15:22

Run make update.

8e0fdf0

Fix documentation add changelog.

b3be6f7

Fix bad change in azure module.

cbc4c67

Run make update.

f9a3f21

Improve reference config.

99be099

Remove the special pins type for ca_sha256 (elastic#16615)

23d7c06

We have changed the type from a special custom type to a slice of string. This reduce the number of exposed types and make it easier in general to work with in the tests.

Remove feature.Describer and enhance Details (elastic#16658)

e79af86

Make memqueue.Broker internal (now memqueue.broker) (elastic#16667)

e4e683b

[docs] Add skeleton files for 7.7 release highlights and breaking cha…

421fd91

…nges (elastic#16584)

[Metricbeat] Add vpc metricset for aws module (elastic#16111)

6e79791

* Add vpn into vpc metricset * Add NATGateway and TrasitGateway

Update cisco.asciidoc (elastic#16434)

ba49a12

* Update cisco.asciidoc Making Elasticsearch explicit in the instructions * Update docs source in the module dir Co-authored-by: Loek van Gool <loek.vangool@elastic.co>

Fix wording in dev guide (elastic#16497)

8d5367a

Added NetBeat (elastic#12612)

60c3df0

Close files earlier when generating fields files (elastic#16681)

37ab97d

Fields generator keeps all open files while generating the global fields.yml file. This is not needed and may lead to "too many open files" errors in some environments.

[Metricbeat] Add new module for Redis Enterprise (elastic#16482)

06c7585

Add metricsets: node, proxy

blakerouse requested review from a team as code owners February 28, 2020 14:26

blakerouse removed request for a team February 28, 2020 14:26

Merge branch 'master' into metricbeat-cf-module

5362d75

andresrc added the [zube]: Inbox label Mar 2, 2020

andresrc added [zube]: In Review and removed [zube]: Inbox labels Mar 2, 2020

kaiyan-sheng reviewed Mar 2, 2020

View reviewed changes

Comment thread x-pack/metricbeat/module/cloudfoundry/_meta/docs.asciidoc

sayden reviewed Mar 2, 2020

View reviewed changes

Add doppler_address, uaa_address, rlp_address to config.reference.yml.

eaae0c6

kaiyan-sheng approved these changes Mar 2, 2020

View reviewed changes

blakerouse merged commit 07b03a7 into elastic:master Mar 2, 2020

blakerouse deleted the metricbeat-cf-module branch March 2, 2020 22:40

zube Bot added [zube]: Done and removed [zube]: In Review labels Mar 2, 2020

blakerouse mentioned this pull request Mar 2, 2020

Cherry-pick #16671 to 7.x: Add cloudfoundry module to metricbeat #16741

Merged

5 tasks

blakerouse added the v7.7.0 label Mar 2, 2020

andresrc removed the [zube]: Done label Mar 4, 2020

jsoriano mentioned this pull request Feb 16, 2021

Reusable/common snippets of filebeat input config #16084

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cloudfoundry module to metricbeat#16671

Add cloudfoundry module to metricbeat#16671
blakerouse merged 22 commits intoelastic:masterfrom
blakerouse:metricbeat-cf-module

blakerouse commented Feb 27, 2020 •

edited by andresrc

Loading

Uh oh!

Uh oh!

sayden left a comment

Uh oh!

blakerouse commented Mar 2, 2020

Uh oh!

kaiyan-sheng left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Conversation

blakerouse commented Feb 27, 2020 • edited by andresrc Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

Uh oh!

sayden left a comment

Choose a reason for hiding this comment

Uh oh!

blakerouse commented Mar 2, 2020

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

blakerouse commented Feb 27, 2020 •

edited by andresrc

Loading