[Elastic Agent] Monitor Filebeat Inputs

[kcreddy]

What does this PR do?

• Add new datastream filebeat_input to collect filebeat input metrics by monitoring filebeat's /inputs/ API endpoint. The endpoint currently supports 4 inputs

• Most metrics/fields are dynamically mapped based on the naming convention.

• Elastic Agent will then monitor the filebeat's /inputs/ when Update agent monitoring config to support /inputs of filebeat elastic-agent#2171 is merged

Local Tests

1. Add an input.
   cat packages/elastic_agent/data_stream/filebeat_input_metrics/agent/stream/stream.yml.hbs

metricsets: ["json"]
namespace: "filebeat_input"
hosts: ["http+unix:///usr/share/elastic-agent/state/data/tmp/aws-s3-default.sock"]
period: 10s
json.is_array: true
path: "/inputs/"
tags:
{{#each tags as |tag|}}
  - {{tag}}
{{/each}}
processors:
- convert:
    mode: rename
    ignore_missing: true
    fields:
      - from: http.filebeat_input
        to: filebeat_input_metrics

2. Append below to packages/elastic_agent/data_stream/filebeat_input_metrics/manifest.yml

streams:
  - input: http/metrics
    vars:
      - name: tags
        type: text
        title: Tags
        multi: true
        required: true
        show_user: true
        default:
          - forwarded
          - elastic-agent-filebeat-input
    template_path: stream.yml.hbs
    title: Filebeat Input Metrics
    description: Collect Filebeat Input Metrics

3. Append below to packages/elastic_agent/manifest.yml:

policy_templates:
  - name: elastic_agent
    title: Elastic Agent Metrics
    description: Elastic Agent Metrics
    inputs:
      - type: http/metrics
        title: "Collect Metrics from Filebeat Inputs"
        description: "Collect Metrics from Filebeat Inputs"

4. Configure any datastream with aws-s3 input type (such as aws.s3access)
5. Add Elastic Agent integration. This should create filebeat input metric events under datastream metrics-elastic_agent.filebeat_input-default

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Relates https://github.com/elastic/security-team/issues/5525

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-02-02T02:58:34.222+0000

• Duration: 14 min 51 sec

Test stats 🧪

Test	Results
Failed	0
Passed	34
Skipped	0
Total	34

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚 12.5
Classes	100.0% (0/0)	💚 12.5
Methods	33.333% (24/72)	👎 -50.183
Lines	100.0% (0/0)	💚 7.286
Conditionals	100.0% (0/0)	💚

[kcreddy]

/test

[ruflin]

@cmacknz Is this just a temporary solution? This seems to be monitoring data of v2 inputs? Will this all go under elastic_agent.inputs or similar in the future?

[kcreddy]

Hey @cmacknz @andrewkroh, I also need suggestion/direction on the name of the datastream as well, whether I need to create this new datastream filebeat_input or instead use existing elastic_agent datastream which is currently being used to send filebeat's /stats metrics.

Basically the field mappings in current PR were supposed to be used in conjunction with agent's inbuilt monitoring using http json metricset which monitors filebeat's /inputs endpoint. The PR is here: elastic/elastic-agent#2171 and would also need review
If I use this new datastream filebeat_input inside the agent's configuration instead of fixedAgentName and run the stack, I get following error during indexing:

{\"type\":\"security_exception\",\"reason\":\"action [indices:admin/auto_create] is unauthorized for API key id [j0XuAYYBIxEvrsej-8Ob] of user [elastic/fleet-server] on indices [metrics-elastic_agent.filebeat_input-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]\"}, dropping event!"

This seems to be related to elastic/elastic-agent#1814

If I use existing datastream elastic_agent and add all the fields mapping from this PR to elastic_agent datastream, it seem to work fine.

Also, whether we want to use TSDS or not since the CI is failing from usage of time_series mode probably due to current minimum version 7.16.0.

[andrewkroh]

It definitely sounds like the path of least resistance would be to add these to the existing metrics-elastic_agent.filebeat-default data stream (without using TSDS). This avoids the issues related to permissions and allows the integration package to continue having a 7.x compatibility constraint.

This seems to be monitoring data of v2 inputs?

It's not strictly FB v2 inputs. Any Filebeat input regardless of the Filebeat interface used is eligible expose metrics. I think that all the ones with metrics happen to be implemented with Filebeat input v2 today. My expectation is that in the future we'll update Filebeat to publish metrics through the Agent V2 control interface at which time this metric collection can be removed.

[cmacknz]

This is hopefully temporary (on a to be determined time scale), we want to change the structure of the monitoring indexes because they are hard coded directly into Fleet right now. The path of least resistance here is definitely to add these to metrics-elastic_agent.filebeat-default.

• Issue discussion how we should change the monitoring index structure V2: Elastic agent monitoring data stream names should not be per process elastic-agent#1814. Read from V2: Elastic agent monitoring data stream names should not be per process elastic-agent#1814 (comment)
• We still need a path forward for collecting application level metrics in V2 without starting more and more Beat inputs to do it. We have an issue for discussing this as well but not concrete decisions yet. Determine path forward for collection of metrics for components/units elastic-agent#2181

[ruflin]

Reading all the above, my current preference would be using filebeat_input as proposed in the PR and fix the permission issue in Fleet (hardcode). I expect as soon as v2 is fully done, data is likely shipped into input data stream but there might be breaking change, because of this the filebeat_input seems to be a good temporary solution. Also lets make sure we treat this as a beta data stream. For 7.16, I would expect this to only land in version > 8.7 or similar.

The part I worry about putting it into filebeat is that it could lead to potential conflicts and we ship 2 different datasets into the same data stream which is a bit against the principles.

[kcreddy]

Thanks for the clarification guys!

So if we intend to use new filebeat_input datastream itself, I have updated this PR and also elastic/elastic-agent#2171 to reflect filebeat_input datastream.

@ruflin @cmacknz to fix the permission issue in Fleet, may I know if this is the right place to add this new datastream?
https://github.com/elastic/kibana/blob/c5f20721e1879f1ebe161b0fa3b207f10ed2b6f7/x-pack/plugins/fleet/common/constants/agent_policy.ts#L15-L28
If so, I can work on a quick PR to add it.

We are targeting the filebeats inputs monitoring feature for 8.7 release, so any help would be appreciated to get it done. Thanks!

[ruflin]

On my end, I'm only "suggestion", I'll leave it to @cmacknz to make decisions ;-) For the permissions, @nchaulet likely know the best place but what you linked above at least looks like part of the puzzle.

[cmacknz]

Mixing multiple datasets into the same datastream is something we should avoid, so I agree with Ruflin here in using a separate filebeat_input Beta datastream.

Here's an example of adding the new index to Fleet elastic/kibana@5d5603a

Note that if you make the change there I believe you will get both a logs and a metrics datastream. The logs datastream will effectively be unused so we may not actually want it created. There's nothing explicitly wrong with it being empty though. FYI @kpollich as well.

[kcreddy]

Thanks for the clarification and sharing the example @cmacknz
I have now added logs datastream to this PR as well.

Also I have created PR elastic/kibana#149974 to add new index. @kpollich could you please review it?

[kcreddy]

Thanks @nchaulet!

@AndersonQ / @michel-laterman Could you please review/approve this PR and also elastic/elastic-agent#2171

[cmacknz]

LGTM, except I think we could bump to the next minor version.

[leehinman]

LGTM

Non-blocking suggestion to use the external definition for the ECS fields.

[leehinman]

Is there a reason not to have ECS fields like cloud.account.id use the external definition? This isn't blocking but it does make long term support easier and helps prevent type conflicts.

[kcreddy]

I need to explore the reason, but it looks like, for this package, all datastreams follow this convention, i.e., defining cloud related fields inside agent.yml. Maybe its worth creating an issue to replace all with ECS external definition instead.

[elasticmachine]

Package elastic_agent - 1.5.0 containing this change is available at https://epr.elastic.co/search?package=elastic_agent