[Agent] Inject index with sane defaults to configuration by michalpristas · Pull Request #16903 · elastic/beats

michalpristas · 2020-03-09T14:48:38Z

Migrated from: #15852
Build on top of: #15809

Added one extra rule for injecting index into inputs based on provided type and detected namespace/dataset type.

Other option was to use decorator but at the time of decorating there are already some pieces of config missing.

Draft until final configuration format is settled and mentioned PR is in, but READY for review.

It overwrites index specified in input.

Fixes: #15690
Fixes: #15691

cc @ruflin @ph

elasticmachine · 2020-03-09T14:48:41Z

Pinging @elastic/ingest-management (Project:fleet)

ruflin

I think it's important that the config we have here is the a config that already provides value to the user. It's not only a sample. I compare to to running metricbeat with ./metricbeat. All the system data is just there and works.

ruflin · 2020-03-09T14:56:09Z

-      - /var/log/hello1.log
-      - /var/log/hello2.log
+datasources:
+  - namespace: sample


Why not using the default namespace here?

ruflin · 2020-03-09T14:56:34Z

+      - type: logs
+        processors:
+        streams:
+          - dataset: sample.acccess


I suggest if we don't have any good default logs here, we just skip the log parts / leave it commented out. I would suggest the system logs here (check logs system module)

system module seems purely linux oriented so i left it out so our future windows test wont fail on something like this

That is where in the future our constraints come into play. I think part of it work also on OS X

The samples you put in will also not work. Suggestion: Lets skip the logs part for now but have a logs sample based on the Linux logs inside but commented out. So users can easily edit it.

ruflin · 2020-03-09T14:56:45Z

+            paths: /var/log/sample/error.log
+      - type: system/metrics
+        streams:
+          - metricset: cpu


Lets add a few more here like memory, disk.

ruflin

Can you change to the default namespace everywhere?

ruflin · 2020-03-10T10:57:11Z

-      - /var/log/hello1.log
-      - /var/log/hello2.log
+datasources:
+  - namespace: sample


Suggested change

- namespace: sample

- namespace: default

ruflin · 2020-03-10T10:57:24Z

-      - /var/log/hello1.log
-      - /var/log/hello2.log
+datasources:
+  - namespace: sample


Suggested change

- namespace: sample

- namespace: default

ruflin · 2020-03-10T10:59:12Z

-      - /var/log/hello1.log
-      - /var/log/hello2.log
+datasources:
+  - namespace: sample


Suggested change

- namespace: sample

- namespace: default

ph · 2020-03-10T14:33:41Z

 github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
 github.com/eclipse/paho.mqtt.golang v1.2.1-0.20200121105743-0d940dd29fd2 h1:DW6WrARxK5J+o8uAKCiACi5wy9EK1UzrsCpGBPsKHAA=
 github.com/eclipse/paho.mqtt.golang v1.2.1-0.20200121105743-0d940dd29fd2/go.mod h1:H9keYFcgq3Qr5OUJm/JZI/i6U7joQ8SYLhZwfeOo6Ts=
+github.com/elastic/beats v7.6.1+incompatible h1:4iPVpFr8BSJW2fUVi+/tYXQ4v/IYVx0k2PPLTg8cfks=


I think this is weird?

ph

code LGTM, I haven't tested it.

Test looks OK to catch the problems.

ph · 2020-03-10T14:34:22Z

+          - enabled: true
+            metricset: info
+
+settings.monitoring:


+1 for changing this.

ph · 2020-03-10T14:40:28Z

@michalpristas lets add this change to the new changelog too, so we know when the new syntax has landed.

michalpristas · 2020-03-10T14:41:38Z

yep i will after we merge changelog in

ruflin · 2020-03-11T10:29:16Z

+settings.monitoring:
+  use_output: monitoring
+
+management:


Not in this PR, but thinking this + config should also be under settings?

ruflin · 2020-03-11T10:30:27Z

+    inputs:
+      - type: system/metrics
+        streams:
+          - metricset: cpu


Had a quick chat with @michalpristas about this on Slack. The problem here is that this will send data to metrics-generic-default but it should end up in metrics-system.cpu-default. We probably have to add the dataset here at the moment and simplify it later.

ruflin

Configs LGTM.

ph

LGTM, lets get that merged in.

ph · 2020-03-11T12:40:02Z

I've added the "need_backport" will merge after #16885

michalpristas · 2020-03-11T14:03:07Z

Jenkins test this

…7579) [Agent] Inject index with sane defaults to configuration (#16903) (cherry picked from commit 96e4506) Co-authored-by: Michal Pristas <michal.pristas@gmail.com>

michalpristas added 2 commits March 9, 2020 15:33

master-feature sync

139e44e

go mod tidy

677736e

michalpristas added Project:fleet labels Mar 9, 2020

michalpristas mentioned this pull request Mar 9, 2020

[Agent] Inject index with sane defaults to configuration #16328

Closed

michalpristas requested a review from ph March 9, 2020 14:49

ruflin reviewed Mar 9, 2020

View reviewed changes

updated configs

85920bc

ruflin reviewed Mar 10, 2020

View reviewed changes

michalpristas added 2 commits March 10, 2020 13:37

sample->default

c31e69c

mage update

bde7801

ph reviewed Mar 10, 2020

View reviewed changes

ph approved these changes Mar 10, 2020

View reviewed changes

removed beat v7

4c33759

ph added the needs_backport PR is waiting to be backported to other branches. label Mar 10, 2020

ruflin reviewed Mar 11, 2020

View reviewed changes

michalpristas added 4 commits March 11, 2020 11:39

Merge branch 'master' into agent-new-config-master

8883b1f

specify dataset

f920dfb

docker config

e6885b3

mage update

c5bd054

ruflin approved these changes Mar 11, 2020

View reviewed changes

ph approved these changes Mar 11, 2020

View reviewed changes

michalpristas self-assigned this Mar 11, 2020

michalpristas merged commit 96e4506 into elastic:master Mar 11, 2020

ph mentioned this pull request Apr 7, 2020

Cherry-pick #16903 to 7.x: [Agent] Inject index with sane defaults to configuration #17579

Merged

ph added v7.8.0 and removed needs_backport PR is waiting to be backported to other branches. labels Apr 7, 2020

Conversation

michalpristas commented Mar 9, 2020

Uh oh!

elasticmachine commented Mar 9, 2020

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ph left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ph commented Mar 10, 2020

Uh oh!

michalpristas commented Mar 10, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

ph left a comment

Choose a reason for hiding this comment

Uh oh!

ph commented Mar 11, 2020

Uh oh!

michalpristas commented Mar 11, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants