You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: libbeat/autodiscover/README.md
+18-18Lines changed: 18 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -53,22 +53,22 @@ metricbeat.autodiscover:
53
53
54
54
##### How it works
55
55
56
-
We will deep dive in the internals of [libbeat kubernetes autodiscover provider](https://github.com/elastic/beats/tree/master/libbeat/autodiscover/providers/kubernetes).
56
+
We will deep dive in the internals of [libbeat kubernetes autodiscover provider](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes).
57
57
58
58
We will use metricbeat as an example.
59
59
60
60
Step-by-step walkthrough
61
-
1. Kubernetes provider `init` function [adds](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L46) the provider in the autodiscover providers registry at startup. For Kubernetes provider an `AutodiscoverBuilder` func is passed as an argument.
62
-
2. Metricbeat calls `NewAutodiscover`[function](https://github.com/elastic/beats/blob/master/metricbeat/beater/metricbeat.go#L183) which checks in the config for enabled providers and [builds](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/provider.go#L90) them one by one, calling the `AutodiscoverBuilder` func.
63
-
3. Kubernetes `AutodiscoverBuilder` creates and returns a [Kubernetes Provider struct](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L132) which is then added to an Autodiscover manager struct.
64
-
4. When unique is set to true [NewLeaderElectionManager](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L141) is set as the eventManager of Kubernetes Provider.
65
-
4. Metricbeat [starts](https://github.com/elastic/beats/blob/master/metricbeat/beater/metricbeat.go#L249) the Autodiscover manager which starts for Kubernets provider the [leaderElectionManager](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L326). Before starting the providers it also starts a worker for listening of events that will be published by the eventers of each provider.
66
-
5.`OnStartedLeading` is executed when the specific metricbeat instance gains the leader election lock. [StartLeading](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L205) creates a bus event with `"start": true,` and publishes it. The template configurations is also added in this event.
67
-
6.[Listener](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/autodiscover.go#L140) of events get the published event and generates [configs](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/autodiscover.go#L185) for it. Configs include the variables and settings from the template set by the user.
68
-
7. For each config the worker checks if it already [exists](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/autodiscover.go#L225)(It was already handled). If at least one of the events config does not exist, then the config is marked as `updated`.
69
-
8. The runners list get [reloaded](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/cfgfile/list.go#L54). It is checked from the list of current runners if each config is handled by one of them.
61
+
1. Kubernetes provider `init` function [adds](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L46) the provider in the autodiscover providers registry at startup. For Kubernetes provider an `AutodiscoverBuilder` func is passed as an argument.
62
+
2. Metricbeat calls `NewAutodiscover`[function](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/metricbeat/beater/metricbeat.go#L183) which checks in the config for enabled providers and [builds](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/provider.go#L90) them one by one, calling the `AutodiscoverBuilder` func.
63
+
3. Kubernetes `AutodiscoverBuilder` creates and returns a [Kubernetes Provider struct](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L131) which is then added to an Autodiscover manager struct.
64
+
4. When unique is set to true [NewLeaderElectionManager](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L141) is set as the eventManager of Kubernetes Provider.
65
+
5. Metricbeat [starts](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/metricbeat/beater/metricbeat.go#L249) the Autodiscover manager which starts for Kubernets provider the [leaderElectionManager](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L326). Before starting the providers it also starts a worker for listening of events that will be published by the eventers of each provider.
66
+
6.`OnStartedLeading` is executed when the specific metricbeat instance gains the leader election lock. [StartLeading](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L205) creates a bus event with `"start": true,` and publishes it. The template configurations is also added in this event.
67
+
7.[Listener](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/autodiscover.go#L140) of events get the published event and generates [configs](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/autodiscover.go#L185) for it. Configs include the variables and settings from the template set by the user.
68
+
8. For each config the worker checks if it already [exists](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/autodiscover.go#L221)(It was already handled). If at least one of the events config does not exist, then the config is marked as `updated`.
69
+
9. The runners list get [reloaded](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/cfgfile/list.go#L54). It is checked from the list of current runners if each config is handled by one of them.
70
70
If no runner is handling that config a new runner will [start](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/cfgfile/list.go#L107). If some runners are no longer needed will be removed.
71
-
9. Starting of a runner starts under the hood the metricbeat [module](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/metricbeat/mb/module/runner.go#L76) with the specific metricsets as they are set in the template. Same also applies for filebeat and heartbeat cases.
71
+
10. Starting of a runner starts under the hood the metricbeat [module](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/metricbeat/mb/module/runner.go#L76) with the specific metricsets as they are set in the template. Same also applies for filebeat and heartbeat cases.
72
72
73
73
### Autodiscover without LeaderElection
74
74
@@ -97,10 +97,10 @@ We will use metricbeat and redis module as example.
97
97
Step-by-step walkthrough
98
98
99
99
Steps 1-3 are exactly the same as with Leader Election.
100
-
4. In the [Kubernetes Provider struct](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L131) created by the Kubernetes AutodiscoverBuilder in case there is a [condition](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L116) a `NewConfigMapper` is created. It contains the condition [map](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/template/config.go#L64) with all the conditions of a given config.
101
-
4. When unique is set to false, the resource is [checked](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L241) from the configuration template(default is pod) and a new PodEventer/NodeEventer/ServiceEventer is set as the eventManager of Kubernetes Provider.
100
+
4. In the [Kubernetes Provider struct](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L131) created by the Kubernetes AutodiscoverBuilder in case there is a [condition](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L116) a `NewConfigMapper` is created. It contains the condition [map](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/template/config.go#L64) with all the conditions of a given config.
101
+
4. When unique is set to false, the resource is [checked](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L239) from the configuration template(default is pod) and a new PodEventer/NodeEventer/ServiceEventer is set as the eventManager of Kubernetes Provider.
102
102
5. The kubernetes node that the metricbeat instance is running on is [discovered](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/pod.go#L72).
103
-
6. A dedicated watcher to watch [pods](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/pod.go#L82),nodes and namespaces get started. Also a [metadata generator](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/pod.go#L113) is started to enrich the events with kubernetes metadata.
103
+
6. A dedicated watcher to watch [pods](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/pod.go#L82),nodes and namespaces get started. Also a [metadata generator](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/pod.go#L113) is started to enrich the events with kubernetes metadata.
104
104
7. Each time a new event is about to be published by the watchers(we will have a dedicated section of how events are published), it is checked whether the condition set by the user [matches](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L182) the event.
105
105
8. If it matches, then a config is created with the configuration set by user and it is added in the event. After that the event gets published.
106
106
9. The worker listening for events receives the start event and checks if there is any update in the configs of the event. (same process with leader election steps 6-9)
@@ -177,12 +177,12 @@ spec:
177
177
178
178
Everything works the same as Autodiscover without LeaderElection until step 8.
179
179
180
-
8. If there is no conditions in the template set by the user, the configs will be generated from [hints](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L186).
181
-
9. Wether hints are enabled or not is part of the [Kubernetes Provider struct](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L121) builders field.
180
+
8. If there is no conditions in the template set by the user, the configs will be generated from [hints](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L186).
181
+
9. Wether hints are enabled or not is part of the [Kubernetes Provider struct](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L121) builders field.
182
182
10.[GenerateHints](https://github.com/elastic/beats/blob/eff92354db783001880f4bade9f59942fca747ba/libbeat/autodiscover/builder/helper.go#L213) function looks into the event's annotations. A [hints map](https://github.com/elastic/beats/blob/eff92354db783001880f4bade9f59942fca747ba/libbeat/autodiscover/builder/helper.go#L226) is created with all hints and returned.
183
-
11. From those hints, configs are [created](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/builder.go#L97) in the same form as in `Autodiscover without LeaderElection` step 8.
183
+
11. From those hints, configs are [created](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/builder.go#L97) in the same form as in `Autodiscover without LeaderElection` step 8.
184
184
They contain the same information as if they where set explicitly in the metricbeat configureation but actually derive from the pod annotations.
185
-
12. Those configs are then [added](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L197) in the event and gets published.
185
+
12. Those configs are then [added](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L197) in the event and gets published.
186
186
13. The process after that is same as in `Autodiscover without LeaderElection` step 9 and onward.
0 commit comments