Skip to content

Commit 4e89dbb

Browse files
Replace code links with permalinks
1 parent 122ef18 commit 4e89dbb

File tree

2 files changed

+26
-26
lines changed
  • libbeat/autodiscover
  • x-pack/elastic-agent/pkg/composable/providers/kubernetes

2 files changed

+26
-26
lines changed

libbeat/autodiscover/README.md

Lines changed: 18 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -53,22 +53,22 @@ metricbeat.autodiscover:
5353

5454
##### How it works
5555

56-
We will deep dive in the internals of [libbeat kubernetes autodiscover provider](https://github.com/elastic/beats/tree/master/libbeat/autodiscover/providers/kubernetes).
56+
We will deep dive in the internals of [libbeat kubernetes autodiscover provider](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes).
5757

5858
We will use metricbeat as an example.
5959

6060
Step-by-step walkthrough
61-
1. Kubernetes provider `init` function [adds](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L46) the provider in the autodiscover providers registry at startup. For Kubernetes provider an `AutodiscoverBuilder` func is passed as an argument.
62-
2. Metricbeat calls `NewAutodiscover` [function](https://github.com/elastic/beats/blob/master/metricbeat/beater/metricbeat.go#L183) which checks in the config for enabled providers and [builds](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/provider.go#L90) them one by one, calling the `AutodiscoverBuilder` func.
63-
3. Kubernetes `AutodiscoverBuilder` creates and returns a [Kubernetes Provider struct](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L132) which is then added to an Autodiscover manager struct.
64-
4. When unique is set to true [NewLeaderElectionManager](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L141) is set as the eventManager of Kubernetes Provider.
65-
4. Metricbeat [starts](https://github.com/elastic/beats/blob/master/metricbeat/beater/metricbeat.go#L249) the Autodiscover manager which starts for Kubernets provider the [leaderElectionManager](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L326). Before starting the providers it also starts a worker for listening of events that will be published by the eventers of each provider.
66-
5. `OnStartedLeading` is executed when the specific metricbeat instance gains the leader election lock. [StartLeading](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L205) creates a bus event with `"start": true,` and publishes it. The template configurations is also added in this event.
67-
6. [Listener](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/autodiscover.go#L140) of events get the published event and generates [configs](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/autodiscover.go#L185) for it. Configs include the variables and settings from the template set by the user.
68-
7. For each config the worker checks if it already [exists](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/autodiscover.go#L225)(It was already handled). If at least one of the events config does not exist, then the config is marked as `updated`.
69-
8. The runners list get [reloaded](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/cfgfile/list.go#L54). It is checked from the list of current runners if each config is handled by one of them.
61+
1. Kubernetes provider `init` function [adds](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L46) the provider in the autodiscover providers registry at startup. For Kubernetes provider an `AutodiscoverBuilder` func is passed as an argument.
62+
2. Metricbeat calls `NewAutodiscover` [function](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/metricbeat/beater/metricbeat.go#L183) which checks in the config for enabled providers and [builds](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/provider.go#L90) them one by one, calling the `AutodiscoverBuilder` func.
63+
3. Kubernetes `AutodiscoverBuilder` creates and returns a [Kubernetes Provider struct](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L131) which is then added to an Autodiscover manager struct.
64+
4. When unique is set to true [NewLeaderElectionManager](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L141) is set as the eventManager of Kubernetes Provider.
65+
5. Metricbeat [starts](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/metricbeat/beater/metricbeat.go#L249) the Autodiscover manager which starts for Kubernets provider the [leaderElectionManager](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L326). Before starting the providers it also starts a worker for listening of events that will be published by the eventers of each provider.
66+
6. `OnStartedLeading` is executed when the specific metricbeat instance gains the leader election lock. [StartLeading](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L205) creates a bus event with `"start": true,` and publishes it. The template configurations is also added in this event.
67+
7. [Listener](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/autodiscover.go#L140) of events get the published event and generates [configs](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/autodiscover.go#L185) for it. Configs include the variables and settings from the template set by the user.
68+
8. For each config the worker checks if it already [exists](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/autodiscover.go#L221) (It was already handled). If at least one of the events config does not exist, then the config is marked as `updated`.
69+
9. The runners list get [reloaded](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/cfgfile/list.go#L54). It is checked from the list of current runners if each config is handled by one of them.
7070
If no runner is handling that config a new runner will [start](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/cfgfile/list.go#L107). If some runners are no longer needed will be removed.
71-
9. Starting of a runner starts under the hood the metricbeat [module](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/metricbeat/mb/module/runner.go#L76) with the specific metricsets as they are set in the template. Same also applies for filebeat and heartbeat cases.
71+
10. Starting of a runner starts under the hood the metricbeat [module](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/metricbeat/mb/module/runner.go#L76) with the specific metricsets as they are set in the template. Same also applies for filebeat and heartbeat cases.
7272

7373
### Autodiscover without LeaderElection
7474

@@ -97,10 +97,10 @@ We will use metricbeat and redis module as example.
9797
Step-by-step walkthrough
9898

9999
Steps 1-3 are exactly the same as with Leader Election.
100-
4. In the [Kubernetes Provider struct](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L131) created by the Kubernetes AutodiscoverBuilder in case there is a [condition](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L116) a `NewConfigMapper` is created. It contains the condition [map](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/template/config.go#L64) with all the conditions of a given config.
101-
4. When unique is set to false, the resource is [checked](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L241) from the configuration template(default is pod) and a new PodEventer/NodeEventer/ServiceEventer is set as the eventManager of Kubernetes Provider.
100+
4. In the [Kubernetes Provider struct](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L131) created by the Kubernetes AutodiscoverBuilder in case there is a [condition](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L116) a `NewConfigMapper` is created. It contains the condition [map](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/template/config.go#L64) with all the conditions of a given config.
101+
4. When unique is set to false, the resource is [checked](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L239) from the configuration template(default is pod) and a new PodEventer/NodeEventer/ServiceEventer is set as the eventManager of Kubernetes Provider.
102102
5. The kubernetes node that the metricbeat instance is running on is [discovered](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/pod.go#L72).
103-
6. A dedicated watcher to watch [pods](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/pod.go#L82),nodes and namespaces get started. Also a [metadata generator](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/pod.go#L113) is started to enrich the events with kubernetes metadata.
103+
6. A dedicated watcher to watch [pods](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/pod.go#L82), nodes and namespaces get started. Also a [metadata generator](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/pod.go#L113) is started to enrich the events with kubernetes metadata.
104104
7. Each time a new event is about to be published by the watchers(we will have a dedicated section of how events are published), it is checked whether the condition set by the user [matches](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L182) the event.
105105
8. If it matches, then a config is created with the configuration set by user and it is added in the event. After that the event gets published.
106106
9. The worker listening for events receives the start event and checks if there is any update in the configs of the event. (same process with leader election steps 6-9)
@@ -177,12 +177,12 @@ spec:
177177

178178
Everything works the same as Autodiscover without LeaderElection until step 8.
179179

180-
8. If there is no conditions in the template set by the user, the configs will be generated from [hints](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L186).
181-
9. Wether hints are enabled or not is part of the [Kubernetes Provider struct](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L121) builders field.
180+
8. If there is no conditions in the template set by the user, the configs will be generated from [hints](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L186).
181+
9. Wether hints are enabled or not is part of the [Kubernetes Provider struct](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L121) builders field.
182182
10. [GenerateHints](https://github.com/elastic/beats/blob/eff92354db783001880f4bade9f59942fca747ba/libbeat/autodiscover/builder/helper.go#L213) function looks into the event's annotations. A [hints map](https://github.com/elastic/beats/blob/eff92354db783001880f4bade9f59942fca747ba/libbeat/autodiscover/builder/helper.go#L226) is created with all hints and returned.
183-
11. From those hints, configs are [created](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/builder.go#L97) in the same form as in `Autodiscover without LeaderElection` step 8.
183+
11. From those hints, configs are [created](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/builder.go#L97) in the same form as in `Autodiscover without LeaderElection` step 8.
184184
They contain the same information as if they where set explicitly in the metricbeat configureation but actually derive from the pod annotations.
185-
12. Those configs are then [added](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L197) in the event and gets published.
185+
12. Those configs are then [added](https://github.com/elastic/beats/blob/4b1f69923b3f2abbbf1860295fe5dbff7db3d63c/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L197) in the event and gets published.
186186
13. The process after that is same as in `Autodiscover without LeaderElection` step 9 and onward.
187187

188188

0 commit comments

Comments
 (0)