Kubernetes annotator #3888

vjsamuel · 2017-04-03T08:28:41Z

This PR adds support for Kubernetes as a processor for Beats. The Kubernetes processor allows Beats to enrich events with metadata coming from the Kubernetes Pod from which the event originated. Metadata might include:

pod name
pod namespace
container name

The Kubernetes processor relies on two constructs:

Indexers - used to generate metadata from Pods and store them with unique keys. Example of keys might be container IDs, IP:Port combinations, PodName.
Matchers - used to generate a lookup key from an event. Example of a matcher might be a log_path matcher that looks at the source field of an event, extract the container ID from the log path and use it as a lookup key to retrieve metadata about the Pod from which the log message originated.

elasticmachine · 2017-04-03T08:32:03Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

elasticmachine · 2017-04-03T08:32:08Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

exekias

Awesome contribution @vjsamuel, thank you!! I have left some questions and comments, I'll try to test this in a while

exekias · 2017-04-03T09:17:48Z

libbeat/processors/annotate/kubernetes/kubernetes.go

+}
+
+func init() {
+	processors.RegisterPlugin("annotate.kubernetes",


I think it would be better to use just kubernetes instead of annotate.kubernetes, until now we have left the processor type as something implicit

exekias · 2017-04-03T09:22:05Z

libbeat/processors/annotate/kubernetes/podwatcher.go

+			commonMeta := common.MapStr{
+				"pod":       name,
+				"namespace": pod.Namespace,
+				"labels":    pod.Labels,


We need to define these fields in _meta folder, check https://github.com/elastic/beats/blob/master/libbeat/processors/add_cloud_metadata/_meta/fields.yml as an example

exekias · 2017-04-03T09:24:08Z

libbeat/processors/annotate/kubernetes/podwatcher.go

+
+		//Pod is in terminating state, remove the metadata
+		if pod.DeletionTimestamp != nil {
+			p.removePodMapEntry(name, endpoints, containers)


I'm wondering, what happens when you delete the pod here but some logs are still being processed?

typically, log flow/metric scraping stops after the pod is marked for terminated. we could also move this to another method so that we delete it after the container is completely destroyed.

exekias · 2017-04-03T09:27:42Z

libbeat/processors/annotate/kubernetes/podwatcher.go

+	default:
+		pod, ok := podObj.(*corev1.Pod)
+		if ok {
+			logp.Info("Processing %s:%s", pod.Namespace, pod.Name)


Maybe this should be Debug?

exekias · 2017-04-03T09:52:38Z

libbeat/processors/annotate/kubernetes/podwatcher.go

+
+			p.h.Reset()
+			fmt.Fprint(p.h, commonMeta)
+			commonMeta["_hash"] = p.h.Sum64()


What are you using this hash for?

exekias · 2017-04-03T10:03:50Z

libbeat/processors/annotate/kubernetes/kubernetes.go

+			return event, nil
+		}
+
+	} else if value, ok := event["pod"]; ok {


I guess this comes from a custom beat or module?

we leave this check to make sure that any beat that writes an event with a key pod, we can still annotate it with the metadata.

ruflin · 2017-04-03T14:25:06Z

libbeat/processors/annotate/kubernetes/kubernetes.go

+)
+
+const (
+	timeout = time.Second * 5


This should probably be configurable.

yes. agreed.

ruflin · 2017-04-03T14:29:43Z

libbeat/processors/annotate/kubernetes/kubernetes.go

+
+		delete(event, "pod")
+	} else {
+		return event, nil


Should we log an error here? Because that means someone configured the kubernetes processor but no field was found to match and no enhancement happened.

i could have a filebeat configuration, that tails /var/log/* which is not managed by kubernetes and /var/lib/docker/containers/*/*log which is managed by kubernetes. the processor can be global and it would not pass through this check when a message from /var/log come in.

That speaks for having local processors.

If you don't want to see the error with the global config, the processor checks could be used to exclude some files from processing.

ruflin · 2017-04-03T14:31:32Z

libbeat/processors/annotate/kubernetes/kubernetes.go

+func (k kubernetesAnnotator) Run(event common.MapStr) (common.MapStr, error) {
+
+	metadata := common.MapStr{}
+	if value, ok := event["source"]; ok {


I'm not sure if this is the right place for this to happen. I would suggest in a first version to make it configurable in the processor config with format strings on which fields should be picked / combined for the lookup.

In a second version we should definitively have some auto detection for example based on the beat type or where it could be passed as a option to Run.

i like the idea of format strings. didnt think about it.

my use case is that I have a source file that has a container id and i need to extract it out. This is not supported by format strings today.

ruflin · 2017-04-03T14:37:23Z

libbeat/processors/annotate/kubernetes/podwatcher.go

+	"github.com/elastic/beats/libbeat/logp"
+	"k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/fields"
+	"k8s.io/apimachinery/pkg/runtime"


I'm curious how many of these imports we will actually need or are empty interfaces we could also have locally. The reason I bring this up because on the docker side we have currently too many dependencies because of this and in the end all we do is reading stats which is only a very small subset. If this also applies to kubernetes, perhaps we can remove most of the dependencies here with our own client or having some empty interfaces in our code.

I think a custom client is not an option here, as this one is more advanced than a stateless REST consumer. Check SharedIndexInformer and Watch mechanism, the client code creates a copy of the state by watching k8s API

@ruflin , im in agreement with @exekias, the kube client is a quite advanced client implementation. its not worth the effort to write our own client. we can have these vendored local to the kubernetes plugin.

vjsamuel · 2017-04-03T22:45:58Z

One issue that I am facing is that logp and glog both use v as the verbosity flag. logp uses it at -v and glog uses it as --v=<verbosity_level (1-10)>. this causes:

panic: ./filebeat flag redefined: v

goroutine 1 [running]:
panic(0x5295c20, 0xc4202ddad0)
	/usr/local/Cellar/go/1.7.4_2/libexec/src/runtime/panic.go:500 +0x1a1
flag.(*FlagSet).Var(0xc42009e000, 0x5f10ea0, 0x5f7f3e8, 0x552b627, 0x1, 0x55465c0, 0x14)
	/usr/local/Cellar/go/1.7.4_2/libexec/src/flag/flag.go:791 +0x43e
flag.Var(0x5f10ea0, 0x5f7f3e8, 0x552b627, 0x1, 0x55465c0, 0x14)
	/usr/local/Cellar/go/1.7.4_2/libexec/src/flag/flag.go:806 +0x69
github.com/golang/glog.init.1()
	/Users/vija/workspace/go/src/github.com/golang/glog/glog.go:401 +0xe5
github.com/golang/glog.init()
	/Users/vija/workspace/go/src/github.com/golang/glog/glog_file.go:125 +0x1c1
k8s.io/apimachinery/pkg/labels.init()
	/Users/vija/workspace/go/src/k8s.io/apimachinery/pkg/labels/selector.go:837 +0x5d
k8s.io/apimachinery/pkg/apis/meta/v1.init()

cc: @urso @ruflin

ruflin · 2017-04-05T12:36:33Z

I remember we had in the past our own FlagSet to prevent overlapping but I think we removed it as it was no issue anymore and made things simpler for having flags in different packages. Perhaps time to reintroduce it? I assume there isn't a trick in golang to disable some of the flags from library on compile time?

exekias · 2017-04-10T13:55:37Z

for the record, after changing to a different k8s client the flags issue is gone. We changed to a more lightweight client to avoid increasing filebeat binary size

ruflin · 2017-04-10T14:39:18Z

NOTICE

+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------
+golang.org/x/text


We have this already in the global vendor directory. Not sure if we can tell govendor somehow about this that it only adds the packages which are not there. Or add the missing packages there? @vjsamuel any idea?

BTW: Same is true for other packages and we have the same issue in the docker module.

@ruflin, if there is a version mismatch between $BEATS_ROOT/vendor and $MODULE/vendor then there would be two copies. if we can upgrade the root vendor to the more recent version this would ideally go away

Ah, that's cool, I personally miss some more verbosity from govendor

@vjsamuel @exekias Could we "sync" the two to have the same commit? You can use @hash in govendor to pick the exact same package. For now I would just pick the same as in the top directory as we just updated to master very recently.

i have moved the dependency to root vendor so now they share the same commit.

@vjsamuel That means now all dependencies are on the top level? Not sure I like that the kubernetes dependencies are on the top level as they are module specific. Any chance to have a mix? Means the common ones on the top level and all the other dependencies in here?

vjsamuel · 2017-04-10T14:57:41Z

libbeat/processors/annotate/kubernetes/vendor/vendor.json

+	"comment": "",
+	"ignore": "test github.com/elastic/beats",
+	"package": [
+		{


appengine can be added to ignore

ruflin · 2017-04-12T06:25:33Z

NOTICE

+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------
+golang.org/x/text


@vjsamuel @exekias Could we "sync" the two to have the same commit? You can use @hash in govendor to pick the exact same package. For now I would just pick the same as in the top directory as we just updated to master very recently.

ruflin · 2017-04-12T06:26:42Z

libbeat/processors/annotate/kubernetes/indexing.go

+	for k, v := range pod.Metadata.Labels {
+		labelMap[k] = v
+	}
+	return common.MapStr{


Under which namespace will this end up?

We still have the namespacing discussion open, I think we should put everything under the same kubernetes namespace but I was waiting to see the rest of modules to know if it makes sense

ruflin · 2017-04-12T06:28:34Z

libbeat/processors/annotate/kubernetes/indexing.go

+
+func newLogsPathMatcher(cfg common.Config) Matcher {
+	config := struct {
+		LogsPath string `config:"logs_path"`


Assuming this is in filebeat, that would mean the filebeat processors have an additional config option?

ruflin · 2017-04-12T06:30:44Z

libbeat/processors/annotate/kubernetes/kubernetes.go

+}
+
+func newKubernetesAnnotator(cfg common.Config) (processors.Processor, error) {
+	config := kubeAnnotatorConfig{


we normally have the defaults as a var external to the method. Something called defaultKubeAnnotatorConfig in a config.go file. Check some other config.go files for an example. This makes it easy to find the config options.

ruflin · 2017-04-12T06:32:30Z

libbeat/processors/annotate/kubernetes/podwatcher.go

+)
+
+const (
+	podCheckPeriod = 20 * time.Second


Couldn't find where this is used?

its unused. i will remove this one.

ruflin · 2017-04-12T06:34:09Z

libbeat/processors/annotate/kubernetes/podwatcher.go

+	}
+}
+
+func (p *PodWatcher) deletePod(pod *corev1.Pod) {


I would call this remove as the Pod will still exist (hopefully) and it's only removed from the list.

this is the naming convention that kube controllers have for various events that are processed which are:

onAdd onUpdate onDelete

we could rename it to onPodAdd, onPodUpdate and onPodDelete. does that work @ruflin ?

I like the onPodDelete but as it is a private method naming is not too important. I just stumbled over it when looking at the code.

ruflin

Right now we have no tests for this processor. Would it be possible to test at least parts of the processor without predefined data structures to not have to start a kubernetes environment.

There are a few new config options. Could we add to the PR docs and example of how the config will look like and also the end output event? I would also be happy to have this in doc.go in the package itself.

vjsamuel · 2017-04-17T07:15:34Z

@ruflin, i have moved the config to a dedicated config.go file so that the global configs are easier to keep track of.

vjsamuel · 2017-04-17T08:41:22Z

@ruflin, basic test cases and processor documentation have been added as well.

exekias · 2017-04-17T08:53:14Z

libbeat/docs/processors-config.asciidoc

+    in_cluster: true
+-------------------------------------------------------------------------------
+
+The configuration below enables the processor on a beat running as a process on the kubernetes node


This would also work outside the kubernetes node, I would just say "outside the cluster"

outside the node is useless for logs as the filebeat needs to sit inside the same node.

I was thinking also in the metricbeat case, but works for me, it's a minor thing anyway

Again, capitalize Beat and Kubernetes.

exekias · 2017-04-17T08:56:45Z

libbeat/processors/annotate/kubernetes/config.go

+	return kubeAnnotatorConfig{
+		InCluster:             true,
+		SyncPeriod:            1 * time.Second,
+		Namespace:             "kube-system",


Perhaps we should suggest default? I guess entry level users are not aware of kube-system and may miss beats when listing pods?

i wouldnt recommend that. default namespace is only for control plane components. kube-system or monitoring are recommended namespaces.

exekias · 2017-04-17T09:09:21Z

libbeat/processors/annotate/kubernetes/indexing.go

+
+// AddIndexer to the register
+func (r *Register) AddIndexer(name string, indexer IndexConstructor) {
+	r.indexers[name] = indexer


I'm missing all RWMutex locking logic in getters & setters, could you please add it? Like in https://github.com/elastic/beats/pull/3888/files#diff-2afe85a18185c27752efda431319dac3R120

exekias · 2017-04-17T09:13:30Z

filebeat/processors/annotate/kubernetes/indexing.go

+	//Add a log path matcher which can extract container ID from the "source" field.
+	logsPathMatcher, err := newLogsPathMatcher(*cfg)
+	if err == nil {
+		kubernetes.Indexing.AddDefaultMatcher(logsPathMatcher)


Why not just adding the registered matcher/indexer name here, instead of registering an instance? like AddDefaultIndexer("logs_path")

This would force us to register it before marking it as default

ruflin

Left some more high level comments. Will have a detailed look again tomorrow morning. In general LGTM. Lets make sure to get it in rather soonish and then iterate on top of it.

@exekias Could you already start a follow up github meta issue to track open points?

ruflin · 2017-04-17T17:42:09Z

filebeat/beater/filebeat.go

@@ -17,6 +17,9 @@ import (
 	"github.com/elastic/beats/filebeat/publisher"
 	"github.com/elastic/beats/filebeat/registrar"
 	"github.com/elastic/beats/filebeat/spooler"
+
+	//Add filebeat level processors
+	_ "github.com/elastic/beats/filebeat/processors/annotate/kubernetes"


I like that approach

ruflin · 2017-04-17T17:45:05Z

filebeat/processors/annotate/kubernetes/indexing.go

@@ -0,0 +1,91 @@
+package kubernetes


We have a little bit a mess with package names singular/plural. We started all new packages to be singular (like module) when it contains multiple packages with each package one module. processors initially contained all processors in one package I think, that is why it was called processors and not processor. It should be also changed in libbeat later but in filebeat we should directly start with processor.

ruflin · 2017-04-17T17:47:32Z

libbeat/docs/processors-config.asciidoc

+ `logs_path` matcher that takes the `source` field, extracts the container ID and uses it to retrieve
+ metadata.
+
+ The configuration below enables the processor when the beat as a pod in kubernetes.


"when the beat as a pod" ? Is a word missing here?

A few minor changes to capitalization, word choice, and commas: "For example, Filebeat defines the container indexer, which indexes pod metadata based on all container IDs, and a logs_path matcher, which takes the source field, extracts the container ID, and uses it to retrieve metadata.

ruflin · 2017-04-17T17:49:09Z

libbeat/docs/processors-config.asciidoc

+    in_cluster: false
+    host: <hostname>
+    kube_config: ~/.kube/config
+    enable_default_indexers: false


I would turn this around to default_indexers.enabled to be consistend with enabled in other places. With ucfg the enabled comes for "free" as it is already part of the config option.

Same below for default_matches.enabled.

ruflin · 2017-04-17T17:49:43Z

libbeat/docs/processors-config.asciidoc

+    matchers:
+      - fields:
+          lookup_fields: ["metricset.host"]
+-------------------------------------------------------------------------------


I miss a note on how where the enriched data for the event will be put. How will the end event look like?

ruflin · 2017-04-17T17:52:09Z

metricbeat/beater/metricbeat.go

@@ -12,6 +12,9 @@ import (

 	"github.com/elastic/beats/libbeat/cfgfile"
 	"github.com/pkg/errors"
+
+	//Add metricbeat specific processors
+	_ "github.com/elastic/beats/metricbeat/processors/annotate/kubernetes"


We should auto generate this in the future for all beats based on the convention for the namespace and make it part of the include package.

ruflin · 2017-04-17T17:52:29Z

metricbeat/processors/annotate/kubernetes/indexing.go

@@ -0,0 +1,126 @@
+package kubernetes


For the package name see comment in filebeat.

dedemorton

Some minor edits and questions about areas of the doc that will be potentially confusing to users.

dedemorton · 2017-04-17T19:21:46Z

libbeat/docs/processors-config.asciidoc

+=== kubernetes
+
+The `kubernetes` processor annotates each event with relevant metadata based on
+which kubernetes pod the event originated from. Each event is annotated with:


Capitalize Kubernetes whenever it's used as a product name.

dedemorton · 2017-04-17T19:23:36Z

libbeat/docs/processors-config.asciidoc

+the `ip_port` indexer can take a kubernetes pod and index the pod metadata based on all
+`pod_ip:container_port` combinations.
+
+Matchers are used to build lookup keys with which indices can be queried. For example, the `fields`


A little hard to follow because of the passive voice. Maybe say, "...build lookup keys for querying indices."

dedemorton · 2017-04-17T19:25:16Z

libbeat/docs/processors-config.asciidoc

+`pod_ip:container_port` combinations.
+
+Matchers are used to build lookup keys with which indices can be queried. For example, the `fields`
+matcher can take a list of fields based on whose values, indices can be queried to retrieve metadata.


I can't follow your meaning here, so you'll need to reword. Might also help to have a specific example. In fact, the concepts here are a bit hard to follow (not sure if it's a problem with the doc or my lack of knowledge regarding Kubernetes). Specific examples somewhere showing how the events are enriched with Kubernetes data would be very helpful.

dedemorton · 2017-04-17T19:25:41Z

libbeat/docs/processors-config.asciidoc

+Matchers are used to build lookup keys with which indices can be queried. For example, the `fields`
+matcher can take a list of fields based on whose values, indices can be queried to retrieve metadata.
+
+Each beat can define its own default indexers and matchers which are enabled by default. For example,


Capitalize Beat and Filebeat.

dedemorton · 2017-04-17T19:31:00Z

libbeat/docs/processors-config.asciidoc

+ `logs_path` matcher that takes the `source` field, extracts the container ID and uses it to retrieve
+ metadata.
+
+ The configuration below enables the processor when the beat as a pod in kubernetes.


A few minor changes to capitalization, word choice, and commas: "For example, Filebeat defines the container indexer, which indexes pod metadata based on all container IDs, and a logs_path matcher, which takes the source field, extracts the container ID, and uses it to retrieve metadata.

dedemorton · 2017-04-17T19:31:46Z

libbeat/docs/processors-config.asciidoc

+[source,yaml]
+-------------------------------------------------------------------------------
+processors:
+- kuberenes:


Should this say kubernetes? This question applies to other places where you use kuberenes instead of kubernetes.

dedemorton · 2017-04-17T19:44:23Z

libbeat/docs/processors-config.asciidoc

+* Namespace
+* Labels
+
+The `kuberentes` process has two basic building blocks which are:


Should this say kubernetes processor instead of process? Note that there is also a typo: kuberentes

dedemorton · 2017-04-17T19:49:58Z

libbeat/docs/processors-config.asciidoc

+Matchers are used to build lookup keys with which indices can be queried. For example, the `fields`
+matcher can take a list of fields based on whose values, indices can be queried to retrieve metadata.
+
+Each beat can define its own default indexers and matchers which are enabled by default. For example,


How do users know which indexers and matchers are enabled for each Beat?

we would need to document individual beats on what indexers/matchers each of them enable by default. having them in libbeat documentation wouldnt be ideal.

dedemorton · 2017-04-17T19:55:51Z

libbeat/docs/processors-config.asciidoc

+    in_cluster: true
+-------------------------------------------------------------------------------
+
+The configuration below enables the processor on a beat running as a process on the kubernetes node


Again, capitalize Beat and Kubernetes.

dedemorton · 2017-04-17T20:01:24Z

libbeat/docs/processors-config.asciidoc

+    kube_config: ~/.kube/config
+-------------------------------------------------------------------------------
+
+The configuration below allows custom indexers and matchers to be enabled.


Does this configuration enable the indexers and matchers, or does it allow you to enable them somewhere else? The passive voice used here makes it sound like the latter.

ruflin

I left a few minor comments but I'm ok with moving forward with the PR as is and address the comments later. Please also check the comment from @dedemorton

Before we merge it, all the commits should be squashed into 1 and the commit message and PR message should be updated.

A few additional things we should do after merging:

Going through the config options again and check the naming.
Discuss again where which part of the indexer / matcher logic should be. I could see that for example the ip:port matcher logic is in libbeat close to the processor, but configuration and activation as default matcher happens in metricbeat. Same for filebeat.
Discuss on how to handle dependencies best, as having it on the global level is not optimal, but also having duplicates is not good.

ruflin · 2017-04-18T05:31:46Z

filebeat/processor/annotate/kubernetes/indexing.go

+		return nil, fmt.Errorf("fail to unpack the `logs_path` configuration: %s", err)
+	}
+
+	logPath := config.LogsPath


Should this be logsPath?

ruflin · 2017-04-18T05:35:48Z

filebeat/processor/annotate/kubernetes/indexing.go

+
+	indexer := kubernetes.Indexing.GetIndexer(kubernetes.ContainerIndexerName)
+	//Add a container indexer by default.
+	if indexer != nil {


If we split the code into 2 func init() functions which I think Golang allows, we could make the code quite a bit nicer. Also it allows to check for err != nil and then return which is more the "standard" golang way.

ruflin · 2017-04-18T05:41:39Z

libbeat/processors/annotate/kubernetes/config.go

+	SyncPeriod      time.Duration `config:"sync_period"`
+	Indexers        PluginConfig  `config:"indexers"`
+	Matchers        PluginConfig  `config:"matchers"`
+	DefaultMatchers Enabled       `config:"default_matchers"`


You could use here config:"default_matchers.enabled" and then just use a bool. No additional type needed.

ruflin · 2017-04-18T05:43:44Z

libbeat/processors/annotate/kubernetes/config.go

+	return kubeAnnotatorConfig{
+		InCluster:       true,
+		SyncPeriod:      1 * time.Second,
+		Namespace:       "kube-system",


I'm trying to understand how this Namespace works. Is that something set in the kubernetes environment? Is there a default when I just set up Kubernetes?

ruflin · 2017-04-18T05:56:54Z

libbeat/processors/annotate/kubernetes/kubernetes.go

+		return nil, fmt.Errorf("fail to unpack the kubernetes configuration: %s", err)
+	}
+
+	err = validate(config)


ucfg checks for a Validate() error interface and calls the method automatically on unpack. See https://github.com/elastic/beats/blob/master/heartbeat/monitors/active/http/config.go#L82 This could be used here (can be done later)

ruflin · 2017-04-18T05:58:11Z

libbeat/processors/annotate/kubernetes/kubernetes.go

+		return nil, err
+	}
+
+	indexers := Indexers{


Do we need the double nesting here? Naming gets kind of interesting like this.

ruflin · 2017-04-18T06:43:39Z

libbeat/processors/annotate/kubernetes/kubernetes.go

+			return kubernetesAnnotator{podWatcher: watcher, matchers: &matchers}, nil
+		}
+
+		return nil, fatalError


this line could be removed because there is the same return in the end.

ruflin · 2017-04-18T06:48:29Z

metricbeat/processor/annotate/kubernetes/indexing.go

+		logp.Err("Unable to get indexer plugin %s", IpPortIndexerName)
+	}
+
+	matcher := kubernetes.Indexing.GetMatcher(kubernetes.FieldMatcherName)


Two init functions would simplify the logic here (see comment in filebeat)

This PR adds support for Kubernetes as a processor for Beats. The Kubernetes processor allows Beats to enrich events with metadata coming from the Kubernetes Pod from which the event originated. Metadata might include: * pod name * pod namespace * container name The Kubernetes processor relies on two constructs: * Indexers - used to generate metadata from Pods and store them with unique keys. Example of keys might be container IDs, IP:Port combinations, PodName. * Matchers - used to generate a lookup key from an event. Example of a matcher might be a log_path matcher that looks at the source field of an event, extract the container ID from the log path and use it as a lookup key to retrieve metadata about the Pod from which the log message originated.

vjsamuel force-pushed the kubernetes_annotator branch from bf4c810 to 1e3d5f4 Compare April 3, 2017 08:30

exekias self-requested a review April 3, 2017 08:40

exekias reviewed Apr 3, 2017

View reviewed changes

ruflin reviewed Apr 3, 2017

View reviewed changes

vjsamuel force-pushed the kubernetes_annotator branch 2 times, most recently from 7bfc4f7 to 84210aa Compare April 7, 2017 15:57

exekias force-pushed the kubernetes_annotator branch 2 times, most recently from f0d45b9 to 020c563 Compare April 10, 2017 13:53

exekias force-pushed the kubernetes_annotator branch 2 times, most recently from 70d7aef to 1c96434 Compare April 10, 2017 14:03

ruflin reviewed Apr 10, 2017

View reviewed changes

vjsamuel commented Apr 10, 2017

View reviewed changes

exekias added in progress Pull request is currently in progress. Metricbeat Metricbeat v6.0.0-alpha1 review labels Apr 11, 2017

ruflin reviewed Apr 12, 2017

View reviewed changes

ruflin reviewed Apr 13, 2017

View reviewed changes

vjsamuel force-pushed the kubernetes_annotator branch from f72a4a1 to 96604da Compare April 17, 2017 07:03

exekias reviewed Apr 17, 2017

View reviewed changes

ruflin reviewed Apr 17, 2017

View reviewed changes

exekias mentioned this pull request Apr 17, 2017

Follow up kubernetes processor #4027

Closed

10 tasks

dedemorton reviewed Apr 17, 2017

View reviewed changes

vjsamuel force-pushed the kubernetes_annotator branch from 20a4e63 to 668e571 Compare April 18, 2017 00:31

ruflin approved these changes Apr 18, 2017

View reviewed changes

vjsamuel force-pushed the kubernetes_annotator branch 2 times, most recently from 21ec33c to f1e4147 Compare April 18, 2017 07:31

vjsamuel force-pushed the kubernetes_annotator branch from f1e4147 to 2b9db99 Compare April 18, 2017 07:53

exekias removed the in progress Pull request is currently in progress. label Apr 18, 2017

exekias merged commit 54d557e into elastic:master Apr 18, 2017

vjsamuel deleted the kubernetes_annotator branch May 2, 2017 15:13

Kubernetes annotator #3888

Kubernetes annotator #3888

Conversation

vjsamuel commented Apr 3, 2017 • edited Loading

elasticmachine commented Apr 3, 2017

elasticmachine commented Apr 3, 2017

exekias left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vjsamuel commented Apr 3, 2017 • edited Loading

ruflin commented Apr 5, 2017

exekias commented Apr 10, 2017

ruflin Apr 10, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

vjsamuel commented Apr 17, 2017

vjsamuel commented Apr 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dedemorton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vjsamuel commented Apr 3, 2017 •

edited

Loading

vjsamuel commented Apr 3, 2017 •

edited

Loading

ruflin Apr 10, 2017 •

edited

Loading