ETCD Metricbeat module needs polishing and grooming

[sayden]

Following comments written here: elastic/beats#10592 ETCD module needs some grooming and polishing to make it follow naming conventions (and probably update some metrics more).

Tasks could be:

• Check naming conventions of all fields: Current fields like etcd.self.recv.pkgrate are hard to read and doesn't show a metric unit (per_sec in this case, if it's a rate that occurs every second). Leaves room for misunderstandings.
• Review of current mapping. It actually uses some dynamic JSON keys in leader metricset as you can see in this event example:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "agent": {
        "hostname": "host.example.com",
        "name": "host.example.com"
    },
    "etcd": {
        "leader": {
            "followers": {
                "5a22bdba1efc5b4a": {
                    "latency": {
                        "average": 0.0024145817307692323,
                        "current": 0.001494,
                        "maximum": 0.061351,
                        "minimum": 0,
                        "standardDeviation": 0.0029017970782575734
                    },
                    "counts": {
                        "success": 1248,
                        "fail": 0
                    }
                },
                "639ec377a30542cf": {
                    "latency": {
                        "average": 0.0026389089456869013,
                        "current": 0.001241,
                        "maximum": 0.233578,
                        "minimum": 0,
                        "standardDeviation": 0.00695758066274549
                    },
                    "counts": {
                        "success": 1252,
                        "fail": 0
                    }
                }
            },
            "leader": "d3cf079af51fa9a8"
        }
    },
    "event": {
        "dataset": "etcd.leader",
        "duration": 115000,
        "module": "etcd"
    },
    "metricset": {
        "name": "leader"
    },
    "service": {
        "address": "127.0.0.1:2379",
        "type": "etcd"
    }
}

As you can imagine, the list of followers may be way too long. In this case, I think that each follower should have its own event so that mapping is consistent, something like this:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "agent": {
        "hostname": "host.example.com",
        "name": "host.example.com"
    },
    "etcd": {
        "follower": {
            "id": "5a22bdba1efc5b4a",
            "latency":{ 
                "ms": 0.001494 
            },
            "success_operations": 1248,
            "failed_operations": 0,
            "leader": "d3cf079af51fa9a8"
    },
    "event": {
        "dataset": "etcd.follower",
        "duration": 115000,
        "module": "etcd"
    },
    "metricset": {
        "name": "follower"
    },
    "service": {
        "address": "127.0.0.1:2379",
        "type": "etcd"
    }
}
{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "agent": {
        "hostname": "host.example.com",
        "name": "host.example.com"
    },
    "etcd": {
        "follower": {
            "id": "639ec377a30542cf",
            "latency":{ 
                "ms": 0.001241
            },
            "success_operations": 1252,
            "failed_operations": 0,
            "leader": "d3cf079af51fa9a8"
    },
    "event": {
        "dataset": "etcd.follower",
        "duration": 115000,
        "module": "etcd"
    },
    "metricset": {
        "name": "follower"
    },
    "service": {
        "address": "127.0.0.1:2379",
        "type": "etcd"
    }
}

[jsoriano]

Agree, we should generate an event per follower.

Related to this, we also need to add support for the current monitoring endpoint and for the v3 store (#9093).

[ruflin]

+1 on having one event per follower. The current event should only contain the ids of the followers. To implement such a change non breaking we could introduce a config option to disable follower data in leader metricset. For the new fields / cleaned up fields, we could introduce them as aliases to then be switched over in the next major.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[sayden]

Mostly fixed here elastic/beats#11280

[gpop63]

The metricset leader should be renamed to follower?

[sayden]

As we talked through the chat, the ideal improvement at this moment is:

• To generate an event for each Follower, either in this same metricset using etcd.leader.follower or in a different one using etcd.follower and...
• To update / upgrade the metrics we collect for the leader, in this metricset.