Report the volume of etcd writes via a diagnostic

[smarterclayton]

New EtcdWriteVolume diagnostic measures the number of writes in a time
period to determine where significant write volume is going.

[test]

@derekwaynecarr @eparis

Will make debugging this easier next time:

$ ETCD_WRITE_VOLUME_DURATION=10s oadm diagnostics EtcdWriteVolume --master-config=openshift.local.config/master/       master-config.yaml
[Note] Determining if client configuration exists for client/cluster diagnostics
debug: Reading client config at /Users/clayton/projects/origin/src/github.com/openshift/origin/openshift.local.        config/master/admin.kubeconfig
Info:  Successfully read a client config file at '/Users/clayton/projects/origin/src/github.com/openshift/origin/      openshift.local.config/master/admin.kubeconfig'

[Note] Running diagnostic: EtcdWriteVolume
       Description: Check the volume of writes against etcd and classify them by operation and key for 10s

Info:  Measured 0.2 writes/sec
       /                                                                          2 100.0%
       /v3:PUT                                                                    2 100.0%
       /v3:PUT/kubernetes.io                                                      2 100.0%
       /v3:PUT/kubernetes.io/events                                               1  50.0%
       /v3:PUT/kubernetes.io/events/default                                       1  50.0%
       /v3:PUT/kubernetes.io/events/default/datadir-mysql-0.14c770c1577b8f64      1  50.0%
       /v3:PUT/kubernetes.io/masterleases                                         1  50.0%
       /v3:PUT/kubernetes.io/masterleases/10.192.209.221                          1  50.0%

[Note] Summary of diagnostics execution (version v3.6.0-alpha.2+021fabc-135-dirty):
[Note] Completed with no errors or warnings seen.

[smarterclayton]

@sosiouxme i added the "default skip" behavior I asked about.

[eparis]

My version worked just fine, geez!
timeout 5m etcdctl --endpoints=https://ip-1-2-3-4.ec2.internal:2379,https://ip-1-2-3-4.ec2.internal:2379,https://ip-1-2-3-5.ec2.internal:2379 --ca-file=/etc/origin/master/master.etcd-ca.crt --cert-file=/etc/origin/master/master.etcd-client.crt --key-file=/etc/origin/master/master.etcd-client.key watch -r -f / | grep '.' | grep -v -i '"kind":"event"' | sort | uniq -c | sort -n

What makes yours so great?

[smarterclayton]

It's Eric in a box and I don't have to explain it

[eparis]

I do actually dislike inscrutable ENV vars. It doesn't look easy to make the timer a flag, but if you can find a way, it would make it a WHOLE lot more discoverable.

[smarterclayton]

Unfortunately we don't have any infrastructure for that today in diagnostics. I went back and forth - the reason i went with env var is that 90% of the time the default is enough. However, if you wanted to get a shorter run (test cases) or longer run (less bursty environments) then you'd have to recompile. So env var is more for the skilled user in that case. I agree that it's not ideal.

[smarterclayton]

Any other comments? I would like to have this tool available, and agree adding args in the future to this would be good.

[eparis]

[merge]
[test]

[openshift-bot]

continuous-integration/openshift-jenkins/merge Waiting: You are in the build queue at position: 18

[openshift-bot]

Evaluated for origin merge up to f74b38b

[smarterclayton]

[severity:bug]

[openshift-bot]

Evaluated for origin test up to 68e1ede

[openshift-bot]

continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/2249/) (Base Commit: 76c0850)