Gather apiserver latency metrics to be able to detect performance regression

[dgrisonnet]

Gather the 99th percentile and average of requests latency to be able to follow the trend of apiserver latency across all CI jobs. With this high-level information, we will be able to create better thresholds for openshift/origin#26739 which will allow us to prevent PR from being merged if they have an impact too important on the apiserver latency.

This additional information will also allow us to investigate whether openshift/kubernetes#1120 resolves this blocker bug https://bugzilla.redhat.com/show_bug.cgi?id=2039539.

/cc @deads2k @tkashem

[deads2k]

/approve

[dgrisonnet]

/retest-required

[openshift-ci]

@dgrisonnet: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/openshift/cluster-kube-controller-manager-operator/release-4.11/e2e-aws-ccm	51bb9f0	link	unknown	/test pj-rehearse
ci/rehearse/openshift/windows-machine-config-operator/release-4.11/azure-e2e-ccm-install	1f455654423bdd2a494fa9733cdd76fdb3233195	link	unknown	/test pj-rehearse
ci/rehearse/openshift/windows-machine-config-operator/release-4.9/azure-e2e-operator	1f455654423bdd2a494fa9733cdd76fdb3233195	link	unknown	/test pj-rehearse
ci/rehearse/openshift/installer/release-4.9/e2e-azure-shared-vpc	51bb9f0	link	unknown	/test pj-rehearse
ci/rehearse/openshift/installer/release-4.9/e2e-azure-resourcegroup	51bb9f0	link	unknown	/test pj-rehearse
ci/rehearse/openshift/windows-machine-config-operator/release-4.11/aws-e2e-ccm-install	1f455654423bdd2a494fa9733cdd76fdb3233195	link	unknown	/test pj-rehearse
ci/rehearse/openshift/installer/release-4.11/e2e-azure-upi	51bb9f0	link	unknown	/test pj-rehearse
ci/rehearse/openshift/windows-machine-config-operator/release-4.9/platform-none-vsphere-e2e-operator	1f455654423bdd2a494fa9733cdd76fdb3233195	link	unknown	/test pj-rehearse
ci/rehearse/openshift/installer/release-4.11/e2e-azurestack	51bb9f0	link	unknown	/test pj-rehearse
ci/rehearse/openshift/windows-machine-config-operator/release-4.9/vsphere-e2e-operator	1f455654423bdd2a494fa9733cdd76fdb3233195	link	unknown	/test pj-rehearse
ci/prow/pj-rehearse	51bb9f0	link	false	/test pj-rehearse
ci/rehearse/openshift/ovirt-csi-driver/release-4.7/e2e-ovirt	51bb9f0	link	unknown	/test pj-rehearse
ci/rehearse/openshift/assisted-service/release-ocm-2.4/e2e-metal-assisted-operator-disconnected	1f455654423bdd2a494fa9733cdd76fdb3233195	link	unknown	/test pj-rehearse

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[tkashem]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, dgrisonnet, tkashem

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/step-registry/gather/OWNERS [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@dgrisonnet: Updated the step-registry configmap in namespace ci at cluster app.ci using the following files:

• key gather-extra-commands.sh using file ci-operator/step-registry/gather/extra/gather-extra-commands.sh

Details

In response to this:

Gather the 99th percentile and average of requests latency to be able to follow the trend of apiserver latency across all CI jobs. With this high-level information, we will be able to create better thresholds for openshift/origin#26739 which will allow us to prevent PR from being merged if they have an impact too important on the apiserver latency.

This additional information will also allow us to investigate whether openshift/kubernetes#1120 resolves this blocker bug https://bugzilla.redhat.com/show_bug.cgi?id=2039539.

/cc @deads2k @tkashem

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.