Skip to content

Add a new baseline performance test #25562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ironcladlou wants to merge 1 commit into from
Closed

Add a new baseline performance test #25562

ironcladlou wants to merge 1 commit into from

Conversation

@ironcladlou
Copy link
Contributor

@ironcladlou ironcladlou commented Sep 28, 2020

Introduce a new baseline performance test which does nothing but sleep for 15
minutes. This test is intended to be executed by itself (or in addition to other
such "passive" tests) against a new cluster.

Periodic data collected in this manner will hopefully be more reliable as
control data for regression analysis and to detect things like slow leaks
without confounding factors introduced by workloads and other invasive testing.

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 28, 2020
@ironcladlou
Copy link
Contributor Author

ironcladlou commented Sep 28, 2020

I'm not sure why there are so many diffs in the annotation generation. I ran:

$ go version
go version go1.14.4 darwin/amd64

hack/update-generated.sh

wking
wking reviewed Sep 28, 2020
View reviewed changes
@ironcladlou
Copy link
Contributor Author

ironcladlou commented Sep 28, 2020

This is now the only test in the existing openshift/scalability suite, so it didn't seem worth making a dedicated suite at the moment. I'm not sure what the bar would be for introducing a new one. This might be a "performance" test but also a "passive" one. Not sure where it fits.

@smarterclayton
Copy link
Contributor

smarterclayton commented Sep 28, 2020

Why is verify breaking? @marun in case this is known fallout from various o/k separation changes

@marun
Copy link
Contributor

marun commented Sep 28, 2020 •
edited
Loading

Why is verify breaking? @marun in case this is known fallout from various o/k separation changes

@smarterclayton The verify failure looks legit. make update computes the latest state of the annotations, and locally for me and in CI those annotations include the [Suite:openshift] tag that the current revision of this PR proposes to strip.

@ironcladlou
Copy link
Contributor Author

ironcladlou commented Sep 28, 2020

@smarterclayton @marun

$ git clone https://github.com/openshift/origin.git                                                                                                                                                             Cloning into 'origin'...
remote: Enumerating objects: 5791, done.
remote: Counting objects: 100% (5791/5791), done.
remote: Compressing objects: 100% (4027/4027), done.
remote: Total 655804 (delta 3417), reused 1769 (delta 1764), pack-reused 650013
Receiving objects: 100% (655804/655804), 1.09 GiB | 15.64 MiB/s, done.
Resolving deltas: 100% (404418/404418), done.
Checking out files: 100% (14558/14558), done.

$ go version
go version go1.14.4 darwin/amd64

$ hack/update-generated.sh
hack/update-generated.sh took 117 seconds

$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   test/extended/util/annotate/generated/zz_generated.annotations.go

And there are all the same unexpected diffs you see here.

@marun
Copy link
Contributor

marun commented Sep 28, 2020

@smarterclayton @marun

$ git clone https://github.com/openshift/origin.git                                                                                                                                                             Cloning into 'origin'...
remote: Enumerating objects: 5791, done.
remote: Counting objects: 100% (5791/5791), done.
remote: Compressing objects: 100% (4027/4027), done.
remote: Total 655804 (delta 3417), reused 1769 (delta 1764), pack-reused 650013
Receiving objects: 100% (655804/655804), 1.09 GiB | 15.64 MiB/s, done.
Resolving deltas: 100% (404418/404418), done.
Checking out files: 100% (14558/14558), done.

$ go version
go version go1.14.4 darwin/amd64

$ hack/update-generated.sh
hack/update-generated.sh took 117 seconds

$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   test/extended/util/annotate/generated/zz_generated.annotations.go

And there are all the same unexpected diffs you see here.

We're on golang 1.15 fwiw. I don't see the behavior you're reporting with 1.15.2.

@ironcladlou
Copy link
Contributor Author

ironcladlou commented Sep 28, 2020

I reproduced just now with go version go1.15.2 darwin/amd64

@ironcladlou
Copy link
Contributor Author

ironcladlou commented Sep 28, 2020

Moving the codegen issue over to #25563

@marun
Copy link
Contributor

marun commented Sep 28, 2020

@ironcladlou tl;dr run make update on a repo in a GOPATH and you should be able to satisfy the verify job.

As per #25565, the problem is that the annotation code only works if the repo is in a GOPATH (i.e. GOPATH/src/github.com/openshift/origin). CI executes in a GOPATH, and my initial test was in a GOPATH too. Only when I cloned outside of a gopath (~/src/origin) was I able to reproduce the symptom.

@ironcladlou ironcladlou force-pushed the steady-test branch 2 times, most recently from cfc2489 to 35800db Compare September 29, 2020 13:10
@ironcladlou
Copy link
Contributor Author

ironcladlou commented Sep 29, 2020

@smarterclayton @marun @wking code gen issues fixed up.

@marun
Copy link
Contributor

marun commented Sep 29, 2020

/lgtm

Unleash the retesting bot!

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 29, 2020
@openshift-ci-robot
Copy link

openshift-ci-robot commented Sep 29, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ironcladlou, marun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-bot
Copy link
Contributor

openshift-bot commented Sep 29, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

8 similar comments
@openshift-bot
Copy link
Contributor

openshift-bot commented Sep 29, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 3, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 3, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 3, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 3, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 3, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 3, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 3, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

11 similar comments
@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

openshift-bot commented Oct 5, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Oct 5, 2020

New changes are detected. LGTM label has been removed.

@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Oct 5, 2020
@ironcladlou
Add a new baseline performance test
89bed26 
Introduce a new baseline performance test which does nothing but sleep for 15
minutes. This test is intended to be executed by itself (or in addition to other
such "passive" tests) against a new cluster.

Periodic data collected in this manner will hopefully be more reliable as
control data for regression analysis and to detect things like slow leaks
without confounding factors introduced by workloads and other invasive testing.
@openshift-ci-robot
Copy link

openshift-ci-robot commented Oct 6, 2020 •
edited
Loading

@ironcladlou: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-gcp 89bed26 link /test e2e-gcp

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ironcladlou
Copy link
Contributor Author

ironcladlou commented Oct 7, 2020

/bugzilla refresh

@openshift-ci-robot
Copy link

openshift-ci-robot commented Oct 7, 2020

@ironcladlou: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ironcladlou
Copy link
Contributor Author

ironcladlou commented Oct 8, 2020

@marun re-tag?

@openshift-merge-robot
Copy link
Contributor

openshift-merge-robot commented Oct 23, 2020

@ironcladlou: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-agnostic-cmd 89bed26 link /test e2e-agnostic-cmd

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-bot
Copy link
Contributor

openshift-bot commented Jan 22, 2021

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci-robot openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 22, 2021
@openshift-bot
Copy link
Contributor

openshift-bot commented Feb 21, 2021

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

@openshift-ci-robot openshift-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 21, 2021
@openshift-bot
Copy link
Contributor

openshift-bot commented Mar 23, 2021

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 23, 2021

@openshift-bot: Closed this PR.

Details

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@deads2k deads2k Awaiting requested review from deads2k

2 more reviewers

@wking wking wking left review comments

@smarterclayton smarterclayton smarterclayton left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@marun marun

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@ironcladlou @smarterclayton @marun @openshift-ci-robot @openshift-bot @openshift-merge-robot @wking