Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jobs: remove jobs that have been continuously failing for over N days #8861

Closed
spiffxp opened this issue Jul 27, 2018 · 25 comments
Closed

jobs: remove jobs that have been continuously failing for over N days #8861

spiffxp opened this issue Jul 27, 2018 · 25 comments
Assignees
@spiffxp
Labels
area/jobs kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/testing Categorizes an issue or PR as relevant to SIG Testing.

Comments

@spiffxp
Copy link
Member

spiffxp commented Jul 27, 2018

I'm going to suggest N=120 for now. I'd like to do this one more time as a human to learn what the right detection methods are for straight-up-failing vs. seriously flaky, how/when to reach people in case there are some brave souls out there who want to save these jobs, etc.

Next step would be to automate at least some portion of this.

ref: #2528 (comment)
@spiffxp
Copy link
Member Author

spiffxp commented Jul 27, 2018

/milestone 1.12
/assign
/area jobs

@k8s-ci-robot k8s-ci-robot added this to the 1.12 milestone Jul 27, 2018
@BenTheElder
Copy link
Member

This is an excellent issue. 👌
/woof

@k8s-ci-robot
Copy link
Contributor

@BenTheElder: dog image

In response to this:

This is an excellent issue. 👌
/woof

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dims
Copy link
Member

dims commented Jul 30, 2018

remove means delete the job or just disable it? (leave it in the prow config, but disabled)

@BenTheElder
Copy link
Member

I'd say remove from everything and use git history, or create a seperate dir for dead jobs. Trimming the config is also desirable, besides saving on runtime resources, and on testgrid at least it doesn't make sense to keep showing jobs we aren't running.

@spiffxp
Copy link
Member Author

spiffxp commented Jul 31, 2018

Horrible bash + jq to get me the list and links to testgrid
https://gist.github.com/spiffxp/1e3ff608a92e8bfc0091a0b2918a11c6

@spiffxp
Copy link
Member Author

spiffxp commented Sep 28, 2018

/milestone v1.13
/priority important-soon
/sig testing
/kind purge
/kind cleanup

@k8s-ci-robot k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Sep 28, 2018
@k8s-ci-robot k8s-ci-robot modified the milestones: v1.12, v1.13 Sep 28, 2018
@k8s-ci-robot k8s-ci-robot added sig/testing Categorizes an issue or PR as relevant to SIG Testing. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Sep 28, 2018
@BenTheElder
Copy link
Member

/assign
Deleting things is fun right? Let's get this in 1.13 :^) 🔥

@spiffxp
Copy link
Member Author

spiffxp commented Dec 26, 2018

/milestone v1.14

For reals this time. Document and then enforce rules.

@k8s-ci-robot k8s-ci-robot modified the milestones: v1.13, v1.14 Dec 26, 2018
@stevekuznetsov
Copy link
Contributor

The grinch is here!

@BenTheElder BenTheElder removed their assignment Mar 16, 2019
@spiffxp
Copy link
Member Author

spiffxp commented May 7, 2019

/milestone v1.15

@k8s-ci-robot k8s-ci-robot modified the milestones: v1.14, v1.15 May 7, 2019
@spiffxp spiffxp removed this from the v1.15 milestone Jul 13, 2019
@spiffxp
Copy link
Member Author

spiffxp commented Aug 8, 2019

/milestone v1.16
Many of the cos jobs that come from generate_tests may be able to be removed, although there is some question from @tpepper over whether those are legitimate failures

#13353 (comment)

@k8s-ci-robot k8s-ci-robot added this to the v1.16 milestone Aug 8, 2019
@tpepper
Copy link
Member

tpepper commented Aug 8, 2019

Following up, if these can be green, they provide useful signal. Looking at the testgrid cos set of k8sbeta aka 1.15 things in particular in the hope that we might make things more green ahead of merging kubernetes/kubernetes#80134 I see:

  • cos-cos1-k8sbeta (this is 1.15)
  • cos-cos2-k8sbeta (this is 1.15)
    • gke-cos2-k8sbeta-slow
    • gke-cos2-k8sbeta-autoscaling
    • gke-cos2-k8sbeta-serial
    • gke-cos2-k8sbeta-reboot
    • gke-cos2-k8sbeta-updown
    • gke-cos2-k8sbeta-default
    • gke-cos2-k8sbeta-gpu
    • gke-cos2-k8sbeta-flakyA
    • gke-cos2-k8sbeta-alphafeatures
      • ^^^ all showing GCE error: "A Shielded VM Config cannot be set when using a source
        image that is not UEFI-compatible."
    • gce-cos2-k8sbeta-serial
      • "FailedMount: MountVolume.SetUp" timeout: multiple cloud provider
        specific issues in this area of code opened this spring.
    • gce-cos2-k8sbeta-default
    • cos2-k8sbeta-gkespec
      • before suite system validation fails misc kernel CONFIG_ checks
      • (same as cos1-k8sbeta-gkespec above)
    • cos2-k8sbeta-serial
      • misc. timeouts/flakes storage
      • (maybe same as cos1-k8sbeta-serial above?)
    • gce-cos2-k8sbeta-slow
      • flex volume timeouts
      • (same as gce-cos1-k8sbeta-slow above)

There's a relatively limited set of symptoms there...

@BenTheElder BenTheElder mentioned this issue Aug 13, 2019
Closed
@spiffxp spiffxp mentioned this issue Aug 15, 2019
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Aug 15, 2019 •
edited
Loading

I confirmed the cosN jobs aren't really relevant to the community (they are more about testing different versions of an underlying image that is GCP-specific, rather than k8s itself), and there are equivalents running inside of google, so think we should remove them #13913

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 13, 2019
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Dec 13, 2019
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@BenTheElder
Copy link
Member

this probably should still be open?

@spiffxp
Copy link
Member Author

spiffxp commented Jan 23, 2020

/reopen
/remove-lifecycle rotten
/lifecycle frozen

@k8s-ci-robot k8s-ci-robot reopened this Jan 23, 2020
@k8s-ci-robot
Copy link
Contributor

@spiffxp: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. labels Jan 23, 2020
@spiffxp
Copy link
Member Author

spiffxp commented Apr 10, 2020

Well, I managed to remove an N=923 job #17156

@BenTheElder
Copy link
Member

BenTheElder commented Apr 10, 2020 via email

@spiffxp spiffxp removed this from the v1.16 milestone Apr 16, 2020
@spiffxp
Copy link
Member Author

spiffxp commented Aug 21, 2020

/close
closing this in favor of #18600

@k8s-ci-robot
Copy link
Contributor

@spiffxp: Closing this issue.

In response to this:

/close
closing this in favor of #18600

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spiffxp spiffxp

Labels
area/jobs kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/testing Categorizes an issue or PR as relevant to SIG Testing.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@dims @spiffxp @BenTheElder @tpepper @stevekuznetsov @k8s-ci-robot @fejta-bot