-
-
Notifications
You must be signed in to change notification settings - Fork 9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[JENKINS-73835] Do not allow builds to be deleted while they are still running and ensure build discarders run after builds are fully complete #9810
Changes from all commits
cf30fcc
eec23fd
97fb6aa
9a6a90a
f676be5
9aaa7ba
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||
---|---|---|---|---|
|
@@ -250,7 +250,7 @@ private boolean shouldKeepRun(Run r, Run lsb, Run lstb) { | |||
LOGGER.log(FINER, "{0} is not to be removed or purged of artifacts because it’s the last stable build", r); | ||||
return true; | ||||
} | ||||
if (r.isBuilding()) { | ||||
if (r.isLogUpdated()) { | ||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is to avoid race conditions involving Pipeline builds. Previously it was possible for log rotation to delete builds which had not yet fully completed, which meant that a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are you confident that this is safe for freestyle builds? I.e., that There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am pretty sure it is safe, but if you prefer we can switch to For The converse For Pipeline things are simpler since Also just as a data point, if There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (And as far as I know the states for a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
That looks right.
Not sure what you mean.
building and ought to be switched to only check logUpdated .
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🤦 I misread There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I fixed that in f676be5. |
||||
LOGGER.log(FINER, "{0} is not to be removed or purged of artifacts because it’s still building", r); | ||||
return true; | ||||
} | ||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -35,7 +35,7 @@ | |
import org.kohsuke.accmod.restrictions.NoExternalUse; | ||
|
||
/** | ||
* Run background build discarders on an individual job once a build is finalized | ||
* Run build discarders on an individual job once a build is finalized | ||
*/ | ||
@Extension | ||
@Restricted(NoExternalUse.class) | ||
|
@@ -46,6 +46,15 @@ | |
@Override | ||
public void onFinalized(Run run) { | ||
Job job = run.getParent(); | ||
BackgroundGlobalBuildDiscarder.processJob(new LogTaskListener(LOGGER, Level.FINE), job); | ||
try { | ||
// Job-level build discarder execution is unconditional. | ||
job.logRotate(); | ||
} catch (Exception e) { | ||
LOGGER.log(Level.WARNING, e, () -> "Failed to rotate log for " + run); | ||
} | ||
// Avoid calling Job.logRotate twice in case JobGlobalBuildDiscarderStrategy is configured globally. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a bit confusing, but essentially the old behavior was that Also, the old behavior is why I am not concerned about removing this call in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (jenkinsci/workflow-job-plugin#70 for better linking) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hi, after some digging into the behavior of globalBuildDiscarder I ended here. As a Jenkins admin, I expected the specific (or simple) global discarder to take precedence over any project build discarder, especially if I remove the global project build discarder. This is also what I implied from reading the descriptions here and here (even though it's not explicitly mentioned there and does not contradict the actual behavior). I also found multiple occurrences on SO and forums mentioning that global specific job discarder will take precedence, which is not the case. This is now obvious with the change line 51 here but was already the behavior before as I tested with 2.462.2. What makes it even more confusing, is the way the discarders are merged and applied. The most aggressive policy takes over. This is confirmed to be an expected behavior from the tests. However, from an admin perspective I'd see the least aggressive discarder to be a safer approach (e.g. for compliance reasons). I'm curious however, to get your input on the topic. Also let me know if I should raise this somewhere else. For context, this investigation started as a result of the security eng. telling me that he can erase his traces by overriding build discarder in a job. Example: normal run -> evil run (set buildDiscarder to 1) -> normal run. Then we can't see what happened in the evil run anymore. We also have some compliance policy that require us to keep production pipelines history up to a year which we would like to enforce. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think the current behavior is as designed; build discarders are mainly meant to trim disk space, so it would be normal to have a blanket policy that builds over a month old are not kept, while a particular job that runs every five minutes for some automation is configured to discard all but the last build since history is trash. If your interest is in auditing, you would better use one of the many plugins which stream events or even whole build logs from Jenkins to external systems, which could be configured to only accept “create/append”-type operations and reject attempts to delete anything even if the Jenkins controller were to be compromised somehow. After all, even without any discarders, your white-hat engineer could simply delete the job after one build, along with all of its builds. You could also make the entire controller view-only to regular developers, using various “as code” systems. |
||
BackgroundGlobalBuildDiscarder.processJob(new LogTaskListener(LOGGER, Level.FINE), job, | ||
GlobalBuildDiscarderConfiguration.get().getConfiguredBuildDiscarders().stream() | ||
.filter(s -> !(s instanceof JobGlobalBuildDiscarderStrategy))); | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -32,21 +32,18 @@ | |
import static org.hamcrest.Matchers.equalTo; | ||
import static org.hamcrest.Matchers.hasSize; | ||
import static org.junit.Assert.assertNotNull; | ||
import static org.junit.Assume.assumeFalse; | ||
|
||
import hudson.Functions; | ||
import hudson.model.ExecutorTest; | ||
import hudson.model.FreeStyleProject; | ||
import hudson.model.Item; | ||
import hudson.model.Run; | ||
import hudson.model.labels.LabelAtom; | ||
import hudson.tasks.Shell; | ||
import java.io.IOException; | ||
import jenkins.model.Jenkins; | ||
import org.junit.AssumptionViolatedException; | ||
import org.junit.Before; | ||
import org.junit.Rule; | ||
import org.junit.Test; | ||
import org.jvnet.hudson.test.Issue; | ||
import org.jvnet.hudson.test.JenkinsRule; | ||
|
||
/** | ||
|
@@ -139,24 +136,18 @@ public class DeleteBuildsCommandTest { | |
assertThat(result.stdout(), containsString("Deleted 0 builds")); | ||
} | ||
|
||
@Test public void deleteBuildsShouldSuccessEvenTheBuildIsRunning() throws Exception { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I did not notice this test before. It was added in #2310 along with all of the other tests for I still think the previous behavior was undesirable - deleting builds that are still running and reachable in memory does not make sense, can lead to tons of errors in the Jenkins logs, and any builds deleted while running are very likely to come back after a Jenkins restart given that build completion will cause I can see the argument though that There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No strong opinion. I think the proposed error behavior is intuitive enough. You can explicitly abort the build first if that is what you intended. |
||
assumeFalse("You can't delete files that are in use on Windows", Functions.isWindows()); | ||
@Issue("JENKINS-73835") | ||
@Test public void deleteBuildsShouldFailIfTheBuildIsRunning() throws Exception { | ||
FreeStyleProject project = j.createFreeStyleProject("aProject"); | ||
ExecutorTest.startBlockingBuild(project); | ||
assertThat(((FreeStyleProject) j.jenkins.getItem("aProject")).getBuilds(), hasSize(1)); | ||
|
||
final CLICommandInvoker.Result result = command | ||
.authorizedTo(Jenkins.READ, Item.READ, Run.DELETE) | ||
.invokeWithArgs("aProject", "1"); | ||
assertThat(result, succeeded()); | ||
assertThat(result.stdout(), containsString("Deleted 1 builds")); | ||
assertThat(((FreeStyleProject) j.jenkins.getItem("aProject")).getBuilds(), hasSize(0)); | ||
assertThat(project.isBuilding(), equalTo(false)); | ||
try { | ||
project.delete(); | ||
} catch (IOException | InterruptedException x) { | ||
throw new AssumptionViolatedException("Could not delete test project (race condition?)", x); | ||
} | ||
assertThat(result, failedWith(1)); | ||
assertThat(result, hasNoStandardOutput()); | ||
assertThat(result.stderr(), containsString("Unable to delete aProject #1 because it is still running")); | ||
} | ||
|
||
@Test public void deleteBuildsShouldSuccessEvenTheBuildIsStuckInTheQueue() throws Exception { | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -50,8 +50,10 @@ | |
import java.util.concurrent.TimeoutException; | ||
import java.util.logging.Level; | ||
import java.util.logging.Logger; | ||
import org.junit.ClassRule; | ||
import org.junit.Rule; | ||
import org.junit.Test; | ||
import org.jvnet.hudson.test.BuildWatcher; | ||
import org.jvnet.hudson.test.FailureBuilder; | ||
import org.jvnet.hudson.test.Issue; | ||
import org.jvnet.hudson.test.JenkinsRule; | ||
|
@@ -62,6 +64,9 @@ | |
*/ | ||
public class LogRotatorTest { | ||
|
||
@ClassRule | ||
public static BuildWatcher watcher = new BuildWatcher(); | ||
|
||
@Rule | ||
public JenkinsRule j = new JenkinsRule(); | ||
|
||
|
@@ -96,6 +101,17 @@ public void successVsFailureWithRemoveLastBuild() throws Exception { | |
assertEquals(2, numberOf(project.getLastFailedBuild())); | ||
} | ||
|
||
@Test | ||
public void ableToDeleteCurrentBuild() throws Exception { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm, this was to check that the switch to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess, in principle, someone could want to keep zero builds if they were using publishers (GitHub Checks, Slack, etc.) to track failures? |
||
var p = j.createFreeStyleProject(); | ||
// Keep 0 builds, i.e. immediately delete builds as they complete. | ||
LogRotator logRotator = new LogRotator(-1, 0, -1, -1); | ||
logRotator.setRemoveLastBuild(true); | ||
p.setBuildDiscarder(logRotator); | ||
j.buildAndAssertStatus(Result.SUCCESS, p); | ||
assertNull(p.getBuildByNumber(1)); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Failure in https://github.com/jenkinsci/jenkins/pull/9921/checks?check_run_id=32166988156 (#9921) looks like a flake? Missing There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't know what we would There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. From some testing, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also the |
||
} | ||
|
||
@Test | ||
@Issue("JENKINS-2417") | ||
public void stableVsUnstable() throws Exception { | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I moved this into
GlobalBuildDiscarderListener
so thatisLogUpdated() == true
when log rotation runs. It still runs synchronously, it just runs a little bit later now down inonEndBuilding
. This allows us to checkisLogUpdated()
elsewhere to avoid race conditions with Pipeline builds.