[JENKINS-65829] Fix WorkspaceCleanupThread to consider suffixed workspaces even if original is inexistent

[Dohbedoh]

See JENKINS-65829. The WorkspaceCleanupThread was not considering workspaces with suffixes if the original workspace did not exist. Typically @libs workspaces on a controller.

Reworked the WorkspaceCleanupThread to reduce the number of remoting calls by creating a MasterToSlaveFileCallable that do filtering + deletion. There used to be multiple calls to check if directory exists, the last modified date, do a delete recursive, do a delete of the suffixes.

• https://github.com/jenkinsci/jenkins/blob/jenkins-2.451/core/src/main/java/hudson/model/WorkspaceCleanupThread.java#L107 (this was actually a problem because the we can have suffixed workspaces that exist but the workspace actually does not)
• https://github.com/jenkinsci/jenkins/blob/jenkins-2.451/core/src/main/java/hudson/model/WorkspaceCleanupThread.java#L114
• https://github.com/jenkinsci/jenkins/blob/jenkins-2.451/core/src/main/java/hudson/model/WorkspaceCleanupThread.java#L93
• https://github.com/jenkinsci/jenkins/blob/jenkins-2.451/core/src/main/java/hudson/model/WorkspaceCleanupThread.java#L94

Now reduced to only one remoting call.

https://github.com/Dohbedoh/jenkins/blob/JENKINS-65829/core/src/main/java/hudson/model/WorkspaceCleanupThread.java#L102

Note: the retention is check on each directory. Before, we used to check the last modified date of the WORKSPACE directory and if older than 30 day, we would delete it and all suffixed workspace. Now we check the last modified date of each directory. Which makes sense to me. WDYT ?

Testing done

Test 1

• Start Jenkins
• Set up a remote agent
• Create a pipeline that clones a library:

@Library("testLibs") _
node ('test') {
    writeFile(file: "test.txt", text: "test")
}

• Build the job
• Run the following in the script console to execute the thread run:

hudson.model.WorkspaceCleanupThread.retainForDays=0
hudson.model.AsyncPeriodicWork.all().get(WorkspaceCleanupThread.class).run()

--> Validate that the $WORKSPACE@libs on the controller is removed

Test 2

• Following Test 1, create suffixed workspaces in the agent workspace directory $WORKSPACE@tmp, $WORKSPACE@test.
• Edit the pipeline to run on the built-in node - we need to to this because the cleanup thread preserve the workspace of the last build. So we want to test that everything on the agent (previous build) gets cleaned up.
• Run the following in the script console to execute the thread run:

hudson.model.WorkspaceCleanupThread.retainForDays=0
hudson.model.AsyncPeriodicWork.all().get(WorkspaceCleanupThread.class).run()

--> Validate that the $WORKSPACE, $WORKSPACE@libs and $WORKSPACE@tmp on the agent are removed

Proposed changelog entries

• JENKINS-65829, Fix WorkspaceCleanupThread to consider workspaces with suffixes even if the original is inexistent
• Reduce number of remoting calls made by WorkspaceCleanupThread

Proposed upgrade guidelines

N/A

Submitter checklist

Beta Give feedback

1. The Jira issue, if it exists, is well-described.
2. The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see examples). Fill in the Proposed upgrade guidelines section only if there are breaking changes or changes that may require extra steps from users during upgrade.
3. There is automated testing or an explanation as to why this change has no tests.
4. New public classes, fields, and methods are annotated with @Restricted or have @since TODO Javadocs, as appropriate.
5. New deprecations are annotated with @Deprecated(since = "TODO") or @Deprecated(forRemoval = true, since = "TODO"), if applicable.
6. New or substantially changed JavaScript is not defined inline and does not call eval to ease future introduction of Content Security Policy (CSP) directives (see documentation).
7. For dependency updates, there are links to external changelogs and, if possible, full differentials.
8. For new APIs and extension points, there is a link to at least one consumer.

Desired reviewers

Before the changes are marked as ready-for-merge:

Maintainer checklist

Beta Give feedback

1. There are at least two (2) approvals for the pull request and no outstanding requests for change.
2. Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
3. Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
4. Proper changelog labels are set so that the changelog can be generated automatically.
5. If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
6. If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).

[mawinter69]

The workspacecleaner is broken in my eyes, already before and this change doesn't fix the problems. It totally neglects the fact of concurrent build directories. And the approach to check the timestamp of the workspace root is also problematic. The timestamp of a directory only changes when you create or delete files but if you just modify files the timestamp might not change. Also adding removing files in subdirectories will not lead to a change of the timestamp.
Another bug is the check if a job is currently building. It uses job.isBuilding() but that only return true if the last build is running. When I have concurrent builds and my last build failed but the previous is still running it will falsely. The correct approach would probably be to loop over the executors of a node and check if any of them corresponds to the current job.

So worst case might be that you delete a workspace that is currently in use. The risk is not very high though, but I've had some situations where jenkins was not properly releasing the lease of workspaces. The result was that a build was running in a workspace with @2 on an agent that has only 1 executor. In such situations the risk of deleting an inuse workspace will be much higher.

[mawinter69]

This would also match against build directories of concurrent builds.

[mawinter69]

Another problem I see is that the deletion is not atomic (or near atomic). For large workspaces the deletion might take quite some time and in the meantime a new could have started.

[basil]

Checking the last modified date of each directory makes sense to me. Thanks for the PR!

[timja]

Thanks!

---

/label ready-for-merge

---

This PR is now ready for merge, after ~24 hours, we will merge it if there's no negative feedback.

Thanks!