Nodes persistence cleanup, APIs to control loading

[jglick]

When running CloudBees CI in high availability mode we need to control which agents are loaded from disk: blocking some from being loaded at startup, loading them later, reloading them from disk, unloading them from memory while they remain on disk. The four new methods here allow that level of fine-grained control. By default the behavior remains unchanged, of course: the list of agents in memory matches the contents of the nodes directory.

Testing done

Exercised by various functional tests in CloudBees CI.

Proposed changelog entries

• Add experimental APIs to control which agents are loaded and when.

Proposed upgrade guidelines

• N/A

Before the changes are marked as ready-for-merge:

Maintainer checklist

Beta Give feedback

1. There are at least two (2) approvals for the pull request and no outstanding requests for change.
2. Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
3. Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
4. Proper changelog labels are set so that the changelog can be generated automatically.
5. If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
6. If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).

[jglick]

Taking the opportunity to simplify the logic here a bit: #5450 (comment)

[timja]

/label ready-for-merge

---

This PR is now ready for merge, after ~24 hours, we will merge it if there's no negative feedback.

Thanks!

[mattiassluis]

I couldn't find an issue number to comment to hence responding here:
Could it be that a side effect of this change is that renaming a node through Groovy (calling setNodeName) causes duplicate nodes?
We have a groovy init script that randomises node names and after updating to 2.449 it suddenly causes nodes to duplicate after running the script (we are aware that we are using API internals that probably shouldn't be used).

[jglick]

@mattiassluis that seems plausible. Renaming a node remains possible through the GUI so I think you were just using the wrong method all along. Try creating a new Slave object and calling Jenkins.removeNode + Jenkins.addNode, or Jenkins.setNodes. (To reconfigure one agent without renaming, create a new Slave and call updateNode. I am not sure if this is safe to use for a rename as well.)