[JENKINS-53958] allow pipeline jobs to run when built-in is offline

[mawinter69]

When a pipeline starts building it creates an OneOffExecutor that takes care of the pipeline execution on the built-in node. The executor has some logic to prevent running things when an agent has gone offline in the timeframe between assigning the task to the executor and the executor actually starting running the task. But this logic falsely leads to a termination of the executor for the pipeline job and the attempts to restart the task fails as the task is no longer in the queue.
This change tries to avoid this by ignoring the online state for the built-in node as it will never be really offline (there is no channel that can be closed). One can take it temporarily offline but this should not prevent pipelines that do not explicitly make use of the built-in to start running.

See JENKINS-53958.

Testing done

Manual testing:

Scenario 1: pipeline not using built-in

1. take the built-in node offline
2. create a second agent
3. Create pipeline job that will run something on the second agent
4. run the pipeline -> pipeline run succeeds

Scenario 2: pipeline explicitly using built-in

1. take the built-in node offline
2. create a second agent
3. Create pipeline job that will run something on the built-in
4. run the pipeline -> pipeline run waits for the built-in node

Proposed changelog entries

• Allow pipeline jobs to run when built-in is offline.

Proposed upgrade guidelines

N/A

Submitter checklist

Beta Give feedback

1. The Jira issue, if it exists, is well-described.
2. The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see examples). Fill in the Proposed upgrade guidelines section only if there are breaking changes or changes that may require extra steps from users during upgrade.
3. There is automated testing or an explanation as to why this change has no tests.
4. New public classes, fields, and methods are annotated with @Restricted or have @since TODO Javadocs, as appropriate.
5. New deprecations are annotated with @Deprecated(since = "TODO") or @Deprecated(forRemoval = true, since = "TODO"), if applicable.
6. New or substantially changed JavaScript is not defined inline and does not call eval to ease future introduction of Content Security Policy (CSP) directives (see documentation).
7. For dependency updates, there are links to external changelogs and, if possible, full differentials.
8. For new APIs and extension points, there is a link to at least one consumer.

Desired reviewers

@mention

Before the changes are marked as ready-for-merge:

Maintainer checklist

Beta Give feedback

1. There are at least two (2) approvals for the pull request and no outstanding requests for change.
2. Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
3. Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
4. Proper changelog labels are set so that the changelog can be generated automatically.
5. If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
6. If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).

[StefanSpieker]

In general, I like the idea, but in the past the current behavior also helped us when the controller disk space ran out and the built-in node was taken offline automatically. We do not run jobs on the built-in node, but since it is needed for starting jobs, it prevented further jobs to run.
We have a better monitoring today, so we hadn't had this issue for quite some time, but it might be worth considering.

[mawinter69]

I think that when a node is offline it should only affect explicit usage of the node i.e. because it is matched by a label expression in a node step or due to a job restriction.
The situation as it is now is bad as pipelines jobs just fail as if they were never triggered, only in the logs is some message where it is not instantly clear what happened. Better have the job fail with disk problems I would say.
Maybe one should check before starting the flyweighttask if the node is online and if not then leave it in the queue. But that is a bigger change and requires thorough testing.

[timja]

/label ready-for-merge

---

This PR is now ready for merge, after ~24 hours, we will merge it if there's no negative feedback.

Thanks!

[alexsch01]

I just updated to a version of Jenkins that now has this behavior. Ughh.

Is there a config option to revert this?

[MarkEWaite]

I just updated to a version of Jenkins that now has this behavior. Ughh.

Is there a config option to revert this?

As far as I know there is not a configuration option to revert this change. Can you provide a detailed description of the problem you are seeing as a result of this change so that it can be evaluated?

The Jenkins issue tracker is the best place to provide that detailed description

[alexsch01]

@MarkEWaite
I used the Disable in built node button to prevent all schedules from running
Now that doesn't work
Is there another way of achieving that?

[daniel-beck]

This is not the place to have that discussion. Please file an issue in Jira or open a topic in the forum.

[alexsch01]

my apologies, I made issue https://issues.jenkins.io/browse/JENKINS-73866