All highmem jobs are stuck in queue

[NotMyFault]

Service(s)

ci.jenkins.io

Summary

Currently, several ATH jobs are building on several highmem nodes which appear all to be stuck. There's no new high memory node coming online either, causing new ATH jobs to be stuck in the build queue while existing ones are stalled.

There's one high memory node in the "launching" phase, but I can't see that there's any progress.

Reproduction steps

No response

[MarkEWaite]

Also shown by the check agent availability job job on ci.jenkins.io

[dduportal]

• Set Jenkins in "safe mode" (builds are queued but not executed
• Had to stop manually all the running builds: they all were stuck trying to contact their executors. It's weird because I expect timeout to kick: smells liek a bug :(
• For each stopped build, triggered a build (that was queued)
• Restarting Jenkins

[dduportal]

Another weird thing: the EC2 higmem agents are not started by the scheduler. Trying to spawn a few manually.

Nevermind: they are started as expected.

[dduportal]

• Ref. support file upload for remote webdriver jenkinsci/acceptance-test-harness#780 (comment) 2 days ago.
• It sounds like that the Azure VM High mem are always the ones stuck

[dduportal]

Side note: there has been issues with repo.jenkins-ci.org during the past 9 hours:

[dduportal]

OH: I missed the new features of the azure-vm-agents plugin about VM retention (the "Idle retention" was the only one available last time I checked) and the ability to limit amount of VMs per kind of template

• We should switch to the "one shot" retention strategy on all our controllers
• That explains the "suspended" state
• We might want to update the azure vm config to only have one azure cloud (requires puppet template overhaul) so only one GC

[timja]

Those features have been there forever, although need to check I didn't break the once one by reconnecting

[dduportal]

I wonder if Olivier and I missed it because of the UI last year then, but I'm 101% sure we did not find it. not sure that it would fix the root issue there but would help.

[dduportal]

@timja Weird: there are a lot of azure agents on ci.j, but on Azure portal I don't see them.
Logs are telling "reconnect", which feels weird, as if the azure VM GC was not picking it up.
I've tried the "save cloud config on UI" but did not change, so I'm going to:

• stop the puppet agent
• apply the "once retention strategy" manually

[timja]

I'm wondering if this had some unintended consequences:
jenkinsci/azure-vm-agents-plugin#359

[timja]

Can revert if needed

[dduportal]

Can revert if needed

I would revert the plugin on ci.jenkins.io instead.

Would you need any logs or debug info to help tracking the plugin behavior?

[timja]

I would revert the plugin on ci.jenkins.io instead.

I can reproduce an issue with one shot strategy anyway, I don't have time to fix properly currently so going to revert for now

[dduportal]

First step on the "retention" cleanup to "once applied in jenkins-infra/jenkins-infra#2200

[dduportal]

Closing as the queue has been cleaned up and agents are now in better shape.