Mute k8s job pod scheduling warnings by default

[BioWilko]

New feature

Currently when many processes are queued via k8s jobs Nextflow will print warnings for every job unable to schedule a pod e.g.

WARN: K8s Job nf-1858f850f1e7e854a6f4695ace684a7c-df144 does not have pod - Not yet scheduled?
WARN: K8s Job nf-24a1c789dc837e54b34c16a08660365d-67a1c does not have pod - Not yet scheduled?
WARN: K8s Job nf-b3ed3f9916b43af5a8196597e9044bdb-48e63 does not have pod - Not yet scheduled?
WARN: K8s Job nf-ec282416a6cd7d600131a0d2685c7951-175d3 does not have pod - Not yet scheduled?

There are cases where this is a useful message but the majority of the time it just means that you don't have a large enough cluster/quota to schedule all pods simultaneously which is fine and normal and shouldn't lead to a warning.

Could this message be hidden by default? it makes watching Nextflow in k8s via the command line quite unpleasant since stderr is totally full of these messages and you only occasionally get to see the status of the pipeline.
An even better feature would be if Nextflow could distinguish between jobs unable to schedule pods because there's insufficient resources available vs jobs unable to schedule pods because the container URL is unreachable or k8s errors etc.

[bentsherman]

I always thought that warning was inconsistent with how other executors behave. Perhaps we could change it to debug1 so that it's logged to the nextflow log but not the console:

nextflow/plugins/nf-k8s/src/main/nextflow/k8s/client/K8sClient.groovy

Line 444 in a76b760

log.warn1("K8s Job $jobName does not have pod - Not yet scheduled?")

[BioWilko]

That would certainly suit me but it would definitely bloat logfiles, if you have a couple of hundred jobs getting checked regularly it can lead to a very large amount of log spam, that's not necessarily an issue of course

[bentsherman]

The warn1 means that it logs each unique line only once, which would be one line per job. Is that what you're seeing in your console output? I figure since Nextflow already logs at least one line per job in the log file, it wouldn't be unreasonable

[BioWilko]

Ah I see, I had assumed that it was repeatedly warning since they stick around for quite a while, they're quite hard to tell apart so I hadn't noticed it was unique.

[BioWilko]

So I tested and these messages are getting logged far more than once:

grep -oE "nf-[[:alnum:]]{1,}-[[:alnum:]]{1,} does not have pod - Not yet scheduled?" .nextflow.log | sort | uniq -c | sort
    100 nf-a069f31f1a99b0213a285b009ade4201-54518 does not have pod - Not yet scheduled
    100 nf-f92b3048a9e4f507957340182ddc89b8-23599 does not have pod - Not yet scheduled
    103 nf-c535090e98ee99af14e7a1cd3ebe2088-8b136 does not have pod - Not yet scheduled
    104 nf-3cec7954146a41c9e1fe452e3700ba05-5dd8f does not have pod - Not yet scheduled
    104 nf-b7b088c4f58014e0ab2948c383f98102-9725c does not have pod - Not yet scheduled
     10 nf-0da239039c2fd42b3f0dd8a2893544be-dd69b does not have pod - Not yet scheduled
     10 nf-6675e344784e0114dc23f0668a4df8b9-fb7b5 does not have pod - Not yet scheduled
    113 nf-310b91ec42bc1f1fd7a2d92e85058d08-50cf4 does not have pod - Not yet scheduled
    114 nf-ac705b9568db7116fd738480977a7837-59c43 does not have pod - Not yet scheduled
    117 nf-621e7c5317491ba04bce40fc4f542042-6ffa6 does not have pod - Not yet scheduled
    117 nf-8ed970d99f627245d40a3860e03e1b46-76784 does not have pod - Not yet scheduled
    122 nf-ca63545762782437fc9b10dfe7cef354-70623 does not have pod - Not yet scheduled
    128 nf-92704256bc28aaf0d654e83d71f1e79f-4e9c9 does not have pod - Not yet scheduled
     14 nf-064817fbe0a9601c5d50ac91646c66c5-ad1f9 does not have pod - Not yet scheduled
     14 nf-14f7219c36f10598665d93107e865eb0-b0b32 does not have pod - Not yet scheduled
     14 nf-65e47cbbb3820c48529c43327702e956-c0b4c does not have pod - Not yet scheduled
     14 nf-93858db28aed4f50fcf18454c684ea7b-51fa4 does not have pod - Not yet scheduled
    151 nf-8e0861b5c6646966fff3747440403cd3-80906 does not have pod - Not yet scheduled
    151 nf-eaa8233d06ff00cfa3c8b7bca6266ecc-e5057 does not have pod - Not yet scheduled
    152 nf-0b280ac6edfc3105a00f7095f4ca08ef-acb19 does not have pod - Not yet scheduled
    152 nf-ebf71fb057f48f21d020ab8863df5a2e-cf95a does not have pod - Not yet scheduled
    153 nf-e9934111c4754162e4c0b85868e32310-2cc09 does not have pod - Not yet scheduled
    157 nf-e8006f040c18dcc9fa599a1d9a639bc1-fb49b does not have pod - Not yet scheduled
    157 nf-fad5acc09ef0134a4908b261d5b5548b-1e3e7 does not have pod - Not yet scheduled
     15 nf-0ae768e277670394f45d2bc5f1c93dbf-0aed2 does not have pod - Not yet scheduled
     15 nf-7d77e054af39482d62677c1751383b7c-2745e does not have pod - Not yet scheduled
     15 nf-f0a2ad4b3ef352050f9f7b0390d83413-f51f4 does not have pod - Not yet scheduled
    160 nf-c2659732a8660fd4ed059b39aa738e56-84569 does not have pod - Not yet scheduled
    163 nf-438dd046a0c8bc51045204ae96b93303-71acd does not have pod - Not yet scheduled
    163 nf-6ee172820fd1a1daa4f788c07ba8ae75-22430 does not have pod - Not yet scheduled
    163 nf-752ea6023ac13ee7ab41b1317cc83591-8cd97 does not have pod - Not yet scheduled