To add support for non-concurrent task execution in a clustered environment

[vondacho]

Description

Considering a clustered setup of Quartz, the need for a recurrent task to be triggered only once its previous execution is terminated can be addressed using the programmatic way i.e. the Quartz API.

final JobDetail job = JobBuilder.newJob(MyTask.class).withIdentity(identity, group).build();
final Trigger trigger = TriggerBuilder.newTrigger()
        .withIdentity(identity, group)
        .withSchedule(CronScheduleBuilder.cronSchedule(cron))
        .build();
scheduler.schedule(job, trigger);

@DisallowConcurrentExecution
public static final class MyTask implements Job {
    @Override
    public void execute(JobExecutionContext context) {
        // my work
    }
}

The declarative way i.e. the @Scheduled annotation is still limited, because its concurrentExecution attribute manages non-concurrency locally on a single node i.e. a single application instance, but not globally, considering every nodes i.e. every application instances.

Therefore, the request is to provide an equivalent solution design using the declarative way i.e. an annotation like the @scheduled one.

Implementation ideas

• To bridge the @DisallowedConcurrentExecution annotation and the concurrentExecution=SKIP attribute of the @scheduled annotation.
• To design another annotation

[geoand]

cc @mkouba

[vsevel]

I may add that the only reason for us to use quartz in the first place is specifically for jobs that are required to execute just once at a time across the whole cluster. for the other timers that execute per pod we use the default quarkus scheduler.
this current limitation requires us to stay on the programmatic approach, which ideally we would like to avoid in simple cases.
the other option is for us to switch on jobrunr. but for legacy eap applications that are migrating to quarkus and are already working with quartz, this would make our life simpler if we could stay with quartz.

[mkouba]

I do understand your needs. In theory, we could map @Scheduled#concurrentExecution=SKIP to a org.quartz.Job implementation class annotated with @DisallowedConcurrentExecution but we'll need to check if this annotation itself is enough. Also it would be a breaking change because AFAIK the FQCN of the job is used in Quartz tables.

FYI I've tried to implement a simple (and naive.. yet working ;-) mutex/non-concurrent job with the default/simple scheduler; it's based on the SkipPredicate API, it leverages SuccessfulExecution/FailedExecution events and Hibernate + Panache + JTA + PostgreSQL: https://github.com/mkouba/scheduler-mutex/blob/main/src/main/java/org/acme/Jobs.java

[mkouba]

I think that we'll need to introduce a new annotation because @Scheduled#concurrentExecution=SKIP does not really map to @DisallowedConcurrentExecution. First of all, we won't be able to fire the SkippedExecution event. Furthermore, Quartz defines various misfire policies whereas ConcurrentExecution#SKIP has single policy - skip execution. I will try to propose a pull request with @io.quarkus.quartz.Nonconcurrent.

[mkouba]

So I have a working solution but it has quite some limitations.

TLDR>

It only works if quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=true (it's false by default) and only for scheduled methods with blocking signature (it won't work for non-blocking scheduled methods and for methods annotated with @RunOnVirtualThread).

Why it won't work with quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=false?

Because we do not execute the scheduled method on a Quartz thread by default. We only trigger the execution from a Quartz thread and return immediately. We offload the execution to the corresponding executor (event loop, worker thread, virtual thread executor). The main reason is consistency and propagation of the Vert.x duplicated context.

What can we do?

We could detect quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=false and fail during startup if @Nonconcurrent is used. We could also detect @Nonconcurrent declared on a non-blocking/virtual thread scheduled method.

Is it really worth the effort?

I'm not really sure...

CC @manovotn

Here's my draft PR: #44224

[manovotn]

Is it really worth the effort?

I'm not really sure...

I took a look at the draft and I think that the solution is too restrictive to be useful, although I understand why it is done in such a way and I sadly don't have a better one up my sleeve.

The initial comment shows a "workaround" that's actually more flexible as you can keep scheduling other jobs on whichever thread pool you need while offloading these non-concurrent jobs directly to Quartz. OTOH, the draft requires you to explicitly set quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread property.

Ultimately, @vsevel and @vondacho should have the last word on that as you are running this use case. So please, let us know if the draft presented here would help you, or not.

And last but not least, Martin's example with mutex implementation should also be taken into consideration as that seems to be applicable to arbitrary scheduled method with Quarkus if need be (or perhaps if you just don't want to go through the Quartz API).

[mkouba]

And last but not least, #44048 (comment) with mutex implementation should also be taken into consideration as that seems to be applicable to arbitrary scheduled method with Quarkus if need be (or perhaps if you just don't want to go through the Quartz API).

In theory, we could also turn this POC into a separate extension... but again, I'm not really sure it would be worth it.

[vsevel]

It only works if quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=true

that would not be an issue. we have our own extensions where we can force the apps to set this property.

only for scheduled methods with blocking signature (it won't work for non-blocking scheduled methods and for methods annotated with @RunOnVirtualThread)

it would not be an issue, since the workaround does not do any better. we run anyway on the quartz thread, and @RunOnVirtualThread typically does not do anything.
there are no thousands of timers typically in an application, so we can spare some blocking threads for this activity.

The main reason is consistency and propagation of the Vert.x duplicated context.

this means that we would not get a duplicated context. I checked with the workaround that we can force a request scope. I suppose we would be able to do the same thing with your code change proposition?
what are other consistency issues?

We could also detect @Nonconcurrent declared on a non-blocking/virtual thread scheduled method.

virtual thread is already covered at startup (the app fails to start with a clear message)
this is a nice to have. with that solution we would plan to force the use of the property from our extensions.

... shows a "workaround" that's actually more flexible as you can keep scheduling other jobs on whichever thread pool you need while offloading these non-concurrent jobs directly to Quartz

our original need was to be able to run exclusive timers on quartz, and anything else on the simple scheduler.
we do not need the extra flexibility offered by the workaround.
furthermore, we actually don't feel we have a better solution with the workaround as:

• we execute on a blocking thread, which is a quartz thread just like in the PR
• the declaration is more cumbersome as we have to create a new job class, and schedule it programmatically at app startup

Ultimately, @vsevel and @vondacho should have the last word on that as you are running this use case.

we appreciate the effort @mkouba . we believe your proposition brings some decent value. if this feature was available, we would certainly use it, as we would avoid a more complex declaration, and the limitations do not seem to apply, at least in our situation.
if you think this is not really worth the effort, we would gladly stay with the workaround. that would not be an issue.

as our utilization of quartz really revolves around non concurrent jobs, I think your proposition is valuable, assuming this is not bringing too much complexity into the quartz extension.

[mkouba]

as our utilization of quartz really revolves around non concurrent jobs, I think your proposition is valuable, assuming this is not bringing too much complexity into the quartz extension.

It does not bring additional complexity. So I will add a check that would fail at runtime if quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=false and @Nonconcurrent is used. And we'll see what CI thinks...

[vsevel]

We only trigger the execution from a Quartz thread and return immediately

would we be able to offload the execution to a quarkus managed thread (potentially a virtual thread), and block the quarkus thread until the processing returns?
that way you would get the duplicated context and consistency.

[mkouba]

We only trigger the execution from a Quartz thread and return immediately

would we be able to offload the execution to a quarkus managed thread (potentially a virtual thread), and block the quarkus thread until the processing returns? that way you would get the duplicated context and consistency.

Hm, that would be possible but I'm not quite sure it's a good idea. I mean, if we do it globally (for all scheduled methods) then we will always block two threads, but most of the time for no good reason. It would also mean that the quartz thread pool would have to be big enough so that it's not a bottleneck. If we only do it for @Nonconcurrent methods it might be confusing for users. We could make it configurable but that's IMO even more confusing.

[vsevel]

If we only do it for @Nonconcurrent methods it might be confusing for users

that is what I was thinking, and may be only when the blocking property is set.
that way you would have consistency for the user application from a threading point of view (always running on a worker thread from quarkus).
you solve the duplicated context and other potential consistency issues.
the fact that it is triggered from quartz thread is an implementation detail, technically required for Nonconcurrent

[mkouba]

If we only do it for @Nonconcurrent methods it might be confusing for users

that is what I was thinking, and may be only when the blocking property is set.

Which property do you have in mind?

that way you would have consistency for the user application from a threading point of view (always running on a worker thread from quarkus). you solve the duplicated context and other potential consistency issues. the fact that it is triggered from quartz thread is an implementation detail, technically required for Nonconcurrent

It is an implementation detail but it might require additional considerations, for example the Quartz thread pool configuration.

That said, we could document the impacts in the javadoc of @Nonconcurrent.

WDYT? @manovotn

[vsevel]

Which property do you have in mind?

quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=true

[manovotn]

Which property do you have in mind?

quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=true

Just to make sure I understand this, you're suggesting that with this property set, we handle blocking scheduled jobs so that:

• We trigger them from a Quartz thread
• However, we still offload them to a thread from Quarkus thread pool
• We block the Quartz thread until the execution of the job completes within Quarkus thread

Is that right?

If so, I think this could work if we only do that for @Nonconcurrent jobs where the price of locking two threads for a single execution is "justifiable". Although I am not sure how to clearly communicate this to users. Especially given that in such a case they should also tweak the Quartz pool config or at least be aware of how big it is by default.

[mkouba]

Which property do you have in mind?

quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=true

Just to make sure I understand this, you're suggesting that with this property set, we handle blocking scheduled jobs so that:

* We trigger them from a Quartz thread

* However, we still offload them to a thread from Quarkus thread pool

* We block the Quartz thread until the execution of the job completes within Quarkus thread

Is that right?

If so, I think this could work if we only do that for @Nonconcurrent jobs where the price of locking two threads for a single execution is "justifiable". Although I am not sure how to clearly communicate this to users. Especially given that in such a case they should also tweak the Quartz pool config or at least be aware of how big it is by default.

I don't think it makes a lot of sense. If quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=true then we should always execute the method on a quartz thread, otherwise we add another exception and modify the existing behavior 👎.

In theory, we could block the quartz thread if quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=false. This way, @Nonconcurrent would work without a config change. And it could be documented in the javadoc of @Nonconcurrent...

[manovotn]

In theory, we could block the quartz thread if quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=false. This way, @Nonconcurrent would work without a config change. And it could be documented in the javadoc of @Nonconcurrent...

Yes, that's essentially the same - they key point is that we want to only have this behavior for @Nonconcurrent and the less special config knobs we need to have for that, the better :)

[mkouba]

In theory, we could block the quartz thread if quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=false. This way, @Nonconcurrent would work without a config change. And it could be documented in the javadoc of @Nonconcurrent...

Yes, that's essentially the same - they key point is that we want to only have this behavior for @Nonconcurrent and the less special config knobs we need to have for that, the better :)

Hm, it's not the same but I get your point. I will adapt my PR then.

[mkouba]

In theory, we could block the quartz thread if quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=false. This way, @Nonconcurrent would work without a config change. And it could be documented in the javadoc of @Nonconcurrent...

Yes, that's essentially the same - they key point is that we want to only have this behavior for @Nonconcurrent and the less special config knobs we need to have for that, the better :)

Hm, it's not the same but I get your point. I will adapt my PR then.

Done.

[vsevel]

the price of locking two threads for a single execution

the second thread may be a virtual thread. in that case we lock just 1 really. and we were going to execute on it anyway. so we are not really locking anything or wasting resources.

what did you end up doing? @mkouba

[mkouba]

what did you end up doing? @mkouba

We only block the quartz thread if quarkus.quartz.run-blocking-scheduled-method-on-quartz-thread=false and @Nonconcurrent is used so that no config change is needed.

[vsevel]

ok. and you execute the business method on that quartz thread, or you offload it to a quarkus thread?

[mkouba]

ok. and you execute the business method on that quartz thread, or you offload it to a quarkus thread?

We offload and block the quartz thread until the execution finished.

[vsevel]

that may have been a good thing to introduce a complete new annotation QuartzSchedule, and keep Scheduler for the normal simple timer.
that way there was no need to add executeWith=QUARTZ, and the non concurrent could have been an attribute on QuartzSchedule

[mkouba]

that may have been a good thing to introduce a complete new annotation QuartzSchedule, and keep Scheduler for the normal simple timer. that way there was no need to add executeWith=QUARTZ, and the non concurrent could have been an attribute on QuartzSchedule

I don't think it would be a good idea. @Nonconcurrent and executeWith=QUARTZ are corner cases. We want to try to keep the API unified where possible so that it's easy to switch the underlying implementation in most use cases.