After sidekiq shutdown (with running job) lock digest is not pushned back

[kirsha2]

Describe the bug
Rails: 7.0.4.3
Ruby: 3.2.2
Sidekiq: 6.2.1

When trying upgrading sidekiq and unique jobs our tests began to fail. Same behavior in manual testing.

1. Enqueue until_executed job
2. start processing
3. shutdown worker (job is still not finished)
4. job is pushed back into queue but not the digest

The bug occurs after 7.1.29 version. Same behavior in 8.
Used lock: until_executed

Expected behavior
After sidekiq shutdown job and digest are pushed back.

Current behavior
After sidekiq shutdown only job is pushed back.

I've looked into diff between version 29 and 30 but I cant see anything suspicious there. Is this expected behavior ?

Thanks :) for all your hard work - love the gem!

[kirsha2]

I've tried to look into the changes more and found out it was due to the ensure in until_executed.rb

https://my.diffend.io/gems/sidekiq-unique-jobs/7.1.29/7.1.30#d2h-630681

So I don't know if this change/ consequence was intentional or not

[mhenrixon]

I've tried to look into the changes more and found out it was due to the ensure in until_executed.rb

https://my.diffend.io/gems/sidekiq-unique-jobs/7.1.29/7.1.30#d2h-630681

So I don't know if this change/ consequence was intentional or not

It sounds like a bug; I'll see about fixing these issues in the next few days. I should use an else rather than an ensure.

[joshuacronemeyer]

what behavior does this cause? Currently i'm facing an issue where my until_executed jobs can't get enqueued anymore and this sounds like it could be related to my problem. if so Is there something i can run from the console that will fix this for me when this happens?

[kirsha2]

@joshuacronemeyer sounds like your locks/jobs are not correctly unlocked.
Behavior that I reproduced only affects re-enqued jobs, so if the job is not finished but pushed back into to queue.

This is issue is about missing lock and yours is probably still there "stuck".
I'd recommend to check your Unique Digests.

[pboling]

@kirsha2 The ensure-based bug was fixed in v8.0.5 and v7.1.31 I think. thread: #800 (comment)

[kirsha2]

@pboling thank you for the info

yes it works now

[kirsha2]

Unfortunally I closed it prematurely, I've forgotten to remove my monkey patch in the code (too many changes in MR :D )

The main issue for me is still the ensure part in

sidekiq-unique-jobs/lib/sidekiq_unique_jobs/lock/until_executed.rb

Line 38 in bdca185

ensure

My expected behavior, when job is in retry set the lock is still kept.

When this job after retries goes to dead, death_handlers takes care of unlocking
https://github.com/mhenrixon/sidekiq-unique-jobs/blob/main/lib/sidekiq_unique_jobs/server.rb#L13

I patched it like this

# lib/sidekiq_unique_jobs/lock/until_executed.rb
def execute
  executed = locksmith.execute do
    yield
    unlock_and_callback # NOTE: override ensure due to lock not pushing back after sidekiq shutdown
  end

  reflect(:execution_failed, item) unless executed

  nil
end

[pboling]

@kirsha2 thanks for the update. I'll need to pull in your monkey patch also, as my production system can't afford to have this issue either, but I hadn't had time to evaluate it closely yet.