Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rejected jobs are still displayed as 'Queued' with gem 'sidekiq-status' on /sidekiq/statuses #564

Closed
jamesst20 opened this issue Jan 18, 2021 · 11 comments · Fixed by #566
Closed

Comments

@jamesst20
Copy link

** Gems (all latest) **

gem 'sidekiq'
gem 'sidekiq-cron', '~> 1.2'
gem 'sidekiq-status'
gem 'sidekiq-unique-jobs', '~> 7.0.0.beta28'

** Description **
When I queue multiple jobs and that a lock rejects them, they are removed from the queue and added to processed as expected.

However, when visiting /sidekiq/statuses which is provided by the gem sidekiq-status, they are still marked as 'queued' excepted for the jobs that are not discarded.

** Expected behaviors **

Set them as completed or remove them

** Current behavior **

Show as queued while they are no longer queued.
Screen Shot 2021-01-18 at 10 01 53 AM

@mhenrixon
Copy link
Owner

I have my own page for viewing statuses. You can find a Locks tab in the Sidekiq Web interface.

I know nothing about Sidekiq::Status so the fastest way to get my help on this would be to give me a failing test case or example where it goes wrong.

The more details the better. Otherwise, I can't really commit to fixing it.

Late last year, someone explained really nicely what went wrong with Sidekiq::Apartment and I was able to quickly create a simple fix for their problem.

@jamesst20
Copy link
Author

jamesst20 commented Jan 18, 2021

Thanks for the fast reply, I can't seem to find the Locks tab you are talking, I will try to look deeper in the documentation.

** Setup **

  1. Install Sidekiq, Sidekiq Unique Jobs, Sidekiq Status
  2. Setup routing:
require 'sidekiq/web'
require 'sidekiq-status/web'

Rails.application.routes.draw do
 mount Sidekiq::Web => '/sidekiq'
end
  1. The route /sidekiq/statuses should be working.

** Reproducing error **

Have a worker:

class MyWorker
  include Sidekiq::Worker
  include Sidekiq::Status::Worker
  
  sidekiq_options(
    lock: :until_executed,
    lock_args_method: ->(args) { [args.first] },
    on_conflict: :reject
  )

  def perform(name)
    puts "Running job #{name}..."
    at(50)
    sleep 10
    at(100)
    puts "Done..."
  end
end
  1. Run Rails Server but not Sidekiq
  2. Queue MyWorker a few times (let's say 3 times)
  3. Run sidekiq
  4. Visit /sidekiq/statuses and notice how there is 2 jobs queued and one completed

@mhenrixon
Copy link
Owner

mhenrixon commented Jan 20, 2021

First of all, you might want to add the web extension for this gem as well:

require 'sidekiq/web'
require 'sidekiq-status/web'
require 'sidekiq_unique_jobs/web'

Rails.application.routes.draw do
 mount Sidekiq::Web => '/sidekiq'
end

@mhenrixon
Copy link
Owner

From my investigation, this is due to the order of the middleware. To avoid confusion and teach people that the order of middleware is super important, I decided to not add the middleware automatically.

This part deserves to be clear and up to the user. If you try the master branch (or clear your middleware first) you should be able to set everything up and verify locally.

I added more information about this to the readme.

@jamesst20
Copy link
Author

jamesst20 commented Jan 20, 2021

@mhenrixon

Thanks for the fast reply / fixes. I updated to 7.0.0 (non-beta).

I had never heard of the web extension you told me to add in routes. Can't seem to find anything about it. Also, it's giving me this error when I try to boot up the server:

/Users/james/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `**require': cannot load such file -- sidekiq-unique-jobs/web (LoadError)**
	from /Users/james/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
	from /Users/james/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:89:in `register'
	from /Users/james/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
	from /Users/james/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:44:in `require'

So in the mean time, I removed the require 'sidekiq-unique-jobs/web'

I updated my middleware chain to this

Sidekiq.client_middleware do |chain|
  chain.add SidekiqUniqueJobs::Middleware::Client
  chain.add Sidekiq::Status::ClientMiddleware, expiration: 12.hours
end

Sidekiq.server_middleware do |chain|
  chain.add SidekiqUniqueJobs::Middleware::Server
  chain.add Sidekiq::Status::ServerMiddleware, expiration: 12.hours
end

but I am still getting the same behavior.

Sidekiq v6.1.2
sidekiq-unique-jobs v7.0.0 (stable)

@jamesst20
Copy link
Author

Update, looks like doing the opposite inside middleware worked !

Sidekiq.client_middleware do |chain|
  chain.add Sidekiq::Status::ClientMiddleware, expiration: 12.hours
  chain.add SidekiqUniqueJobs::Middleware::Client
end

Sidekiq.server_middleware do |chain|
  chain.add Sidekiq::Status::ServerMiddleware, expiration: 12.hours
  chain.add SidekiqUniqueJobs::Middleware::Server
end

Sidekiq turned off, I queued 4 workers that sleeps for 5 seconds. I ran sidekiq and I got instantly this in the ui:

Screen Shot 2021-01-20 at 7 37 53 AM

5 seconds later:

Screen Shot 2021-01-20 at 7 38 07 AM

Looks like it's all good :) !

@mhenrixon
Copy link
Owner

Yeah sorry, I sloppy pasted the wrong code. There is a section in the README that has been corrected.

@mhenrixon
Copy link
Owner

@jamesst20 require 'sidekiq_unique_jobs/web'

@ArturT
Copy link
Contributor

ArturT commented Jan 22, 2021

Hi @mhenrixon

I believe the code example from readme is invalid. https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/README.md#sidekiq-status
I tried it and it left multiple jobs with queued status in sidekiq-status web UI (the same as on screenshot in the first post).

I noticed you said in readme something opposite to the code presented: The status middleware should run after uniqueness on client and before on server. .
So I reverted the order of middlewares and now it works fine for me.

Here is my working config:

Sidekiq.configure_server do |config|
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
    chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
  end

  config.server_middleware do |chain|
    chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes # default
    chain.add SidekiqUniqueJobs::Middleware::Server
  end

  SidekiqUniqueJobs::Server.configure(config)
end


Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
    chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
  end
end

@mhenrixon
Copy link
Owner

Dangit @ArturT! I knew I rushed that a bit much. I'll copy your example to the README.md, thanks a bunch for letting me know!

@mhenrixon
Copy link
Owner

Ok, I updated the README again 😶

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants