Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Confusing behavior when trying to [1,2,3].each { |n| SomeJob.perform_in(n.seconds.from_now, n) } never running, logging as duplicate value #98

Closed
just3ws opened this issue Aug 3, 2015 · 1 comment

Comments

@just3ws
Copy link

just3ws commented Aug 3, 2015

I'm in a Rails 4.2 project running Sidekiq 3.4.2, Redis-rb 3.2.1, Hiredis 0.6.0, Sidekiq-Unique-Jobs 3.0.14 and am running Redis server version 3.0.3.

Step to repo:

  • Stop Sidekiq
  • Stop Console
  • Execute flushall in redis-cli (and verify that database has been cleaned)
  • Start Sidekiq (and see that Sidekiq has set up it's state in Redis)
  • Have job that looks like
class MainJob
  include Sidekiq::Worker
  sidekiq_options unique: true, log_duplicate_payload: true
  def perform(status_id)
    Rails.logger.debug("RUN #{status_id} - #{Time.zone.now}")
  end
end
  • Run in console [ 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20].each { |x| MainJob.perform_in(x.seconds.from_now, x);}
  • See the logs report everything as being duplicate
22:03:03 sidekiq.1 | started with pid 52254
22:03:05 sidekiq.1 | 2015-08-03T03:03:05.658Z 52254 TID-ow0dcxq00 INFO: Booting Sidekiq 3.4.2 with redis options {:url=>"redis://127.0.0.1:6379", :namespace=>"sidekiq_twitterstats_development"}
22:03:05 sidekiq.1 | 2015-08-03T03:03:05.967Z 52254 TID-ow0dcxq00 INFO: Running in ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-darwin14]
22:03:05 sidekiq.1 | 2015-08-03T03:03:05.967Z 52254 TID-ow0dcxq00 INFO: See LICENSE and the LGPL-3.0 for licensing details.
22:03:05 sidekiq.1 | 2015-08-03T03:03:05.967Z 52254 TID-ow0dcxq00 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org/pro
22:03:05 sidekiq.1 | 2015-08-03T03:03:05.973Z 52254 TID-ow0dcxq00 INFO: Starting processing, hit Ctrl-C to stop
22:03:39 sidekiq.1 | 2015-08-03T03:03:39.104Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[1], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"efad4d336d01a3518cde29f1", "created_at"=>1438571014.925308, "unique_hash"=>"sidekiq_unique:873fe317f6ae04a7309a367ae26a3258"}
22:03:39 sidekiq.1 | 2015-08-03T03:03:39.105Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[2], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"06b97376f1eb339b1b1959e4", "created_at"=>1438571014.929945, "unique_hash"=>"sidekiq_unique:65ba40524c1d2e350f2991240fe10604"}
22:03:39 sidekiq.1 | 2015-08-03T03:03:39.106Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[3], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"34484c5b551960d559d3a825", "created_at"=>1438571014.930542, "unique_hash"=>"sidekiq_unique:7bdc68d5d53d6f78dd7cc33e3734f333"}
22:03:39 sidekiq.1 | 2015-08-03T03:03:39.107Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[4], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"b21e4da4e4801dd333ad5838", "created_at"=>1438571014.931111, "unique_hash"=>"sidekiq_unique:4b1ac39c4d31f36df1b7214e71aa72b5"}
22:03:47 sidekiq.1 | 2015-08-03T03:03:47.229Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[5], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"c32fc156bae613c9d8c3cf3a", "created_at"=>1438571014.931665, "unique_hash"=>"sidekiq_unique:6f3546ae3ce28289f4e5eb942fe0e26c"}
22:03:47 sidekiq.1 | 2015-08-03T03:03:47.230Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[6], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"d8136a7c7444dc9ff8c26893", "created_at"=>1438571014.932162, "unique_hash"=>"sidekiq_unique:b68b7fceb9dfa03e2be725d4090579e5"}
22:03:47 sidekiq.1 | 2015-08-03T03:03:47.231Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[7], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"6ad3f13f5c1da0aea06c027a", "created_at"=>1438571014.932631, "unique_hash"=>"sidekiq_unique:b823e01bfcdf1152d1875a92c10d6aa3"}
22:03:47 sidekiq.1 | 2015-08-03T03:03:47.232Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[8], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"d249de55cc32200a1056043b", "created_at"=>1438571014.933073, "unique_hash"=>"sidekiq_unique:1c400012ff673638d6fa213bd69f3fd7"}
22:03:47 sidekiq.1 | 2015-08-03T03:03:47.233Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[9], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"172307fc65e660777ec77304", "created_at"=>1438571014.93351, "unique_hash"=>"sidekiq_unique:76f8b429b0ec936bb8b6d02d23cc9fcd"}
22:03:47 sidekiq.1 | 2015-08-03T03:03:47.234Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[10], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"c2ec311b3a45b1fbb96ba0d2", "created_at"=>1438571014.933995, "unique_hash"=>"sidekiq_unique:562562164181fc9aeb96c7bee71daa7c"}
22:03:47 sidekiq.1 | 2015-08-03T03:03:47.234Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[11], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"fcbb8d112c410f529fcaa2c7", "created_at"=>1438571014.934447, "unique_hash"=>"sidekiq_unique:744a12ddc633952956dfa0995936c225"}
22:03:47 sidekiq.1 | 2015-08-03T03:03:47.235Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[12], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"0f53bc92836fac8bbc5adc32", "created_at"=>1438571014.93488, "unique_hash"=>"sidekiq_unique:b4b9426e77203f59098525185ea1f5e2"}
22:04:05 sidekiq.1 | 2015-08-03T03:04:05.981Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[13], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"38989335206c8f52e5049b26", "created_at"=>1438571014.935255, "unique_hash"=>"sidekiq_unique:02ede2c4f7389c795f3dc7f8d65862cb"}
22:04:05 sidekiq.1 | 2015-08-03T03:04:05.982Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[14], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"3cd77ba139f99b3b1f034b7e", "created_at"=>1438571014.93568, "unique_hash"=>"sidekiq_unique:3be8926ed8ec50b7c5345b9f4d59cd38"}
22:04:05 sidekiq.1 | 2015-08-03T03:04:05.983Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[15], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"ff9f3d06aba63505acbe16f1", "created_at"=>1438571014.936095, "unique_hash"=>"sidekiq_unique:b7100995e9b7b3e812a6f413a3513552"}
22:04:05 sidekiq.1 | 2015-08-03T03:04:05.984Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[16], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"b8c84fda7b9018a8f00057a0", "created_at"=>1438571014.936539, "unique_hash"=>"sidekiq_unique:0353b7bede8407ac96ef65c7ea9f9638"}
22:04:05 sidekiq.1 | 2015-08-03T03:04:05.985Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[17], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"de69a3acf461236935c1701b", "created_at"=>1438571014.937022, "unique_hash"=>"sidekiq_unique:04ed1f749f09d16af9cb6cdd46315b4d"}
22:04:05 sidekiq.1 | 2015-08-03T03:04:05.985Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[18], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"c5d33eb3df62420d58440242", "created_at"=>1438571014.937494, "unique_hash"=>"sidekiq_unique:e5272283aebbfe8c98d80605285e40a7"}
22:04:05 sidekiq.1 | 2015-08-03T03:04:05.986Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[19], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"431958f101b3ad88e7dab73d", "created_at"=>1438571014.937935, "unique_hash"=>"sidekiq_unique:10edae4af2f53068e6b9bf3296e4fd9a"}
22:04:05 sidekiq.1 | 2015-08-03T03:04:05.987Z 52254 TID-ow0dex9k0 WARN: payload is not unique {"class"=>"MainJob", "args"=>[20], "retry"=>true, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"29ca67869cc262a76744143d", "created_at"=>1438571014.938309, "unique_hash"=>"sidekiq_unique:bd2611e46e1d682efa04c61bab5f064b"}

Each `unique_hash is unique but as you can see the values are being marked as duplicates.

Any ideas? If I run the same console commands using perform_async with no offset then the project behaves as expected.

@mhenrixon
Copy link
Owner

The time to perform is not part of the unique hash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants