Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Que integration can fail #545

Closed
mikz opened this issue Jun 11, 2019 · 6 comments · Fixed by 3scale/zync#209
Closed

Que integration can fail #545

mikz opened this issue Jun 11, 2019 · 6 comments · Fixed by 3scale/zync#209
Labels
released This feature/bug fix has been released

Comments

@mikz
Copy link
Contributor

mikz commented Jun 11, 2019

Description

Some of my jobs fail because of a statement timeout and bugsnag que integration fails to report that.

Issue

Failed to notify Bugsnag of error in Que job (NoMethodError): undefined method `[]' for nil:NilClass
--
  | /opt/app-root/src/vendor/bundle/ruby/2.4.0/gems/bugsnag-6.11.1/lib/bugsnag/integrations/que.rb:7:in `block (2 levels) in <main>'
  | /opt/app-root/src/vendor/bundle/ruby/2.4.0/gems/bugsnag-6.11.1/lib/bugsnag.rb:73:in `notify'
  | /opt/app-root/src/vendor/bundle/ruby/2.4.0/gems/bugsnag-6.11.1/lib/bugsnag/integrations/que.rb:6:in `block in <main>'
  | /opt/app-root/src/vendor/bundle/ruby/2.4.0/gems/que-1.0.0.beta3/lib/que/utils/error_notification.rb:17:in `notify_error'
  | /opt/app-root/src/vendor/bundle/ruby/2.4.0/gems/que-1.0.0.beta3/lib/que/worker.rb:136:in `rescue in work_job'
  | /opt/app-root/src/vendor/bundle/ruby/2.4.0/gems/que-1.0.0.beta3/lib/que/worker.rb:90:in `work_job'
  | /opt/app-root/src/vendor/bundle/ruby/2.4.0/gems/que-1.0.0.beta3/lib/que/worker.rb:69:in `work_loop'
  | /opt/app-root/src/vendor/bundle/ruby/2.4.0/gems/que-1.0.0.beta3/lib/que/worker.rb:46:in `block in initialize'

Environment

Error triggered is PG::QueryCanceled, which makes Que to pass a job nil to bugsnag.
tobyhs added a commit that referenced this issue Jun 11, 2019
@tobyhs
Check for nil job in Que integration
f667e50 
This might fix #545

https://github.com/chanks/que/blob/master/docs/error_handling.md#error-notifications
mentions that the job block parameter may be nil for connection failures
and similar issues.
tobyhs added a commit that referenced this issue Jun 11, 2019
@tobyhs
Check for nil job in Que integration
9a3168d 
This might fix #545

https://github.com/chanks/que/blob/master/docs/error_handling.md#error-notifications
mentions that the job block parameter may be nil for connection failures
and similar issues.
tobyhs added a commit that referenced this issue Jun 11, 2019
@tobyhs
Check for nil job in Que integration
00dac89 
This might fix #545

https://github.com/chanks/que/blob/master/docs/error_handling.md#error-notifications
mentions that the job block parameter may be nil for connection failures
and similar issues.
@tobyhs
Copy link
Contributor

tobyhs commented Jun 11, 2019

Hi @mikz

Would you mind checking if the branch at https://github.com/bugsnag/bugsnag-ruby/tree/tobyhs/check_nil_job_in_que solves your issue?

@mikz
Copy link
Contributor Author

mikz commented Jun 12, 2019

I'll try to reproduce it & verify the fix, but it might take a few days. Thanks!

@tobyhs tobyhs added the awaiting feedback Awaiting a response from a customer. Will be automatically closed after approximately 2 weeks. label Jun 12, 2019
mikz added a commit to 3scale/zync that referenced this issue Jun 14, 2019
@mikz
[gemfile] try bugsnag hotfix
5f7839c 
should fix bugsnag/bugsnag-ruby#545
@mikz mikz mentioned this issue Jun 14, 2019
Merged
@mikz mikz reopened this Jun 14, 2019
@mikz
Copy link
Contributor Author

mikz commented Jun 14, 2019

@tobyhs I can't really reproduce those conditions anymore. It was under extremely high load and Que started getting database timeouts. I can't reproduce that with plain locking the table. But the change looks ok 👍

@tobyhs tobyhs removed the awaiting feedback Awaiting a response from a customer. Will be automatically closed after approximately 2 weeks. label Jun 17, 2019
tobyhs pushed a commit that referenced this issue Jun 17, 2019
Handle cases where job is nil in Que error notifier
a28bd43 
This should fix #545
@tobyhs tobyhs mentioned this issue Jun 17, 2019
Merged
15 tasks
@tobyhs
Copy link
Contributor

tobyhs commented Jun 18, 2019

@mikz I merged a change into the next branch at #548. If you can't wait until the next release of bugsnag-ruby, can you use the next branch (at commit 90df996 ) in your app as I'm planning on deleting the tobyhs/check_nil_job_in_que soon (to keep things tidy).

@tobyhs tobyhs closed this as completed Jun 18, 2019
@mikz
Copy link
Contributor Author

mikz commented Jun 20, 2019

@tobyhs very thoughtful of you, fixed! Thank you.

tomlongridge pushed a commit that referenced this issue Aug 23, 2019
@tomlongridge
Handle cases where job is nil in Que error notifier
afca717 
This should fix #545
@abigailbramble
Copy link

Released in https://github.com/bugsnag/bugsnag-ruby/releases/tag/v6.12.0

@abigailbramble abigailbramble added the released This feature/bug fix has been released label Sep 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
released This feature/bug fix has been released
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[gemfile] try bugsnag hotfix 3scale/zync
3 participants
@mikz @tobyhs @abigailbramble