Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Celery Beat auto monitoring #1967

Merged
merged 33 commits into from
Apr 4, 2023
Merged

Conversation

antonpirker
Copy link
Member

@antonpirker antonpirker commented Mar 21, 2023

Automatically monitor Celery Beat tasks with Sentry Crons.

How we do this:

  • we dynamically create a function that listens to the celery_beat_init hook. In the hook we do two things:
  • 1.) patch existing scheduled tasks (in sender.scheduler.schedule):
    • Each scheduled task is patched to contain information about the Sentry monitor (the monitor slug and config (timezone, schedule, ...) in its headers.
    • We then stop Celery Beat and replace the scheduled tasks with the new patched scheduled tasks
    • We restart Celery Beat to enable our patched tasks
  • 2.) Connect each task to the following hooks to send information about the task to sentry: task_prerun, task_success, task_failure, task_retry. (config is sent by the tasks in its headers we set up in 1))

Docs: getsentry/sentry-docs#6526

@antonpirker antonpirker self-assigned this Mar 22, 2023
Copy link
Member

@sl0thentr0py sl0thentr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please summarize the important moving parts in the PR description? at least a high level overview of what we're patching etc

sentry_sdk/integrations/celery.py Outdated Show resolved Hide resolved
Copy link
Member

@HazAT HazAT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rubberstamp ✅

@antonpirker antonpirker dismissed sl0thentr0py’s stale review April 4, 2023 09:22

Changes to PR description done

@antonpirker antonpirker merged commit d4bbd85 into master Apr 4, 2023
@antonpirker antonpirker deleted the antonpirker/celery-auto-monitor-tasks branch April 4, 2023 10:28
nijel added a commit to WeblateOrg/weblate that referenced this pull request Apr 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants