fix: Signature of Celery pruner jobs #32699

michael-s-molina · 2025-03-17T14:20:34Z

SUMMARY

This PR changes the signatures of the pruner jobs to accept key-word arguments and also binds the task to the current task instance to allows the task to access its own state and request information. Previously, the kwargs passed in the configuration were not accessible when running the task.

TESTING INSTRUCTIONS

CI should be sufficient.

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

michael-s-molina · 2025-03-17T14:20:57Z

superset/commands/logs/prune.py

        total_rows = len(ids_to_delete)

-        logger.info("Total rows to be deleted: %s", total_rows)
+        logger.info("Total rows to be deleted: %s", f"{total_rows:,}")


bycatch: Just making logs more readable.

michael-s-molina · 2025-03-17T14:22:05Z

cc @Usiel

korbit-ai

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Category	Issue	Fix Detected
	Incorrect Celery Task Type Hint ▹ view	✅

Files scanned

File Path	Reviewed
superset/commands/logs/prune.py	✅
superset/commands/sql_lab/query.py	✅
superset/tasks/scheduler.py	✅

Explore our documentation to understand the languages and file types we support and the files we ignore.

Need a new review? Comment /korbit-review on this PR and I'll review your latest changes.

Korbit Guide: Usage and Customization

Interacting with Korbit

You can manually ask Korbit to review your PR using the /korbit-review command in a comment at the root of your PR.

You can ask Korbit to generate a new PR description using the /korbit-generate-pr-description command in any comment on your PR.

Too many Korbit comments? I can resolve all my comment threads if you use the /korbit-resolve command in any comment on your PR.

On any given comment that Korbit raises on your pull request, you can have a discussion with Korbit by replying to the comment.

Help train Korbit to improve your reviews by giving a 👍 or 👎 on the comments Korbit posts.

Customizing Korbit

Check out our docs on how you can make Korbit work best for you and your team.

Customize Korbit for your organization through the Korbit Console.

Feedback and Support

Tell us what you think of Korbit

Schedule a call with our team

Email us @ [email protected]

superset/tasks/scheduler.py

codecov · 2025-03-17T14:27:37Z

Codecov Report

❌ Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.43%. Comparing base (76d897e) to head (67e9584).
⚠️ Report is 2419 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/commands/logs/prune.py	0.00%	1 Missing ⚠️
superset/commands/sql_lab/query.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #32699       +/-   ##
===========================================
+ Coverage   60.48%   83.43%   +22.94%     
===========================================
  Files        1931      549     -1382     
  Lines       76236    39421    -36815     
  Branches     8568        0     -8568     
===========================================
- Hits        46114    32892    -13222     
+ Misses      28017     6529    -21488     
+ Partials     2105        0     -2105

Flag	Coverage Δ
hive	`∅ <80.00%> (∅)`
javascript	`?`
mysql	`∅ <80.00%> (?)`
postgres	`∅ <80.00%> (?)`
presto	`∅ <80.00%> (∅)`
python	`∅ <80.00%> (∅)`
sqlite	`∅ <80.00%> (?)`
unit	`∅ <0.00%> (∅)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

villebro

LGTM with a few code quality considerations.

villebro · 2025-03-17T16:20:50Z

superset/commands/logs/prune.py

        total_rows = len(ids_to_delete)

-        logger.info("Total rows to be deleted: %s", total_rows)
+        logger.info("Total rows to be deleted: %s", f"{total_rows:,}")


optional thought: it could be a good idea to have a prettify_number or similar to centralize this logic, with a few unit tests to verify/demonstrate how the value is formatted.

I think this idea is good but it would be nice to do that as a general logging utility and review our logs to use the same standards. For that reason, I'll leave it out of the scope of this PR.

superset/tasks/scheduler.py

(cherry picked from commit df06bdf)

Usiel · 2025-03-18T02:39:07Z

cc @Usiel

Thanks for the ping. Both versions (pre fix and post fix) work for me, in the sense that retention_period_days is set to the value we pass in. Is this issue specific to a broker or celery version?

(tested with celery 5.4.0 * sqlite and ampq)

michael-s-molina · 2025-03-18T11:14:45Z

Is this issue specific to a broker or celery version?

I previously tested with 5.4.0 too.

(cherry picked from commit df06bdf)

* chore: bump base image in Dockerfile with `ARG PY_VER=3.11.11-slim-bookworm` (apache#32780) * chore: Revert "chore: bump base image in Dockerfile with `ARG PY_VER=3.11.11-slim-bookworm`" (apache#32782) * fix(chart data): removing query from /chart/data payload when accessing as guest user (apache#30858) (cherry picked from commit dd39138) * fix: upgrade to 3.11.11-slim-bookworm to address critical vulnerabilities (apache#32240) (cherry picked from commit ad05732) * fix(model/helper): represent RLS filter clause in proper textual SQL string (apache#32406) Signed-off-by: hainenber <[email protected]> (cherry picked from commit ff0529c) * fix: Log table retention policy (apache#32572) (cherry picked from commit 89b6d7f) * fix(welcome): perf on distinct recent activities (apache#32608) (cherry picked from commit 832e028) * fix(log): Update recent_activity by event name (apache#32681) (cherry picked from commit 449f51a) * fix: Signature of Celery pruner jobs (apache#32699) (cherry picked from commit df06bdf) * fix(logging): missing path in event data (apache#32708) (cherry picked from commit cd5a943) * fix(fe/dashboard-list): display modifier info for `Last modified` data (apache#32035) Signed-off-by: hainenber <[email protected]> (cherry picked from commit 88cf2d5) * fix: make packages PEP 625 compliant (apache#32866) Co-authored-by: Michael S. Molina <[email protected]> (cherry picked from commit 6e02d19) * all cccs changes * fix: Downgrade to marshmallow<4 (apache#33216) * fix(log): store navigation path to get correct logging path (apache#32795) (cherry picked from commit 4a70065) * fix(pivot-table): Revert "fix(Pivot Table): Fix column width to respect currency config (apache#31414)" (apache#32968) (cherry picked from commit a36e636) * fix: improve error type on parse error (apache#33048) (cherry picked from commit ed0cd5e) * fix(plugin-chart-echarts): remove erroneous upper bound value (apache#32473) (cherry picked from commit 5766c36) * fix(pinot): revert join and subquery flags (apache#32382) (cherry picked from commit 822d72c) * fix: loading examples from raw.githubusercontent.com fails with 429 errors (apache#33354) (cherry picked from commit f045a73) * chore: creating 4.1.3rc1 change log and updating frontend json (cherry picked from commit 72cf9b6) * chore(🦾): bump python sqlglot 26.1.3 -> 26.11.1 (apache#32745) Co-authored-by: GitHub Action <[email protected]> (cherry picked from commit 66c1a6a) * chore(🦾): bump python h11 0.14.0 -> 0.16.0 (apache#33339) Co-authored-by: GitHub Action <[email protected]> (cherry picked from commit 8252686) * docs: CVEs fixed on 4.1.2 (apache#33435) (cherry picked from commit 8a8fb49) * feat(api): Added uuid to list api calls (apache#32414) (cherry picked from commit 8decc9e) * fix(table-chart): time shift is not working (apache#33425) (cherry picked from commit dc44748) * fix(Sqllab): Autocomplete got stuck in UI when open it too fast (apache#33522) (cherry picked from commit b4e2406) * chore: update Dockerfile - Upgrade to 3.11.12 (apache#33612) (cherry picked from commit f0b6e87) * chore: updating 4.1.3rc2 change log * Select all Drag and Drop (#546) * add a select all button for the dnd select * remove cypress * chore(deps): bump cryptography from 43.0.3 to 44.0.1 (apache#32236) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit fa09d81) * fix: Adds missing __init__ file to commands/logs (apache#33059) (cherry picked from commit c1159c5) * fix: Saved queries list break if one query can't be parsed (apache#34289) (cherry picked from commit 1e5a4e9) * chore: Adds 4.1.4RC1 data to CHANGELOG.md and UPDATING.md * tag bump for select all drag and drop * Fix package-lock.json * Add db migration, bump Docker image base * gevent for gunicorn * remove threads and make worker-connections configurable * Fix package-lock.json * tag bump for cccs build * Remove CCCS Dataset Explorer (#550) * tag bump for CCCS build --------- Signed-off-by: hainenber <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: gpchandran <[email protected]> Co-authored-by: Joe Li <[email protected]> Co-authored-by: Jack <[email protected]> Co-authored-by: Đỗ Trọng Hải <[email protected]> Co-authored-by: Michael S. Molina <[email protected]> Co-authored-by: JUST.in DO IT <[email protected]> Co-authored-by: Michael S. Molina <[email protected]> Co-authored-by: Andreas Motl <[email protected]> Co-authored-by: Ville Brofeldt <[email protected]> Co-authored-by: Yuri <[email protected]> Co-authored-by: Maxime Beauchemin <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: GitHub Action <[email protected]> Co-authored-by: sha174n <[email protected]> Co-authored-by: Paul Rhodes <[email protected]> Co-authored-by: Rafael Benitez <[email protected]> Co-authored-by: cccs-RyanK <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: cyber-jessie <[email protected]>

fix: Signature of Celery pruner jobs

38a98c6

michael-s-molina requested a review from villebro March 17, 2025 14:20

pull-request-size bot added the size/S label Mar 17, 2025

michael-s-molina commented Mar 17, 2025

View reviewed changes

korbit-ai bot reviewed Mar 17, 2025

View reviewed changes

superset/tasks/scheduler.py Show resolved Hide resolved

Fix mypy

a675236

pull-request-size bot added size/M and removed size/S labels Mar 17, 2025

villebro approved these changes Mar 17, 2025

View reviewed changes

Remove optional

67e9584

michael-s-molina merged commit df06bdf into apache:master Mar 17, 2025
44 checks passed

michael-s-molina added a commit that referenced this pull request Mar 17, 2025

fix: Signature of Celery pruner jobs (#32699)

be9d7d0

(cherry picked from commit df06bdf)

michael-s-molina added the v5.0 Label added by the release manager to track PRs to be included in the 5.0 branch label Mar 20, 2025

michael-s-molina added a commit that referenced this pull request Mar 20, 2025

fix: Signature of Celery pruner jobs (#32699)

d2ba0fc

(cherry picked from commit df06bdf)

sadpandajoe added the v4.1 Label added by the release manager to track PRs to be included in the 4.1 branch label Mar 26, 2025

sadpandajoe pushed a commit that referenced this pull request Mar 27, 2025

fix: Signature of Celery pruner jobs (#32699)

ade8abb

(cherry picked from commit df06bdf)

mistercrunch added 🍒 4.1.3 Cherry-picked to 4.1.3 🍒 5.0.0 Cherry-picked to 5.0.0 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels labels Jul 29, 2025

github-actions bot added the 🍒 4.1.4 label Aug 28, 2025

silveira-js mentioned this pull request Jan 7, 2026

Add MapLibre support to charts PortalTelemedicina/superset#4

Merged

9 tasks

This was referenced Jan 15, 2026

Point explicit service account to client PortalTelemedicina/superset#9

Merged

Fix margin parameters PortalTelemedicina/superset#10

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: Signature of Celery pruner jobs #32699

fix: Signature of Celery pruner jobs #32699

Uh oh!

michael-s-molina commented Mar 17, 2025

Uh oh!

michael-s-molina Mar 17, 2025

Uh oh!

michael-s-molina commented Mar 17, 2025

Uh oh!

korbit-ai bot left a comment •

edited

Loading

Uh oh!

Uh oh!

codecov bot commented Mar 17, 2025 •

edited

Loading

Uh oh!

villebro left a comment

Uh oh!

villebro Mar 17, 2025

Uh oh!

michael-s-molina Mar 17, 2025

Uh oh!

Uh oh!

Uh oh!

Usiel commented Mar 18, 2025 •

edited

Loading

Uh oh!

michael-s-molina commented Mar 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fix: Signature of Celery pruner jobs #32699

fix: Signature of Celery pruner jobs #32699

Uh oh!

Conversation

michael-s-molina commented Mar 17, 2025

SUMMARY

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Uh oh!

michael-s-molina Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

michael-s-molina commented Mar 17, 2025

Uh oh!

korbit-ai bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Interacting with Korbit

Customizing Korbit

Feedback and Support

Uh oh!

Uh oh!

codecov bot commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

villebro left a comment

Choose a reason for hiding this comment

Uh oh!

villebro Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

michael-s-molina Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Usiel commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michael-s-molina commented Mar 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

korbit-ai bot left a comment •

edited

Loading

codecov bot commented Mar 17, 2025 •

edited

Loading

Usiel commented Mar 18, 2025 •

edited

Loading