Skip to content

Conversation

@mistercrunch
Copy link
Member

Recently we've been getting HTTP Error 429: Too Many Requests in CI while loading examples.

Historically we fetched the sample files straight from raw.githubusercontent.com. Anonymous requests are limited to 60 per hour per IP on GitHub, so our busy CI matrix started to hit 429s.

We now point to jsDelivr – a multi‑CDN cache in front of every public GitHub repo. It doesn’t consume the GitHub API quota and advertises unlimited bandwidth for OSS.

…rrors

Been gettting `HTTP Error 429: Too Many Requests` in CI while loading examples.

```
Historically we fetched the sample files straight from
``raw.githubusercontent.com``.  Anonymous requests are limited to **60
per
hour per IP** on GitHub, so our busy CI matrix started to hit 429s.

We now point to **jsDelivr** – a multi‑CDN cache in front of every
public
GitHub repo.  It doesn’t consume the GitHub API quota and advertises
unlimited bandwidth for OSS.
```
@dosubot dosubot bot added the doc:examples Related to example datasets and dashboards label May 4, 2025
Copy link

@korbit-ai korbit-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've completed my review and didn't find any issues.

Files scanned
File Path Reviewed
superset/examples/helpers.py

Explore our documentation to understand the languages and file types we support and the files we ignore.

Check out our docs on how you can make Korbit work best for you and your team.

Loving Korbit!? Share us on LinkedIn Reddit and X

@geido geido merged commit f045a73 into master May 5, 2025
46 checks passed
@geido geido deleted the fix_examples_rate_limit branch May 5, 2025 11:07
@IdlePhysicist
Copy link

@mistercrunch @geido is there a plan to cut a release with this fix in the near term future?

sadpandajoe pushed a commit that referenced this pull request May 5, 2025
@sadpandajoe sadpandajoe added the v4.1 Label added by the release manager to track PRs to be included in the 4.1 branch label May 5, 2025
kbsteere pushed a commit to wolfi-dev/os that referenced this pull request May 6, 2025
This commit adds a patch to allow use to fetch the test data from
github, other wise we get 429s from github.

See apache/superset#33354
sadpandajoe pushed a commit that referenced this pull request May 6, 2025
sadpandajoe pushed a commit that referenced this pull request May 6, 2025
sadpandajoe pushed a commit that referenced this pull request May 6, 2025
@michael-s-molina michael-s-molina added the v5.0 Label added by the release manager to track PRs to be included in the 5.0 branch label May 12, 2025
michael-s-molina pushed a commit that referenced this pull request May 12, 2025
michael-s-molina pushed a commit that referenced this pull request May 12, 2025
michael-s-molina pushed a commit that referenced this pull request May 12, 2025
mistercrunch added a commit that referenced this pull request May 12, 2025
Recently in #33354 I tried to address http 429 "too many attempts that we started getting in CI. This PR adds a bit of custom logic with retries and expodential backoffs to work around the issue. Ideally we'd have another approach using a proper CDN that supports our needs or use a github repo to clone locally, but this should help relieve the issue.
LevisNgigi pushed a commit to LevisNgigi/superset that referenced this pull request Jun 18, 2025
@mistercrunch mistercrunch added 🍒 4.1.3 Cherry-picked to 4.1.3 🍒 5.0.0 Cherry-picked to 5.0.0 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels labels Jul 29, 2025
cyber-jessie added a commit to CybercentreCanada/superset that referenced this pull request Jan 8, 2026
* chore: bump base image in Dockerfile with `ARG PY_VER=3.11.11-slim-bookworm` (apache#32780)

* chore: Revert "chore: bump base image in Dockerfile with `ARG PY_VER=3.11.11-slim-bookworm`" (apache#32782)

* fix(chart data): removing query from /chart/data payload when accessing as guest user (apache#30858)

(cherry picked from commit dd39138)

* fix: upgrade to 3.11.11-slim-bookworm to address critical vulnerabilities (apache#32240)

(cherry picked from commit ad05732)

* fix(model/helper): represent RLS filter clause in proper textual SQL string (apache#32406)

Signed-off-by: hainenber <[email protected]>
(cherry picked from commit ff0529c)

* fix: Log table retention policy (apache#32572)

(cherry picked from commit 89b6d7f)

* fix(welcome): perf on distinct recent activities (apache#32608)

(cherry picked from commit 832e028)

* fix(log): Update recent_activity by event name (apache#32681)

(cherry picked from commit 449f51a)

* fix: Signature of Celery pruner jobs (apache#32699)

(cherry picked from commit df06bdf)

* fix(logging): missing path in event data (apache#32708)

(cherry picked from commit cd5a943)

* fix(fe/dashboard-list): display modifier info for `Last modified` data (apache#32035)

Signed-off-by: hainenber <[email protected]>
(cherry picked from commit 88cf2d5)

* fix: make packages PEP 625 compliant (apache#32866)

Co-authored-by: Michael S. Molina <[email protected]>
(cherry picked from commit 6e02d19)

* all cccs changes

* fix: Downgrade to marshmallow<4 (apache#33216)

* fix(log): store navigation path to get correct logging path (apache#32795)

(cherry picked from commit 4a70065)

* fix(pivot-table): Revert "fix(Pivot Table): Fix column width to respect currency config (apache#31414)" (apache#32968)

(cherry picked from commit a36e636)

* fix: improve error type on parse error (apache#33048)

(cherry picked from commit ed0cd5e)

* fix(plugin-chart-echarts): remove erroneous upper bound value (apache#32473)

(cherry picked from commit 5766c36)

* fix(pinot): revert join and subquery flags (apache#32382)

(cherry picked from commit 822d72c)

* fix: loading examples from raw.githubusercontent.com fails with 429 errors (apache#33354)

(cherry picked from commit f045a73)

* chore: creating 4.1.3rc1 change log and updating frontend json

(cherry picked from commit 72cf9b6)

* chore(🦾): bump python sqlglot 26.1.3 -> 26.11.1 (apache#32745)

Co-authored-by: GitHub Action <[email protected]>
(cherry picked from commit 66c1a6a)

* chore(🦾): bump python h11 0.14.0 -> 0.16.0 (apache#33339)

Co-authored-by: GitHub Action <[email protected]>
(cherry picked from commit 8252686)

* docs: CVEs fixed on 4.1.2 (apache#33435)

(cherry picked from commit 8a8fb49)

* feat(api): Added uuid to list api calls (apache#32414)

(cherry picked from commit 8decc9e)

* fix(table-chart): time shift is not working (apache#33425)

(cherry picked from commit dc44748)

* fix(Sqllab):  Autocomplete got stuck in UI when open it too fast (apache#33522)

(cherry picked from commit b4e2406)

* chore: update Dockerfile - Upgrade to 3.11.12 (apache#33612)

(cherry picked from commit f0b6e87)

* chore: updating 4.1.3rc2 change log

* Select all Drag and Drop (#546)

* add a select all button for the dnd select

* remove cypress

* chore(deps): bump cryptography from 43.0.3 to 44.0.1 (apache#32236)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit fa09d81)

* fix: Adds missing __init__ file to commands/logs (apache#33059)

(cherry picked from commit c1159c5)

* fix: Saved queries list break if one query can't be parsed (apache#34289)

(cherry picked from commit 1e5a4e9)

* chore: Adds 4.1.4RC1 data to CHANGELOG.md and UPDATING.md

* tag bump for select all drag and drop

* Fix package-lock.json

* Add db migration, bump Docker image base

* gevent for gunicorn

* remove threads and make worker-connections configurable

* Fix package-lock.json

* tag bump for cccs build

* Remove CCCS Dataset Explorer (#550)

* tag bump for CCCS build

---------

Signed-off-by: hainenber <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: gpchandran <[email protected]>
Co-authored-by: Joe Li <[email protected]>
Co-authored-by: Jack <[email protected]>
Co-authored-by: Đỗ Trọng Hải <[email protected]>
Co-authored-by: Michael S. Molina <[email protected]>
Co-authored-by: JUST.in DO IT <[email protected]>
Co-authored-by: Michael S. Molina <[email protected]>
Co-authored-by: Andreas Motl <[email protected]>
Co-authored-by: Ville Brofeldt <[email protected]>
Co-authored-by: Yuri <[email protected]>
Co-authored-by: Maxime Beauchemin <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: GitHub Action <[email protected]>
Co-authored-by: sha174n <[email protected]>
Co-authored-by: Paul Rhodes <[email protected]>
Co-authored-by: Rafael Benitez <[email protected]>
Co-authored-by: cccs-RyanK <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: cyber-jessie <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels doc:examples Related to example datasets and dashboards size/M v4.1 Label added by the release manager to track PRs to be included in the 4.1 branch v5.0 Label added by the release manager to track PRs to be included in the 5.0 branch 🍒 4.1.3 Cherry-picked to 4.1.3 🍒 4.1.4 🍒 5.0.0 Cherry-picked to 5.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants