wp-build dashboards: throttle i18n catalog downloads and load widget catalogs on demand

[dhasilva]

Fixes # N/A (reported in WOOA7S-1822, follow-up to #50762 — see this comment)

Proposed changes

On any non-English locale, loadI18nCatalogs() issued one translation-catalog request per bundle in the build's i18n manifest — 100 requests for the current Premium Analytics build — all in one unthrottled loop at boot. On HTTP/2 origins nothing caps that burst, and a slow origin serves nothing else until it drains (measured at ~48s to first render in WOOA7S-1822). Widget catalogs (84 of the 100) were "non-blocking" only in the sense of not being awaited; they were still all requested for widgets that may never render.

• wp-build-polyfills: loadI18nCatalogs() now downloads only route/module/script catalogs at boot, through a 6-slot concurrency queue, so the burst can never starve the origin regardless of protocol. Widget bundles are no longer requested at boot. A new loadBundleI18nCatalog( domain, bundlePath, timeoutMs? ) export downloads a single bundle's catalog on demand — it checks manifest membership (bundles without gettext calls are skipped without a request), dedupes repeat requests, and shares the queue. Shared state lives on window, since each wp-build bundle inlines its own copy of the helper.
• premium-analytics: each widget bundle's catalog now loads as part of importing that bundle. Render modules go through a custom resolveWidgetModule on WidgetDashboard; metadata (widget.js) records are gated in a useWidgetTypes wrapper until their catalogs are installed. Both are awaited (bounded) before the import because widget bundles evaluate __() at module scope — a catalog that arrives after evaluation leaves those strings untranslated for the session.

The other five packages sharing loadI18nCatalogs (videopress, forms, activity-log, newsletter, publicize, backup) have no widget bundles; they pick up the boot throttle with no behavior change.

Measured (local Docker site via a tunnel, pt_BR, one full dashboard load each)

Metric	trunk	this PR
Catalog requests (+ manifest)	100 + 1	56 + 1 (−44%)
Catalog drain window	~23s	~17s
Last dashboard REST data request	page+33s	page+23s
Requests when switching to the Insights tab	0 (all pre-paid at boot)	9 (only newly rendered widgets)

Note: over HTTP/1.1 the browser's 6-connections-per-host cap already serializes the old burst, which hides the pathology locally. The unthrottled burst manifests on HTTP/2 (as in WOOA7S-1822), where the browser multiplexes all 100 requests at once; the new queue bounds concurrency in both cases.

Related product discussion/links

• WOOA7S-1822
• Follow-up to wp-build dashboards: translate JavaScript strings (load catalogs + stamp text domains) #50762

Does this pull request change what data or activity we track or use?

No.

Testing instructions

1. On a Jetpack site with Premium Analytics, set the site (or user) language to any non-en_US locale, e.g. Español.
2. Open DevTools → Network, filter by languages, and load wp-admin/admin.php?page=jetpack-premium-analytics-wp-admin.
3. Expect roughly 55–70 catalog requests (16 route/module catalogs plus one per widget module the dashboard actually imports), with at most 6 in flight at a time — instead of ~100 issued simultaneously right after boot.
4. Switch to a dashboard tab whose widgets haven't rendered yet (e.g. Insights): a small batch of new catalog requests should fire, one per newly rendered widget.
5. Check the console: there should be no [jetpack-i18n] warnings.
6. Requests 404ing is expected on a test site — no language packs exist for these bundles yet, and the UI falls back to English. On en_US no catalog requests should fire at all.

JS tests: jp test js packages/wp-build-polyfills and jp test js packages/premium-analytics.

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ Add a "[Status]" label (In Progress, Needs Review, ...).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

Follow this PR Review Process:

1. Ensure all required checks appearing at the bottom of this PR are passing.
2. Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
3. You can use GitHub's Reviewers functionality to request a review.
4. When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

[github-actions]

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

• To test on WoA, go to the Plugins menu on a WoA dev site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin (Jetpack or WordPress.com Site Helper), and enable the fix/wp-build-i18n-catalog-burst branch.
• To test on Simple, run the following command on your sandbox:

bin/jetpack-downloader test jetpack fix/wp-build-i18n-catalog-burst

bin/jetpack-downloader test jetpack-mu-wpcom-plugin fix/wp-build-i18n-catalog-burst

Interested in more tips and information?

• In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
• Read more about our development workflow here: PCYsg-eg0-p2
• Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

[jp-launch-control]

Code Coverage Summary

Coverage changed in 1 file.

File	Coverage	Δ%	Δ Uncovered
projects/packages/wp-build-polyfills/src/js/load-i18n-catalogs.ts	471/488 (96.52%)	-3.48%	17 💔

1 file is newly checked for coverage.

File	Coverage
projects/packages/premium-analytics/routes/widget-module-i18n.ts	30/31 (96.77%) 💚

Full summary · PHP report · JS report

_{If appropriate, add one of these labels to override the failing coverage check: Covered by non-unit tests Use to ignore the Code coverage requirement check when E2Es or other non-unit tests cover the code Coverage tests to be added later Use to ignore the Code coverage requirement check when tests will be added in a follow-up PR I don't care about code coverage for this PR Use this label to ignore the check for insufficient code coveage.}

[chihsuan]

Thanks for the fix! @dhasilva I tested this locally and the dashboard renders fine. Confirmed catalog requests dropped from 100 to 57, peak concurrency never exceeded 6, and the queue always drained.

There's still some delay before the grid fills in, and a batch of [jetpack-i18n] "still pending" warnings, but both look expected rather than regressions to me.

I think the residual cost really wants an upstream fix rather than more work in this PR. And useWidgetTypes looks worth improving too. cc @retrofox

So nothing blocking except one small thing I'd like before merge: the missing .catch() on the metadata gate, whose failure mode is a dashboard that never finishes loading. The rest are docs and wording. Thanks!

[chihsuan]

Could we add a .catch() here to fall back to English if this ever rejects?

[dhasilva]

Added in b3ab026 — .catch( () => undefined ) before the .then(), so the gate opens either way.

For the record it should never fire: every rejection path inside preloadWidgetModuleCatalogs is already sealed (the manifest promise and downloadOnce both carry their own .catch, and each load is bounded). But the failure mode if that ever changes is a grid stuck on placeholders with nothing in the console, so the guard is worth the line.

Also covered it with a test that rejects the catalog load and asserts the records still reach useWidgetTypes.

Generated by Claude.

[chihsuan]

Small one: docblock still says widget catalogs are "downloaded in the background and never awaited"

[dhasilva]

Fixed in b3ab026. It now reads:

Widget catalogs (<build>/widgets/**) are not requested here at all; they load through loadBundleI18nCatalog() when a widget's module is imported.

Generated by Claude.

[chihsuan]

nit: could we move these imports under Internal dependencies?

[dhasilva]

Moved in b3ab026. This was a regression from this PR — the new import landed above the Internal dependencies header and dragged useDashboardGridSettings up with it, orphaning the comment that explains why video-detail reuses the dashboard's hook. Both are back under the header, with the comment sitting directly above the import it describes.

Generated by Claude.

[LiamSarsfield]

Boot half looks right, widget bundles genuinely aren't requested now, not just un-awaited.

Small stuff first, mostly one-liners:

• preloadWidgetModuleCatalogs( records ).then( ... ) has no .catch and no bound, so only success opens the gate. One rejected catalog strands the grid on placeholders for the page.
• path.includes( '/widgets/' ) should be path.startsWith( 'build/widgets/' ). build/routes/widgets/foo.js gets skipped at boot and the anchored on-demand regex can't recover it.
• throttled() could use Promise.resolve().then( task ).finally( () => releaseSlot( state ) ) and lose both release branches.
• loadI18nCatalogs()'s JSDoc still promises background widget downloads.
• A repeat loadI18nCatalogs() publishes its manifest promise before resolving, so a failed refetch swaps a good manifest for an empty set and later widgets skip their catalogs quietly.
• The gate has no real coverage. I set gatedRecords = records to open it unconditionally and all 9 tests still passed. Upstream useWidgetTypes starts resolving on its own, so that assertion holds either way. Mock it and assert null, then the exact array.

Worth flagging: CATALOG_TIMEOUT_MS = 5000 and the Promise.race predate this PR, and the ticket measured 48 seconds with them active. So the gain appears to be the concurrency cap, not first paint.

On the local numbers above, 100 to 57 is the part worth pulling on. The ticket's split was 84 widget bundles and 16 route/module/script bundles, so a load with widget catalogs genuinely deferred should land near 16. 57 means roughly 41 came back anyway, because preloadWidgetModuleCatalogs gates the whole registry rather than the widgets actually on screen. That's very likely the same delay reported before the grid fills, and it's new: before this PR useWidgetTypes got its records straight away. Agreed the clean fix is upstream in useWidgetTypes so it isn't blocking here, but it's worth a follow-up issue rather than losing it.

Two things worth checking while the setup is still around:

• Were the "still pending" warnings from the boot path or the per-bundle one? Per-bundle means those widgets rendered untranslated, which is a bit more than log noise.
• Was the local site the MAMP and Jurassic Tube one from the ticket? On a fast origin the burst drains either way, so "queue always drained" wouldn't tell us much about the case in the report.

[dhasilva]

@LiamSarsfield — all six addressed in b3ab026.

.catch() on the gate. Added, plus a test that rejects the catalog load and asserts the records still reach useWidgetTypes. On "no bound": each loadBundleI18nCatalog call carries METADATA_CATALOG_TIMEOUT_MS (15s) and they all start in the same synchronous map, so the Promise.all is bounded at ~15s. The missing .catch() was the real gap.

path.includes( "/widgets/" ). Now an anchored /^[^/]+\/widgets\//. Anchored rather than a literal build/widgets/ because the manifest prefix is path.basename( buildDir ), not a constant. Latent today (no route named widgets), but exactly as you describe: skipped at boot and unrecoverable on demand, since the resolvers construct build/widgets/<name>/<file>.js themselves. Covered with a test asserting build/routes/widgets/content.js and build/scripts/widgets/index.js still download at boot while build/widgets/** does not.

throttled(). Now resolve( task().finally( () => releaseSlot( state ) ) ) — one release branch, and resolve() adopting the promise keeps rejections reaching the caller.

Stale JSDoc. Fixed.

Manifest clobber. The fetch is now deduped per domain: loadI18nCatalogs() reuses a cached manifest promise instead of overwriting it. The manifest is a static build artifact, so a repeat boot has nothing to gain from refetching and everything to lose if it fails. Test asserts a repeat call with a failing fetch keeps the good bundle set and issues no second request.

Gate coverage. You were right, and the reason is worse than it looks: upstream useWidgetTypes initialises isResolvingWidgetTypes to true and only clears it in an async .then(), so result.current[ 1 ] === true on first render was guaranteed by upstream regardless of what we passed it. The test asserted a value the code under test could not influence.

Rewrote it to assert the contract directly — what argument the gate hands useWidgetTypes, and when — with the hook mocked as a synchronous stand-in (records => [ [], ! records ]) so upstream's async resolution stops being a confound. Three tests now: null while pending then the exact array by identity, a new records array closing the gate again, and the rejection case. I ran your mutation (gatedRecords = records) against them and two fail.

On the residual 57 requests and the useWidgetTypes gating — agreed that is the real remaining cost and that the fix belongs upstream rather than here. Will open a follow-up issue so it does not get lost.

Generated by Claude.

[LiamSarsfield]

Thanks, b3ab026 covers everything I raised. Two small things left and then I'm happy.

resolveWidgetModuleWithI18n didn't get the .catch() that useWidgetTypesWithI18n did.
The comment above it says it never rejects, but if loadBundleI18nCatalog ever does, the .then() never runs and the widget doesn't import at all, which is worse than rendering it in English.

sharedState() also adopts whatever is already on window.__jetpackWpBuildI18nCatalogs without checking the shape. Two separately-inlined copies from different builds on the same page is enough to break that assumption, and then state.manifests.get() throws, loadI18nCatalogs() rejects, and runInitModules has no catch around it. That's a blank dashboard rather than an untranslated one. A shape check, or a try/catch around the body, would keep the "always falls back to English" promise true. slots for the rest of the page. downloadI18n has no AbortSignal so it isn't really fixable from here, but it'd be worth folding into the same follow-up as the registry gate.

[dhasilva]

@LiamSarsfield — both fixed in 3c4e906.

resolveWidgetModuleWithI18n. Right, and the asymmetry was worse than it looked from the outside: your two findings are the same failure chained. sharedState() is the only synchronous throw site in the loader, and loadBundleI18nCatalog() calls it — so the missing .catch() here was the propagation path for exactly the shape mismatch you describe below. Guarded the same way the metadata gate is, and the comment above it no longer claims it never rejects.

sharedState() shape. Confirmed the blast radius before fixing: eight init modules across the wp-build dashboards (premium-analytics, forms, backup, newsletter, videopress, publicize, activity-log, social) do a bare await loadI18nCatalogs( … ), and boot awaits those with no catch of its own. So a throw is a blank dashboard, as you said.

It now validates the parked value and, when it isn't recognized, falls back to state private to this copy rather than replacing what's there. Replacing looked simpler but is the wrong move: the copy that parked it is still reading the same global on every call, so the two would overwrite each other's state indefinitely and neither would end up with a usable dedupe map. Unshared state costs a few repeat downloads and splits the concurrency budget — both cheaper than the alternatives, and it warns once so the situation is visible.

I also took the second half of your suggestion rather than choosing between them: both exported helpers now catch around their own body. The "always resolves, worst case English" contract was true but argued — it held only as long as every internal path stayed individually sealed, which is precisely why this round found two more. It's structural now.

On coverage, given last round: both new tests fail without their fix (sharedState mutation → 1 fail of 86; the resolver mutation → 1 fail of 12). The shape test is deliberately the only one driving that path — the fallback state is module-level by design, and the helper module is cached across tests in the file, so a second one would inherit the first's download map and see no warning. There's a comment on the test saying so.

Follow-up: JETPACK-2095, covering both open items — the registry gate (100 → 57 measured, ~16 if only on-screen widgets loaded, so ~41 still arriving through the metadata gate, plus the grid delay that didn't exist before this PR) and the slot leak.

One pushback on the slot leak, for the record: I don't think it's unfixable from here. throttled() could release the slot on a timeout and let the orphaned request finish in the background — that fixes the deadlock without touching jetpack-assets, at the cost of the strict ≤6-in-flight guarantee (you'd get transient overshoot rather than a queue that never drains). Adding AbortSignal to downloadI18n() is the correct fix and cancels the wasted bytes too, but it's a cross-package change. Both options are written up in the ticket; no strong preference from me.

Generated by Claude.