Skip to content

refactor: bb.js non-inlined web workers#13736

Merged
Thunkar merged 8 commits intomasterfrom
gj/bb_js_web_workers
Apr 22, 2025
Merged

refactor: bb.js non-inlined web workers#13736
Thunkar merged 8 commits intomasterfrom
gj/bb_js_web_workers

Conversation

@Thunkar
Copy link
Contributor

@Thunkar Thunkar commented Apr 22, 2025

Inlined web workers are completely unsupported in browser extensions, which created problems for external teams. This PR moves to a more modern approach, leveraging the integrated support in latest versions of webpack for web workers.

Unfortunately, I've run into quite a few problems implementing this approach:

  • Webpack didn't support proper module workers up until recently, forcing me to update it (so it wouldn't transpile the imports to the unsupported importScripts). Took the opportunity to equalize versions accross the board.
  • There's a bug in webpack that forbids loading them asynchronously and threw me for a while: Generated code not run when outputModule: true and entry chunk dependent on common chunk webpack/webpack#17014. Solution is to carefully handle our dynamic imports so they're only applied to the wasm files.
  • We were using worker-loader in webpack 5, which is explicitly deprecated, but we needed it for the inlining. This gets rid of it.
  • Unfortunately again, new webpack handles .d.ts and .d.ts.map files differently now, which was causing trouble for downstream apps that also used webpack. Test apps and boxes have been updated to ignore them at build time (since they're only needed during development). Vite has no issues.
  • Took the opportunity to clean up our API, since we were never initializing the SyncApiwith a worker and conversely, the Async API with a plain wasm module.

@Thunkar Thunkar requested review from ludamad, nventuro and saleel April 22, 2025 15:30
@Thunkar Thunkar self-assigned this Apr 22, 2025
nventuro
nventuro approved these changes Apr 22, 2025
View reviewed changes
barretenberg/ts/webpack.config.js
Comment on lines +20 to +22
// // Force inclusion of worker threads without mangling worker import statements.
// main: { import: './src/barretenberg_wasm/barretenberg_wasm_main/factory/browser/main.worker.ts', filename: 'main.worker.js' },
// thread: { import: './src/barretenberg_wasm/barretenberg_wasm_thread/factory/browser/thread.worker.ts', filename: 'thread.worker.js' },
Copy link
Contributor

@nventuro nventuro Apr 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to be deleted?

Copy link
Contributor Author

@Thunkar Thunkar Apr 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I'll follow this up with some more cleanup I wanted to do

@nventuro nventuro added the ci-full Run all master checks. label Apr 22, 2025
@Thunkar Thunkar added this pull request to the merge queue Apr 22, 2025
@ledwards2225
Copy link
Contributor

ledwards2225 commented Apr 22, 2025

@Thunkar Flagging this as it appears to have made some "flaky" tests fail consistently: https://aztecprotocol.slack.com/archives/C053490AV6V/p1745358607367319

Merged via the queue into master with commit 0e604a1 Apr 22, 2025
14 checks passed
@Thunkar Thunkar deleted the gj/bb_js_web_workers branch April 22, 2025 22:22
@AztecBot AztecBot mentioned this pull request Apr 22, 2025
Merged
github-merge-queue bot pushed a commit that referenced this pull request Apr 24, 2025
@AztecBot
chore(master): release 0.86.0 (#13579)
2d8e0b5 
🤖 I have created a new Aztec Packages release
---


##
[0.86.0](v0.85.0...v0.86.0)
(2025-04-24)


### ⚠ BREAKING CHANGES

* Use combined p2p and http prover coordination
([#13760](#13760))
* use single extended viem client
([#13715](#13715))
* remove slice read from CALL
([#13729](#13729))

### Features

* `#[allow(dead_code)]` (noir-lang/noir#8066)
([366d980](366d980))
* add aztec-up amm test w/ proving & sponsoredfpc
([#13209](#13209))
([f54c8a1](f54c8a1))
* Add Rollup to registry and produce blocks
([#13556](#13556))
([9ed10d3](9ed10d3))
* adding devnet tests
([#12808](#12808))
([7687a1f](7687a1f))
* audit tracking tooling
([#13639](#13639))
([e50d8e0](e50d8e0))
* **avm:** fast entity indexing without macros
([#13737](#13737))
([5c8a993](5c8a993))
* **avm:** quick n dirty memory trace
([#13659](#13659))
([062c6a9](062c6a9))
* **avm:** support col arrays and add keccak cols
([#13711](#13711))
([9c4345a](9c4345a))
* **avm:** tagged value type in C++
([#13540](#13540))
([b30b5b3](b30b5b3))
* avoid unnecessary zero check in brillig overflow check
(noir-lang/noir#8109)
([366d980](366d980))
* **contracts:** static + snapshotted validator set
([#13046](#13046))
([90033f2](90033f2))
* enable mempool limiter on devnet
([#13722](#13722))
([36fb0a1](36fb0a1))
* exec opcode spec table
([#13594](#13594))
([d4d503a](d4d503a))
* **experimental:** Implement separate `-Zownership` analysis for
ownership pass (noir-lang/noir#7861)
([366d980](366d980))
* expose mempool limit through helm chart
([#13616](#13616))
([822e8d9](822e8d9))
* Garaga UltraStarknet[Zk]Honk flavours
([#11489](#11489))
([6bc34a1](6bc34a1))
* IPA claim as public component
([#13429](#13429))
([1ce3c43](1ce3c43))
* no longer recompute vk's in CIVC proofs
([#13590](#13590))
([66a61ba](66a61ba))
* **p2p:** add private peers
([#12585](#12585))
([4264c8c](4264c8c))
* **p2p:** optional P2P_BROADCAST_PORT
([#13525](#13525))
([f709fab](f709fab))
* Port callstack resolution from aztec to noirjs
(noir-lang/noir#7642)
([366d980](366d980))
* Refactor IPA claim handling in acir format to support them for AVM
([#13547](#13547))
([df73c05](df73c05))
* remove slice read from CALL
([#13729](#13729))
([7e7eb85](7e7eb85))
* replace field divisions by constants with multiplication by inv…
(noir-lang/noir#8053)
([366d980](366d980))
* report world state size on disk
([#13706](#13706))
([0dd8a7e](0dd8a7e))
* skip inverting field elements with values +-1 in ACVM
(noir-lang/noir#8049)
([44c1347](44c1347))
* SMT Verificaiton Module: Data Structures
([#13658](#13658))
([2de3bc5](2de3bc5))
* SMT verification module updates
([#13551](#13551))
([f02123d](f02123d))
* ssa fuzzer (noir-lang/noir#7641)
([366d980](366d980))
* Tighter timing on ACVM
([#13743](#13743))
([45cd39b](45cd39b))
* track rewards and slots
([#13546](#13546))
([489c6cf](489c6cf))
* Use combined p2p and http prover coordination
([#13760](#13760))
([4d04e62](4d04e62))
* VK generation test for HonkRecursionConstraint
([#13637](#13637))
([71e81ce](71e81ce))


### Bug Fixes

* **acir:** Check whether opcodes were laid down for non-equality check
before fetching payload locations
(noir-lang/noir#8133)
([366d980](366d980))
* add proper handling for `u128` in comptime interpreter
(noir-lang/noir#8079)
([366d980](366d980))
* Add version to gossipsub protocol
([#13567](#13567))
([8ce4829](8ce4829))
* allow keywords in function names in SSA parser
(noir-lang/noir#8063)
([44c1347](44c1347))
* Allow more slack in AST calibration for CI
(noir-lang/noir#8076)
([366d980](366d980))
* amm bot
([#13553](#13553))
([6b93832](6b93832)),
closes
[#13544](#13544)
* **avm:** cpp addressing
([#13652](#13652))
([9d941b6](9d941b6))
* blob sink waits for reg address when needed
([#13628](#13628))
([486a42a](486a42a))
* **brillig:** SliceRefCount reads from the appropriate pointer
(noir-lang/noir#8148)
([366d980](366d980))
* Check against multiple aws profiles in build instance credentails
([#13421](#13421))
([71a408a](71a408a))
* dependency cycles in public simulator - part 0 (sim -> context)
([#13678](#13678))
([e00089f](e00089f))
* dependency cycles in public simulator - part 1 (errors/revertReason)
([#13679](#13679))
([c75e365](c75e365))
* dependency cycles in public simulator part 2 (serializable bytecode)
([#13680](#13680))
([5d3e24c](5d3e24c))
* discv5 test failure
([#13653](#13653))
([58c143b](58c143b))
* **docs:** Make viewing (outdated) protocol specs work locally
([#13534](#13534))
([694537d](694537d))
* Increase timeout for p2p integration test
([#13720](#13720))
([c244b2e](c244b2e))
* mac bb --version
([#13690](#13690))
([9f8b64f](9f8b64f))
* make translator use ultra rather than eccvm ops
([#13489](#13489))
([53c070d](53c070d))
* more worker fixes
([#13759](#13759))
([080485a](080485a))
* move helm lint to build
([#13532](#13532))
([7768ff8](7768ff8))
* native ivc benches not publishing
([#13665](#13665))
([7ab712d](7ab712d))
* no exports of simulator should depend on jest-mock-extended
([#13694](#13694))
([65a9f33](65a9f33))
* only clear trailing zeros on logs
([#13727](#13727))
([e392d7c](e392d7c))
* **p2p:** better batch connection sampling
([#13674](#13674))
([9c1d9f1](9c1d9f1))
* **p2p:** reqresp types + batch request tx pool filtering
([#13666](#13666))
([8e2a3c9](8e2a3c9))
* **parser:** error on missing let semicolon in trait (and others)
(noir-lang/noir#8101)
([366d980](366d980))
* pass along coinbase
([#13560](#13560))
([593f810](593f810)),
closes
[#13643](#13643)
* prover-node start
([#13627](#13627))
([5d87f7a](5d87f7a))
* quote numbers
([#13654](#13654))
([f0d1ec1](f0d1ec1))
* remove all txs from a failed epoch
([#13771](#13771))
([abc4623](abc4623))
* remove insecure dummy round derivation from sumcheck and shplemini
([#13488](#13488))
([9a3bb46](9a3bb46))
* replace values in data_bus too
(noir-lang/noir#8086)
([366d980](366d980))
* retry deploy npm
([#13691](#13691))
([76cab3f](76cab3f))
* run yarn
([#13713](#13713))
([0e51259](0e51259))
* squash nightly foundry warning
([#13640](#13640))
([e21f9fd](e21f9fd))
* SSA pass print filter to include the count
(noir-lang/noir#8074)
([366d980](366d980))
* **ssa:** Do not inline simple recursive functions
(noir-lang/noir#8127)
([366d980](366d980))
* **ssa:** Do not unroll loop with break
(noir-lang/noir#8090)
([366d980](366d980))
* **ssa:** Do not use inner else condition when collapsing IfElse based
upon the then condition (noir-lang/noir#8040)
([44c1347](44c1347))
* **ssa:** fix possibility to `Field % Field` operaions in Brillig from
SSA (noir-lang/noir#8105)
([366d980](366d980))
* **ssa:** Loop range with u1
(noir-lang/noir#8131)
([366d980](366d980))
* update metric name to avoid conflicts
([#13629](#13629))
([70c58ab](70c58ab)),
closes
[#13626](#13626)
* wasm memory benchmark
([#13573](#13573))
([9a73c4a](9a73c4a))
* wrapping mul support for u128
(noir-lang/noir#7941)
([366d980](366d980))


### Miscellaneous

* Accept optional test path for emitting integration tests
(noir-lang/noir#8062)
([366d980](366d980))
* Add `GITHUB_TOKEN` to more workflow steps
(noir-lang/noir#8046)
([44c1347](44c1347))
* add a benchmark for opcodes which need a batchable inversion
(noir-lang/noir#8110)
([366d980](366d980))
* add hacky faster bootstrap for bb-centric e2e flow
([#13587](#13587))
([498d433](498d433))
* address some visibility warnings
([#13728](#13728))
([35dec90](35dec90))
* **avm:** less codegen for lookups
([#13741](#13741))
([ec468d9](ec468d9))
* **avm:** move interaction jobs to trace builders
([#13621](#13621))
([caac1c9](caac1c9))
* bb.js non-inlined web workers
([#13736](#13736))
([0e604a1](0e604a1))
* **bb:** debugging helpers
([#13584](#13584))
([0ebb29e](0ebb29e))
* **bb:** Make goblin a proper source module
([#13580](#13580))
([3048a14](3048a14))
* bespoke export for client native prover / PXE server store lazy load
([#13783](#13783))
([640dd08](640dd08))
* Blob and archiver syncing improvements
([#13542](#13542))
([629aa2a](629aa2a)),
closes
[#13530](#13530)
* blob archive client factory code
([#13648](#13648))
([7b16791](7b16791))
* bug list
([#13773](#13773))
([c6d4fb8](c6d4fb8))
* bump axois
([#13453](#13453))
([1c20920](1c20920))
* bump bb to 0.84.0 (noir-lang/noir#8047)
([44c1347](44c1347))
* bump bignum timeout (noir-lang/noir#8080)
([366d980](366d980))
* bump external pinned commits
(noir-lang/noir#8045)
([44c1347](44c1347))
* **ci:** run non dependent bootstrap build steps in parallel
([#13454](#13454))
([b17e18d](b17e18d))
* cleanup aztec up amm test
([#13754](#13754))
([3d92afe](3d92afe))
* Cleanup scripts in package jsons in yarn-project
([#13527](#13527))
([a296945](a296945))
* **contracts:** core/staking -> core/slashing
([#13748](#13748))
([5d93a0e](5d93a0e))
* create module for array handling in acirgen
(noir-lang/noir#8119)
([366d980](366d980))
* dashboards use ${data_source} variable
([#13613](#13613))
([45533d9](45533d9))
* delete honk_recursion for building ACIR
([#13664](#13664))
([ac95729](ac95729))
* delete Ultra Vanilla CIVC
([#13669](#13669))
([ae578a2](ae578a2))
* delete zeromorph
([#13667](#13667))
([c8acae0](c8acae0))
* Disable blobscan by default
([#13742](#13742))
([c55088e](c55088e))
* **docs:** `aztec` and `aztec-wallet` cli reference updates
([#13692](#13692))
([d11d154](d11d154))
* **docs:** add 0.85.0 docs
([#13578](#13578))
([cf07358](cf07358))
* **docs:** add AND/XOR bug to bug list
([#13784](#13784))
([797ae6d](797ae6d))
* **docs:** Add architecture docs
(noir-lang/noir#7992)
([366d980](366d980))
* **docs:** adding redirects for node guide
([#13768](#13768))
([e96e5bf](e96e5bf))
* **docs:** bugfix + deploy on master
([#13717](#13717))
([010975a](010975a))
* **docs:** Fix some stuff on fees, rm confusing storage example
([#13501](#13501))
([c00b20b](c00b20b))
* **docs:** Sequencer and Prover Guide
([#13433](#13433))
([55b9a2b](55b9a2b))
* **docs:** update bb commands to match 0.84.0
(noir-lang/noir#8050)
([366d980](366d980))
* **docs:** Update docs bootrap to to include release
([#13615](#13615))
([2154d5e](2154d5e))
* don't use `set_from_value_id` in `remove_if_else`
(noir-lang/noir#8070)
([366d980](366d980))
* don't use `set_value_from_id` in `as_slice_length`
(noir-lang/noir#8039)
([44c1347](44c1347))
* don't use `set_value_from_id` in `constant_folding`
(noir-lang/noir#8091)
([366d980](366d980))
* don't use `set_value_from_id` in `loop_invariant`
(noir-lang/noir#8085)
([366d980](366d980))
* don't use `set_value_from_id` in `remove_bit_shifts`
(noir-lang/noir#8071)
([366d980](366d980))
* don't use `set_value_from_id` in `simplify_cfg`
(noir-lang/noir#8072)
([366d980](366d980))
* **experimental:** Function::simple_optimization for SSA optimizations
(noir-lang/noir#8102)
([366d980](366d980))
* Fetch rollup address using version as index
([#13620](#13620))
([60e73f9](60e73f9))
* Fix license files
([#13695](#13695))
([9931fb8](9931fb8))
* mempool limit
([#13735](#13735))
([def1287](def1287))
* mint block rewards for 200K blocks at deployment
([#13537](#13537))
([1a3a326](1a3a326))
* move check_circuit functionality from `TranslatorCircuitBuilder` into
a `TranslatorCircuitChecker`
([#13761](#13761))
([e4e7fca](e4e7fca))
* noir test flake addition
([#13663](#13663))
([c302fdd](c302fdd))
* **noir-contracts:** update readme
([#13563](#13563))
([42dfbbf](42dfbbf))
* **optimization:** Enable experimental ownership clone scheme by
default (noir-lang/noir#8097)
([366d980](366d980))
* parse nop in SSA parser (noir-lang/noir#8141)
([366d980](366d980))
* ping luke instead of cody on bb benchmark regressions
([#13718](#13718))
([2802f5a](2802f5a))
* playground improvements
([#13588](#13588))
([4c3453b](4c3453b))
* **playground:** bump vite
([#13750](#13750))
([7e72420](7e72420))
* Print `unsafe { ... }` around calls to Brillig from ACIR in AST
(noir-lang/noir#8077)
([366d980](366d980))
* **public/avm:** from hints to the end of the world
([#13459](#13459))
([a89de5d](a89de5d))
* redo typo PR by dizer-ti
([#13757](#13757))
([075d3a8](075d3a8))
* release playground on s3.
([#13582](#13582))
([2f9458a](2f9458a))
* **release:** give aztec release image sensible default entrypoint
([#13517](#13517))
([a864c06](a864c06))
* remove circuit simulator
([#13689](#13689))
([a00566c](a00566c))
* Remove code of the fake AVM recursive verifier
([#13614](#13614))
([048dfed](048dfed))
* remove msm sorter
([#13668](#13668))
([ff29d86](ff29d86))
* remove old terraform configs
([#13716](#13716))
([49ac1db](49ac1db)),
closes
[#13651](#13651)
* remove omit param from serialize derivation
([#13703](#13703))
([6af5943](6af5943))
* remove pcs vk from vk
([#13638](#13638))
([0e941ef](0e941ef))
* remove try_merge_only_changed_indices
(noir-lang/noir#8142)
([366d980](366d980))
* replace relative paths to noir-protocol-circuits
([78c87b5](78c87b5))
* replace relative paths to noir-protocol-circuits
([4e53cd1](4e53cd1))
* replace relative paths to noir-protocol-circuits
([3a33c47](3a33c47))
* replace relative paths to noir-protocol-circuits
([eaaa85b](eaaa85b))
* replace relative paths to noir-protocol-circuits
([648a4fe](648a4fe))
* replace relative paths to noir-protocol-circuits
([b3f4745](b3f4745))
* replace relative paths to noir-protocol-circuits
([269b316](269b316))
* replace relative paths to noir-protocol-circuits
([1330efe](1330efe))
* replace relative paths to noir-protocol-circuits
([a739284](a739284))
* Sanity check block number from archiver before returning it
([#13631](#13631))
([e459b2a](e459b2a))
* simpler `make_mutable` in `array_set` optimization
(noir-lang/noir#8106)
([366d980](366d980))
* **spartan:** bump default kind resources
([#13747](#13747))
([8ca828f](8ca828f))
* **ssa:** Additional unrolling and mem2reg passes
(noir-lang/noir#8017)
([44c1347](44c1347))
* **ssa:** Remove IfElse pass basic SSA tests
(noir-lang/noir#8058)
([44c1347](44c1347))
* **ssa:** Test terminator value constant folding and resolve cache for
data bus (noir-lang/noir#8132)
([366d980](366d980))
* starknet feature flag
([#13681](#13681))
([2b6c627](2b6c627))
* stop prover node swallowing start prover job errors
([#13676](#13676))
([90c2b7b](90c2b7b))
* **test:** AST generation calibration test
(noir-lang/noir#8054)
([44c1347](44c1347))
* update ACVM doc (noir-lang/noir#8004)
([366d980](366d980))
* Update docs snippet
([#13739](#13739))
([7fb333a](7fb333a))
* update staking asset handler address
([#13566](#13566))
([daa09f2](daa09f2))
* Updated contract addresses for alpha-testnet
([#13585](#13585))
([9f0ff4a](9f0ff4a))
* Use chain monitor to sync system time in p2p tests
([#13632](#13632))
([abbad4c](abbad4c))
* use public component key for pairing inputs
([#13705](#13705))
([467166f](467166f))
* use PublicComponentKeys
([#13686](#13686))
([68e4b6f](68e4b6f))
* use single extended viem client
([#13715](#13715))
([cf637dc](cf637dc))
* validate block header hash in circuits
([#13094](#13094))
([f19c182](f19c182))
* wasm proof verifying with native bb
([#13499](#13499))
([1ad9249](1ad9249))


### Documentation

* fees tutorial
([#13357](#13357))
([c580657](c580657))
* move snippets all versions
([#13650](#13650))
([a2f5553](a2f5553))
* Update glossary & snippets
([#13661](#13661))
([d55816d](d55816d))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: AztecBot <tech@aztecprotocol.com>
Thunkar added a commit that referenced this pull request Apr 28, 2025
@Thunkar
refactor: bb.js non-inlined web workers (#13736)
893e9d2 
Inlined web workers are completely unsupported in browser extensions,
which created problems for external teams. This PR moves to a more
modern approach, leveraging the integrated support in latest versions of
`webpack` for web workers.

Unfortunately, I've run into quite a few problems implementing this
approach:

* Webpack didn't support proper module workers up until recently,
forcing me to update it (so it wouldn't transpile the imports to the
unsupported `importScripts`). Took the opportunity to equalize versions
accross the board.
* There's a bug in webpack that forbids loading them asynchronously and
threw me for a while: webpack/webpack#17014.
Solution is to carefully handle our dynamic imports so they're only
applied to the wasm files.
* We were using `worker-loader` in `webpack` 5, which is explicitly
deprecated, but we needed it for the inlining. This gets rid of it.
* Unfortunately again, new webpack handles `.d.ts` and `.d.ts.map` files
differently now, which was causing trouble for downstream apps that also
used webpack. Test apps and boxes have been updated to ignore them at
build time (since they're only needed during development). Vite has no
issues.
* Took the opportunity to clean up our API, since we were never
initializing the `SyncApi`with a worker and conversely, the `Async` API
with a plain wasm module.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nventuro nventuro nventuro approved these changes

@ludamad ludamad Awaiting requested review from ludamad

@saleel saleel Awaiting requested review from saleel

Assignees

@Thunkar Thunkar

Labels

ci-full Run all master checks.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Thunkar @ledwards2225 @nventuro