feat(DRAFT): Save the noir-repo content hash after each sync to a file#12858
Closed
feat(DRAFT): Save the noir-repo content hash after each sync to a file#12858
noir-repo content hash after each sync to a file#12858Conversation
ludamad
reviewed
Mar 18, 2025
| @@ -0,0 +1 @@ | |||
| 5a2582f95f4f49a9 | |||
Collaborator
There was a problem hiding this comment.
We can probably get rid of the test distinction imho
Collaborator
|
For posterity: I don't like things relying on people committing files written out by bootstrap, we've gone through lengths to avoid that, hence asking for (for CI purposes) to just use the noir ref and patches |
ludamad
pushed a commit
that referenced
this pull request
Mar 18, 2025
…d `noir-repo.patch` (#12861) Alternative to #12858 Changes `noir/bootstrap.sh` to use `cache_content_hash .rebuild_patterns` if * we have specified an `AZTEC_COMMIT_HASH`, looking for historical values * there are no commits in `noir-repo` that could be added to a patch file, indicating changes that aren't captured by the `.rebuild_patterns` and the `noir-repo-ref` file, and * `noir-repo` isn't on a _branch_ which could evolve all the time; if it's on a branch then use the `.noir-repo.rebuild_patterns` in `noir-repo` itself to figure out the hash The exception for feature branches should not affect normal aztec workflow, which is based on _tags_, it's only there to support temporary work noir developers do across both repos at the same time, and it's not something that will be merged into the `master` branch of `aztec-packages` (it would defeat repeatable builds). With this change it should be possible to use `AZTEC_COMMIT_HASH` to query historical queries, as seen in the 2nd example below. ### Example 1 This shows that if we're on a tag, then adding a new commit to the noir-repo will disable caching. This is the normal case in `aztec-packages`. Otherwise the hash is based on just `noir`. But if we switch to a feature branch, caching is back on to support faster builds, incorporating the latest commits into the hash. ```console # We are on a tag % ./noir/scripts/sync.sh info Repo exists: yes Fixup commit: 9dcbd3fbf47f886821ac71a647d4f9712ab1eb1d Checkout commit: 45ad637273cef317eba42feaf2be0e59d34065ed Wanted: nightly-2025-03-18 Needs switch: no Needs patch: no Detached: yes On branch: no Branch name: n/a Has wanted tag: yes Has tag commit: yes Has patch commit: yes Last commit is patch: yes Has fixup and patch: yes Has uncommitted changes: no Latest nightly: nightly-2025-03-18 Cache mode: noir % ./noir/scripts/sync.sh cache-mode noir % ./noir/bootstrap.sh hash eb08d2624603de97 # Make a change in the noir-repo % cd noir/noir-repo % echo "Foo" > compiler/foo.txt && git add compiler/foo.txt && git commit -m "Foo" [detached HEAD 23d1d2ac6b] Foo 1 file changed, 1 insertion(+) create mode 100644 compiler/foo.txt % cd ../.. # The extra commit disables the cache % ./noir/scripts/sync.sh cache-mode disabled-cache % ./noir/bootstrap.sh hash disabled-cache # Now switch to a feature branch % echo af/msgpack-codegen > noir/noir-repo-ref % ./noir/bootstrap.sh hash Error: noir-repo is on a detached HEAD and the last commit is not the patch marker commit; switching to af/msgpack-codegen could mean losing those commits. Please use the 'make-patch' command to create a noir-repo.patch file and commit it in aztec-packages, so that it is re-applied after each checkout. Make sure to commit the patch on the branch where it should be. # Get rid of the foo commit, so we can switch away % cd noir/noir-repo % git reset --hard HEAD~1 HEAD is now at 8b88883d58 Noir local patch commit. % cd ../.. # Hashing still involves syncing; we need the noir-repo to decide how to hash % ./noir/bootstrap.sh hash remote: Enumerating objects: 187, done. ... Switched to branch 'af/msgpack-codegen' Your branch is up to date with 'upstream/af/msgpack-codegen'. ... [af/msgpack-codegen 98f9a3cc43] Noir local fixup commit. 4 files changed, 108 insertions(+), 2 deletions(-) create mode 100755 acvm-repo/acvm_js/build.sh.bak create mode 100755 tooling/noirc_abi_wasm/build.sh.bak Applying: patch: delete honk example programs Applying: chore: turn on `skipLibCheck` Applying: chore: delete leftover file Applying: Ignore package.tgz [af/msgpack-codegen 182331696a] Noir local patch commit. disabled-cache # ^ The cache is disabled becase we have a pending change in `noir/noir-repo-ref`, # but `cache-mode` indicates we need to cache based on the `noir-repo` contents. % ./noir/scripts/sync.sh cache-mode noir-repo # Commit the noir-repo-ref file, so we have a clean state in aztec-packages % git add noir/noir-repo-ref && git commit -m "Switched to a feature branch" Formatting barretenberg staged files... [af/noir-repo-ref-hash 62fc17a03b] Switched to a feature branch 1 file changed, 1 insertion(+), 1 deletion(-) # Now we hash based on the code in `noir-repo`, *and* the contents of `noir` % ./noir/bootstrap.sh hash 798ec85262e6133a ``` ### Example 2 This one shows that even after switching to a different tag, we can use the `AZTEC_CACHE_COMMIT` feature to go back in the commit log to get a historical hash. ```console # We're on a tag % git rev-parse HEAD 517f246 % cat noir/noir-repo-ref nightly-2025-03-18 % ./noir/bootstrap.sh hash bf9bcc2b4f6046c7 # Want to roll back to a previous nightly tag % echo nightly-2025-03-11 > noir/noir-repo-ref % git add noir % git commit -m "Roll back nightly" Formatting barretenberg staged files... [af/noir-repo-ref-hash 858d98f42b] Roll back nightly 1 file changed, 1 insertion(+), 1 deletion(-) # Recalculating the hash will sync the repo, but the hash will be based on the ref, not the content % ./noir/bootstrap.sh hash remote: Enumerating objects: 66481, done. remote: Counting objects: 100% (61479/61479), done. remote: Compressing objects: 100% (36681/36681), done. remote: Total 57993 (delta 21540), reused 52084 (delta 16927), pack-reused 0 (from 0) Receiving objects: 100% (57993/57993), 158.39 MiB | 33.13 MiB/s, done. Resolving deltas: 100% (21540/21540), completed with 557 local objects. remote: Enumerating objects: 23, done. remote: Counting objects: 100% (16/16), done. remote: Compressing objects: 100% (3/3), done. remote: Total 3 (delta 2), reused 1 (delta 0), pack-reused 0 (from 0) Unpacking objects: 100% (3/3), 1.66 KiB | 1.66 MiB/s, done. From https://github.com/noir-lang/noir * tag nightly-2025-03-11 -> FETCH_HEAD Warning: you are leaving 6 commits behind, not connected to any of your branches: ba0ad9e5f2 Noir local patch commit. 41425e3e82 Ignore package.tgz e5942f5295 chore: delete leftover file b98e22d860 chore: turn on `skipLibCheck` ... and 2 more. HEAD is now at 1fa0dd95a9 chore: bump external pinned commits (#7640) [detached HEAD 96b25f1022] Noir local fixup commit. 4 files changed, 108 insertions(+), 2 deletions(-) create mode 100755 acvm-repo/acvm_js/build.sh.bak create mode 100755 tooling/noirc_abi_wasm/build.sh.bak Applying: patch: delete honk example programs Applying: chore: turn on `skipLibCheck` Applying: chore: delete leftover file Applying: Ignore package.tgz [detached HEAD 2e591ecb7c] Noir local patch commit. cb3599adbc8ee588 # We can still query the hash as it was before % AZTEC_CACHE_COMMIT=HEAD~1 ./noir/bootstrap.sh hash 13s bf9bcc2b4f6046c7 ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates
noir/bootstrap.shto:noir-repoaccording tonoir/.noir-repo.rebuild_patternsintonoir/.noir-repo.hashevery time there is a call to thenoir_syncfunction. Do the same for thenoir/.noir-repo.rebuild_patterns_tests, saving it intonoir/.noir-repo.hash_testsaztec-packages, in tandem withnoir/.rebuild_patterns, to return the hash innoir/bootstrap.sh hashandnoir/bootstrap.sh hash-tests.This way if we pass
AZTEC_COMMIT_HASHthen it will correctly roll back time and return a historical content hash. If the files change during sync, then the cache is disabled until their content is committed toaztec-packages.This means that a PR to bump the version of noir must also update these, which we can add to the workflow that does it.
Example