Skip to content

feat(DRAFT): Save the noir-repo content hash after each sync to a file#12858

Closed
aakoshh wants to merge 1 commit intomasterfrom
af/noir-repo-hash-file
Closed

feat(DRAFT): Save the noir-repo content hash after each sync to a file#12858
aakoshh wants to merge 1 commit intomasterfrom
af/noir-repo-hash-file

Conversation

@aakoshh
Copy link
Contributor

@aakoshh aakoshh commented Mar 18, 2025

Updates noir/bootstrap.sh to:

  • Save the content hash of noir-repo according to noir/.noir-repo.rebuild_patterns into noir/.noir-repo.hash every time there is a call to the noir_sync function. Do the same for the noir/.noir-repo.rebuild_patterns_tests, saving it into noir/.noir-repo.hash_tests
  • Use the above two files, which are to be committed to aztec-packages, in tandem with noir/.rebuild_patterns, to return the hash in noir/bootstrap.sh hash and noir/bootstrap.sh hash-tests.

This way if we pass AZTEC_COMMIT_HASH then it will correctly roll back time and return a historical content hash. If the files change during sync, then the cache is disabled until their content is committed to aztec-packages.

This means that a PR to bump the version of noir must also update these, which we can add to the workflow that does it.

Example

% noir/bootstrap.sh hash                                                                                                                                                                            
c13aa0d774a1beba
% noir/bootstrap.sh hash-tests                                                                                                                                                                      
9db50268d214b4b5
% cd noir/noir-repo                                                                                                                                                                                 
% git status                                                                                                                                                                                    
HEAD detached from nightly-2025-03-18
nothing to commit, working tree clean
% echo "Hello" > hello.txt                                                                                                                                                                      
% git add hello.txt && git commit -m "Hello"                                                                                                                                                    
[detached HEAD 8d3b3a6f2a] Hello
 1 file changed, 1 insertion(+)
 create mode 100644 hello.txt
% cd ../..      

# This file was not actually part of the rebuild patterns
% noir/bootstrap.sh hash                                                                                                                                                                            
c13aa0d774a1beba
% cd noir/noir-repo                                                                                                                                                                                 
% echo "World" > compiler/world.txt                                                                                                                                                             
% git add compiler/world.txt && git commit -m "World"                                                                                                                                           
[detached HEAD 6ac59ad1c0] World
 1 file changed, 1 insertion(+)
 create mode 100644 compiler/world.txt
% cd ../..                                                                                                                                                                                      
% noir/bootstrap.sh hash                                                                                                                                                                            
disabled-cache
% git status                                                                                                                                                                                        
On branch af/noir-repo-hash-file
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
	modified:   noir/.noir-repo.hash
	modified:   noir/.noir-repo.hash_tests

no changes added to commit (use "git add" and/or "git commit -a")
% git add noir                                                                                                                                                                                      
% git commit -m "Updated the hashes"          
[af/noir-repo-hash-file e9d9bcb391] Updated the hashes
 2 files changed, 2 insertions(+), 2 deletions(-)
% noir/bootstrap.sh hash                                                                                                                                                                            
e76134e49b62a0e1
% noir/bootstrap.sh hash-tests                                                                                                                                                                      
5bec282920ce6da5
% git log --oneline            

# historical query recovers the original hash
% AZTEC_CACHE_COMMIT=e0d40d95eb noir/bootstrap.sh hash                                                                                                                                              
c13aa0d774a1beba

@aakoshh aakoshh requested a review from charlielye as a code owner March 18, 2025 19:51
@@ -0,0 +1 @@
5a2582f95f4f49a9
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably get rid of the test distinction imho

@ludamad
Copy link
Collaborator

ludamad commented Mar 18, 2025

For posterity: I don't like things relying on people committing files written out by bootstrap, we've gone through lengths to avoid that, hence asking for (for CI purposes) to just use the noir ref and patches

ludamad pushed a commit that referenced this pull request Mar 18, 2025
…d `noir-repo.patch` (#12861)

Alternative to
#12858

Changes `noir/bootstrap.sh` to use `cache_content_hash
.rebuild_patterns` if
* we have specified an `AZTEC_COMMIT_HASH`, looking for historical
values
* there are no commits in `noir-repo` that could be added to a patch
file, indicating changes that aren't captured by the `.rebuild_patterns`
and the `noir-repo-ref` file, and
* `noir-repo` isn't on a _branch_ which could evolve all the time; if
it's on a branch then use the `.noir-repo.rebuild_patterns` in
`noir-repo` itself to figure out the hash

The exception for feature branches should not affect normal aztec
workflow, which is based on _tags_, it's only there to support temporary
work noir developers do across both repos at the same time, and it's not
something that will be merged into the `master` branch of
`aztec-packages` (it would defeat repeatable builds).

With this change it should be possible to use `AZTEC_COMMIT_HASH` to
query historical queries, as seen in the 2nd example below.

### Example 1

This shows that if we're on a tag, then adding a new commit to the
noir-repo will disable caching. This is the normal case in
`aztec-packages`. Otherwise the hash is based on just `noir`. But if we
switch to a feature branch, caching is back on to support faster builds,
incorporating the latest commits into the hash.

```console
# We are on a tag
% ./noir/scripts/sync.sh info                                                                                                                                                                        
Repo exists:              yes
Fixup commit:             9dcbd3fbf47f886821ac71a647d4f9712ab1eb1d
Checkout commit:          45ad637273cef317eba42feaf2be0e59d34065ed
Wanted:                   nightly-2025-03-18
Needs switch:             no
Needs patch:              no
Detached:                 yes
On branch:                no
Branch name:              n/a
Has wanted tag:           yes
Has tag commit:           yes
Has patch commit:         yes
Last commit is patch:     yes
Has fixup and patch:      yes
Has uncommitted changes:  no
Latest nightly:           nightly-2025-03-18
Cache mode:               noir
% ./noir/scripts/sync.sh cache-mode                                                                                                                                                                  
noir
% ./noir/bootstrap.sh hash                                                                                                                                                                           
eb08d2624603de97

# Make a change in the noir-repo
% cd noir/noir-repo                                                                                                                                                                                  
% echo "Foo" > compiler/foo.txt && git add compiler/foo.txt && git commit -m "Foo"                                                                                                              
[detached HEAD 23d1d2ac6b] Foo
 1 file changed, 1 insertion(+)
 create mode 100644 compiler/foo.txt
% cd ../..        

# The extra commit disables the cache
% ./noir/scripts/sync.sh cache-mode                                                                                                                                                                  
disabled-cache
% ./noir/bootstrap.sh hash                                                                                                                                                                           
disabled-cache

# Now switch to a feature branch
% echo af/msgpack-codegen > noir/noir-repo-ref                                                                                                                                                       
% ./noir/bootstrap.sh hash                                                                                                                                                                          
Error: noir-repo is on a detached HEAD and the last commit is not the patch marker commit;
switching to af/msgpack-codegen could mean losing those commits.
Please use the 'make-patch' command to create a noir-repo.patch file and commit it in aztec-packages, 
so that it is re-applied after each checkout. Make sure to commit the patch on the branch where it should be.

# Get rid of the foo commit, so we can switch away
% cd noir/noir-repo                                                                                                                                                                                 
% git reset --hard HEAD~1                                                                                                                                                                       
HEAD is now at 8b88883d58 Noir local patch commit.
% cd ../..                   

# Hashing still involves syncing; we need the noir-repo to decide how to hash
% ./noir/bootstrap.sh hash                                                                                                                                                                          
remote: Enumerating objects: 187, done.
...
Switched to branch 'af/msgpack-codegen'
Your branch is up to date with 'upstream/af/msgpack-codegen'.
...
[af/msgpack-codegen 98f9a3cc43] Noir local fixup commit.
 4 files changed, 108 insertions(+), 2 deletions(-)
 create mode 100755 acvm-repo/acvm_js/build.sh.bak
 create mode 100755 tooling/noirc_abi_wasm/build.sh.bak
Applying: patch: delete honk example programs
Applying: chore: turn on `skipLibCheck`
Applying: chore: delete leftover file
Applying: Ignore package.tgz
[af/msgpack-codegen 182331696a] Noir local patch commit.
disabled-cache

# ^ The cache is disabled becase we have a pending change in `noir/noir-repo-ref`, 
# but `cache-mode` indicates we need to cache based on the `noir-repo` contents.
% ./noir/scripts/sync.sh cache-mode                                                                                                                                                                 
noir-repo

# Commit the noir-repo-ref file, so we have a clean state in aztec-packages                                                                                                                                                                              
% git add noir/noir-repo-ref && git commit -m "Switched to a feature branch"                                                                                                                        
Formatting barretenberg staged files...
[af/noir-repo-ref-hash 62fc17a03b] Switched to a feature branch
 1 file changed, 1 insertion(+), 1 deletion(-)

# Now we hash based on the code in `noir-repo`, *and* the contents of `noir`
% ./noir/bootstrap.sh hash                                                                                                                                                                          
798ec85262e6133a

```

### Example 2

This one shows that even after switching to a different tag, we can use
the `AZTEC_CACHE_COMMIT` feature to go back in the commit log to get a
historical hash.

```console
# We're on a tag
% git rev-parse HEAD                                                                                                                                                                                
517f246
% cat noir/noir-repo-ref                                                                                                                                                                            
nightly-2025-03-18
% ./noir/bootstrap.sh hash                                                                                                                                                                          
bf9bcc2b4f6046c7

# Want to roll back to a previous nightly tag
% echo nightly-2025-03-11 > noir/noir-repo-ref                                                                                                                                                      
% git add noir                                                                                                                                                                                      
% git commit -m "Roll back nightly"                                                                                                                                                                 
Formatting barretenberg staged files...
[af/noir-repo-ref-hash 858d98f42b] Roll back nightly
 1 file changed, 1 insertion(+), 1 deletion(-)

# Recalculating the hash will sync the repo, but the hash will be based on the ref, not the content
% ./noir/bootstrap.sh hash                                                                                                                                                                          
remote: Enumerating objects: 66481, done.
remote: Counting objects: 100% (61479/61479), done.
remote: Compressing objects: 100% (36681/36681), done.
remote: Total 57993 (delta 21540), reused 52084 (delta 16927), pack-reused 0 (from 0)
Receiving objects: 100% (57993/57993), 158.39 MiB | 33.13 MiB/s, done.
Resolving deltas: 100% (21540/21540), completed with 557 local objects.
remote: Enumerating objects: 23, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 2), reused 1 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (3/3), 1.66 KiB | 1.66 MiB/s, done.
From https://github.com/noir-lang/noir
 * tag                     nightly-2025-03-11 -> FETCH_HEAD
Warning: you are leaving 6 commits behind, not connected to
any of your branches:

  ba0ad9e5f2 Noir local patch commit.
  41425e3e82 Ignore package.tgz
  e5942f5295 chore: delete leftover file
  b98e22d860 chore: turn on `skipLibCheck`
 ... and 2 more.

HEAD is now at 1fa0dd95a9 chore: bump external pinned commits (#7640)
[detached HEAD 96b25f1022] Noir local fixup commit.
 4 files changed, 108 insertions(+), 2 deletions(-)
 create mode 100755 acvm-repo/acvm_js/build.sh.bak
 create mode 100755 tooling/noirc_abi_wasm/build.sh.bak
Applying: patch: delete honk example programs
Applying: chore: turn on `skipLibCheck`
Applying: chore: delete leftover file
Applying: Ignore package.tgz
[detached HEAD 2e591ecb7c] Noir local patch commit.
cb3599adbc8ee588

# We can still query the hash as it was before
% AZTEC_CACHE_COMMIT=HEAD~1 ./noir/bootstrap.sh hash                                                                                                                                            13s 
bf9bcc2b4f6046c7

```
@aakoshh aakoshh closed this Mar 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants