diff --git a/CI.md b/CI.md index c6c05d1a9cec..290b16759f42 100644 --- a/CI.md +++ b/CI.md @@ -341,15 +341,11 @@ The release image is created from a bootstrap, by the `release-image/Dockerfile` ## Releases -Releases can be performed directly from the terminal if necessary. However at present this will require `NPM_TOKEN` which is a secret restricted to a few people. In future we may provide a "staging organistion" for less secure unofficial releases. +Release please is used and will automatically tag the commit e.g. `v1.2.3`. The project will subsequently be released under that version. -One example might be to do an arbitrary commit release: +You can also trigger pre and post releases using extended semver notation such as `v1.2.3-nightly.20250101` or `v1.2.3-devnet.0`. This are made simply by tagging the appropriate master commit. -``` -./bootstrap.sh release_commit -``` - -Which will take the current working tree build, and release it under the current HEAD commit hash. Do this with caution as it's releasing your working tree, but it won't upgrade the official release. For example the npm packages will be published under the `next` distribution tag. +Releases can be performed directly from the terminal if necessary. However at present this will require `NPM_TOKEN` which is a secret restricted to a few people. In future we may provide a "staging organization" for less secure unofficial releases. One can also side-step Release Please automation by updating the version number in the root `.release-please-manifest.json`, committing, tagging the repository with e.g. `v1.2.3`, checking out the tag, and running: @@ -386,6 +382,7 @@ CI_FULL=1 ci ec2-test This will create a new instance, bootstrap, and run all tests that would run on master. ### How does swc compare to tsc (typescript compiler?) + 1. swc is stricter than tsc when it comes to hoisting ESM imports. This means that circular dependencies that were not causing issues previously may now do. madge seems like a good tool for spotting them (npx madge --circular path-to-file), since eslint no-circular-imports not always spots them. When dealing with circular deps, keep in mind type imports are removed, so you don't need to worry about those. 2. swc is a lot faster than tsc, but it does not type check. When running bootstrap eg after a rebase, a successful run does not mean the project types are correct. You need to run bootstrap with TYPECHECK=1 for that. If you want to keep a running process that alerts you of type errors, do yarn tsc -b -w --emitDeclarationOnly at the root of yarn project. 3. There is some combination of swc+tsc+jest that breaks things. If you happen to build your project with `tsc -b`, some test suites (such as prover-client or e2e) will fail to run with a parse error. Workaround is to re-build with swc before running tests by running `./bootstrap.sh compile` on yarn-project, and make sure you are not running `tsc -b -w` (or if you do, you set `--emitDeclarationOnly` as described above). diff --git a/barretenberg/cpp/.rebuild_patterns b/barretenberg/cpp/.rebuild_patterns index c6dd24549afc..4c7ba2459ea5 100644 --- a/barretenberg/cpp/.rebuild_patterns +++ b/barretenberg/cpp/.rebuild_patterns @@ -3,3 +3,4 @@ ^barretenberg/cpp/.*Dockerfile.*$ ^barretenberg/cpp/scripts/ ^barretenberg/cpp/bootstrap.sh +^barretenberg/cpp/CMakePresets.json diff --git a/barretenberg/cpp/bootstrap.sh b/barretenberg/cpp/bootstrap.sh index 5e01e28bc181..a4b06a337b9c 100755 --- a/barretenberg/cpp/bootstrap.sh +++ b/barretenberg/cpp/bootstrap.sh @@ -8,6 +8,24 @@ export preset=clang16-assert export pic_preset="clang16-pic" export hash=$(cache_content_hash .rebuild_patterns) +# Injects version number into a given bb binary. +# Means we don't actually need to rebuild bb to release a new version if code hasn't changed. +function inject_version { + local binary=$1 + local version=$(jq -r '."."' ../../.release-please-manifest.json) + local placeholder='00000000.00000000.00000000' + if [ ${#version} -gt ${#placeholder} ]; then + echo "Error: version ($version) is longer than placeholder. Cannot update bb binaries." + exit 1 + fi + local offset=$(grep -aobF "$placeholder" $binary | head -n 1 | cut -d: -f1) + if [ -z "$offset" ]; then + echo "Placeholder not found in $binary, can't inject version." + exit 1 + fi + printf "$version\0" | dd of=$binary bs=1 seek=$offset conv=notrunc 2>/dev/null +} + # Build all native binaries, including tests. function build_native { set -eu @@ -98,46 +116,26 @@ function download_old_crs { } function build_release { + local arch=$(arch) rm -rf build-release mkdir build-release - local version=$(jq -r '."."' ../../.release-please-manifest.json) - local version_placeholder='00000000.00000000.00000000' - local version_regex='00000000\.00000000\.00000000' - local arch=$(arch) - # We pad version to the length of our version_placeholder, adding null bytes to the end. - # We then write out to this version to our artifacts, using sed to replace the version placeholder in our binaries. - # Calculate lengths (both version and placeholder) - local placeholder_length=${#version_placeholder} - local version_length=${#version} - if (( version_length > placeholder_length )); then - echo "Error: version ($version) is longer than placeholder ($version_placeholder). Cannot update bb binaries." - exit 1 - fi - # Create a string for use in sed that will write the appropriate number of null bytes. - local N=$(( placeholder_length - version_length )) - local nullbytes="$(printf '\\x00%.0s' $(seq 1 "$N"))" - function update_bb_version { - local file=$1 - if ! grep $version_regex "$file" 2>/dev/null; then - echo_stderr "Error: $file does not have placeholder ($version_placeholder). Cannot update bb binaries." - exit 1 - fi - # Perform the actual replacement on a file. - sed "s/$version_regex/$version$nullbytes/" "$file" - } - update_bb_version build/bin/bb > build-release/bb - chmod +x build-release/bb - tar -czf build-release/barretenberg-$arch-linux.tar.gz -C build-release bb - # WASM binaries do not currently report version. - tar -czf build-release/barretenberg-wasm.tar.gz -C build-wasm/bin barretenberg.wasm - tar -czf build-release/barretenberg-debug-wasm.tar.gz -C build-wasm/bin barretenberg-debug.wasm - tar -czf build-release/barretenberg-threads-wasm.tar.gz -C build-wasm-threads/bin barretenberg.wasm - tar -czf build-release/barretenberg-threads-debug-wasm.tar.gz -C build-wasm-threads/bin barretenberg-debug.wasm + cp build/bin/bb build-release/bb + inject_version build-release/bb + tar -czf build-release/barretenberg-$arch-linux.tar.gz -C build-release --remove-files bb + if [ "$CI_FULL" -eq 1 ]; then - update_bb_version build-darwin-$arch/bin/bb > build-darwin-$arch/bin/bb.replaced - chmod +x build-darwin-$arch/bin/bb.replaced - tar -czf build-release/barretenberg-$arch-darwin.tar.gz -C build-darwin-$arch/bin --transform 's/.replaced//' bb.replaced + cp build-darwin-$arch/bin/bb build-release/bb + inject_version build-release/bb + tar -czf build-release/barretenberg-$arch-darwin.tar.gz -C build-release --remove-files bb + fi + + # Only release wasms built on amd64. + if [ "$arch" == "amd64 "]; then + tar -czf build-release/barretenberg-wasm.tar.gz -C build-wasm/bin barretenberg.wasm + tar -czf build-release/barretenberg-debug-wasm.tar.gz -C build-wasm/bin barretenberg-debug.wasm + tar -czf build-release/barretenberg-threads-wasm.tar.gz -C build-wasm-threads/bin barretenberg.wasm + tar -czf build-release/barretenberg-threads-debug-wasm.tar.gz -C build-wasm-threads/bin barretenberg-debug.wasm fi } @@ -152,16 +150,12 @@ function build { build_wasm_threads download_old_crs ) - if [ "$(arch)" == "amd64" ] && [ "${CI:-0}" = 1 ]; then + if [ "$(arch)" == "amd64" ] && [ "$CI" -eq 1 ]; then # TODO figure out why this is failing on arm64 with ultra circuit builder string op overflow. - builds+=( - build_gcc_syntax_check_only - ) + builds+=(build_gcc_syntax_check_only) fi if [ "$CI_FULL" -eq 1 ]; then - builds+=( - build_darwin - ) + builds+=(build_darwin) fi parallel --line-buffered --tag --halt now,fail=1 denoise {} ::: ${builds[@]} build_release @@ -280,10 +274,6 @@ function release { do_or_dryrun gh release upload $REF_NAME build-release/* --clobber } -function release_commit { - release -} - case "$cmd" in "clean") git clean -fdx @@ -304,7 +294,7 @@ case "$cmd" in "hash") echo $hash ;; - test|test_cmds|bench|release|release_commit|build_native|build_wasm|build_wasm_threads|build_darwin|build_release) + test|test_cmds|bench|release|build_native|build_wasm|build_wasm_threads|build_darwin|build_release|inject_version) $cmd "$@" ;; *) diff --git a/barretenberg/ts/bootstrap.sh b/barretenberg/ts/bootstrap.sh index 58c3ad4838cd..ee3bdc50092c 100755 --- a/barretenberg/ts/bootstrap.sh +++ b/barretenberg/ts/bootstrap.sh @@ -41,13 +41,7 @@ function test { } function release { - local version=${REF_NAME#v} - deploy_npm $(dist_tag) $version -} - -function release_commit { - local version="$CURRENT_VERSION-commit.$COMMIT_HASH" - deploy_npm next $version + deploy_npm $(dist_tag) ${REF_NAME#v} } case "$cmd" in @@ -67,7 +61,7 @@ case "$cmd" in "bench") echo "ts/bootstrap.sh bench is empty" ;; - test|test_cmds|release|release_commit) + test|test_cmds|release) $cmd ;; *) diff --git a/bootstrap.sh b/bootstrap.sh index 56a8022409a3..99ddacf810e9 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -276,29 +276,6 @@ function release_dryrun { DRY_RUN=1 release } -function release_commit { - export REF_NAME="commit-$COMMIT_HASH" - - release_github - - projects=( - barretenberg/cpp - barretenberg/ts - noir - l1-contracts - yarn-project - boxes - aztec-up - playground - docs - release-image - ) - - for project in "${projects[@]}"; do - $project/bootstrap.sh release_commit - done -} - case "$cmd" in "clean") echo "WARNING: This will erase *all* untracked files, including hooks and submodules." @@ -338,7 +315,7 @@ case "$cmd" in release fi ;; - test|test_cmds|bench|release|release_dryrun|release_commit) + test|test_cmds|bench|release|release_dryrun) $cmd "$@" ;; *) diff --git a/boxes/bootstrap.sh b/boxes/bootstrap.sh index 69da2bda9221..bc2069bd90f2 100755 --- a/boxes/bootstrap.sh +++ b/boxes/bootstrap.sh @@ -102,11 +102,6 @@ function release { release_git_push $branch $REF_NAME ${REF_NAME#v} } -function release_commit { - echo_header "boxes release commit" - release_git_push "$CURRENT_VERSION" "commit-$COMMIT_HASH" "$CURRENT_VERSION-commit.$COMMIT_HASH" -} - case "$cmd" in "clean") git clean -fdx @@ -118,7 +113,7 @@ case "$cmd" in ""|"fast"|"full") build ;; - test|test_cmds|release|release_commit) + test|test_cmds|release) $cmd ;; "hash") diff --git a/ci.sh b/ci.sh index 6d8eb3787d55..07f7e40fb9ae 100755 --- a/ci.sh +++ b/ci.sh @@ -91,7 +91,7 @@ case "$cmd" in export DENOISE=1 num=${1:-5} seq 0 $((num - 1)) | parallel --tag --line-buffered \ - 'INSTANCE_POSTFIX={} bootstrap_ec2 "USE_TEST_CACHE=0 ./bootstrap.sh ci" | cache_log "Grind {}"' + 'INSTANCE_POSTFIX={} bootstrap_ec2 "USE_TEST_CACHE=0 ./bootstrap.sh ci" 2>&1 | cache_log "Grind {}"' ;; "local") # Create container with clone of local repo and bootstrap. diff --git a/ci3/log_ci_run b/ci3/log_ci_run index 62707235c3a1..dbaac5d188b5 100755 --- a/ci3/log_ci_run +++ b/ci3/log_ci_run @@ -13,6 +13,11 @@ if [ -z "$key" ]; then name=$REF_NAME [ "$(aws_get_meta_data instance-life-cycle)" == "spot" ] && spot=true || spot=false + # If this is github merge queue, just keep the queue name. + if [[ "$name" =~ ^gh-readonly-queue/([^/]+)/ ]]; then + name=${BASH_REMATCH[1]} + fi + json=$(jq -c -j -n \ --argjson timestamp "$key" \ --arg status "$status" \ diff --git a/ci3/source_refname b/ci3/source_refname index 7a86eef636e4..3893ddb1159e 100644 --- a/ci3/source_refname +++ b/ci3/source_refname @@ -25,14 +25,6 @@ if [ -z "${REF_NAME:-}" ]; then REF_NAME="${REF_NAME#heads/}" fi - # Strip off any leading "release/" - REF_NAME="${REF_NAME#release/}" - - # If this is github merge queue, just keep the queue name. - if [[ "$REF_NAME" =~ ^gh-readonly-queue/([^/]+)/ ]]; then - REF_NAME=${BASH_REMATCH[1]} - fi - [ -z "$REF_NAME" ] && exit 1 export REF_NAME diff --git a/docs/bootstrap.sh b/docs/bootstrap.sh index a6e1d2bb2d3f..4ff98a6a0391 100755 --- a/docs/bootstrap.sh +++ b/docs/bootstrap.sh @@ -83,10 +83,6 @@ function release { fi } -function release_commit { - yarn netlify deploy --site aztec-docs-dev -} - case "$cmd" in "clean") git clean -fdx diff --git a/l1-contracts/bootstrap.sh b/l1-contracts/bootstrap.sh index 44384c8e8a13..5d4624c843d8 100755 --- a/l1-contracts/bootstrap.sh +++ b/l1-contracts/bootstrap.sh @@ -138,11 +138,6 @@ function release { release_git_push $branch $REF_NAME ${REF_NAME#v} } -function release_commit { - echo_header "l1-contracts release commit" - release_git_push "$CURRENT_VERSION" "commit-$COMMIT_HASH" "$CURRENT_VERSION-commit.$COMMIT_HASH" -} - case "$cmd" in "clean") git clean -fdx @@ -157,7 +152,7 @@ case "$cmd" in "test") test ;; - test_cmds|release|release_commit) + test_cmds|release) $cmd ;; "hash") diff --git a/noir/bootstrap.sh b/noir/bootstrap.sh index bc6d5b3e2106..c41aa588e42e 100755 --- a/noir/bootstrap.sh +++ b/noir/bootstrap.sh @@ -179,10 +179,6 @@ function release { release_packages $(dist_tag) ${REF_NAME#v} } -function release_commit { - release_packages next "$CURRENT_VERSION-commit.$COMMIT_HASH" -} - case "$cmd" in "clean") git clean -fdx @@ -194,7 +190,7 @@ case "$cmd" in ""|"fast"|"full") build ;; - test_cmds|build_native|build_packages|format|test|release|release_commit|test_example) + test_cmds|build_native|build_packages|format|test|release|test_example) $cmd "$@" ;; "hash") diff --git a/release-image/bootstrap.sh b/release-image/bootstrap.sh index 9ec4c56f86a9..5034e18d2dd5 100755 --- a/release-image/bootstrap.sh +++ b/release-image/bootstrap.sh @@ -56,10 +56,6 @@ case "$cmd" in docker manifest push aztecprotocol/aztec:$(dist_tag) fi ;; - "release_commit") - echo_header "release-image release commit" - docker push aztecprotocol/aztec:$(git rev-parse HEAD) - ;; *) echo "Unknown command: $cmd" exit 1 diff --git a/yarn-project/bootstrap.sh b/yarn-project/bootstrap.sh index 944c3b2a7da3..47acf0e9fb16 100755 --- a/yarn-project/bootstrap.sh +++ b/yarn-project/bootstrap.sh @@ -166,12 +166,7 @@ function release_packages { function release { echo_header "yarn-project release" # WORKTODO latest is only on master, otherwise use ref name - release_packages $(dist_tag) ${REF_NAME#v} -} - -function release_commit { - echo_header "yarn-project release commit" - release_packages next "$CURRENT_VERSION-commit.$COMMIT_HASH" + release_packages $(dist-tag) ${REF_NAME#v} } case "$cmd" in @@ -205,7 +200,7 @@ case "$cmd" in "lint") lint "$@" ;; - test|test_cmds|hash|release|release_commit|format) + test|test_cmds|hash|release|format) $cmd ;; *)