Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 4 additions & 7 deletions CI.md
Original file line number Diff line number Diff line change
Expand Up @@ -341,15 +341,11 @@ The release image is created from a bootstrap, by the `release-image/Dockerfile`

## Releases

Releases can be performed directly from the terminal if necessary. However at present this will require `NPM_TOKEN` which is a secret restricted to a few people. In future we may provide a "staging organistion" for less secure unofficial releases.
Release please is used and will automatically tag the commit e.g. `v1.2.3`. The project will subsequently be released under that version.

One example might be to do an arbitrary commit release:
You can also trigger pre and post releases using extended semver notation such as `v1.2.3-nightly.20250101` or `v1.2.3-devnet.0`. This are made simply by tagging the appropriate master commit.

```
./bootstrap.sh release_commit
```

Which will take the current working tree build, and release it under the current HEAD commit hash. Do this with caution as it's releasing your working tree, but it won't upgrade the official release. For example the npm packages will be published under the `next` distribution tag.
Releases can be performed directly from the terminal if necessary. However at present this will require `NPM_TOKEN` which is a secret restricted to a few people. In future we may provide a "staging organization" for less secure unofficial releases.

One can also side-step Release Please automation by updating the version number in the root `.release-please-manifest.json`, committing, tagging the repository with e.g. `v1.2.3`, checking out the tag, and running:

Expand Down Expand Up @@ -386,6 +382,7 @@ CI_FULL=1 ci ec2-test
This will create a new instance, bootstrap, and run all tests that would run on master.

### How does swc compare to tsc (typescript compiler?)

1. swc is stricter than tsc when it comes to hoisting ESM imports. This means that circular dependencies that were not causing issues previously may now do. madge seems like a good tool for spotting them (npx madge --circular path-to-file), since eslint no-circular-imports not always spots them. When dealing with circular deps, keep in mind type imports are removed, so you don't need to worry about those.
2. swc is a lot faster than tsc, but it does not type check. When running bootstrap eg after a rebase, a successful run does not mean the project types are correct. You need to run bootstrap with TYPECHECK=1 for that. If you want to keep a running process that alerts you of type errors, do yarn tsc -b -w --emitDeclarationOnly at the root of yarn project.
3. There is some combination of swc+tsc+jest that breaks things. If you happen to build your project with `tsc -b`, some test suites (such as prover-client or e2e) will fail to run with a parse error. Workaround is to re-build with swc before running tests by running `./bootstrap.sh compile` on yarn-project, and make sure you are not running `tsc -b -w` (or if you do, you set `--emitDeclarationOnly` as described above).
Expand Down
1 change: 1 addition & 0 deletions barretenberg/cpp/.rebuild_patterns
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
^barretenberg/cpp/.*Dockerfile.*$
^barretenberg/cpp/scripts/
^barretenberg/cpp/bootstrap.sh
^barretenberg/cpp/CMakePresets.json
86 changes: 38 additions & 48 deletions barretenberg/cpp/bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,24 @@ export preset=clang16-assert
export pic_preset="clang16-pic"
export hash=$(cache_content_hash .rebuild_patterns)

# Injects version number into a given bb binary.
# Means we don't actually need to rebuild bb to release a new version if code hasn't changed.
function inject_version {
local binary=$1
local version=$(jq -r '."."' ../../.release-please-manifest.json)
local placeholder='00000000.00000000.00000000'
if [ ${#version} -gt ${#placeholder} ]; then
echo "Error: version ($version) is longer than placeholder. Cannot update bb binaries."
exit 1
fi
local offset=$(grep -aobF "$placeholder" $binary | head -n 1 | cut -d: -f1)
if [ -z "$offset" ]; then
echo "Placeholder not found in $binary, can't inject version."
exit 1
fi
printf "$version\0" | dd of=$binary bs=1 seek=$offset conv=notrunc 2>/dev/null
}

# Build all native binaries, including tests.
function build_native {
set -eu
Expand Down Expand Up @@ -98,46 +116,26 @@ function download_old_crs {
}

function build_release {
local arch=$(arch)
rm -rf build-release
mkdir build-release
local version=$(jq -r '."."' ../../.release-please-manifest.json)
local version_placeholder='00000000.00000000.00000000'
local version_regex='00000000\.00000000\.00000000'
local arch=$(arch)
# We pad version to the length of our version_placeholder, adding null bytes to the end.
# We then write out to this version to our artifacts, using sed to replace the version placeholder in our binaries.
# Calculate lengths (both version and placeholder)
local placeholder_length=${#version_placeholder}
local version_length=${#version}
if (( version_length > placeholder_length )); then
echo "Error: version ($version) is longer than placeholder ($version_placeholder). Cannot update bb binaries."
exit 1
fi
# Create a string for use in sed that will write the appropriate number of null bytes.
local N=$(( placeholder_length - version_length ))
local nullbytes="$(printf '\\x00%.0s' $(seq 1 "$N"))"

function update_bb_version {
local file=$1
if ! grep $version_regex "$file" 2>/dev/null; then
echo_stderr "Error: $file does not have placeholder ($version_placeholder). Cannot update bb binaries."
exit 1
fi
# Perform the actual replacement on a file.
sed "s/$version_regex/$version$nullbytes/" "$file"
}
update_bb_version build/bin/bb > build-release/bb
chmod +x build-release/bb
tar -czf build-release/barretenberg-$arch-linux.tar.gz -C build-release bb
# WASM binaries do not currently report version.
tar -czf build-release/barretenberg-wasm.tar.gz -C build-wasm/bin barretenberg.wasm
tar -czf build-release/barretenberg-debug-wasm.tar.gz -C build-wasm/bin barretenberg-debug.wasm
tar -czf build-release/barretenberg-threads-wasm.tar.gz -C build-wasm-threads/bin barretenberg.wasm
tar -czf build-release/barretenberg-threads-debug-wasm.tar.gz -C build-wasm-threads/bin barretenberg-debug.wasm
cp build/bin/bb build-release/bb
inject_version build-release/bb
tar -czf build-release/barretenberg-$arch-linux.tar.gz -C build-release --remove-files bb

if [ "$CI_FULL" -eq 1 ]; then
update_bb_version build-darwin-$arch/bin/bb > build-darwin-$arch/bin/bb.replaced
chmod +x build-darwin-$arch/bin/bb.replaced
tar -czf build-release/barretenberg-$arch-darwin.tar.gz -C build-darwin-$arch/bin --transform 's/.replaced//' bb.replaced
cp build-darwin-$arch/bin/bb build-release/bb
inject_version build-release/bb
tar -czf build-release/barretenberg-$arch-darwin.tar.gz -C build-release --remove-files bb
fi

# Only release wasms built on amd64.
if [ "$arch" == "amd64 "]; then
tar -czf build-release/barretenberg-wasm.tar.gz -C build-wasm/bin barretenberg.wasm
tar -czf build-release/barretenberg-debug-wasm.tar.gz -C build-wasm/bin barretenberg-debug.wasm
tar -czf build-release/barretenberg-threads-wasm.tar.gz -C build-wasm-threads/bin barretenberg.wasm
tar -czf build-release/barretenberg-threads-debug-wasm.tar.gz -C build-wasm-threads/bin barretenberg-debug.wasm
fi
}

Expand All @@ -152,16 +150,12 @@ function build {
build_wasm_threads
download_old_crs
)
if [ "$(arch)" == "amd64" ] && [ "${CI:-0}" = 1 ]; then
if [ "$(arch)" == "amd64" ] && [ "$CI" -eq 1 ]; then
# TODO figure out why this is failing on arm64 with ultra circuit builder string op overflow.
builds+=(
build_gcc_syntax_check_only
)
builds+=(build_gcc_syntax_check_only)
fi
if [ "$CI_FULL" -eq 1 ]; then
builds+=(
build_darwin
)
builds+=(build_darwin)
fi
parallel --line-buffered --tag --halt now,fail=1 denoise {} ::: ${builds[@]}
build_release
Expand Down Expand Up @@ -280,10 +274,6 @@ function release {
do_or_dryrun gh release upload $REF_NAME build-release/* --clobber
}

function release_commit {
release
}

case "$cmd" in
"clean")
git clean -fdx
Expand All @@ -304,7 +294,7 @@ case "$cmd" in
"hash")
echo $hash
;;
test|test_cmds|bench|release|release_commit|build_native|build_wasm|build_wasm_threads|build_darwin|build_release)
test|test_cmds|bench|release|build_native|build_wasm|build_wasm_threads|build_darwin|build_release|inject_version)
$cmd "$@"
;;
*)
Expand Down
10 changes: 2 additions & 8 deletions barretenberg/ts/bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,7 @@ function test {
}

function release {
local version=${REF_NAME#v}
deploy_npm $(dist_tag) $version
}

function release_commit {
local version="$CURRENT_VERSION-commit.$COMMIT_HASH"
deploy_npm next $version
deploy_npm $(dist_tag) ${REF_NAME#v}
}

case "$cmd" in
Expand All @@ -67,7 +61,7 @@ case "$cmd" in
"bench")
echo "ts/bootstrap.sh bench is empty"
;;
test|test_cmds|release|release_commit)
test|test_cmds|release)
$cmd
;;
*)
Expand Down
25 changes: 1 addition & 24 deletions bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -276,29 +276,6 @@ function release_dryrun {
DRY_RUN=1 release
}

function release_commit {
export REF_NAME="commit-$COMMIT_HASH"

release_github

projects=(
barretenberg/cpp
barretenberg/ts
noir
l1-contracts
yarn-project
boxes
aztec-up
playground
docs
release-image
)

for project in "${projects[@]}"; do
$project/bootstrap.sh release_commit
done
}

case "$cmd" in
"clean")
echo "WARNING: This will erase *all* untracked files, including hooks and submodules."
Expand Down Expand Up @@ -338,7 +315,7 @@ case "$cmd" in
release
fi
;;
test|test_cmds|bench|release|release_dryrun|release_commit)
test|test_cmds|bench|release|release_dryrun)
$cmd "$@"
;;
*)
Expand Down
7 changes: 1 addition & 6 deletions boxes/bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -102,11 +102,6 @@ function release {
release_git_push $branch $REF_NAME ${REF_NAME#v}
}

function release_commit {
echo_header "boxes release commit"
release_git_push "$CURRENT_VERSION" "commit-$COMMIT_HASH" "$CURRENT_VERSION-commit.$COMMIT_HASH"
}

case "$cmd" in
"clean")
git clean -fdx
Expand All @@ -118,7 +113,7 @@ case "$cmd" in
""|"fast"|"full")
build
;;
test|test_cmds|release|release_commit)
test|test_cmds|release)
$cmd
;;
"hash")
Expand Down
2 changes: 1 addition & 1 deletion ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ case "$cmd" in
export DENOISE=1
num=${1:-5}
seq 0 $((num - 1)) | parallel --tag --line-buffered \
'INSTANCE_POSTFIX={} bootstrap_ec2 "USE_TEST_CACHE=0 ./bootstrap.sh ci" | cache_log "Grind {}"'
'INSTANCE_POSTFIX={} bootstrap_ec2 "USE_TEST_CACHE=0 ./bootstrap.sh ci" 2>&1 | cache_log "Grind {}"'
;;
"local")
# Create container with clone of local repo and bootstrap.
Expand Down
5 changes: 5 additions & 0 deletions ci3/log_ci_run
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ if [ -z "$key" ]; then
name=$REF_NAME
[ "$(aws_get_meta_data instance-life-cycle)" == "spot" ] && spot=true || spot=false

# If this is github merge queue, just keep the queue name.
if [[ "$name" =~ ^gh-readonly-queue/([^/]+)/ ]]; then
name=${BASH_REMATCH[1]}
fi

json=$(jq -c -j -n \
--argjson timestamp "$key" \
--arg status "$status" \
Expand Down
8 changes: 0 additions & 8 deletions ci3/source_refname
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,6 @@ if [ -z "${REF_NAME:-}" ]; then
REF_NAME="${REF_NAME#heads/}"
fi

# Strip off any leading "release/"
REF_NAME="${REF_NAME#release/}"

# If this is github merge queue, just keep the queue name.
if [[ "$REF_NAME" =~ ^gh-readonly-queue/([^/]+)/ ]]; then
REF_NAME=${BASH_REMATCH[1]}
fi

[ -z "$REF_NAME" ] && exit 1

export REF_NAME
Expand Down
4 changes: 0 additions & 4 deletions docs/bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -83,10 +83,6 @@ function release {
fi
}

function release_commit {
yarn netlify deploy --site aztec-docs-dev
}

case "$cmd" in
"clean")
git clean -fdx
Expand Down
7 changes: 1 addition & 6 deletions l1-contracts/bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -138,11 +138,6 @@ function release {
release_git_push $branch $REF_NAME ${REF_NAME#v}
}

function release_commit {
echo_header "l1-contracts release commit"
release_git_push "$CURRENT_VERSION" "commit-$COMMIT_HASH" "$CURRENT_VERSION-commit.$COMMIT_HASH"
}

case "$cmd" in
"clean")
git clean -fdx
Expand All @@ -157,7 +152,7 @@ case "$cmd" in
"test")
test
;;
test_cmds|release|release_commit)
test_cmds|release)
$cmd
;;
"hash")
Expand Down
9 changes: 4 additions & 5 deletions noir/bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ function build_packages {
mv packages/package packages/${project#*/}
done

# Find all files in packages dir and use sed to in-place replace @noir-lang with @aztec/noir-
find packages -type f -exec sed -i 's|@noir-lang/|@aztec/noir-|g' {} \;

cache_upload noir-packages-$hash.tar.gz \
packages \
noir-repo/acvm-repo/acvm_js/nodejs \
Expand Down Expand Up @@ -179,10 +182,6 @@ function release {
release_packages $(dist_tag) ${REF_NAME#v}
}

function release_commit {
release_packages next "$CURRENT_VERSION-commit.$COMMIT_HASH"
}

case "$cmd" in
"clean")
git clean -fdx
Expand All @@ -194,7 +193,7 @@ case "$cmd" in
""|"fast"|"full")
build
;;
test_cmds|build_native|build_packages|format|test|release|release_commit|test_example)
test_cmds|build_native|build_packages|format|test|release|test_example)
$cmd "$@"
;;
"hash")
Expand Down
4 changes: 0 additions & 4 deletions release-image/bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,6 @@ case "$cmd" in
docker manifest push aztecprotocol/aztec:$(dist_tag)
fi
;;
"release_commit")
echo_header "release-image release commit"
docker push aztecprotocol/aztec:$(git rev-parse HEAD)
;;
*)
echo "Unknown command: $cmd"
exit 1
Expand Down
4 changes: 2 additions & 2 deletions yarn-project/bb-prover/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -70,14 +70,14 @@
"@aztec/bb.js": "portal:../../barretenberg/ts",
"@aztec/constants": "workspace:^",
"@aztec/foundation": "workspace:^",
"@aztec/noir-noirc_abi": "portal:../../noir/packages/noirc_abi",
"@aztec/noir-protocol-circuits-types": "workspace:^",
"@aztec/noir-types": "portal:../../noir/packages/types",
"@aztec/simulator": "workspace:^",
"@aztec/stdlib": "workspace:^",
"@aztec/telemetry-client": "workspace:^",
"@aztec/world-state": "workspace:^",
"@msgpack/msgpack": "^3.0.0-beta2",
"@noir-lang/noirc_abi": "portal:../../noir/packages/noirc_abi",
"@noir-lang/types": "portal:../../noir/packages/types",
"commander": "^12.1.0",
"pako": "^2.1.0",
"source-map-support": "^0.5.21",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
import { runInDirectory } from '@aztec/foundation/fs';
import { type Logger, createLogger } from '@aztec/foundation/log';
import { serializeWitness } from '@aztec/noir-noirc_abi';
import { BundleArtifactProvider } from '@aztec/noir-protocol-circuits-types/client/bundle';
import type { WitnessMap } from '@aztec/noir-types';
import type { SimulationProvider } from '@aztec/simulator/server';
import type { ClientIvcProof } from '@aztec/stdlib/proofs';

import { encode } from '@msgpack/msgpack';
import { serializeWitness } from '@noir-lang/noirc_abi';
import type { WitnessMap } from '@noir-lang/types';
import { promises as fs } from 'fs';
import path from 'path';

Expand Down
Loading