chore(fuzz): Compare print output on failed programs#8921
Merged
Conversation
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Execution Time'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: a603ece | Previous: 2e36b30 | Ratio |
|---|---|---|---|
rollup-merge |
0.004 s |
0.003 s |
1.33 |
This comment was automatically generated by workflow using github-action-benchmark.
CC: @TomAFrench
rkarabut
approved these changes
Jun 17, 2025
github-merge-queue bot
pushed a commit
to AztecProtocol/aztec-packages
that referenced
this pull request
Jun 18, 2025
Automated pull of nightly from the [noir](https://github.com/noir-lang/noir) programming language, a dependency of Aztec. BEGIN_COMMIT_OVERRIDE fix(LSP): suggest generic type methods (noir-lang/noir#8948) fix: Fix if/match tracking in last uses pass (noir-lang/noir#8935) fix(ssa): Swap Brillig index shift and DIE in minimal pipeline (noir-lang/noir#8946) fix: when macro parse error happens, discard warnings; also preserve unquoted token locations (noir-lang/noir#8944) chore: disable noncritical workflow steps which fail on external prs (noir-lang/noir#8939) chore(test): Add `--minimal-ssa` to integration tests (noir-lang/noir#8938) fix(noirc_evaluator): U128 Binary::And simplification (noir-lang/noir#8940) chore: Improve access to data required to extract Noir (noir-lang/noir#8793) chore: bump external pinned commits (noir-lang/noir#8920) chore(fuzz): Compare print output on failed programs (noir-lang/noir#8921) chore(fuzz): Display nightly fuzz status badge (noir-lang/noir#8932) fix(fuzz): Consider values returned from Brillig to ACIR as dynamic (noir-lang/noir#8931) fix(fuzz): Fix env var name in fuzzing workflow (noir-lang/noir#8929) chore(fuzz): Add nightly fuzz workflow with 5 minute budget (noir-lang/noir#8925) fix(mem2reg): Keep last store for a used nested array (noir-lang/noir#8917) chore(fuzz): Increase the number of deterministic test cases (noir-lang/noir#8872) fix: Create calls to `apply` before function values are changed to fields in defunctionalize (noir-lang/noir#8916) chore: Add various regression tests for mutable references (noir-lang/noir#8915) fix: handle return_data in the interpreter SSA CLI (noir-lang/noir#8914) feat(fuzz): Generate references in the AST fuzzer (noir-lang/noir#8728) fix(fuzz): Use an inline block to circumvent negation with overflow (noir-lang/noir#8911) fix: Inline global arrays with functions at their call site (noir-lang/noir#8905) chore: Casting ACIR vs Brillig regression tests (noir-lang/noir#8903) fix(mem2reg): Keep last store for reference in array used only in an array get (noir-lang/noir#8877) fix: more SSA interpreter fixes (noir-lang/noir#8904) chore: Ban dynamic array indices with reference elements (noir-lang/noir#8888) chore(ssa_fuzzer): Add SSA validation to the SSA fuzzer (noir-lang/noir#8901) chore(fuzz): More metamorphic rules (noir-lang/noir#8857) fix: catch unbound type variables during frontend compilation (noir-lang/noir#8686) fix: preserve functions which are used in `array_set` instructions (noir-lang/noir#8891) fix: assorted SSA interpreter fixes (noir-lang/noir#8893) fix(ssa): Signed cast simplification (noir-lang/noir#8862) feat: simplify apply function cfg immediately (noir-lang/noir#8895) fix: do not hoist control dependent cast (noir-lang/noir#8886) fix(licm): Account for negative bounds when checking whether a loop executes (noir-lang/noir#8889) chore: Release Noir(1.0.0-beta.7) (noir-lang/noir#8520) chore: test noir-ecdsa library in CI (noir-lang/noir#8859) chore: Incorrect clippy warning blocking PRs (noir-lang/noir#8861) END_COMMIT_OVERRIDE --------- Co-authored-by: AztecBot <tech@aztecprotocol.com> Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>
danielntmd
pushed a commit
to danielntmd/aztec-packages
that referenced
this pull request
Jul 16, 2025
Automated pull of nightly from the [noir](https://github.com/noir-lang/noir) programming language, a dependency of Aztec. BEGIN_COMMIT_OVERRIDE fix(LSP): suggest generic type methods (noir-lang/noir#8948) fix: Fix if/match tracking in last uses pass (noir-lang/noir#8935) fix(ssa): Swap Brillig index shift and DIE in minimal pipeline (noir-lang/noir#8946) fix: when macro parse error happens, discard warnings; also preserve unquoted token locations (noir-lang/noir#8944) chore: disable noncritical workflow steps which fail on external prs (noir-lang/noir#8939) chore(test): Add `--minimal-ssa` to integration tests (noir-lang/noir#8938) fix(noirc_evaluator): U128 Binary::And simplification (noir-lang/noir#8940) chore: Improve access to data required to extract Noir (noir-lang/noir#8793) chore: bump external pinned commits (noir-lang/noir#8920) chore(fuzz): Compare print output on failed programs (noir-lang/noir#8921) chore(fuzz): Display nightly fuzz status badge (noir-lang/noir#8932) fix(fuzz): Consider values returned from Brillig to ACIR as dynamic (noir-lang/noir#8931) fix(fuzz): Fix env var name in fuzzing workflow (noir-lang/noir#8929) chore(fuzz): Add nightly fuzz workflow with 5 minute budget (noir-lang/noir#8925) fix(mem2reg): Keep last store for a used nested array (noir-lang/noir#8917) chore(fuzz): Increase the number of deterministic test cases (noir-lang/noir#8872) fix: Create calls to `apply` before function values are changed to fields in defunctionalize (noir-lang/noir#8916) chore: Add various regression tests for mutable references (noir-lang/noir#8915) fix: handle return_data in the interpreter SSA CLI (noir-lang/noir#8914) feat(fuzz): Generate references in the AST fuzzer (noir-lang/noir#8728) fix(fuzz): Use an inline block to circumvent negation with overflow (noir-lang/noir#8911) fix: Inline global arrays with functions at their call site (noir-lang/noir#8905) chore: Casting ACIR vs Brillig regression tests (noir-lang/noir#8903) fix(mem2reg): Keep last store for reference in array used only in an array get (noir-lang/noir#8877) fix: more SSA interpreter fixes (noir-lang/noir#8904) chore: Ban dynamic array indices with reference elements (noir-lang/noir#8888) chore(ssa_fuzzer): Add SSA validation to the SSA fuzzer (noir-lang/noir#8901) chore(fuzz): More metamorphic rules (noir-lang/noir#8857) fix: catch unbound type variables during frontend compilation (noir-lang/noir#8686) fix: preserve functions which are used in `array_set` instructions (noir-lang/noir#8891) fix: assorted SSA interpreter fixes (noir-lang/noir#8893) fix(ssa): Signed cast simplification (noir-lang/noir#8862) feat: simplify apply function cfg immediately (noir-lang/noir#8895) fix: do not hoist control dependent cast (noir-lang/noir#8886) fix(licm): Account for negative bounds when checking whether a loop executes (noir-lang/noir#8889) chore: Release Noir(1.0.0-beta.7) (noir-lang/noir#8520) chore: test noir-ecdsa library in CI (noir-lang/noir#8859) chore: Incorrect clippy warning blocking PRs (noir-lang/noir#8861) END_COMMIT_OVERRIDE --------- Co-authored-by: AztecBot <tech@aztecprotocol.com> Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Problem*
Follow up for a discussion about whether we can treat failures as equivalents.
Summary*
So far the fuzzer considered the programs to either succeed with an optional return value and the printed output, or fail with an error. Now the printed output is also considered on the failure side: if the errors are considered equivalent, then we also demand that the printed output is the same, for the result to be "both failed the same way, therefore the test passes".
Additional Context
I didn't go all the say to say all ACIR and Brillig errors are equivalent just yet. Let's relax them as they crop up, knowing that the backstop of comparing the prints is there.
I relaxed the
ConstantDoesNotFitTypeerror in the SSA interpreter to be considered equivalent if the constant is the same and the type is the same. I didn't do it for other errors for now, because the SSA interpreter does not capture print outputs yet.Documentation*
Check one:
PR Checklist*
cargo fmton default settings.