Skip to content

chore(fuzz): Compare print output on failed programs#8921

Merged
aakoshh merged 2 commits intomasterfrom
af/fuzz-track-prints-in-failure
Jun 17, 2025
Merged

chore(fuzz): Compare print output on failed programs#8921
aakoshh merged 2 commits intomasterfrom
af/fuzz-track-prints-in-failure

Conversation

@aakoshh
Copy link
Contributor

@aakoshh aakoshh commented Jun 16, 2025

Description

Problem*

Follow up for a discussion about whether we can treat failures as equivalents.

Summary*

So far the fuzzer considered the programs to either succeed with an optional return value and the printed output, or fail with an error. Now the printed output is also considered on the failure side: if the errors are considered equivalent, then we also demand that the printed output is the same, for the result to be "both failed the same way, therefore the test passes".

Additional Context

I didn't go all the say to say all ACIR and Brillig errors are equivalent just yet. Let's relax them as they crop up, knowing that the backstop of comparing the prints is there.

I relaxed the ConstantDoesNotFitType error in the SSA interpreter to be considered equivalent if the constant is the same and the type is the same. I didn't do it for other errors for now, because the SSA interpreter does not capture print outputs yet.

Documentation*

Check one:

  • No documentation needed.
  • Documentation included in this PR.
  • [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist*

  • I have tested the changes locally.
  • I have formatted the changes with Prettier and/or cargo fmt on default settings.

@aakoshh aakoshh requested a review from a team June 16, 2025 10:14
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Execution Time'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.

Benchmark suite Current: a603ece Previous: 2e36b30 Ratio
rollup-merge 0.004 s 0.003 s 1.33

This comment was automatically generated by workflow using github-action-benchmark.

CC: @TomAFrench

@aakoshh aakoshh requested a review from rkarabut June 16, 2025 11:27
@aakoshh aakoshh added this pull request to the merge queue Jun 17, 2025
Merged via the queue into master with commit d068d38 Jun 17, 2025
118 checks passed
@aakoshh aakoshh deleted the af/fuzz-track-prints-in-failure branch June 17, 2025 09:25
github-merge-queue bot pushed a commit to AztecProtocol/aztec-packages that referenced this pull request Jun 18, 2025
Automated pull of nightly from the
[noir](https://github.com/noir-lang/noir) programming language, a
dependency of Aztec.
BEGIN_COMMIT_OVERRIDE
fix(LSP): suggest generic type methods
(noir-lang/noir#8948)
fix: Fix if/match tracking in last uses pass
(noir-lang/noir#8935)
fix(ssa): Swap Brillig index shift and DIE in minimal pipeline
(noir-lang/noir#8946)
fix: when macro parse error happens, discard warnings; also preserve
unquoted token locations (noir-lang/noir#8944)
chore: disable noncritical workflow steps which fail on external prs
(noir-lang/noir#8939)
chore(test): Add `--minimal-ssa` to integration tests
(noir-lang/noir#8938)
fix(noirc_evaluator): U128 Binary::And simplification
(noir-lang/noir#8940)
chore: Improve access to data required to extract Noir
(noir-lang/noir#8793)
chore: bump external pinned commits
(noir-lang/noir#8920)
chore(fuzz): Compare print output on failed programs
(noir-lang/noir#8921)
chore(fuzz): Display nightly fuzz status badge
(noir-lang/noir#8932)
fix(fuzz): Consider values returned from Brillig to ACIR as dynamic
(noir-lang/noir#8931)
fix(fuzz): Fix env var name in fuzzing workflow
(noir-lang/noir#8929)
chore(fuzz): Add nightly fuzz workflow with 5 minute budget
(noir-lang/noir#8925)
fix(mem2reg): Keep last store for a used nested array
(noir-lang/noir#8917)
chore(fuzz): Increase the number of deterministic test cases
(noir-lang/noir#8872)
fix: Create calls to `apply` before function values are changed to
fields in defunctionalize (noir-lang/noir#8916)
chore: Add various regression tests for mutable references
(noir-lang/noir#8915)
fix: handle return_data in the interpreter SSA CLI
(noir-lang/noir#8914)
feat(fuzz): Generate references in the AST fuzzer
(noir-lang/noir#8728)
fix(fuzz): Use an inline block to circumvent negation with overflow
(noir-lang/noir#8911)
fix: Inline global arrays with functions at their call site
(noir-lang/noir#8905)
chore: Casting ACIR vs Brillig regression tests
(noir-lang/noir#8903)
fix(mem2reg): Keep last store for reference in array used only in an
array get (noir-lang/noir#8877)
fix: more SSA interpreter fixes
(noir-lang/noir#8904)
chore: Ban dynamic array indices with reference elements
(noir-lang/noir#8888)
chore(ssa_fuzzer): Add SSA validation to the SSA fuzzer
(noir-lang/noir#8901)
chore(fuzz): More metamorphic rules
(noir-lang/noir#8857)
fix: catch unbound type variables during frontend compilation
(noir-lang/noir#8686)
fix: preserve functions which are used in `array_set` instructions
(noir-lang/noir#8891)
fix: assorted SSA interpreter fixes
(noir-lang/noir#8893)
fix(ssa): Signed cast simplification
(noir-lang/noir#8862)
feat: simplify apply function cfg immediately
(noir-lang/noir#8895)
fix: do not hoist control dependent cast
(noir-lang/noir#8886)
fix(licm): Account for negative bounds when checking whether a loop
executes (noir-lang/noir#8889)
chore: Release Noir(1.0.0-beta.7)
(noir-lang/noir#8520)
chore: test noir-ecdsa library in CI
(noir-lang/noir#8859)
chore: Incorrect clippy warning blocking PRs
(noir-lang/noir#8861)
END_COMMIT_OVERRIDE

---------

Co-authored-by: AztecBot <tech@aztecprotocol.com>
Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>
danielntmd pushed a commit to danielntmd/aztec-packages that referenced this pull request Jul 16, 2025
Automated pull of nightly from the
[noir](https://github.com/noir-lang/noir) programming language, a
dependency of Aztec.
BEGIN_COMMIT_OVERRIDE
fix(LSP): suggest generic type methods
(noir-lang/noir#8948)
fix: Fix if/match tracking in last uses pass
(noir-lang/noir#8935)
fix(ssa): Swap Brillig index shift and DIE in minimal pipeline
(noir-lang/noir#8946)
fix: when macro parse error happens, discard warnings; also preserve
unquoted token locations (noir-lang/noir#8944)
chore: disable noncritical workflow steps which fail on external prs
(noir-lang/noir#8939)
chore(test): Add `--minimal-ssa` to integration tests
(noir-lang/noir#8938)
fix(noirc_evaluator): U128 Binary::And simplification
(noir-lang/noir#8940)
chore: Improve access to data required to extract Noir
(noir-lang/noir#8793)
chore: bump external pinned commits
(noir-lang/noir#8920)
chore(fuzz): Compare print output on failed programs
(noir-lang/noir#8921)
chore(fuzz): Display nightly fuzz status badge
(noir-lang/noir#8932)
fix(fuzz): Consider values returned from Brillig to ACIR as dynamic
(noir-lang/noir#8931)
fix(fuzz): Fix env var name in fuzzing workflow
(noir-lang/noir#8929)
chore(fuzz): Add nightly fuzz workflow with 5 minute budget
(noir-lang/noir#8925)
fix(mem2reg): Keep last store for a used nested array
(noir-lang/noir#8917)
chore(fuzz): Increase the number of deterministic test cases
(noir-lang/noir#8872)
fix: Create calls to `apply` before function values are changed to
fields in defunctionalize (noir-lang/noir#8916)
chore: Add various regression tests for mutable references
(noir-lang/noir#8915)
fix: handle return_data in the interpreter SSA CLI
(noir-lang/noir#8914)
feat(fuzz): Generate references in the AST fuzzer
(noir-lang/noir#8728)
fix(fuzz): Use an inline block to circumvent negation with overflow
(noir-lang/noir#8911)
fix: Inline global arrays with functions at their call site
(noir-lang/noir#8905)
chore: Casting ACIR vs Brillig regression tests
(noir-lang/noir#8903)
fix(mem2reg): Keep last store for reference in array used only in an
array get (noir-lang/noir#8877)
fix: more SSA interpreter fixes
(noir-lang/noir#8904)
chore: Ban dynamic array indices with reference elements
(noir-lang/noir#8888)
chore(ssa_fuzzer): Add SSA validation to the SSA fuzzer
(noir-lang/noir#8901)
chore(fuzz): More metamorphic rules
(noir-lang/noir#8857)
fix: catch unbound type variables during frontend compilation
(noir-lang/noir#8686)
fix: preserve functions which are used in `array_set` instructions
(noir-lang/noir#8891)
fix: assorted SSA interpreter fixes
(noir-lang/noir#8893)
fix(ssa): Signed cast simplification
(noir-lang/noir#8862)
feat: simplify apply function cfg immediately
(noir-lang/noir#8895)
fix: do not hoist control dependent cast
(noir-lang/noir#8886)
fix(licm): Account for negative bounds when checking whether a loop
executes (noir-lang/noir#8889)
chore: Release Noir(1.0.0-beta.7)
(noir-lang/noir#8520)
chore: test noir-ecdsa library in CI
(noir-lang/noir#8859)
chore: Incorrect clippy warning blocking PRs
(noir-lang/noir#8861)
END_COMMIT_OVERRIDE

---------

Co-authored-by: AztecBot <tech@aztecprotocol.com>
Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants