Skip to content

feat(test): Check that nargo::ops::transform_program is idempotent#6694

Merged
aakoshh merged 15 commits intomasterfrom
6670-test-transform-is-idempotent
Dec 9, 2024
Merged

feat(test): Check that nargo::ops::transform_program is idempotent#6694
aakoshh merged 15 commits intomasterfrom
6670-test-transform-is-idempotent

Conversation

@aakoshh
Copy link
Contributor

@aakoshh aakoshh commented Dec 3, 2024

Description

Problem*

Resolves #6670

Summary*

Added a test in the compile_cmd module to run through all tests in test_programs/execution_success, compile them in-memory, then run them through the optimisation transformation twice, comparing the hashes after the first and the second pass, which should be equal unless the optimiser is not idempotent for one of these examples.

The idempotency test is currently failing for most test programs.

Testing

An example of a failing run:

cargo test -q -p nargo_cli --bins -- test_transform_program_is_idempotent

running 1 test
F

failures:

---- cli::compile_cmd::tests::test_transform_program_is_idempotent stdout ----
thread '<unnamed>' panicked at tooling/nargo_cli/src/cli/compile_cmd.rs:439:25:
optimization not idempotent for test program 'to_le_bytes'
...
thread '<unnamed>' panicked at tooling/nargo_cli/src/cli/compile_cmd.rs:439:25:
optimization not idempotent for test program 'ram_blowup_regression'

failures:
    cli::compile_cmd::tests::test_transform_program_is_idempotent

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 2 filtered out; finished in 9.02s

To see why a test fails, we can pass its name as an extra argument:

cargo test -q -p nargo_cli --bins -- test_transform_program_is_idempotent slice_loop

When an argument is given, the test compares the actual Programs so we can see the difference in opcodes, otherwise it just compares hashes to have a list of failing tests that we can potentially look at. Some have really long bytecode which would produce walls of text if the detailed output was always shown.

Applied various fixes in #6695

Additional Context

I added the test in compile_cmd because this is where the transformation is applied, where we are defending against idempotency problems; it is a defensive test to verify that we're right to assume this property, even though we're cautious and avoid running it on cached programs twice. The module also contained code to read manifests and parse files into a workspace, which could be conveniently reused in the test.

I thought about trying to generate a separate #[test] per directory in build.rs, but instead of running them as an integration test, splice them into a unit test module to get access to the utility functions for parsing the compiled artefacts. However, in the end I thought this test is less about the test programs themselves as just a sanity check that the optimiser produces stable results, ie. we expect it to be true for all programs, therefore having dozens of tests doing nothing that is really about the test program itself would be just noisy. One benefit of separate tests would have been that cargo would run them in parallel; I hope that using rayon reclaimed enough of this efficiency to make this a non-issue.

I am only testing on execution_success, but we could add the other test directories which are supposed to compile, and potentially even the stdlib tests I suppose. Let me know if you think more coverage is needed.

Documentation*

Check one:

  • No documentation needed.
  • Documentation included in this PR.
  • [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist*

  • I have tested the changes locally.
  • I have formatted the changes with Prettier and/or cargo fmt on default settings.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add some CI to test that nargo::ops::transform_program is idempotent.

2 participants