feat: ux fuzz invariant

[0xMelkor]

Motivation

Provide a solution for #3062, #4085 and updates #3411.

Some tests need more/less fuzzing than others, so it would be great to configure this on a per-test basis.

As described in the issue we are going to support Solidity comments as a new configuration source.

contract MyTest is Test {
  /// forge-config: default.fuzz.runs = 100
  /// forge-config: ci.fuzz.runs = 500
  function test_SimpleFuzzTest(uint256 x) public {
    // --- snip ---
  }

  /// forge-config: default.fuzz.runs = 500
  /// forge-config: ci.fuzz.runs = 10000
  function test_ImportantFuzzTest(uint256 x) public {
    // --- snip ---
  }
}

Solution

Tasks

• Create a parser able to convert structured text into configuration structs (FuzzConfig, InvariantConfig)
• Parser should take care of the current execution profile (ci, default, ecc.)
• Use ProjectCompileOutput to extract inline per-test configs.
• Provide new configs to the test runner extending the behavior of TestOptions.
• Override project config at runtime if required by the specific test

NOTE:
For the sake of KISS I used a standalone parser (ConfParser) instead of a figment::Provider. If you prefer the second approach just tell me.

Cheers

[mattsse]

really appreciate this.

before we proceed with this, we should discuss how exactly this would look like in the toml file.

could you please add same examples so we can discuss,

cc @mds1

[0xMelkor]

could you please add same examples so we can discuss

Hi @mattsse,

This works inlining config comments in solidity test functions, to override wider scoped toml params.

E2E tests can help to understand:

1. Annotated sol tests FuzzInlineConf.t.sol, InvariantInlineConf.t.sol;
2. E2E tests show how to extract FuzzConfig and InvariantConfig from compiled inline comments;

Let's discuss this.

[mds1]

before we proceed with this, we should discuss how exactly this would look like in the toml file.

@mattsse I don't think we should need any config file changes here. Like @0xMelkor said, the idea is that the toml file's profiles would give the default config, and you can override the config at the file, contract, or test level using comments like this

---

Hey @0xMelkor, should this work yet? I setup a new project and modified the default tests to look like this:

pragma solidity ^0.8.13;

import "forge-std/Test.sol";
import "../src/Counter.sol";

contract CounterTest is Test {
    Counter public counter;

    function setUp() public {
        counter = new Counter();
        counter.setNumber(0);
    }

    function testSetNumberNoChange(uint256 x) public {
        counter.setNumber(x);
        assertEq(counter.number(), x);
    }

    // forge-config: default.fuzz.runs = 111
    // forge-config: ci.fuzz.runs = 222
    function testSetNumberCustom(uint256 x) public {
        counter.setNumber(x);
        assertEq(counter.number(), x);
    }
}

And the config file looks like this:

[profile.ci]
  fuzz = { runs = 1000 }

But when I run this branch, testSetNumberCustom and testSetNumberNoChange always have the same number of fuzz runs. I tried using both 2 and 3 leading slashes for the comment, with the same result

[0xMelkor]

Hey @0xMelkor, should this work yet?

Hi @mds1,
For now we can load the configs but the execution part is not yet integrated.

If you agree with the proposed design I would continue with the execution part.
Just let me know.

Thank you guys

[mds1]

Ah got it, sorry didn't realize! This design lgtm, cc @mattsse @PaulRBerg @PatrickAlphaC for any final thoughts on UX here.

Per the discussion in #3062 I think one important part is throwing an error if we see a comment that starts with forge-config that we're not able to parse properly, to ensure you don't accidentally test with the wrong config due a typo

[PaulRBerg]

The proposed design looks great, thanks for picking up this task @0xMelkor!

I agree with @mds1 that it would be helpful to throw an error when the comment contains an erroneous config. If we don't do this, there will be lots and lots of red herrings in Foundry code bases, i.e. comments that mislead readers by making them believe there is a bespoke fuzzing configuration when in fact there isn't any.

Regarding execution - the comment config will be disregarded if it refers to a profile other than the one used by Forge during runtime, correct?

[PatrickAlphaC]

Nice!!
Does this work for fail_on_revert for invariants?

pragma solidity ^0.8.13;

import "forge-std/Test.sol";
import "../src/Counter.sol";

contract CounterTest is Test {
    Counter public counter;

    function setUp() public {
        counter = new Counter();
        counter.setNumber(0);
    }

    // forge-config: default.invariant.fail_on_revert = false
    function invariant_SetNumberCustom() public {
        assert(counter.number() < 100);
    }
}

[0xMelkor]

Does this work for fail_on_revert for invariants?

Yes this field is mapped

foundry/config/src/invariant.rs

Line 62 in 7d540c0

"fail-on-revert" => conf.fail_on_revert = value.parse()?,

The difference is that it must be provided in kebab case:

    // forge-config: default.invariant.fail-on-revert = false
    function invariant_SetNumberCustom() public {
        assert(counter.number() < 100);
    }

This PR is still in draft. Very soon these configs will be applied at execution time.

[0xMelkor]

Hi guys,

Fuzz inline configs are interpreted and applied at runtime. I added some tests that confirm my assertion.
Invariant integration is coming soon.

[0xMelkor]

Suggestions applied @Evalir
We will bring light where is missing mate.

[0xMelkor]

Some tests were failing because the new logic cannot find a foundry.toml to extract profiles:

foundry/cli/src/cmd/forge/test/mod.rs

Line 155 in fa97f9a

let profiles = get_available_profiles(toml)?;

For such reason I added a fallback value in the case the file doesn't exist:

foundry/config/src/utils.rs

Lines 182 to 187 in fa97f9a

	pub fn get_available_profiles(toml_path: impl AsRef<Path>) -> eyre::Result<Vec<String>> {
	let mut result = vec![Config::DEFAULT_PROFILE.to_string()];
	if !toml_path.as_ref().exists() {
	return Ok(result)
	}

Is this acceptable?

Failing tests:

• can test cmd::can_check_snapshot
• test test_cmd::can_fuzz_array_params
• test test_cmd::suggest_when_no_tests_match
• test test_cmd::warn_no_tests
• test test_cmd::warn_no_tests_match
• test test_cmd::can_test_with_match_path
• test test_cmd::can_test_pre_bytecode_hash

[mattsse]

Changes look good, but failed tests are ominous,

passing on master

some of them look config-related I think.

what's the reason they're failing?

[0xMelkor]

No more problems with tests.
If a foundry.toml is not available, as it happens for some e2e tests, we return vec![Config::DEFAULT_PROFILE] as a fallback for all available profiles.
Discussed with @mattsse

[mattsse]

okay, @0xMelkor answered a few questions, the change regarding profiles fixed the failing tests.

it looks pretty good.

mind giving this another look as well @Evalir ?

[Evalir]

this lgtm—just the nit on the InlineConfigParser impls

[Evalir]

ok so if i understand correctly this is a copy that we're later mutating and returning? can we do self.clone()? This doesn't seem too idiomatic

[mds1]

Right now the error looks like this:

Inline config Error detected at test/Counter.t.sol:CounterTest:testSetNumber1:"288:82:20"

Not sure if useful, but I think @mattsse added an offset_to_line to the forge fmt error messages which may useful to get a line number here. Or if we can get rid of the quotes around the 288:82:20 that might look a bit nicer

[mds1]

Related, it looks like the error is actually duplicated, once in red, and again in the "context" section. Is that intentional?

[mds1]

Two small comments otherwise this is awesome and looks great!

[0xMelkor]

Two small comments otherwise this is awesome and looks great!

Nice error now

[0xMelkor]

Hi guys,
Any chances to have this merged?
CI is green and last @mds1 comments are fixed.

[mattsse]

awesome!

everything lgtm!

@Evalir let's merge?

[mattsse]

awesome!

everything lgtm!

@Evalir let's merge?

[Evalir]

Sick! Happy with the changes too—merging!

Amazing work @0xMelkor ❤️

[0xMelkor]

Awesome!
We have also this docs PR
foundry-rs/book#880