Invariants feat: add config option to turn off shrinking by grandizzy · Pull Request #4868 · foundry-rs/foundry

grandizzy · 2023-05-02T19:11:42Z

Motivation

See #4867, when running invariants test with many runs and depth (1000) the test hangs in shrinking scenario (performed for each invariant). In this case sometimes is preferable to fail fast and address the issue.

Solution

add invariant_shrink_sequence option, default true, tries to reduce number of calls in scenario to min
when set to false test will fail fast and output the original number of calls in scenario

…uce number of calls in scenario to min - when set to false test will just error with the original number of calls in scenario

mds1 · 2023-05-03T12:51:55Z

@0xMelkor note that once this is merged we'll need to update #4744 to support this option in there too

0xMelkor · 2023-05-03T17:57:14Z

@0xMelkor note that once this is merged we'll need to update #4744 to support this option in there too

Hi @mds1, sure.

mattsse

smol nit, otherwise lgtm

mattsse · 2023-05-04T17:41:03Z

evm/src/fuzz/invariant/error.rs

+        let _calls = match self.test_error {
            // Don't use at the moment.
            TestError::Abort(_) => return Ok(None),
-            TestError::Fail(_, ref calls) => calls,
+            TestError::Fail(_, ref _calls) => _calls,
        };

-        let calls = self.try_shrinking(calls, &executor);
+        if self.shrink {
+            let _calls = self.try_shrinking(_calls, &executor);
+        } else {
+            trace!(target: "forge::test", "Shrinking disabled.");


please remove _ prefix

grandizzy · 2023-05-04T17:55:18Z

smol nit, otherwise lgtm

yep, added such to avoid warning: unused variable: if this is intentional, prefix it with an underscore is there a different preferred way to disable this?

mattsse · 2023-05-04T18:01:32Z

but

let _calls =

let _calls = self.try_shrinking(_calls, &executor);

so I think it should be let calls = ...; let _ self.try_shrinking(calls,&executor)

grandizzy · 2023-05-04T18:05:18Z

but

let _calls =

let _calls = self.try_shrinking(_calls, &executor);

so I think it should be let calls = ...; let _ self.try_shrinking(calls,&executor)

oops, right, changing

grandizzy · 2023-05-04T18:30:58Z

but

let _calls =

let _calls = self.try_shrinking(_calls, &executor);
so I think it should be let calls = ...; let _ self.try_shrinking(calls,&executor)

oops, right, changing

fixed with 0226461

grandizzy · 2023-05-10T13:41:50Z

but

let _calls =

let _calls = self.try_shrinking(_calls, &executor);
so I think it should be let calls = ...; let _ self.try_shrinking(calls,&executor)

oops, right, changing

fixed with 0226461

hey @mattsse just bumping this, when you have time please check the change, would really love to get this in. thank you!

mattsse

cool, tysm!

lgtm!

Evalir

LGTM! Just pushed something upstream to make it compatible with #4744. Nice work!

deferring to @mattsse for merge

Evalir · 2023-05-12T00:01:21Z

config/src/invariant.rs

                "depth" => conf_clone.depth = parse_config_u32(key, value)?,
                "fail-on-revert" => conf_clone.fail_on_revert = parse_config_bool(key, value)?,
                "call-override" => conf_clone.call_override = parse_config_bool(key, value)?,
+                "shrink-sequence" => conf_clone.shrink_sequence = parse_config_bool(key, value)?,


added the new shrink-sequence key to the invariant inlineconfigparser so it works with #4744

grandizzy · 2023-05-12T15:22:00Z

thank you all!

grandizzy added 2 commits May 2, 2023 22:05

- add try_shrinking config in [invariant], default true, tries to red…

d340bfb

…uce number of calls in scenario to min - when set to false test will just error with the original number of calls in scenario

Add comment

6d8bf9e

grandizzy mentioned this pull request May 3, 2023

feat: add config option to turn off shrinking #4867

Closed

Improve option name: shrink_sequence

651cd26

grandizzy force-pushed the git-4867 branch from d4a4546 to 651cd26 Compare May 3, 2023 06:35

mattsse requested changes May 4, 2023

View reviewed changes

Changes after request: proper way to fix warnings

0226461

grandizzy requested a review from mattsse May 4, 2023 18:31

Merge branch 'master' into git-4867

e4d2188

mattsse approved these changes May 11, 2023

View reviewed changes

add shrink-sequence to valid config keys for the parser

bbdd5bc

Evalir approved these changes May 12, 2023

View reviewed changes

mattsse approved these changes May 12, 2023

View reviewed changes

mattsse merged commit 9eb8974 into foundry-rs:master May 12, 2023

grandizzy mentioned this pull request Jul 7, 2023

Invariant testing: read shrink sequence config when assert invariants that aren't broken yet #5323

Merged

Conversation

grandizzy commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Solution

Uh oh!

mds1 commented May 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0xMelkor commented May 3, 2023

Uh oh!

mattsse left a comment

Choose a reason for hiding this comment

Uh oh!

mattsse May 4, 2023

Choose a reason for hiding this comment

Uh oh!

grandizzy commented May 4, 2023

Uh oh!

mattsse commented May 4, 2023

Uh oh!

grandizzy commented May 4, 2023

Uh oh!

grandizzy commented May 4, 2023

Uh oh!

grandizzy commented May 10, 2023

Uh oh!

mattsse left a comment

Choose a reason for hiding this comment

Uh oh!

Evalir left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Evalir May 12, 2023

Choose a reason for hiding this comment

Uh oh!

grandizzy commented May 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

grandizzy commented May 2, 2023 •

edited

Loading

mds1 commented May 3, 2023 •

edited

Loading

Evalir left a comment •

edited

Loading