From 0eecb549b00bb8c477b6f6a1e775a0cac58fe2b9 Mon Sep 17 00:00:00 2001 From: Andrea Simeoni Date: Fri, 28 Apr 2023 12:36:05 +0200 Subject: [PATCH 1/5] Docs for in-line test configuration. --- src/reference/config/README.md | 1 + src/reference/config/inline-test-config.md | 63 ++++++++++++++++++++++ 2 files changed, 64 insertions(+) create mode 100644 src/reference/config/inline-test-config.md diff --git a/src/reference/config/README.md b/src/reference/config/README.md index 8d36432fc..214e14f6d 100644 --- a/src/reference/config/README.md +++ b/src/reference/config/README.md @@ -4,6 +4,7 @@ - [Project](./project.md) - [Solidity Compiler](./solidity-compiler.md) - [Testing](./testing.md) +- [In-line test configuration](./inline-test-config.md) - [Formatter](./formatter.md) - [Documentation Generator](./doc-generator.md) - [Etherscan](./etherscan.md) diff --git a/src/reference/config/inline-test-config.md b/src/reference/config/inline-test-config.md new file mode 100644 index 000000000..445bf6342 --- /dev/null +++ b/src/reference/config/inline-test-config.md @@ -0,0 +1,63 @@ +## In-line test configuration +Foundry users are enabled to specify overall test configurations, using a combination of ENV variables and config statements in the `foundry.toml`. Checkout the [📚 Testing reference](./testing) for a detailed description. + +Despite this may work in the general case, some tests may need finer control over their configuration. For such reason Forge provides a way to specify per-test configs for invariant and fuzz testing scenarios. + +Users can in-line test config statements directly in Solidity comments. This would affect the behavior of the `forge test` command for a specific test instance, as illustrated in the example below. + +```solidity +contract MyTest is Test { + /// forge-config: default.fuzz.runs = 100 + /// forge-config: ci.fuzz.runs = 500 + function test_SimpleFuzzTest(uint256 x) public { + // --- snip --- + } +} +``` + +What we are asking here is to run our fuzzer `100` and `500` times for the `default` and `ci` profiles respectively. + +### In-line fuzz configs +Users can specify the configs described in the table. Each statement must have a prefix of the form `forge-config: ${PROFILE}.fuzz.` + +| Parameter | Type | Description | +|-|-|-| +|`runs`|integer|The amount of fuzz runs to perform for this specific test case [📚 ref](./testing#runs).| +|`max-test-rejects`|integer|The maximum number of combined inputs that may be rejected before the test as a whole aborts [📚 ref](./testing#max_test_rejects).| + +Fuzz config example +```solidity +contract MyFuzzTest is Test { + /// forge-config: default.fuzz.runs = 100 + /// forge-config: default.fuzz.max-test-rejects = 2 + function test_InlineConfig(uint256 x) public { + // --- snip --- + } +} +``` + +### In-line invariant configs +Users can specify the configs described in the table. Each statement must have a prefix of the form `forge-config: ${PROFILE}.invariant.` + +| Parameter | Type | Description | +|-|-|-| +|`runs`|integer|The amount of invariant runs to perform for this specific test case [📚 ref](./testing#runs-1). +|`depth`|integer|The number of calls executed to attempt to break invariant in one run [📚 ref](./testing#depth). +|`fail-on-revert`|boolean|Fails the invariant fuzzing if a revert occurs [📚 ref](./testing#fail_on_revert). +|`call-override`|boolean|Overrides unsafe external calls when running invariant test [📚 ref](./testing#call_override). + +Invariant config example +```solidity +contract MyInvariantTest is Test { + /// forge-config: default.invariant.runs = 100 + /// forge-config: default.invariant.depth = 2 + /// forge-config: default.invariant.fail-on-revert = false + /// forge-config: default.invariant.call-override = true + function invariant_InlineConfig() public { + // --- snip --- + } +} +``` + + + From 54e67405070f2d9f54ae6454008a1400e69ed2d1 Mon Sep 17 00:00:00 2001 From: Andrea Simeoni Date: Fri, 28 Apr 2023 13:10:14 +0200 Subject: [PATCH 2/5] Added description about base config inheritance --- src/reference/config/inline-test-config.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/reference/config/inline-test-config.md b/src/reference/config/inline-test-config.md index 445bf6342..a7d87be98 100644 --- a/src/reference/config/inline-test-config.md +++ b/src/reference/config/inline-test-config.md @@ -15,7 +15,8 @@ contract MyTest is Test { } ``` -What we are asking here is to run our fuzzer `100` and `500` times for the `default` and `ci` profiles respectively. +What we are asking here is to run our fuzzer `100` and `500` times for the `default` and `ci` profiles respectively. The interesting fact is that this would override any fuzz `runs` setup existing at a broader context. Such a context would still provide configs for parameters not explicitely reported in in-line statements, acting then as a fallback. + ### In-line fuzz configs Users can specify the configs described in the table. Each statement must have a prefix of the form `forge-config: ${PROFILE}.fuzz.` From 0c549edc0c6bcb0e1afe6e8c584ee92f38029977 Mon Sep 17 00:00:00 2001 From: Andrea Simeoni Date: Fri, 28 Apr 2023 13:17:47 +0200 Subject: [PATCH 3/5] Rewording --- src/reference/config/inline-test-config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/reference/config/inline-test-config.md b/src/reference/config/inline-test-config.md index a7d87be98..6510f644c 100644 --- a/src/reference/config/inline-test-config.md +++ b/src/reference/config/inline-test-config.md @@ -15,7 +15,7 @@ contract MyTest is Test { } ``` -What we are asking here is to run our fuzzer `100` and `500` times for the `default` and `ci` profiles respectively. The interesting fact is that this would override any fuzz `runs` setup existing at a broader context. Such a context would still provide configs for parameters not explicitely reported in in-line statements, acting then as a fallback. +What we are asking here is to run our fuzzer `100` and `500` times for the `default` and `ci` profiles respectively. The interesting fact is that this would override any fuzz `runs` setup existing at a global level. All other configs would be inherited from the global context, making this acting as a fallback for all possible configurations. ### In-line fuzz configs From d165214fa4179b5aedbcb752fc1b9f184a1b9509 Mon Sep 17 00:00:00 2001 From: Andrea Simeoni Date: Fri, 28 Apr 2023 15:20:34 +0200 Subject: [PATCH 4/5] Pages that describe "tests" now reference pages that describe how to config tests. --- src/forge/fuzz-testing.md | 7 ++++++- src/forge/invariant-testing.md | 7 +++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/src/forge/fuzz-testing.md b/src/forge/fuzz-testing.md index 1a9f588a4..106d4deb2 100644 --- a/src/forge/fuzz-testing.md +++ b/src/forge/fuzz-testing.md @@ -64,6 +64,11 @@ There are different ways to run property-based tests, notably parametric testing You might have noticed that fuzz tests are summarized a bit differently compared to unit tests: -- "runs" refers to the amount of scenarios the fuzzer tested. By default, the fuzzer will generate 256 scenarios, however, this can be configured using the [`FOUNDRY_FUZZ_RUNS`](../reference/config/testing.md#runs) environment variable. +- "runs" refers to the amount of scenarios the fuzzer tested. By default, the fuzzer will generate 256 scenarios, but this and other test execution parameters can be setup by the user. Fuzzer configuration details are provided [`here`](#configuring-fuzz-test-execution). - "μ" (Greek letter mu) is the mean gas used across all fuzz runs - "~" (tilde) is the median gas used across all fuzz runs + +### Configuring fuzz test execution + +Fuzz tests execution is governed by parameters that can be controlled by users via Forge configuration primitives. Configs can be applied globally or on a per-test basis. For details on this topic please refer to + 📚 [Global config](../reference/config/testing.md) and 📚 [In-line config](../reference/config/inline-test-config.md). diff --git a/src/forge/invariant-testing.md b/src/forge/invariant-testing.md index cfb220b04..666f1fc8a 100644 --- a/src/forge/invariant-testing.md +++ b/src/forge/invariant-testing.md @@ -9,8 +9,15 @@ Invariant testing campaigns have two dimensions, `runs` and `depth`: - `runs`: Number of times that a sequence of function calls is generated and run. - `depth`: Number of function calls made in a given `run`. All defined invariants are asserted after each function call is made. If a function call reverts, the `depth` counter still increments. +These and other invariant configuration aspects are explained [`here`](#configuring-invariant-test-execution). + Similar to how standard tests are run in Foundry by prefixing a function name with `test`, invariant tests are denoted by prefixing the function name with `invariant` (e.g., `function invariant_A()`). +### Configuring invariant test execution + +Invariant tests execution is governed by parameters that can be controlled by users via Forge configuration primitives. Configs can be applied globally or on a per-test basis. For details on this topic please refer to + 📚 [Global config](../reference/config/testing.md) and 📚 [In-line config](../reference/config/inline-test-config.md). + ## Defining Invariants Invariants are conditions expressions that should always hold true over the course of a fuzzing campaign. A good invariant testing suite should have as many invariants as possible, and can have different testing suites for different protocol states. From aa6ff26daf3e90b836bee56c1e57e006adabbd37 Mon Sep 17 00:00:00 2001 From: Andrea Simeoni Date: Sat, 29 Apr 2023 14:39:17 +0200 Subject: [PATCH 5/5] Config in block comments explained --- src/reference/config/inline-test-config.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/reference/config/inline-test-config.md b/src/reference/config/inline-test-config.md index 6510f644c..705ccbff1 100644 --- a/src/reference/config/inline-test-config.md +++ b/src/reference/config/inline-test-config.md @@ -17,6 +17,20 @@ contract MyTest is Test { What we are asking here is to run our fuzzer `100` and `500` times for the `default` and `ci` profiles respectively. The interesting fact is that this would override any fuzz `runs` setup existing at a global level. All other configs would be inherited from the global context, making this acting as a fallback for all possible configurations. +### Block comments +In-line test configurations can also be expressed in block comments, as illustrated in the example. + +```solidity +contract MyTest is Test { + /** + * forge-config: default.fuzz.runs = 1024 + * forge-config: default.fuzz.max-test-rejects = 500 + */ + function test_SimpleFuzzTest(uint256 x) public { + // --- snip --- + } +} +``` ### In-line fuzz configs Users can specify the configs described in the table. Each statement must have a prefix of the form `forge-config: ${PROFILE}.fuzz.`