feat(test): only compile files needed for tests

[klkvr]

Motivation

Compile contracts requesting only abi output and filter out abis which match given filters or just start with test/invariant

This allowed to move some checks to be performed earlier ("No tests match the provided pattern")

This might be breaking for some codebases using vm.getDeployedCode as we won't recompile contracts which are not appearing as dependecy of test contract, however if contract being fetched through vm.getDeployedCode is imported from test file, then it will still be recompiled and I believe that it's a common pattern used, for example, in optimism codebase (https://github.com/ethereum-optimism/optimism/blob/53573e0ea6a807a125784cc5c7df07cbb4dbe3bc/packages/contracts-bedrock/scripts/Deployer.sol#L25)

Closes #7045

cc @mds1

[mds1]

Awesome, will test this!

This might be breaking for some codebases using vm.getDeployedCode as we won't recompile contracts which are not appearing as dependecy of test contract, however if contract being fetched through vm.getDeployedCode is imported from test file, then it will still be recompiled

Good callout. Is a potential fix here for forge to error if it sees a contract name in getDeployedCode that is not seen as an import? This would have to be optional though, as such a safety check is not always applicable (e.g. vyper contracts compiled in a prior step). Having that safety check as opt-out would probably be the conservative default

[klkvr]

Good callout. Is a potential fix here for forge to error if it sees a contract name in getDeployedCode that is not seen as an import? This would have to be optional though, as such a safety check is not always applicable (e.g. vyper contracts compiled in a prior step). Having that safety check as opt-out would probably be the conservative default

Oh, that's a nice idea. Yeah, we can manage the list of contracts which were compiled before current test run and only allow getDeployedCode invocations for freshly compiled/cached contracts.

I don't think that this will cause issues with Vyper: afaik people are using ffi to compile and get bytecode of Vyper contract (like here, for example: https://github.com/pcaversaccio/snekmate/blob/main/lib/utils/VyperDeployer.sol#L53-L80)

[mds1]

I still think there are cases where it might cause issues, like seaport, where they use one profile to compile, then another to test: https://github.com/ProjectOpenSea/seaport?tab=readme-ov-file#foundry-tests (cc @emo-eth to confirm since you used to work on seaport)

[mattsse]

+1 on @mds1 suggestion

this is now always invoked?
the overhead should be minimal?
perhaps we can skip this if there's no filter?

[mattsse]

I think there's an easier way to do this, if not we should make this easier in foundry-compilers

[klkvr]

After #87 we'll spend additional time only obtaining abi for dirty sources. If there's no filters we will filеer out contracts which don't have any invariant/test functions which reduces the full compilation time and is reasonable imo

This additional step results in ~4-5% build time increase for normal builds and in <1% increase for via-ir builds, so it should worth it

[klkvr]

I've added logic for runtime validation of getCode and getDeployedCode inputs. We are now collecting ArtifactIds after compilation and find match across them depending on user input.

This should make those cheatcodes impl more stable as we won't make any assumptions on artifact location.

I've also added ability to pass artifacts in the format of vm.getCode("src/Counter.sol:Counter:0.X.Y") which should resolve and provide a clear API for #7045.

This check can be disabled with unchecked_cheatcode_artifacts = true config flag (perhaps there is a better name for this?)

[onbjerg]

looks good to me, remember to close #7045 (and other issues if you find some) with this pr

[mattsse]

amazing

[mds1]

Testing on optimism codebase using a test suite that is pretty self contained and does not require most of the codebase:

Before this PR

Using 9f6bb3b

$ time forge t --mt testFuzz_handleCall_initiation_succeeds                    
[⠒] Compiling...
[⠑] Compiling 1 files with 0.5.17
[⠘] Compiling 16 files with 0.8.19
[⠃] Compiling 122 files with 0.8.23
[⠊] Compiling 347 files with 0.8.15
[⠘] Solc 0.5.17 finished in 48.71ms
[⠒] Solc 0.8.19 finished in 723.05ms
[⠒] Solc 0.8.23 finished in 4.66s
[⠒] Solc 0.8.15 finished in 163.61s
Compiler run successful!

Ran 1 test for test/L1/DelayedVetoable.t.sol:DelayedVetoable_HandleCall_Test
[PASS] testFuzz_handleCall_initiation_succeeds(bytes) (runs: 64, μ: 41080, ~: 40741)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 4.80ms (4.25ms CPU time)

Ran 1 test suite in 142.81ms (4.80ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)
forge t --mt testFuzz_handleCall_initiation_succeeds  168.29s user 2.35s system 102% cpu 2:46.42 total

After this PR

Using e758bd2, looks like it could not find the test

time forge t --mt testFuzz_handleCall_initiation_succeeds
No tests match the provided pattern:
        match-test: `testFuzz_handleCall_initiation_succeeds`
Error: 
No tests to run
forge t --mt testFuzz_handleCall_initiation_succeeds  0.04s user 0.07s system 65% cpu 0.182 total

[klkvr]

@mds1 can't really reproduce this failure locally, getting:

$ time forge t --mt testFuzz_handleCall_initiation_succeeds
[⠒] Compiling...
[⠆] Compiling 23 files with 0.8.15
[⠔] Solc 0.8.15 finished in 2.24s
Compiler run successful!

Ran 1 test for test/L1/DelayedVetoable.t.sol:DelayedVetoable_HandleCall_Test
[PASS] testFuzz_handleCall_initiation_succeeds(bytes) (runs: 64, μ: 41133, ~: 40746)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 23.72ms (20.42ms CPU time)

Ran 1 test suite in 163.19ms (23.72ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)
forge t --mt   7.58s user 0.54s system 114% cpu 7.092 total

can you check if issue still persists with purged cache and share logs with RUST_LOG=foundry_compilers if so?

[mds1]

Confirmed it is working now 👍

time forge t --mt testFuzz_handleCall_initiation_succeeds 
[⠒] Compiling...
[⠢] Compiling 23 files with 0.8.15
[⠆] Solc 0.8.15 finished in 2.11s
Compiler run successful!

Ran 1 test for test/L1/DelayedVetoable.t.sol:DelayedVetoable_HandleCall_Test
[PASS] testFuzz_handleCall_initiation_succeeds(bytes) (runs: 64, μ: 41089, ~: 40736)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 5.68ms (4.36ms CPU time)

Ran 1 test suite in 137.69ms (5.68ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)
forge t --mt testFuzz_handleCall_initiation_succeeds  6.54s user 0.45s system 111% cpu 6.296 total