Fix flaky test by a test env setup refactor

[komikat]

Fixes #4200 by creating a separate temporary .cache directory for every test instead of having all tests share the same .cache directory. This makes sure the test iface-error-test-1 is run with a clean directory — causing the diagnostic to be generated every time.

[soulomoon]

It seems hls-refactor-plugin-tests depend on the cache directory.🤔

/tmp/hls-test-root/.cache-b4d2c0447616dd51: createDirectory: does not exist (No such file or directory)

[fendor]

That seems impossible to me, there must be another race condition :D

[komikat]

Weird because hls-refactor-plugin-tests seems to pass on my system (MacOS, ghc9.8)

[komikat]

The setup wasn't creating the hls-test-root directory.
ghcide-tests is still failing, will look into this soon.

[komikat]

definition.Imported symbol (reexported) fails without $TEMPDIR/hls-test-root

running HLS_TEST_HARNESS_NO_TESTDIR_CLEANUP=1 HLS_TEST_LOG_STDERR=1 TASTY_PATTERN="/definition.Imported symbol (reexported)/" cabal test ghcide-tests
FAILS (>=GHC94)

expected: Location {_uri = Uri {getUri = "$TMPDIR/hls-test-root/extra-dir-85406152527/Bar.hs"}, _range = Range {_start = Position {_line = 3, _character = 5}, _end = Position {_line = 3, _character = 8}}}
 but got: Location {_uri = Uri {getUri = "$TMPDIR/hls-test-root/extra-dir-85406152527/Bar.hs"}, _range = Range {_start = Position {_line = 3, _character = 0}, _end = Position {_line = 3, _character = 14}}}

Notes

• AtPoint.hs.gotoDefinition calls locationsAtPoint
• it is using nameToLocation when failing
• itExists <- liftIO $ doesFileExist fs returns true
• doesn't seem to be falling back to db
• in srcSpanToLocation sp
rng <- srcSpanToRange src
  reports a different location from what's expected in the test
• the sp comes from
  nameSrcSpan name
  in that branch.

[fendor]

This is good!

Let's merge this as is, add the comment to the test case, create an issue containing the knowledge you have gained so far, and then accept the test case changes. E.g., update the assertions in the test. Even though we have discovered an interesting race (?) condition, the test case itself will be stable.

@komikat Could you do that, please?

[fendor]

Please add a comment to the test case that fails now and create a new tracking issue! LGTM otherwise

[soulomoon]

definition.Imported symbol (reexported) fails without $TEMPDIR/hls-test-root

running HLS_TEST_HARNESS_NO_TESTDIR_CLEANUP=1 HLS_TEST_LOG_STDERR=1 TASTY_PATTERN="/definition.Imported symbol (reexported)/" cabal test ghcide-tests FAILS (>=GHC94)

expected: Location {_uri = Uri {getUri = "$TMPDIR/hls-test-root/extra-dir-85406152527/Bar.hs"}, _range = Range {_start = Position {_line = 3, _character = 5}, _end = Position {_line = 3, _character = 8}}}
 but got: Location {_uri = Uri {getUri = "$TMPDIR/hls-test-root/extra-dir-85406152527/Bar.hs"}, _range = Range {_start = Position {_line = 3, _character = 0}, _end = Position {_line = 3, _character = 14}}}

Notes

• AtPoint.hs.gotoDefinition calls locationsAtPoint
• it is using nameToLocation when failing
• itExists <- liftIO $ doesFileExist fs returns true
• doesn't seem to be falling back to db
• in srcSpanToLocation sp
rng <- srcSpanToRange src
  reports a different location from what's expected in the test
• the sp comes from
  nameSrcSpan name
  in that branch.

This is a bug in our code, It worth a seperate issue to fix this.

[komikat]

The test has this condition which seems to be wrong

if ghcVersion >= GHC94 && ghcVersion < GHC910 then mkL bar 3 5 3 8 else mkL bar 3 0 3 14

I've set it to just be 3 0 3 14 for all tests for now, since that's what the new test suite seems to accept.

[komikat]

Hm, still failing a few windows tests.