Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Order of test transforms influences flakiness of tests #903

Open
Gedochao opened this issue Feb 28, 2025 · 0 comments
Open

Order of test transforms influences flakiness of tests #903

Gedochao opened this issue Feb 28, 2025 · 0 comments

Comments

@Gedochao
Copy link

This works:

//> using scala 2.13
//> using dep org.scalameta::munit:1.1.0

import munit.Tag

class CustomFlakyTests extends munit.FunSuite {
  override def munitFlakyOK: Boolean = true

  private val CustomFlakyTag = new Tag("CustomFlaky")

  override def munitTestTransforms: List[TestTransform] =
    List(
      new TestTransform(
        "Custom Flaky",
        { test =>
          if (test.tags.contains(CustomFlakyTag)) test.tag(munit.Flaky)
          else test
        }
      )
    ) ++ super.munitTestTransforms

  test("fail".tag(CustomFlakyTag)) {
    assert(false)
  }
}

This doesn't:

//> using scala 2.13
//> using dep org.scalameta::munit:1.1.0

import munit.Tag

class CustomFlakyTests extends munit.FunSuite {
  override def munitFlakyOK: Boolean = true

  private val CustomFlakyTag = new Tag("CustomFlaky")

  override def munitTestTransforms: List[TestTransform] =
    super.munitTestTransforms ++ List(
      new TestTransform(
        "Custom Flaky",
        { test =>
          if (test.tags.contains(CustomFlakyTag)) test.tag(munit.Flaky)
          else test
        }
      )
    )

  test("fail".tag(CustomFlakyTag)) {
    assert(false)
  }
}

Took me a good bit to figure out it's just the matter of order of transforms, and by the time I add the Flaky tag in mine, the tests have already been evaluated.
At least a warning would go a long way here, I think.
Discovered while working on: VirtusLab/scala-packager#213

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant