Skip to content

Conversation

@newhoggy
Copy link
Contributor

@newhoggy newhoggy commented Aug 10, 2025

Changelog

- description: |
    Migrate test suite to use the newer tasty-discover API which supports
    Flavored test monads for platform-specific test filtering
# uncomment types applicable to the change:
  type:
  # - feature        # introduces a new feature
  # - breaking       # the API has changed in a breaking way
  # - compatible     # the API has changed but is non-breaking
  # - optimisation   # measurable performance improvements
  # - refactoring    # QoL changes
  # - bugfix         # fixes a defect
    - test           # fixes/modifies tests
    - maintenance    # not directly related to the code
  # - release        # related to a new release preparation
  # - documentation  # change in code docs, haddocks...

Motivation

Background: Platform-Specific Testing Challenges

The cardano-cli help text tests needed to be skipped on Windows because the CLI executable has platform-specific differences in its output:

  • On Unix-like systems: cardano-cli
  • On Windows: cardano-cli.exe

Previous Approach (Manual Runtime Check)

The old implementation used a manual runtime platform check:

test_golden_HelpCmds :: IO TestTree
test_golden_HelpCmds =
  if isWin32
    then return $ testGroup "help-commands" []
    else do
      -- ... generate tests

Problems with this approach:

  1. Required importing Hedgehog.Extras.Stock.OS (isWin32) for platform detection
  2. Manual conditional logic scattered throughout test code
  3. Less declarative - the platform constraint is hidden in imperative code
  4. Doesn't leverage tasty's built-in test filtering capabilities

New Approach (Flavored API with Platform Filter)

The new implementation uses tasty-discover's Flavored type and platform function:

tasty_golden_HelpCmds :: Flavored (IO TestTree)
tasty_golden_HelpCmds =
  flavored (platform "!windows") $ do
    -- ... generate tests

Benefits:

  1. Declarative Platform Filtering: The platform constraint "!windows" is expressed declaratively
  2. Standardized API: Uses tasty-discover's standard platform expression syntax supporting complex logic (!, &, | operators)
  3. Consistent Skipping Behavior: Skipped tests show uniformly as [SKIPPED] in yellow in test output
  4. Better Integration: Works seamlessly with tasty's option system and test tree transformations
  5. Naming Convention: The tasty_ prefix (vs test_) signals that this test uses the custom Tasty instance mechanism, which is appropriate for Flavored transformations
  6. The type of the second argument of Flavored is IO TestTree so the test monad hasn't changed for the actual test.

Additional Context from tasty-discover 5.1.0

The tasty-discover 5.1.0 release added:

  • Platform-specific test filtering with platform function and logical expressions
  • Flavored type for general-purpose test transformations with extensible design
  • Support for complex platform expressions like "!windows", "linux | darwin", "!windows & !darwin"

The Flavored pattern is designed to be extensible for future use cases like:

  • Setting test timeouts
  • Adding test metadata
  • Applying multiple transformations in sequence
  • Custom test grouping with resource dependencies

Context

This PR upgrades the test infrastructure to use the newer tasty-discover API with Flavored test support. The key changes include:

  • cardano-cli.cabal: Added explicit tasty-discover dependency to the cardano-cli-golden test suite
  • Test/Golden/Help.hs: Migrated from the legacy test_* naming convention to tasty_* with the new Flavored API for platform-specific test filtering

The migration replaces the manual platform check (isWin32) with the cleaner flavored (platform "!windows") API, which provides better declarative platform filtering. Tests are conditionally executed based on platform constraints, preserving the existing behavior of skipping help text tests on Windows due to platform-specific output differences (e.g., cardano-cli.exe vs cardano-cli).

How to trust this PR

To verify this change works correctly:

  1. Build the project: cabal build all --enable-tests
  2. Run the golden tests: cabal test cardano-cli-golden --enable-tests
  3. Verify platform filtering: The help command golden tests should be skipped on Windows and run on other platforms

The tests should pass with the same coverage as before, but now use the updated test discovery mechanism with cleaner platform filtering.

Checklist

  • Commit sequence broadly makes sense and commits have useful messages
  • New tests are added if needed and existing tests are updated. See Running tests for more details
  • Self-reviewed the diff

@newhoggy newhoggy marked this pull request as draft August 10, 2025 02:52
@newhoggy newhoggy force-pushed the newhoggy/upgrade-tasty-discover branch 4 times, most recently from 6807ac9 to 46e041a Compare August 14, 2025 12:27
@newhoggy newhoggy force-pushed the newhoggy/upgrade-tasty-discover branch from 46e041a to 48d33d4 Compare September 11, 2025 22:27
@newhoggy newhoggy force-pushed the newhoggy/upgrade-tasty-discover branch from 48d33d4 to 541e030 Compare October 15, 2025 11:15
Migrates test suite to use the newer tasty-discover API which supports
Flavored test monads for platform-specific test filtering. This allows tests
to be conditionally executed based on platform constraints.

- Add tasty-discover dependency to cardano-cli.cabal
- Update Test.Golden.Help to use Flavored API
- Rename test_golden_HelpCmds to tasty_golden_HelpCmds
- Replace manual Windows platform check with flavored platform filter
- Import Test.Tasty.Discover for Flavored type and utilities

The platform "!windows" filter preserves the existing behavior of
skipping help text tests on Windows due to platform-specific output
differences (e.g., cardano-cli.exe vs cardano-cli).
@newhoggy newhoggy force-pushed the newhoggy/upgrade-tasty-discover branch from 541e030 to 14a386c Compare October 15, 2025 11:29
@newhoggy newhoggy changed the title Upgrade tasty-discover chore(test): upgrade tasty-discover dependency Oct 15, 2025
@newhoggy newhoggy marked this pull request as ready for review October 15, 2025 11:58
@newhoggy newhoggy changed the title chore(test): upgrade tasty-discover dependency chore(test): migrate to tasty-discover Flavored API for platform-specific test filtering Oct 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants