Change backend threading behavior

[afilogo]

This PR is based on the discussion #2159, having into account your feedback. I apologize for the late reply and hope did not make any mistake.

[github-actions]

Review checklist

This checklist is meant to assist creators of PRs (to let them know what reviewers will typically look for) and reviewers (to guide them in a structured review process). Items do not need to be checked explicitly for a PR to be eligible for merging.

Purpose and scope

• The PR has a single goal that is clear from the PR title and/or description.
• All code changes represent a single set of modifications that logically belong together.
• No more than 500 lines of code are changed or there is no obvious way to split the PR into multiple PRs.

Code quality

• The code can be understood easily.
• Newly introduced names for variables etc. are self-descriptive and consistent with existing naming conventions.
• There are no redundancies that can be removed by simple modularization/refactoring.
• There are no leftover debug statements or commented code sections.
• The code adheres to our conventions and style guide, and to the Julia guidelines.

Documentation

• New functions and types are documented with a docstring or top-level comment.
• Relevant publications are referenced in docstrings (see example for formatting).
• Inline comments are used to document longer or unusual code sections.
• Comments describe intent ("why?") and not just functionality ("what?").
• If the PR introduces a significant change or new feature, it is documented in NEWS.md with its PR number.

Testing

• The PR passes all tests.
• New or modified lines of code are covered by tests.
• New or modified tests run in less then 10 seconds.

Performance

• There are no type instabilities or memory allocations in performance-critical parts.
• If the PR intent is to improve performance, before/after time measurements are posted in the PR.

Verification

• The correctness of the code was verified using appropriate tests.
• If new equations/methods are added, a convergence test has been run and the results
  are posted in the PR.

Created with ❤️ by the Trixi.jl community.

[DanielDoehring]

Thanks for your contribution! I am not so sure if we really want to offer other threading strategies (namely dynamic) in the main repository. Same holds also for the serial option - IMO, this is not really necessary, as one could just run the program with a single thread.

[afilogo]

Thanks for the feedback @DanielDoehring and @vchuravy. Appreciate you taking the time to review it. I thought it might increase user control, but I also understand your perspective.

[vchuravy]

Thanks that seems like a good improvement.

[codecov]

Codecov Report

Attention: Patch coverage is 40.00000% with 6 lines in your changes missing coverage. Please review.

Project coverage is 96.79%. Comparing base (55d44d9) to head (657311f).
Report is 41 commits behind head on main.

Files with missing lines	Patch %	Lines
src/auxiliary/math.jl	0.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2417      +/-   ##
==========================================
- Coverage   96.80%   96.79%   -0.01%     
==========================================
  Files         495      495              
  Lines       40917    40919       +2     
==========================================
- Hits        39608    39607       -1     
- Misses       1309     1312       +3     

Flag	Coverage Δ
unittests	96.79% <40.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ranocha]

CI is failing, so it appears that something is off. Could you please take a look and see what's going on?

[afilogo]

CI is failing, so it appears that something is off. Could you please take a look and see what's going on?

I believe the issue is the in static macro "@static if _PREFERENCE_THREADING === :polyester && LoopVectorization.check_args(u_ode)" code line. It should be fine now.

[afilogo]

I've also missed a previous _PREFERENCE_POLYESTER line. Sorry about that.

[ranocha]

Could you please add an entry to the NEWS.md file since this is a breaking change?

[ranocha]

This will introduce bugs for code like

Trixi.jl/src/solvers/dgsem_tree/dg_2d.jl

Lines 945 to 950 in f35d749

	@threaded for mortar in eachmortar(dg, cache)
	# Choose thread-specific pre-allocated container
	fstar_primary_upper = fstar_primary_upper_threaded[Threads.threadid()]
	fstar_primary_lower = fstar_primary_lower_threaded[Threads.threadid()]
	fstar_secondary_upper = fstar_secondary_upper_threaded[Threads.threadid()]
	fstar_secondary_lower = fstar_secondary_lower_threaded[Threads.threadid()]

won't it? @vchuravy

[vchuravy]

Oof, yes it will. @benegee and I were just talking about #2212 and the fact that creating or adapting the cache is a weird operation due to these backend specific containers. We are baking quite a bit of structure that is dependent on how we are performing the computation into the cache object.

So eventually we might need something like create_cache(..., backend), but even that is fraught since with #2212 I am able to change the storage type and thus the backend.

Maybe, we need struct StaticThreadIDCache end and struct GPUCache end...

[ranocha]

So I think we will need to remove the "dynamic" option for now from this PR.

[ranocha]

Suggested change

	elseif _PREFERENCE_THREADING === :dynamic
	quote
	let
	if $Threads.nthreads() == 1
	$(expr)
	else
	$Threads.@threads :dynamic $(expr)
	end
	end
	end

[vchuravy]

@afilogo thank you very much for your contribution. I am going to bring this PR to the finish line in #2476