Print global number of cells and dofs

[benegee]

Resolves #1616

[github-actions]

Review checklist

This checklist is meant to assist creators of PRs (to let them know what reviewers will typically look for) and reviewers (to guide them in a structured review process). Items do not need to be checked explicitly for a PR to be eligible for merging.

Purpose and scope

• The PR has a single goal that is clear from the PR title and/or description.
• All code changes represent a single set of modifications that logically belong together.
• No more than 500 lines of code are changed or there is no obvious way to split the PR into multiple PRs.

Code quality

• The code can be understood easily.
• Newly introduced names for variables etc. are self-descriptive and consistent with existing naming conventions.
• There are no redundancies that can be removed by simple modularization/refactoring.
• There are no leftover debug statements or commented code sections.
• The code adheres to our conventions and style guide, and to the Julia guidelines.

Documentation

• New functions and types are documented with a docstring or top-level comment.
• Relevant publications are referenced in docstrings (see example for formatting).
• Inline comments are used to document longer or unusual code sections.
• Comments describe intent ("why?") and not just functionality ("what?").
• If the PR introduces a significant change or new feature, it is documented in NEWS.md.

Testing

• The PR passes all tests.
• New or modified lines of code are covered by tests.
• New or modified tests run in less then 10 seconds.

Performance

• There are no type instabilities or memory allocations in performance-critical parts.
• If the PR intent is to improve performance, before/after time measurements are posted in the PR.

Verification

• The correctness of the code was verified using appropriate tests.
• If new equations/methods are added, a convergence test has been run and the results
  are posted in the PR.

Created with ❤️ by the Trixi.jl community.

[codecov]

Codecov Report

Attention: Patch coverage is 93.88464% with 88 lines in your changes are missing coverage. Please review.

Project coverage is 96.11%. Comparing base (909abb4) to head (bded959).
Report is 58 commits behind head on main.

Files	Patch %	Lines
...ation/paired_explicit_runge_kutta/methods_PERK2.jl	85.44%	23 Missing ⚠️
src/callbacks_step/analysis_surface_integral_2d.jl	91.04%	12 Missing ⚠️
src/equations/linearized_euler_1d.jl	76.60%	11 Missing ⚠️
src/equations/linearized_euler_3d.jl	94.74%	5 Missing ⚠️
src/Trixi.jl	0.00%	4 Missing ⚠️
src/meshes/structured_mesh_view.jl	91.30%	4 Missing ⚠️
src/meshes/t8code_mesh.jl	96.04%	4 Missing ⚠️
src/solvers/dgsem_tree/subcell_limiters.jl	88.89%	4 Missing ⚠️
src/callbacks_step/analysis.jl	86.96%	3 Missing ⚠️
...em/elixir_linearizedeuler_characteristic_system.jl	93.10%	2 Missing ⚠️
... and 10 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1865      +/-   ##
==========================================
- Coverage   96.30%   96.11%   -0.19%     
==========================================
  Files         440      460      +20     
  Lines       35793    36913    +1120     
==========================================
+ Hits        34470    35478    +1008     
- Misses       1323     1435     +112     

Flag	Coverage Δ
unittests	96.11% <93.88%> (-0.19%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ranocha]

Thanks! In #1616, you mentioned that the analysis callback also prints only the local information. Does this PR fix this as well?

[JoshuaLampert]

What about other mesh types like the TreeMesh? Does it print the local or global number of cells (see here)?

[ranocha]

The TreeMesh replicates all cell info on all ranks. Thus, it prints the global info.

[sloede]

@benegee Please note that you should also adapt the output of the AMR output, probably in these three functions:

Trixi.jl/src/callbacks_step/analysis.jl

Line 494 in 8a9fc7b

function print_amr_information(callbacks, mesh, solver, cache)

Trixi.jl/src/callbacks_step/analysis.jl

Line 520 in 8a9fc7b

function print_amr_information(callbacks, mesh::P4estMesh, solver, cache)

Trixi.jl/src/callbacks_step/analysis.jl

Line 554 in 8a9fc7b

function print_amr_information(callbacks, mesh::T8codeMesh, solver, cache)

Otherwise we get a global element count but only rank-0 information on AMR, which is bound to cause confusion IMHO

[benegee]

True! I realized this in the meantime as well, but have not finished the MPI syncing of element counts.

[sloede]

True! I realized this in the meantime as well, but have not finished the MPI syncing of element counts.

Optimally, you'll use an implementation that only requires a single additional MPI_Reduce call.

[sloede]

Suggested change

	@inline function ndofsglobal(semi::SemidiscretizationCoupled)
	sum(ndofsglobal, semi.semis)
	end
	"""
	ndofsglobal(semi::SemidiscretizationCoupled)
	Return the global number of degrees of freedom associated with each scalar variable across all MPI ranks, and summed up over all coupled systems.
	This is the same as [`ndofs`](@ref) for simulations running in serial or
	parallelized via threads. It will in general be different for simulations
	running in parallel with MPI.
	"""
	@inline function ndofsglobal(semi::SemidiscretizationCoupled)
	sum(ndofsglobal, semi.semis)
	end

[ranocha]

I would actually argue that this docstring doesn't really help in practice since it's the same as the one for the AbstractSemidiscretization - and SemidiscretizationCoupled <: AbstractSemidiscretization. But I don't have a strong opinion on this. Shall we keep it, @sloede?

[sloede]

Well, it explicitly mentions that it is the number of DOFs over all coupled systems. I don't have a strong opinion either, so I'll leave it up to @benegee to decide 🙂

[benegee]

Summary

Global number of elements and dofs

nodofsglobal and nelementsglobal are now used in the Base.show output of semidiscretizations and meshes, and of the AnalysisCallback.

For DG, nodofsglobal already relied on nelementsglobal:

Trixi.jl/src/solvers/dg.jl

Lines 461 to 463 in 76719a8

	@inline function ndofsglobal(mesh, dg::DG, cache)
	nelementsglobal(dg, cache) * nnodes(dg)^ndims(mesh)
	end

and nelementsglobal was already MPI aware:

Trixi.jl/src/solvers/dg.jl

Lines 520 to 522 in 76719a8

	@inline function nelementsglobal(dg::DG, cache)
	mpi_isparallel() ? cache.mpi_cache.n_elements_global : nelements(dg, cache)
	end

For DGMulti, ndofsglobal was already MPI aware as well:

Trixi.jl/src/callbacks_step/analysis_dgmulti.jl

Lines 188 to 194 in 76719a8

	function ndofsglobal(mesh::DGMultiMesh, solver::DGMulti, cache)
	if mpi_isparallel()
	error("`ndofsglobal` is not implemented for `DGMultiMesh` when used in parallel with MPI")
	else
	return ndofs(mesh, solver, cache)
	end
	end

I now added nelementsglobal in analogy.

New solver types would now have to implement ndofsglobal and nelementsglobal.

Global number of elements per level

The local element numbers per level are now summed up across all ranks before being printed on rank 0.
In order to not having to synchronize the minimum and maximum element levels in advance, I took this information from the AMRController. This currently works but would require new controllers to provide the same information.
If you see a better solution, please give me a hint.

[ranocha]

Thanks a lot! Looks quite good to me 👍

[ranocha]

I would actually argue that this docstring doesn't really help in practice since it's the same as the one for the AbstractSemidiscretization - and SemidiscretizationCoupled <: AbstractSemidiscretization. But I don't have a strong opinion on this. Shall we keep it, @sloede?

[sloede]

Thanks a lot for tackling this @benegee! This makes Trixi.jl much more usable in parallel 💪