Skip to content

Add synchronization statements to ensure timer output correctness on the GPU #2892

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ranocha merged 7 commits into from
Mar 29, 2026
+35 −12
Merged

Add synchronization statements to ensure timer output correctness on the GPU #2892

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b7348fd
Add synchronization statements to ensure timer output correctness on …
vchuravy Mar 27, 2026
d91cd5b
use trixi_timit macro with a backend
vchuravy Mar 27, 2026
8dcbdac
don't commit piracy
vchuravy Mar 27, 2026
c75e385
Update src/Trixi.jl
vchuravy Mar 27, 2026
713a5e2
Merge branch 'main' into vc+bg/gpu_timeroutputs
ranocha Mar 29, 2026
b690653
fixup! Add synchronization statements to ensure timer output correctn…
vchuravy Mar 29, 2026
4f4e48e
only synchronize when timers are enabled
vchuravy Mar 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions src/auxiliary/auxiliary.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,26 @@ end
return ncalls_first
end

"""
@trixi_timeit_ext backend timer() "some label" expression

This macro is an extension of [`@trixi_timeit`](@ref) that also synchronizes the given `backend` after executing the given `expression`.
This is useful to get accurate timing measurements for GPU backends, where the execution of kernels is asynchronous.
The synchronization ensures that all GPU operations are completed before the timer is stopped.
Copy link
Copy Markdown
Member

@ranocha ranocha Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this impact runtime performance?

Copy link
Copy Markdown
Member Author

@vchuravy vchuravy Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does add pipeline bubbles into the execution. So with CUDA@profile we can measure the device side activity and get a measure of how active the GPU was (this does not capture fully how well we used the GPU, but does give us a first order estimate of if we have large pipeline bubbles).

So here we get a change of 99.40% -> 94.15% activity of the GPU over the whole trace on the TGV benchmark. The wall-clock time change is small (70.3s -> 70.5s)

Copy link
Copy Markdown
Member

@ranocha ranocha Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thanks. So it's a reasonable compromise to get correct data by default with slightly reduced runtime performance. If people are interested in raw performance, they should disable the timings, correct?

Copy link
Copy Markdown
Member Author

@vchuravy vchuravy Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah and my latest commit ensure that when the timing is disabled then the synchronization statements are removed.


See also [`@trixi_timeit`](@ref).
"""
macro trixi_timeit_ext(backend, timer_output, label, expr)
expr = quote
local val = $(esc(expr))
if $(esc(backend)) !== nothing && $(TrixiBase).timeit_debug_enabled()
$(KernelAbstractions.synchronize)($(esc(backend)))
end
val
end
return :(@trixi_timeit($(esc(timer_output)), $(esc(label)), $(expr)))
end

"""
examples_dir()

Expand Down
27 changes: 15 additions & 12 deletions src/solvers/dgsem_tree/dg_2d.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,63 +113,66 @@ function rhs!(du, u, t,
backend = trixi_backend(u)

# Reset du
@trixi_timeit timer() "reset ∂u/∂t" set_zero!(du, dg, cache)
@trixi_timeit_ext backend timer() "reset ∂u/∂t" begin
set_zero!(du, dg, cache)
end

# Calculate volume integral
@trixi_timeit timer() "volume integral" begin
@trixi_timeit_ext backend timer() "volume integral" begin
calc_volume_integral!(backend, du, u, mesh,
have_nonconservative_terms(equations), equations,
dg.volume_integral, dg, cache)
end

# Prolong solution to interfaces
@trixi_timeit timer() "prolong2interfaces" begin
@trixi_timeit_ext backend timer() "prolong2interfaces" begin
prolong2interfaces!(backend, cache, u, mesh, equations, dg)
end

# Calculate interface fluxes
@trixi_timeit timer() "interface flux" begin
@trixi_timeit_ext backend timer() "interface flux" begin
calc_interface_flux!(backend, cache.elements.surface_flux_values, mesh,
have_nonconservative_terms(equations), equations,
dg.surface_integral, dg, cache)
end

# Prolong solution to boundaries
@trixi_timeit timer() "prolong2boundaries" begin
@trixi_timeit_ext backend timer() "prolong2boundaries" begin
prolong2boundaries!(cache, u, mesh, equations, dg)
end

# Calculate boundary fluxes
@trixi_timeit timer() "boundary flux" begin
@trixi_timeit_ext backend timer() "boundary flux" begin
calc_boundary_flux!(cache, t, boundary_conditions, mesh, equations,
dg.surface_integral, dg)
end

# Prolong solution to mortars
@trixi_timeit timer() "prolong2mortars" begin
@trixi_timeit_ext backend timer() "prolong2mortars" begin
prolong2mortars!(cache, u, mesh, equations,
dg.mortar, dg)
end

# Calculate mortar fluxes
@trixi_timeit timer() "mortar flux" begin
@trixi_timeit_ext backend timer() "mortar flux" begin
calc_mortar_flux!(cache.elements.surface_flux_values, mesh,
have_nonconservative_terms(equations), equations,
dg.mortar, dg.surface_integral, dg, cache)
end

# Calculate surface integrals
@trixi_timeit timer() "surface integral" begin
@trixi_timeit_ext backend timer() "surface integral" begin
calc_surface_integral!(backend, du, u, mesh, equations,
dg.surface_integral, dg, cache)
end

# Apply Jacobian from mapping to reference element
@trixi_timeit timer() "Jacobian" apply_jacobian!(backend, du, mesh, equations, dg,
cache)
@trixi_timeit_ext backend timer() "Jacobian" begin
apply_jacobian!(backend, du, mesh, equations, dg, cache)
end

# Calculate source terms
@trixi_timeit timer() "source terms" begin
@trixi_timeit_ext backend timer() "source terms" begin
calc_sources!(du, u, t, source_terms, equations, dg, cache)
end

Expand Down
Loading