Test in docs/.../differentiable_programming.jl sometimes fails

[ArseniyKholod]

I noticed that sometimes Documentation CI fails due to the error in test inside docs/literate/src/files/differentiable_programming.jl.
Four recent failures in four different PRs
https://github.com/trixi-framework/Trixi.jl/actions/runs/6745400585
https://github.com/trixi-framework/Trixi.jl/actions/runs/6747174166
https://github.com/trixi-framework/Trixi.jl/actions/runs/6743238156/job/18330780159
https://github.com/trixi-framework/Trixi.jl/actions/runs/6747950967

Failures happen in the following lines
@test 3.0e-10 < relative_maximum < 8.0e-10 #src
and
@test 1.0e-17 < relative_maximum < 1.0e-15 #src
All failures that I saw have the same calculated values causing problems.
3.0e-10 < 8.869894585050838e-10 < 8.0e-10
1.0e-17 < 1.170548031529666e-15 < 1.0e-15
Local on Windows tests are always passed.

Different seems to be quite small, does make sense to expand acceptable values?

[JoshuaLampert]

Looks like the error is related to other CI failures we recently have in various PRs. I think, we should investigate this further instead of just adapting the tolerances. Locally, I don't get the errors on Ubuntu.

[JoshuaLampert]

What is especially strange is that the errors seem to occur stochastically, see e.g.
https://github.com/trixi-framework/Trixi.jl/actions/runs/6747950983/job/18345234543?pr=1701
https://github.com/trixi-framework/Trixi.jl/actions/runs/6747464311/job/18343680417?pr=1702,
where some tests fail for both runs, but some only for one of the two runs.

[ArseniyKholod]

Looks like the error is related to other CI failures we recently hawve in various PRs. I think, we should investigate this further instead of just adapting the tolerances. Locally, I don't get the errors on Ubuntu.

Yeah, I tried on my PR, where I only add one new tutorial, that is executed after this one, so seems to be no way, that it can interfere somehow

[ArseniyKholod]

What is especially strange is that the errors seem to occur stochastically, see e.g. https://github.com/trixi-framework/Trixi.jl/actions/runs/6747950983/job/18345234543?pr=1701 https://github.com/trixi-framework/Trixi.jl/actions/runs/6747464311/job/18343680417?pr=1702, where some tests fail for both runs, but some only for one of the two runs.

For me also sometimes tests are passing, sometimes not, could it depend on runner that GitHub use...?

[JoshuaLampert]

Yeah, I don't know. That is very weird..