CI Use statement coverage rather than branch coverage

[lesteve]

Follow-up of #29444 (comment), trying to see if using statement coverage makes a difference.

	ref	statement coverage	+ sysmon	+ pylatest_pip_openblas_pandas to Python 3.12
pylatest_conda_forge_mkl (Python 3.12)	24m 12s	25m 23s	14m 17s	13m 56s
pymin_conda_forge_openblas_ubuntu_2204 (no coverage)	19m 19s	11m 19s	12m 54s	15m 26s
ubuntu_atlas (no coverage)	16m 16	15m 42s	11m 15s	16m 50s
pymin_conda_defaults_openblas	20m	14m 6s	14m 6s	13m 41s
pylatest_pip_openblas_pandas (Python 3.11 or 3.12)	27m 9s	35m 19s	31m 53s	22m 19s
debian_atlas_32bit	34m 42s	34m 5s	46m 31s	44m 23s
macOS pylatest_conda_forge_mkl (Python 3.12)	19m 11s	15m 49s	10m 45s	11m 16s
macOS pylatest_conda_mkl_no_openmp (Python 3.12)	20m 57s	18m 53s	18m 35s	25m 3s
Windows pymin_conda_forge_mkl (no coverage)	18m 59s	24m 10s	20m 13s	32m 2s

Reference run (same as main) build log

Run with using statement coverage rather than branch coverage build log

Run with using statement coverage + COVERAGE_CORE=sysmon, this should make Python 3.12 builds faster build log

Run with using statement coverage + COVERAGE_CORE=sysmon, this should make Python 3.12 builds + updating openblas_pandas to Python 3.12 build log

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 576e7f7. Link to the linter CI: here}

[thomasjpfan]

@lesteve Have you experimented with https://github.com/plasma-umass/slipcover on this repo before?

The author did run benchmarks with scikit-learn's testing: https://github.com/plasma-umass/slipcover?tab=readme-ov-file#performance I tried slipcover ~2 years ago and did see some improvements, but did not have time to purse it.

[lesteve]

@lesteve Have you experimented with plasma-umass/slipcover on this repo before?

I have heard of it recently but not run anything. It seems like it is missing a few integration that we need e.g. pytest-xdist support, XML results file.

There are some discussions about merging SlipCover and coverage according to this slide of the PyConUS 2024 SlipCover talk.

coveragepy with Python 3.12 can use sys.monitoring which is maybe as fast as SlipCover, at least this is what I gathered from the slides after this one but not 100% sure.

Right now coveragepy does not really support branch coverage with COVERAGE_CORE=sysmon, it actually makes it slower see nedbat/coveragepy#1812. To be honest I don't think we care that about branch coverage vs statement coverage and this PR was a way to test how much using statement coverage for all the builds and COVERAGE_CORE=sysmon for the Python 3.12 build would help making the CI faster.

The summary so far:

• using statement coverage vs branch coverage maybe is a bit faster (for example the pymin_conda_defaults_openblas build goes from 20 minutes to 14 minutes)
• using statement coverage + COVERAGE_CORE=sysmon for the Python 3.12 is definitely faster
• some build timings are quite variable so take this with a pinch of salt. For example Windows which have coverage disabled and should not care about all of the different settings varies between 19 minutes and 32 minutes

[ogrisel]

Could you please trigger another run for debian_atlas_32bit with statement coverage + sysmon? The 44 min runtime is a bit worrying. I wonder if it's random or not.

[lesteve]

The 44 min runtime is a bit worrying.

Could you please trigger another run for debian_atlas_32bit with statement coverage + sysmon? The 44 min runtime is a bit worrying. I wonder if it's random or not.

Note that in a67708e, Debian 32bit build took 35 minutes see build log.

I think since we upgraded the Debian version for the 32bit CI (11->12) this build is a lot slower with quite some variability (the Debian 11->12 upgrade was done when getting rid of setuptools PR #29400). My hunch is that this is due to upgrading Python to 3.11 and that this might be related to coverage too but I am not 100% sure.

For example on main, you can easily find some Debian 32bit runs that took ~45 minutes e.g. this one or this one. To find them you look at all the recent runs on main and pick the ones that take more than 1h.

[ogrisel]

@lesteve this PR is still draft but I have the impression that it's a net improvement, no?