Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

C++ Coverage on MacOS is completely broken #10457

Open
ulfjack opened this issue Dec 19, 2019 · 5 comments
Open

C++ Coverage on MacOS is completely broken #10457

ulfjack opened this issue Dec 19, 2019 · 5 comments
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) team-Rules-CPP Issues for C++ rules type: bug

Comments

@ulfjack
Copy link
Contributor

ulfjack commented Dec 19, 2019

Coverage with gcov:

  1. GCOV_PREFIX_STRIP is set incorrectly
    With a value of 3, the paths end up like this:
    bazel-testlogs/cpp/foo_test/_coverage/_bazel_ulfjack/312a82d9f5d7857ee9e5303755329142/sandbox/darwin-sandbox/4/execroot/main/bazel-out/darwin-fastbuild/bin/cpp/_objs/foo_test/main.gcda

The collect-coverage.sh script doesn't allow overriding GCOV_PREFIX_STRIP. I have a patch to not set it if it's already set in the test env. On this machine, I have to set it to 10.

  1. The latest gcov does not support the -i flag
    The -i flag is hard-coded in collect_cc_coverage.sh with no override possible.

  2. The gcov log uses lower-case "creating"
    The collect_cc_coverage.sh only checks for upper-case "Creating".

  3. The generated output is not in intermediate gcov format as expected by lcov_merger.

Coverage with llvm-profdata (--features=llvm_coverage_map_format):

  1. COVERAGE_GCOV_PATH is set to gcov
    This is obviously wrong. It needs to be set to llvm-profdata. However, llvm-profdata is not on the path, and there does not seem to be a way to override the location of gcov. I tried setting GCOV=/path/to/llvm-profdata before calling Bazel, but that had no effect.

  2. The lcov_merger simply copies the .profdata file to coverage.dat
    At least this allows users to post-process manually, but it's still unfortunate.
@ulfjack ulfjack mentioned this issue Dec 19, 2019
Closed
@jin jin added z-team-Apple Deprecated. Send to rules_apple, or label team-Rules-CPP + platform:apple untriaged team-Rules-CPP Issues for C++ rules and removed z-team-Apple Deprecated. Send to rules_apple, or label team-Rules-CPP + platform:apple labels Dec 20, 2019
@ulfjack ulfjack mentioned this issue Jan 23, 2020
Closed
bazel-io pushed a commit that referenced this issue Jan 27, 2020
@ulfjack @copybara-github
Partially fix MacOS llvm coverage collection
6089b6a 
This allows setting GCOV on the command line to overwrite the default setting, which points to /usr/bin/gcov. In order to use this, you also need to disable the gcov coverage feature and enable the llvm coverage feature instead. The full command-line looks like this:

GCOV=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/llvm-profdata bazel coverage --features=llvm_coverage_map_format --features=-gcc_coverage_map_format //cpp:test

Progress on #10457.

Change-Id: Iae80e3a5830711b6402a3e015bf9ae12b536f429

Closes #10640.

Change-Id: Iae80e3a5830711b6402a3e015bf9ae12b536f429
PiperOrigin-RevId: 291701448
bazel-io pushed a commit that referenced this issue Jan 27, 2020
@ulfjack @copybara-github
MacOS C++ coverage: preserve GCOV_PREFIX_STRIP
b3763e9 
On MacOS, setting GCOV_PREFIX_STRIP to 3 is generally incorrect. This
setting works on Linux, because we set PWD=/proc/self/cwd/, which makes
all paths have a fixed 3-segment prefix. However, this is not possible
on MacOS. On MacOS, the correct setting depends on the path to the exec
root or the sandbox root (if sandboxing is enabled). On my machine, the
proper value is 10.

While this change doesn't fix the underlying issue, at least it allows
users to work around it without having to modify and compile Bazel from
source.

Related to #10457.

PiperOrigin-RevId: 291701562
@oquenchil oquenchil added P3 We're not considering working on this, but happy to review a PR. (No assignee) type: bug and removed untriaged labels Feb 18, 2020
bazel-io pushed a commit that referenced this issue Mar 12, 2020
@oquenchil @copybara-github
Partially fix MacOS llvm coverage collection
ba2ccb1 
This allows setting GCOV on the command line to overwrite the default setting, which points to /usr/bin/gcov. In order to use this, you also need to disable the gcov coverage feature and enable the llvm coverage feature instead. The full command-line looks like this:

GCOV=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/llvm-profdata bazel coverage --features=llvm_coverage_map_format --features=-gcc_coverage_map_format //cpp:test

Progress on #10457.

Original change by ulfjack
RELNOTES: None.
PiperOrigin-RevId: 300583709
oliverlee added a commit to oliverlee/recorder that referenced this issue Jul 14, 2020
@oliverlee
Define Bazel coverage config
b9f1d46 
Define configuration options for collecting coverage data with Bazel.

The coverage command for Bazel is fairly fragile and many configurations
do not work [1,2]. The `--javabase` option is set as suggested in this
bug report[3]. Coverage has been manually tested with GCC 7 on Ubuntu
18.04.

[1]: bazelbuild/bazel#9406
[2]: bazelbuild/bazel#10457
[3]: bazelbuild/bazel#7953
oliverlee added a commit to oliverlee/recorder that referenced this issue Jul 14, 2020
@oliverlee
Define Bazel coverage config
00d826c 
Define configuration options for collecting coverage data with Bazel.

The coverage command for Bazel is fairly fragile and many configurations
do not work [1,2]. The `--javabase` option is set as suggested in this
bug report[3]. Coverage has been manually tested with GCC 7 on Ubuntu
18.04.

[1]: bazelbuild/bazel#9406
[2]: bazelbuild/bazel#10457
[3]: bazelbuild/bazel#7953
@junr03 junr03 mentioned this issue Oct 27, 2020
Merged
oliverlee added a commit to oliverlee/recorder that referenced this issue Nov 21, 2020
@oliverlee
Define Bazel coverage config
fb45ffe 
Define configuration options for collecting coverage data with Bazel.

The coverage command for Bazel is fairly fragile and many configurations
do not work [1,2]. The `--javabase` option is set as suggested in this
bug report[3]. Coverage has been manually tested with GCC 7 on Ubuntu
18.04.

[1]: bazelbuild/bazel#9406
[2]: bazelbuild/bazel#10457
[3]: bazelbuild/bazel#7953
This was referenced Mar 5, 2022
Closed
Closed
Open
@jeffwheeler
Copy link

For anyone else that tries to run coverage on Mac, try putting this in your ~/.bazelrc file:

coverage --action_env=GCOV_PREFIX_STRIP=10

Thanks to @ulfjack, it is possible to set the correct prefix size through this argument. It'd be nice to fix this bug in Bazel.

@ulfjack
Copy link
Contributor Author

ulfjack commented Oct 25, 2023 •
edited
Loading

I'm currently trying to collect coverage data from C++ binaries on MacOS, and it's not generating any .gcda files. When I compile by hand following these instructions then I do get gcda output.

That seems to have happened because I was using the wrong GCOV_PREFIX_STRIP.

@c-mita c-mita mentioned this issue Nov 10, 2023
Open
@ulfjack
Copy link
Contributor Author

ulfjack commented Nov 11, 2023

It bugs me that you have to guess the correct value for GCOV_PREFIX_STRIP on MacOS in order to use it. I looked into it, but so far haven't been able to come up with an automated method.

@ulfjack
Copy link
Contributor Author

ulfjack commented Nov 11, 2023

IIUC, the gcno and gcda files have to be next to each other, which is why Bazel copies them into the same tree; however, matching up the names in shell is not straightforward to do in O(n) time.

@josh-swerdlow
Copy link

Is there any update on this? I'm still hoping I can get coverage reports on my mac using bazel instead of switching to some other compiler mechanism (as done in the aforementioned link).

Thanks for all your work ulfjack...think I've read all of your updates on this since 2018 trying to get a sense of the issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) team-Rules-CPP Issues for C++ rules type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jeffwheeler @jin @ulfjack @oquenchil @josh-swerdlow