Introduce reproducible branch-based coverage #1375
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
aaronmondal
commented
Sep 29, 2024
There was a problem hiding this comment.
+@adam-singer +@allada +@blakehatch +@caass +@SchahinRohani
Reviewable status: 0 of 5 LGTMs obtained, and 0 of 8 files reviewed, and pending CI: Bazel Dev / macos-13, Bazel Dev / macos-14, Bazel Dev / ubuntu-24.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, NativeLink.com Cloud / Remote Cache / macos-14, NativeLink.com Cloud / Remote Cache / ubuntu-24.04, Publish image, Publish nativelink-worker-init, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, Web Platform Deployment / macos-14, Web Platform Deployment / ubuntu-24.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (22.04), integration-tests (22.04), macos-13, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable (waiting on @adam-singer, @allada, @blakehatch, @caass, and @SchahinRohani)
aaronmondal
force-pushed
the
coverage
branch
from
dae772a
to
bf29068
Compare
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
aaronmondal
had a problem deploying
to
production
GitHub Actions
Error
|
Preview at https://aaronmondal.github.io/nativelink/ |
aaronmondal
had a problem deploying
to
production
GitHub Actions
Error
aaronmondal
force-pushed
the
coverage
branch
from
bf29068
to
329889d
Compare
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
aaronmondal
had a problem deploying
to
production
GitHub Actions
Error
aaronmondal
had a problem deploying
to
production
GitHub Actions
Error
Reports may now be build via: ```nix nix build .#nativelinkCoverageForHost ``` The `result` symlink then contains the contents of a webpage to view the existing reports. Pushes to main publish the site at tracemachina.github.io/nativelink. Reports are built in release mode to closely resemble production coverage of the testsuite. This also means that most worker tests are ignored via a new `nix` feature to make the testsuite run in nix sandboxes. It's not ideal, but accurately reflects our production coverage guarantees. A future resolution for this might be to implement more elaborate mocking functionality for `nativelink-worker`. Coverage leverages nix caching, but not Bazel caching. While Bazel has builtin support for coverage, the reports were not satisfactory as they rely on outdated gcov toolchains with vague hermeticity guarantees and unsatisfactory implementations for llvm-cov-based workflows (e.g. no branch-based coverage for rust and no story around coverage for heterogeneous code). Mid-term we should implement "fast development" coverage via Bazel alongside the "slow production" coverage via Nix.
aaronmondal
force-pushed
the
coverage
branch
from
329889d
to
f3c8caf
Compare
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
aaronmondal
temporarily deployed
to
production
GitHub Actions
Inactive
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reports may now be build via:
The
resultsymlink then contains the contents of a webpage to view theexisting reports.
Pushes to main publish the site at tracemachina.github.io/nativelink.
Reports are built in release mode to closely resemble production coverage
of the testsuite. This also means that most worker tests are ignored via
a new
nixfeature to make the testsuite run in nix sandboxes. It's notideal, but accurately reflects our production coverage guarantees. A
future resolution for this might be to implement more elaborate mocking
functionality for
nativelink-worker.Coverage leverages nix caching, but not Bazel caching. While Bazel has
builtin support for coverage, the reports were not satisfactory as they
rely on outdated gcov toolchains with vague hermeticity guarantees and
unsatisfactory implementations for llvm-cov-based workflows (e.g. no
branch-based coverage for rust and no story around coverage for
heterogeneous code). Mid-term we should implement "fast development"
coverage via Bazel alongside the "slow production" coverage via Nix.
This change is