[6.4.0] Add support for the BLAKE3 digest function #19191
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
iancha1992
added
team-Remote-Exec
|
(Disabling auto-merge because I personally want to run some local tests on this. I'll re-enable it after my tests.) |
(cherry picked from commit e382abf)
(cherry picked from commit 0ff4f6d)
This PR adds the BLAKE3 C and asm sources to third_party, and includes a BUILD file to build them. PiperOrigin-RevId: 541539341 Change-Id: I49b1edce20a7d0f986e29712e6050e4e0b9c1d44 (cherry picked from commit a3a569e)
Support new-style digest functions. This PR adds support for new-style digest functions to the remote execution library code. The remote-apis spec says: ``` // * `digest_function` is a lowercase string form of a `DigestFunction.Value` // enum, indicating which digest function was used to compute `hash`. If the // digest function used is one of MD5, MURMUR3, SHA1, SHA256, SHA384, SHA512, // or VSO, this component MUST be omitted. In that case the server SHOULD // infer the digest function using the length of the `hash` and the digest // functions announced in the server's capabilities. ``` PiperOrigin-RevId: 543691155 Change-Id: If8c386d923db1b24dff6054c8ab3f783409b7f13 (cherry picked from commit 88412ce)
This PR adds the Blake3Hasher and Blake3HashFunction classes to vfs and makes them available under the flag --digest_function=BLAKE3. PiperOrigin-RevId: 550525978 Change-Id: Iedc0886c51755585d56b4d8f47676d3be5bbedba (cherry picked from commit cc49d68)
Improve BLAKE3 performance by reducing copies. Rather than buffering bytes, it's faster to just update the blake3 hasher as bytes are hashed. Performance with buffering: ``` Benchmark (inputSize) Mode Cnt Score Error Units Blake3Benchmark.blake3Hash 1 avgt 10 1766.697 ± 709.515 ns/op Blake3Benchmark.blake3Hash 10 avgt 10 1466.253 ± 19.474 ns/op Blake3Benchmark.blake3Hash 100 avgt 10 1522.845 ± 15.480 ns/op Blake3Benchmark.blake3Hash 1000 avgt 10 2254.156 ± 8.588 ns/op Blake3Benchmark.blake3Hash 10000 avgt 10 4660.881 ± 28.637 ns/op Blake3Benchmark.blake3Hash 100000 avgt 10 24283.191 ± 32.754 ns/op Blake3Benchmark.sha256Hash 1 avgt 10 742.091 ± 6.307 ns/op Blake3Benchmark.sha256Hash 10 avgt 10 757.844 ± 12.042 ns/op Blake3Benchmark.sha256Hash 100 avgt 10 942.902 ± 555.874 ns/op Blake3Benchmark.sha256Hash 1000 avgt 10 1208.336 ± 508.392 ns/op Blake3Benchmark.sha256Hash 10000 avgt 10 4871.231 ± 494.507 ns/op Blake3Benchmark.sha256Hash 100000 avgt 10 40686.231 ± 63.814 ns/op ``` Performance without buffering (after this CL): ``` Benchmark (inputSize) Mode Cnt Score Error Units Blake3Benchmark.blake3Hash 1 avgt 10 1021.075 ± 11.640 ns/op Blake3Benchmark.blake3Hash 10 avgt 10 1029.561 ± 19.850 ns/op Blake3Benchmark.blake3Hash 100 avgt 10 1070.509 ± 12.140 ns/op Blake3Benchmark.blake3Hash 1000 avgt 10 1685.043 ± 13.963 ns/op Blake3Benchmark.blake3Hash 10000 avgt 10 3939.516 ± 13.212 ns/op Blake3Benchmark.blake3Hash 100000 avgt 10 21730.550 ± 22.976 ns/op Blake3Benchmark.sha256Hash 1 avgt 10 745.943 ± 9.853 ns/op Blake3Benchmark.sha256Hash 10 avgt 10 747.649 ± 17.381 ns/op Blake3Benchmark.sha256Hash 100 avgt 10 962.802 ± 590.879 ns/op Blake3Benchmark.sha256Hash 1000 avgt 10 1189.069 ± 412.327 ns/op Blake3Benchmark.sha256Hash 10000 avgt 10 4594.978 ± 21.833 ns/op Blake3Benchmark.sha256Hash 100000 avgt 10 39224.248 ± 229.265 ns/op ``` PiperOrigin-RevId: 551225043 Change-Id: I57dc0700b2f44d6faf75ffbd29f1607544e54f29 (cherry picked from commit d922ab3)
I noticed when testing the BLAKE3 digest function that uploaded files were being referenced incorrectly in the BES because they were missing the digest type. This CL fixes that. Before: https://github.com/bazelbuild/bazel/assets/141737/52781f1b-b897-48f0-8956-f63c57b59436 After: https://github.com/bazelbuild/bazel/assets/141737/01ebc61b-3512-4ca5-8e2d-f47ad5f086b7 PiperOrigin-RevId: 553405394 Change-Id: Ic976e5a58f80ab8b5270b4aedc6204c22562533a (cherry picked from commit 1929367) Signed-off-by: Brentley Jones <[email protected]>
DiskCache always stores files in /root/{cas/ac}/digestHash. This change keeps the current behavior, but for new style digest functions inserts a directory between /root/ and {cas/ac} to disambiguate the digest function type.
This prevents issues that could theoretically happen if there were hash collisions between two digest functions sharing the same cache directory.
PiperOrigin-RevId: 554477775
Change-Id: Ibef994e432764c260a3cab821ca6583c231c5b50
(cherry picked from commit f17b280)
Signed-off-by: Brentley Jones <[email protected]>
brentleyjones
force-pushed
the
bj/6.4.0-blake3
branch
from
5a5b6e0 to
1286538
Compare
|
@iancha1992 Can you re-enable the auto-merge? My testing is done. Thank you. |
coeuvre
approved these changes
Aug 8, 2023
iancha1992
removed
the
awaiting-review
copybara-service bot
pushed a commit
that referenced
this pull request
Oct 19, 2023
Baseline: 0f231ac Release Notes: + Support multiple remote execution digest functions (#19042) + Release 6.4.0 remote (#18959) + Move BazelFileSystemModule into bazel package (#19043) + Fix a bug where frozen targets list was mutated while expanding env attribute (#19053) + Mark isolated extension usages as experimental (#19065) + Add the remote_require_cached flag (#19075) + Advertise CcInfo from cc_import (#19086) (#19088) + Update java_tools version to 12.6 (#19092) + Set the digest_function field as part of all relevant gRPC requests (#19049) + Merge `use_repo` buildifier fixups into a single command (#19134) + Ensure that extension unique names followed by `~` are prefix-free (#19164) + Lockfile updates & fixes (#19153) + Switch xcode_autoconf to use 'configure = True' (#19174) + Cherry pick Add a Starlark flag that allows disabling proguard. This will be useful for testing later. (#19179) + Update CODEOWNERS for 6.4.0 (#19194) + Friendlier error message for `bazel_dep`s without `version` (#19196) + Always check `$config_dependencies` visibility at use (#19197) + Add support for the BLAKE3 digest function (#19191) + Enable cc toolchain resolution when cross compiling to windows arm64. (#19198) + Ensure disk cache root exists (#19225) + Fix valid json when using jsonproto output in queries with new `--ouput=streamed_jsonproto` implementation. (#19226) + Add toolchain type for Java bootstrap runtime (#19220) + Add Starlark implementation for several CcCommon methods. (#19076) + Rename `cc_test_wrapper` to `cc_test` (#19231) + Cherry-pick commits to fix a Windows issue (#19232) + Add support for more workspace boundary files to bash completion (#19281) + Use `debugPrint` instead of `str` for `fail` arguments (#19283) + Include name in `repr` of exported `rule`s (#19229) + Download `BazelRegistryJson` only once per registry (#19300) + Make module extension tag's `debugPrint` useful for error messages (#19285) + Intern repository mapping entries (#19293) + Add `additional_linker_inputs` option to `cc_library` rule (#19264) + Do not rerun module extensions when only imports or locations change (#19284) + Add profiling for Bzlmod operations (#19313) + Retry on javax.net.ssl.SSLException ... BAD_DECRYPT (#19346) + Fetch `RepoSpecs` in parallel (#19354) + Make `MODULE.bazel.lock` deterministic (#19370) + Ensure lockfile is updated after reset to pre-build state (#19371) + build-runfiles: remove temporary file prior to creating it (#19386) + Always fail on unknown attributes (#19404) + Ignore Starlark options on commands with `allowResidue = False` (#19417) + Separate PackageSpecificationProvider from its target (PackageGroupConfiguredTarget) (#19420) + Expose PackageSpecificationInfo provider as a top level symbol (#19422) + Revert "Report remote execution messages as events" (#19415) + [6.4] Add --incompatible_disable_objc_library_transition (#19393) + Create .bazelversion to address postsubmit timeout issues (#19435) + Add `contains` method inside `PackageSpecificationProvider` (#19425) + Wrong include path to Clang 16 on Windows (#19430) + Simplify release notes by just printing the first line of the commit … (#19448) + Remove PackageGroupConfiguredTarget.isAvailableFor function (#19444) + Remove default -s flag from macOS libtool invocation (#19454) + Turn off lockfile feature by default (#19462) + Take the no-remote-exec tag into account when computing the action salt (#19457) + Add `--incompatible_merge_fixed_and_default_shell_env` (#19319) + Improve error when a label is provided in `config_setting`'s `values` (#19484) + Mark tool inputs in the execution log. (#19483) + Add visionOS support (#19436) + Intern empty `Depset`s (#19443) + Do not allow applicable_licenses on platform. (#19426) + Cherry pick Bzlmod fixes (#19494) + Optimize classpath pre-processing in java_stub_template.txt (#19491) + Add output name to CacheNotFoundException (#19452) + feat: add option to exit early if analysis cache is discarded (#19503) + Cherry pick platform dependent lockfile (#19498) + Print Passed and Failed methods in detailed test summary (#19505) + Add formatted timestamp entries to volatile workspace status file. (#19499) + Write an explicit line ending to the lockfile (#19519) + Only use `/showIncludes` if supported (#19521) + Also apply `NestedSet` optimizations to `Depset` (#19492) + Add diff_against_dynamic_baseline option to experimental_output_direc… (#19514) + Update java_tools to v12.7 (#19522) + Advertise CcInfo from cc_proto_library (#19534) + Update unknown Xcode version error message and provide an environment variable to force re-evaluation (#19540) + Print dep chain leading to a module that is in error (#19543) + Show fetch progress for the `mod` command (#19542) + Revert "Switch xcode_autoconf to use 'configure = True' (#19174)" (#19550) + Explain the use of `str(Label(...))` in the docs (#19554) + Add `--consistent_labels` flag to all query commands (#19567) + Inject builtin modules at the end of the MODULE.bazel file (#19573) + Disable bzlmod_query_test for RBE build (#19585) + Clear runfiles environment variables for `bazel run` (#19606) + cc_library: propagate data dependencies via implementation_deps. (#19590) + Error on potential unsupported `/showIncludes` lines (#19611) + Use case-insensitive comparison for Windows paths in `runfiles.bash` (#19626) + Show test labels in summaries in display form (#19625) + Remove stale extension entries from lockfile (#19683) + Keep leading zero in formatted date (#19694) + Bzlmod lockfile: fix pretty printing for attributes (#19691) + MODULE.bazel.lock file contains user specific paths (#19698) + Consider RCs equivalent to release for `bazel_compatibility` (#19689) + Use `Label` in `@bazel_tools//tools/jdk` macros (#19675) + Remove stale extension entries from lockfile if module order changes (#19730) + Update rules_java 5.5.1 (#19701) + Fix Java compilation for jdk21 (#19735) + Fix output materialized as symlink when building without the bytes. (#19739) + Make lockfile's `RepoSpec` attributes more readable (#19748) + Merge rule and aspect validation output groups (#19745) + Fix handling of non-ASCII characters in archive entry file names (#19765) + Raise an early error on invalid labels in transitions inputs/outputs (#19764) + Handle synthetic method parameters entries that don't have names (#19758) + Bazel release process: Fix push justification. (#19768) + Flip --experimental_cc_implementation_deps (#19751) + Add blake3 NEON instructions on linux arm64 (#19804) + Fix crash when `environ` contains duplicate entries (#19827) + Bump c++ standard to c++14 per default (#19794) + Collect debug info context from implementation deps (#19836) + Fix unconditional Skyframe invalidation with --lockfile_mode=… (#19848) Acknowledgements: This release contains contributions from many people at Google, as well as Andreas Herrmann, bazel.build machine account, Brentley Jones, buildbreaker2021, Chirag Ramani, David Ostrovsky, Ed Schouten, Fabian Meumertzheim, George Gensure, Greg, John Laxson, Julio Merino, Keith Smiley, Matt Mackay, Mauricio G, NelsonLi0701, nglevin, Nicholas Junge, oquenchil, Orion Hodson, Roman Salvador, Ted Kaplan, Thi Doan, Thi Don, Tyler Williams, Xùdōng Yáng.
tdibattista
pushed a commit
to SibrosTech/bazel
that referenced
this pull request
Oct 19, 2023
Baseline: 0f231ac Release Notes: + Support multiple remote execution digest functions (bazelbuild#19042) + Release 6.4.0 remote (bazelbuild#18959) + Move BazelFileSystemModule into bazel package (bazelbuild#19043) + Fix a bug where frozen targets list was mutated while expanding env attribute (bazelbuild#19053) + Mark isolated extension usages as experimental (bazelbuild#19065) + Add the remote_require_cached flag (bazelbuild#19075) + Advertise CcInfo from cc_import (bazelbuild#19086) (bazelbuild#19088) + Update java_tools version to 12.6 (bazelbuild#19092) + Set the digest_function field as part of all relevant gRPC requests (bazelbuild#19049) + Merge `use_repo` buildifier fixups into a single command (bazelbuild#19134) + Ensure that extension unique names followed by `~` are prefix-free (bazelbuild#19164) + Lockfile updates & fixes (bazelbuild#19153) + Switch xcode_autoconf to use 'configure = True' (bazelbuild#19174) + Cherry pick Add a Starlark flag that allows disabling proguard. This will be useful for testing later. (bazelbuild#19179) + Update CODEOWNERS for 6.4.0 (bazelbuild#19194) + Friendlier error message for `bazel_dep`s without `version` (bazelbuild#19196) + Always check `$config_dependencies` visibility at use (bazelbuild#19197) + Add support for the BLAKE3 digest function (bazelbuild#19191) + Enable cc toolchain resolution when cross compiling to windows arm64. (bazelbuild#19198) + Ensure disk cache root exists (bazelbuild#19225) + Fix valid json when using jsonproto output in queries with new `--ouput=streamed_jsonproto` implementation. (bazelbuild#19226) + Add toolchain type for Java bootstrap runtime (bazelbuild#19220) + Add Starlark implementation for several CcCommon methods. (bazelbuild#19076) + Rename `cc_test_wrapper` to `cc_test` (bazelbuild#19231) + Cherry-pick commits to fix a Windows issue (bazelbuild#19232) + Add support for more workspace boundary files to bash completion (bazelbuild#19281) + Use `debugPrint` instead of `str` for `fail` arguments (bazelbuild#19283) + Include name in `repr` of exported `rule`s (bazelbuild#19229) + Download `BazelRegistryJson` only once per registry (bazelbuild#19300) + Make module extension tag's `debugPrint` useful for error messages (bazelbuild#19285) + Intern repository mapping entries (bazelbuild#19293) + Add `additional_linker_inputs` option to `cc_library` rule (bazelbuild#19264) + Do not rerun module extensions when only imports or locations change (bazelbuild#19284) + Add profiling for Bzlmod operations (bazelbuild#19313) + Retry on javax.net.ssl.SSLException ... BAD_DECRYPT (bazelbuild#19346) + Fetch `RepoSpecs` in parallel (bazelbuild#19354) + Make `MODULE.bazel.lock` deterministic (bazelbuild#19370) + Ensure lockfile is updated after reset to pre-build state (bazelbuild#19371) + build-runfiles: remove temporary file prior to creating it (bazelbuild#19386) + Always fail on unknown attributes (bazelbuild#19404) + Ignore Starlark options on commands with `allowResidue = False` (bazelbuild#19417) + Separate PackageSpecificationProvider from its target (PackageGroupConfiguredTarget) (bazelbuild#19420) + Expose PackageSpecificationInfo provider as a top level symbol (bazelbuild#19422) + Revert "Report remote execution messages as events" (bazelbuild#19415) + [6.4] Add --incompatible_disable_objc_library_transition (bazelbuild#19393) + Create .bazelversion to address postsubmit timeout issues (bazelbuild#19435) + Add `contains` method inside `PackageSpecificationProvider` (bazelbuild#19425) + Wrong include path to Clang 16 on Windows (bazelbuild#19430) + Simplify release notes by just printing the first line of the commit … (bazelbuild#19448) + Remove PackageGroupConfiguredTarget.isAvailableFor function (bazelbuild#19444) + Remove default -s flag from macOS libtool invocation (bazelbuild#19454) + Turn off lockfile feature by default (bazelbuild#19462) + Take the no-remote-exec tag into account when computing the action salt (bazelbuild#19457) + Add `--incompatible_merge_fixed_and_default_shell_env` (bazelbuild#19319) + Improve error when a label is provided in `config_setting`'s `values` (bazelbuild#19484) + Mark tool inputs in the execution log. (bazelbuild#19483) + Add visionOS support (bazelbuild#19436) + Intern empty `Depset`s (bazelbuild#19443) + Do not allow applicable_licenses on platform. (bazelbuild#19426) + Cherry pick Bzlmod fixes (bazelbuild#19494) + Optimize classpath pre-processing in java_stub_template.txt (bazelbuild#19491) + Add output name to CacheNotFoundException (bazelbuild#19452) + feat: add option to exit early if analysis cache is discarded (bazelbuild#19503) + Cherry pick platform dependent lockfile (bazelbuild#19498) + Print Passed and Failed methods in detailed test summary (bazelbuild#19505) + Add formatted timestamp entries to volatile workspace status file. (bazelbuild#19499) + Write an explicit line ending to the lockfile (bazelbuild#19519) + Only use `/showIncludes` if supported (bazelbuild#19521) + Also apply `NestedSet` optimizations to `Depset` (bazelbuild#19492) + Add diff_against_dynamic_baseline option to experimental_output_direc… (bazelbuild#19514) + Update java_tools to v12.7 (bazelbuild#19522) + Advertise CcInfo from cc_proto_library (bazelbuild#19534) + Update unknown Xcode version error message and provide an environment variable to force re-evaluation (bazelbuild#19540) + Print dep chain leading to a module that is in error (bazelbuild#19543) + Show fetch progress for the `mod` command (bazelbuild#19542) + Revert "Switch xcode_autoconf to use 'configure = True' (bazelbuild#19174)" (bazelbuild#19550) + Explain the use of `str(Label(...))` in the docs (bazelbuild#19554) + Add `--consistent_labels` flag to all query commands (bazelbuild#19567) + Inject builtin modules at the end of the MODULE.bazel file (bazelbuild#19573) + Disable bzlmod_query_test for RBE build (bazelbuild#19585) + Clear runfiles environment variables for `bazel run` (bazelbuild#19606) + cc_library: propagate data dependencies via implementation_deps. (bazelbuild#19590) + Error on potential unsupported `/showIncludes` lines (bazelbuild#19611) + Use case-insensitive comparison for Windows paths in `runfiles.bash` (bazelbuild#19626) + Show test labels in summaries in display form (bazelbuild#19625) + Remove stale extension entries from lockfile (bazelbuild#19683) + Keep leading zero in formatted date (bazelbuild#19694) + Bzlmod lockfile: fix pretty printing for attributes (bazelbuild#19691) + MODULE.bazel.lock file contains user specific paths (bazelbuild#19698) + Consider RCs equivalent to release for `bazel_compatibility` (bazelbuild#19689) + Use `Label` in `@bazel_tools//tools/jdk` macros (bazelbuild#19675) + Remove stale extension entries from lockfile if module order changes (bazelbuild#19730) + Update rules_java 5.5.1 (bazelbuild#19701) + Fix Java compilation for jdk21 (bazelbuild#19735) + Fix output materialized as symlink when building without the bytes. (bazelbuild#19739) + Make lockfile's `RepoSpec` attributes more readable (bazelbuild#19748) + Merge rule and aspect validation output groups (bazelbuild#19745) + Fix handling of non-ASCII characters in archive entry file names (bazelbuild#19765) + Raise an early error on invalid labels in transitions inputs/outputs (bazelbuild#19764) + Handle synthetic method parameters entries that don't have names (bazelbuild#19758) + Bazel release process: Fix push justification. (bazelbuild#19768) + Flip --experimental_cc_implementation_deps (bazelbuild#19751) + Add blake3 NEON instructions on linux arm64 (bazelbuild#19804) + Fix crash when `environ` contains duplicate entries (bazelbuild#19827) + Bump c++ standard to c++14 per default (bazelbuild#19794) + Collect debug info context from implementation deps (bazelbuild#19836) + Fix unconditional Skyframe invalidation with --lockfile_mode=… (bazelbuild#19848) Acknowledgements: This release contains contributions from many people at Google, as well as Andreas Herrmann, bazel.build machine account, Brentley Jones, buildbreaker2021, Chirag Ramani, David Ostrovsky, Ed Schouten, Fabian Meumertzheim, George Gensure, Greg, John Laxson, Julio Merino, Keith Smiley, Matt Mackay, Mauricio G, NelsonLi0701, nglevin, Nicholas Junge, oquenchil, Orion Hodson, Roman Salvador, Ted Kaplan, Thi Doan, Thi Don, Tyler Williams, Xùdōng Yáng.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick of the following changes:
RELNOTES: BLAKE3 can be used as a digest function with the
--digest_function=blake3startup flag.