Toolchains on Bazel-CI are outdated

[davido]

In light of recent breakages for new and shiny compiler versions, would it be possible to have experimental CI toolchains with the latest and greatest compiler versions?

Currently there are 3 different gcc compiler versions (only care about Linux here) covered by Bazel-CI: [1]

• gcc 10.2.1
• gcc 9.4.0
• gcc 7.5.0

Current gcc version is 12: [2], that was released almost one year ago: August 19, 2022
and is missing in the above list.

Moreover, some Linux distributions switched to shipping gcc 13 per default: [3]
GCC 13 Stage 4 (starts 2023-01-16)

Given that the major Bazel development dependencies, like abseil-cpp, gRpc and protobuf
are hopelessly outdated (and are hard to upgrade and patch), and are years behind of upstream
the chances are high, that Bazel would have issues building from source on modern toolchains.

Having experimental Bazel-CI coverage on modern toolchains would help to earlier detect and fix
those breakages and/or update development dependencies earlier.

[1] https://buildkite.com/bazel/bazel-bazel-github-presubmit/builds/15498
[2] https://gcc.gnu.org/releases.html
[3] https://gcc.gnu.org/develop.html#timeline

[meteorcloudy]

@davido Can you install your desired version of gcc in the shell_command section in your pipeline configuration file?

[meteorcloudy]

But meanwhile, we do need to add Ubuntu 22.04 LTS support. /cc @fweikert

[meteorcloudy]

@davido If we build Bazel with modern gcc version, is it still going to be compatible with older Linux distributions (e.g. older libc versions)? I think that's a major breaking change that would break many users.

[davido]

@meteorcloudy

Misunderstanding?

If we build Bazel with modern gcc version, is it still going to be compatible with older Linux distributions (e.g. older libc versions)?

I don't ask to wipe out the CI verification on outdated toolchains. All what I'm saying, is that coverage for the current toolchains is entirely missing on Bazel-CI. And as you know, it's hard to upgrade the external dependencies in Bazel build toolchain.

Thus the breakages like: [1] are not detected on Bazel-CI and this is pity. Moreover, because the Bazel maintainers (apparently?) do not have access to new and shiny compiler versions, they cannot easily review/approve toolchain related PRs and this is frustrating.

[1] bazelbuild/bazel#18167

[meteorcloudy]

Does shell_commands help? You can run whatever command you need to install the toolchains or packages required, for example https://github.com/bazelbuild/continuous-integration/blob/master/pipelines/tensorflow.yml#L14

[davido]

@meteorcloudy I would rather not mess around with trying to install new compiler toolchains on outdated linux distribution versions. I also don't specifically want to add new pipeline for gerrit or other downstream projects using Bazel. I just want to add pipeline for modern toolchain for Bazel itself. Bazel and Gerrit projects have many external dependencies in common, like gRpc, Protobuf, Upb, Absel-cpp, zlib, etc. Once we can build Bazel from scratch I am very confident we can build Gerrit as well...

Alternative suggestion: Fedora project is known to ship newest toolchains. Fedora 39 is shipping:

  $ java -fullversion
openjdk full version "20.0.1+9"

  $ g++ --version
g++ (GCC) 13.1.1 20230426 (Red Hat 13.1.1-1)

See also my attempt to through together minimal Dockerfile to build Bazel: [1]. Would it be an option to add this Dockerfile (or something similar) and add Buildkite pipeline to build Bazel based on it as well?

The toolchain upgrade should be straight forward, once Fedora 40 is released, we could bump it and verify that we can still build Bazel on new and shiny compiler versions...

[1] bazelbuild/bazel#18309

[meteorcloudy]

I see, sorry for the previous misunderstanding, I think we can do this

• Add new platform support here for Bazel CI which ships the latest toolchain. Would you be able to contribute a similar docker file for the new platform?
• Then we can set up a minimal bazel build job for the new platform to guard it from regression. We can first enable it in Bazel postsubmit, if no major issues, we can also consider enabling it in presubmit.

[davido]

Add new platform support here for Bazel CI which ships the latest toolchain.

Done in: #1639.

Would you be able to contribute a similar docker file for the new platform?

Done in: #1638