config: add rust build configurations

[10ne1]

config: add rust build configurations

Starting with the upcoming v6.1, mainline Linux has merged the
initial Rust infrastructure so this adds some configs for build
testing it together with some sample modules.

The kernel requires a specific version of rustc, so we add the
rustc-1.62 build environment which derives from clang-15, since
a C compiler is still required to build the kernel and the
supported kernel version is 15 (we might bump this later).

Obviously GCC can be used as well but for now testing all the
toolchain combinations does not add significant value. In the
future more toolchain combinations can be used as needed.

The official "offline" toolchain installation method is used as
documented at [1] with sha256sum and because some distros like
Debian stable might not provide up to date toolchain and crates
to keep up with the mainline kernel.

Only the x86_64 architecture is supported by the kernel for now.

We also add the Rust-for-Linux kernel maintainer trees which
contain additional modules and bindings.

Signed-off-by: Adrian Ratiu <[email protected]>

[nuclearcat]

Testing it, might take a while, i will report progress here.
Docker image rustc-1.62:kernelci built successfully

[mgalka]

In general it looks good. I'd probably go with 2 separate names for the .config files (mentioned above)

[mgalka]

Some changes are required to make the image build by kernelci Jenkins.

[gctucker]

See also the email thread with @ojeda from January 2022 https://groups.io/g/kernelci/message/1336

[ojeda]

Thanks for the Cc!

The official "rustup" toolchain installation method is used as
documented in the upstream kernel

The current commits seem to be using the standalone installers, not rustup. Is the commit description correct?

[mgalka]

Tested builds on staging:
https://bot.staging.kernelci.org/job/kernel-build/91146/console
https://bot.staging.kernelci.org/job/kernel-build/91160/consoleFull

Builds OK

[10ne1]

Hi @ojeda If you are ok with the current state of the PR can you please leave a comment so I can add a reviewed-by tag for you? (I also updated the commit msg & PR description to mention the offline install method).

[nuclearcat]

I'm seeing some problems with rust on staging:

+ kci_build pull_tarball --url http://storage.staging.kernelci.org/rust-for-linux/rust/v6.1-rc1-1486-g729c03011c60/linux-src_rust-for-linux.tar.gz --retries 3 --delete
+ kci_build generate_fragments --build-config=rust-for-linux
kernel/configs/rust.config
kernel/configs/rust-for-linux-samples.config
kernel/configs/amdgpu.config
kernel/configs/crypto.config
kernel/configs/ima.config
kernel/configs/kvm_guest.config
kernel/configs/x86-chromebook.config
kernel/configs/xfstests.config
+ kci_build init_bmeta --build-config=rust-for-linux --commit=729c03011c60382030520f84904a85e8a5b3cfb9 --describe=v6.1-rc1-1486-g729c03011c60 --describe-verbose=v6.1-rc1-1486-g729c03011c60
Initialising build meta-data in /tmp/kci/linux/build
Install directory: /tmp/kci/linux/build/_install_
+ kci_build make_config --defconfig=x86_64_defconfig+x86-chromebook+kselftest
Traceback (most recent call last):
  File "/usr/local/bin/kci_build", line 4, in <module>
    __import__('pkg_resources').run_script('kernelci==1.1', 'kci_build')
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 651, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1448, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/EGG-INFO/scripts/kci_build", line 451, in <module>
    status = opts.command(configs, opts)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/EGG-INFO/scripts/kci_build", line 344, in __call__
    res = step.run(args.j, args.verbose, opts)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/kernelci/build.py", line 1204, in run
    self._gen_kci_frag(configs, fragments, kci_frag_name)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/kernelci/build.py", line 1118, in _gen_kci_frag
    with open(path) as frag:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/kci/linux/kernel/configs/kselftest.config'
Build result: 1

bmeta.json

Several rust builds failed.

[10ne1]

@nuclearcat thanks for the heads-up, I've forgot to add the new ${CARGO_HOME}/bin to the path, so bindgen couldn't be found. It's fixed now.

[ojeda]

can you please leave a comment so I can add a reviewed-by tag for you?

I wrote a nit above and also I left a comment a few days ago above on minimizing the RUN layers, but that is just an optimization, and if you are not doing it in other files, I would ignore it here (I saw it in qemu.ninja2, but not in others, so I don't know if you are following that practice in KernelCI or not...).

Other than those, it looks good to me!

Reviewed-by: Miguel Ojeda [email protected]

One final note: the rust branch will likely go away or be repurposed at some point in the future -- I assume this is not a problem, but just so that you are aware since you were testing it here :)

Thanks a lot for this!

[nuclearcat]

It seems build still failed recently on staging due kfselftest fragment missing (not sure why):

+ kci_build init_bmeta --build-config=rust-for-linux --commit=729c03011c60382030520f84904a85e8a5b3cfb9 --describe=v6.1-rc1-1486-g729c03011c60 --describe-verbose=v6.1-rc1-1486-g729c03011c60
Initialising build meta-data in /tmp/kci/linux/build
Install directory: /tmp/kci/linux/build/_install_
+ kci_build make_config --defconfig=x86_64_defconfig+x86-chromebook+kselftest
Traceback (most recent call last):
  File "/usr/local/bin/kci_build", line 4, in <module>
    __import__('pkg_resources').run_script('kernelci==1.1', 'kci_build')
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 651, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1448, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/EGG-INFO/scripts/kci_build", line 451, in <module>
    status = opts.command(configs, opts)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/EGG-INFO/scripts/kci_build", line 344, in __call__
    res = step.run(args.j, args.verbose, opts)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/kernelci/build.py", line 1204, in run
    self._gen_kci_frag(configs, fragments, kci_frag_name)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/kernelci/build.py", line 1118, in _gen_kci_frag
    with open(path) as frag:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/kci/linux/kernel/configs/kselftest.config'
+ export KERNEL_BUILD_RESULT=1
+ KERNEL_BUILD_RESULT=1
+ set +x
Build result: 1

bmeta.json
{
    "build": {
        "duration": 0.03600597381591797,
        "status": "PASS"
    },
    "environment": {
        "arch": "x86_64",
        "compiler": "clang",
        "compiler_version": "15",
        "compiler_version_full": "Debian clang version 15.0.4-++20221102053030+5c68a1cb1231-1~exp1~20221102053120.92",
        "cross_compile": "",
        "cross_compile_compat": "",
        "make_opts": {
            "KBUILD_BUILD_USER": "KernelCI"
        },
        "name": "rustc-1.62",
        "platform": {
            "uname": [
                "Linux",
                "build-j92564-x86-64-rustc-1-62-x86-64-defconfig-x86-chromed54hv",
                "5.10.127+",
                "#1 SMP Fri Jul 8 00:39:24 UTC 2022",
                "x86_64",
                ""
            ]
        },
        "use_ccache": true
    },
    "revision": {
        "branch": "rust",
        "commit": "729c03011c60382030520f84904a85e8a5b3cfb9",
        "describe": "v6.1-rc1-1486-g729c03011c60",
        "describe_verbose": "v6.1-rc1-1486-g729c03011c60",
        "tree": "rust-for-linux",
        "url": "https://github.com/Rust-for-Linux/linux.git"
    }
}
_install_

[10ne1]

@nuclearcat This fails because the rust configurations do not depend on kselftest config fragment generation, but apparently kselftest is used by the staging tests? Why do they do that?

We could fix it the following way, but I don't think it makes sense because kselftests should not be required for this PR.

diff --git a/config/core/build-configs.yaml b/config/core/build-configs.yaml
index 98809ab7..8cf16c8c 100644
--- a/config/core/build-configs.yaml
+++ b/config/core/build-configs.yaml
@@ -1023,7 +1023,7 @@ build_configs:
     variants:
       rustc-1.62:
         build_environment: rustc-1.62
-        fragments: [rust, rust-for-linux-samples]
+        fragments: [rust, rust-for-linux-samples, kselftest]
         architectures:
           x86_64: *x86_64_arch

Then the following works:

./kci_build init_bmeta --build-config=rust-for-linux
./kci_build generate_fragments --build-config=rust-for-linux
./kci_build make_config --defconfig=x86_64_defconfig+x86-chromebook+kselftest

But again, I repeat, kselftest should not be required. Is there any way to avoid it? To drop that hardcoded +kselftest from the kci_build make_config invocation.

[10ne1]

But again, I repeat, kselftest should not be required. Is there any way to avoid it? To drop that hardcoded +kselftest from the kci_build make_config invocation.

@nuclearcat @mgalka Gentle ping. Any thoughts about dropping the unnecessary +kselftest from the staging run kci_build make_config invocation?

[nuclearcat]

I am not sure why it is hardcoded, @mgalka @gctucker ?

[mgalka]

I am not sure why it is hardcoded, @mgalka @gctucker ?

I don't know why +keselftest is part of the x86_64_arch. I think we can just add config without kselftest for rust builds.

  rust:
    tree: mainline
    branch: 'master'
    variants:
      rustc-1.62:
        build_environment: rustc-1.62
        fragments: [rust, rust-samples]
        architectures:
          x86_64: &x86_64_arch_nokselftest
            base_defconfig: 'x86_64_defconfig'
            extra_configs:
              - 'allmodconfig'
              - 'allnoconfig'
              - 'x86_64_defconfig+x86-chromebook'
              - 'x86_64_defconfig+x86-chromebook+amdgpu'
            fragments: [amdgpu, crypto, ima, x86_kvm_guest, x86-chromebook]

  rust-for-linux:
    tree: rust-for-linux
    branch: 'rust'
    variants:
      rustc-1.62:
        build_environment: rustc-1.62
        fragments: [rust, rust-for-linux-samples]
        architectures:
          x86_64: *x86_64_arch_nokselftest

I'm giving a try to at the moment on staging.

[gctucker]

Well staging build configurations are pretty arbitrary and aim at getting the highest tests / build ratio. Building with kselftest fragment enabled is required to run kselftests, and we need that on staging. There's no point building without kselftest on staging as that's not adding any real coverage. If we can't build kselftest with rust then we just need to fix that in the rust build config options, no need to disable kselftest builds when rust is not enabled.

[10ne1]

Well staging build configurations are pretty arbitrary and aim at getting the highest tests / build ratio. Building with kselftest fragment enabled is required to run kselftests, and we need that on staging. There's no point building without kselftest on staging as that's not adding any real coverage. If we can't build kselftest with rust then we just need to fix that in the rust build config options, no need to disable kselftest builds when rust is not enabled.

Thanks for the context, that is what I was missing, so we do have to hardcode something either way to pass the staging tests.

Between the choice of the two hardcoding options - the rust build config options (eg x86_64_arch_nokselftest above) and the kselftest fragment, I'd like us to go with just adding the kselftest fragment.

@mgalka do you agree?

EDIT: I've pushed the change adding the keselftest fragment.

[mgalka]

@mgalka do you agree?

I'm fine with it. Giving it a try on staging.

[mgalka]

Tested OK on staging. A build which previously failed, works fine now.

https://bot.staging.kernelci.org/job/kernel-build/95009/console

[mgalka]

LGTM

[aliceinwire]

Thanks LGTM