Internal(KvmContext(KvmCap(Xcrs))) on an old server

[bcspragu]

Hi there!

I'm trying to run Firecracker VMs on an old (okay, really old) bare metal blade server. I've enabled virtualization in the BIOS, and have set up /dev/kvm as instructed, but I can't get VMs to run, they fail with the following:

$ firecracker --no-api --config-file ~/vmconfig.json
2023-02-22T19:06:20.817652022 [anonymous-instance:main:ERROR:src/firecracker/src/main.rs:480] Building VMM configured from cmdline json failed: Internal(KvmContext(KvmCap(Xcrs)))

I'm not sure if the server is literally just too old and lacking capabilities to run Firecracker, but according to the datasheet it does support VT-x.

Some debugging info:

$ ./tools/devtool checkenv
[Firecracker devtool] Checking prerequisites for running Firecracker.
[Firecracker devtool] Please check ../docs/getting-started.md#prerequisites in case of any error.
[Firecracker devtool] Checking Host Security Configuration.
[Firecracker devtool] Please check ../docs/prod-host-setup.md in case of any error.
[Firecracker devtool] WARNING: retpoline, IBPB, IBRS: DISABLED.
[Firecracker devtool] WARNING: VMX: cache flushes: DISABLED
[Firecracker devtool] WARNING: Hyperthreading ENABLED.

$ uname -a
Linux localhost 5.15.89-flatcar #1 SMP Wed Feb 15 18:00:42 -00 2023 x86_64 Intel(R) Xeon(R) CPU X5650 @ 2.67GHz GenuineIntel GNU/Linux

$ firecracker --version
Firecracker v1.1.4

Supported snapshot data format versions: v0.23.0, v0.24.0, v0.25.0, v1.0.0, v1.1.0

Not sure if the above WARNINGs are actually a problem, or really just warnings.

[zulinx86]

Hi @bcspragu ,

The error seems to occur due to the lack of a KVM capability.
Firecracker requires KVM_CAP_XCRS capability to call KVM_GET_XCRS and KVM_SET_XCRS [1][2][3][4].

And KVM_CAP_XCRS capability depends on available features of CPU that you're using [5][6].
Could you check if your CPU support XSAVE instructions with the following command?

$ lscpu | grep xsave

I hope this information is helpful for you.
Thanks,

[zulinx86]

Thanks for your confirmation! Sounds great!

I haven't got time to do yet and we need to sort out what capabilities are used only in snapshot feature first. I hope you will continue to use your nice hacky patch if you use firecracker for test purpose.

Thanks,

[travisghansen]

I applied the patch on top of the weaveworks macvtap fork and am using it successfully now with flintlock. Similar situation, older amd hardware that was previously crashing with the error mentioned above.

[mattiaforc]

@travisghansen I've been trying to make it work with flintlock too - could you provide some guidance on how to apply the patch? Did you just rebuild it from source and replace the firecracker binary?

[travisghansen]

Yeah, simple as that.

[mattiaforc]

Yeah, simple as that.

Thanks, it worked just fine!

[zulinx86]

PR #3967 added an ability to add/remove KVM capabilities to be checked with custom CPU templates. This feature will be available in the next minor release (v1.5.0). Regarding how to use the custom CPU template feature, please refer to the public doc. Closing this discussion.