Testing

[kloenk]

Create some way to run rust test.

Bindgen creates tests, for allignment. maybe we can test them somehow.

[stappersg]

Few days after this issue was created, was merge request #52, "use rustc directly, created".

With this comment are now those two connected. How helpfull this comment (this connection) is something that can be told in a next comment.

Thing I'm aim for is a comment that tells which how helpful cargo test is, when builds are done without cargo. Or even a rename of this issue from cargo test to rust test.

[ojeda]

Yeah, we can use this issue for testing in general. It's something we should probably take a look at relatively soon.

[adamrk]

Seems like there was a sporadic test failure here: https://github.com/Rust-for-Linux/linux/runs/1921879052 (the only change from the previous commit which passed was a comment). Not sure if it's something to worry about.

[ojeda]

Hmm... It doesn't look like it is us, since it happens before we run any Rust code, but we could have broken some invariant somewhere nevertheless. Do you want to post it to the LKML? (It would be best to reproduce it first without the Rust stuff compiled in, though).

Related: perhaps we should add a CI run to compile things without enabling Rust but with our changes, and start using fail-fast: false to keep all the CI results.

[ojeda]

As I mentioned elsewhere, I will be splitting rust_example.rs into different kernel modules as poor man's tests.

Later on, after the RFC most likely, I will be working on setting up a proper test framework so that we can write tests as easy as in normal Rust.

[alastairreid]

[@wedsonaf pointed me at this issue because I am looking at the closely related question of 'how can we use formal verification tools with Rust?' - and Rust-for-Linux might be a really good use for the tools. In particular, I want to create a verification harness that can also be used as a test-harness or fuzzing harness.]

When looking at the current samples, it looks as if there are two important decisions about testing (or verification):

1. Should a test for the Rust code call into any of the C parts of the Linux kernel or should parts of rust/kernel/*.rs be replaced by a mocking library that mocks memory allocation, printk, device registration, mutexes, etc.?
2. Should a test for a device driver access the driver by invoking something that looks like Linux syscalls or should it access the driver by invoking Rust methods?

I am currently exploring the No/No choice. That is, I am modifying rust/kernel/.rs so that it does not rely on bindings:: types/functions and I am trying to write tests that use Rust types like UserSlicePtr, File, etc. and invoke Rust methods to manipulate those objects instead of trying to invoke drivers by creating inodes with the right major/minor device numbers and invoking the standard C-shaped system calls.

I'm making the No/No choice because I specifically want to focus on the Rust code and because I want to be able to write generic tests that benefit from Rust's traits, etc.
But note that other choices in this design space would also make sense. e.g., Tests that invoke both the Rust and the the C parts of the Linux kernel, have the potential to detect mismatches/misunderstandings between the Rust code and the C code whereas the tests I am writing could never find those because they ignore all the C code.

So far I have exactly one test working. It checks that samples/rust_chrdev.rs rejects attempts to read from the file. Not a particularly strenuous test for now: my focus is still on mocking enough of rust/kernel/*.rs that I can tackle more tricky code.

Once I have more of the mocking layer implemented, my plan is to use the KLEE verification tool to look for bugs such as integer overflows, array bounds violations, etc. in the Rust code. I will also use the same test harness with a fuzzer as an alternative way of looking for bugs. I am hoping that being able to verify/fuzz for these low level bugs would make it less painful/worrying to disable runtime checks in production builds.

Questions:

• Is anybody else working on tests?
• Does the No/No choice I am making fit with your thoughts on testing or do you think I should shift my focus?

[kloenk]

Just a quick idea. You could use no mangling and extern fn to create a mock library which shadows every C function used by the kernel rust crate. So you use the normal rust/kernel but that links against some custom kernel with shadows functions

[alastairreid]

Thanks. I think there's several places where I could cut the dependencies and insert my mock. The C interface has the huge advantage of being (I think) fairly stable. But it's also a bit low level.

Some places where I think it would be easier to modify rust/kernel/*.rs are as follows. (Where modify probably means 'refactor to expose the right interfaces and then use conditional compilation.)

First: to call FileOperations::read, you need to pass a &File. The File struct contains a pointer to bindings::file so I need to create a pointer to a file somehow. I think that the only way to create a file object is to invoke C code in the kernel. At the moment, I am YOLOing it and passing a null pointer but this only works because (AFAIK) the code I am testing does not use the two public methods on File both of which dereference it - but that is not going to work in the long term.
The cleanest solution that I can see seems to be to have an alternative implementation of File. (The underlying data structure has about 15-20 fields, lots of invariants, etc. and I think that replicating that code in Rust would be a mistake and that trying to invoke the C code would pull in a lot of dependencies.

Second: There are Rust functions that I think can be simplified down to no-ops (for testing purposes). For example, the Registration::register function in rust/kernel/miscdev.rs jumps through a bunch of hoops to initialize a C device object, construct a VTABLE, etc. so that it can call bindings::misc_register. If I understand this correctly, the only purpose for this is so that the C part of the Linux kernel can use the major/minor device numbers on an inode to decide which device driver to invoke. Since my plan for testing sidesteps that mechanism, this is not needed. (This is a bit less clear cut - and, in this case, your suggestion also works and is simpler in some ways.)

Third: In part because formal verification of Rust is in its infancy, there don't seem to be any verifiers that support concurrency at the moment. So the entire content of rust/kernel/sync is a problem because the verifier does not support threads, mutexes, semaphores, etc. Getting the right behaviour (or, at least, a sufficiently good approximation) here will almost certainly require some major surgery. This won't necessarily affect conventional testing and fuzzing - but it is a major issue for me.

At the moment, my feeling is that I am going to have to use several different approaches here: sometimes making the cut at the Rust-C interface as you suggest, sometimes making the cut in the Rust code. But it's early days and I expect that I will end up having to discard and redo my first attempt or two...

[ojeda]

@alastairreid

Is anybody else working on tests?

My plan is to work on supporting unit tests. I am not aware of anybody working on anything else.

At the moment, we only have a few sanity functional tests for the CI.

Does the No/No choice I am making fit with your thoughts on testing or do you think I should shift my focus?

Hard to answer :) Perhaps having a meeting for this would be best.

My gut feeling is that userspace functional tests are currently the most valuable given it is still early days. They are likely the easiest to write and the most stable ones.

Some places where I think it would be easier to modify rust/kernel/*.rs are as follows. (Where modify probably means 'refactor to expose the right interfaces and then use conditional compilation.)

In general, I would be cautious about changing interfaces to accommodate testing/mocking.

This is not to say we should not do it: if we can show it has a big impact, we should definitely push for it. However, we first need to get Rust proven within the kernel.

I am hoping that being able to verify/fuzz for these low level bugs would make it less painful/worrying to disable runtime checks in production builds.

There may be some particular places where we want to do unchecked operations due to performance reasons, and the policy is to justify every unsafe block in the form of a comment (and, of course, ideally they should be proven/tested/fuzzed/etc.). But, in general, I do not think we should encourage users to run kernels with checks disabled.

The C interface has the huge advantage of being (I think) fairly stable.

Please note that the kernel internal APIs are not stable. However, it is true that, at the moment, the Rust ones are likely to change more than the C ones (by virtue of being new). But longer-term there would be nothing that would make the Rust ones more or less stable than the C ones.

my plan is to use the KLEE verification tool to look for bugs such as integer overflows, array bounds violations, etc. in the Rust code.

That sounds great. Has there been any project on using KLEE for the C side? Any showstoppers?

Another thing I would like to have is Miri/KASAN/UBSAN etc. for the Rust side.

[alastairreid]

@ojeda

Thanks for your comments. All makes sense.
I'll see how far I can get with @kloenk's suggestion of implementing the bindings:: API - once I have a bit more experience and have some sample code to share (i.e., not the very small hack-fest I currently have), a meeting would probably be good.

There may be some particular places where we want to do unchecked operations due to performance reasons, and the policy is to justify every unsafe block in the form of a comment (and, of course, ideally they should be proven/tested/fuzzed/etc.). But, in general, I do not think we should encourage users to run kernels with checks disabled.

My interpretation of Linus Torvalds' comment on your RFC was that any panics from the Rust code were unacceptable to him. He focused on the out-of-memory panic that can be fixed by using a panic-free memory API but I'd think that he would have the same reaction to the 'invisible' panics inserted by the compiler to implement runtime overflow checks and runtime array bounds checks because they also cause the kernel to halt with loss of service and potential loss of data.
Of course, I may be over-generalizing his statement and/or he may shift his position...

Has there been any project on using KLEE for the C side? Any showstoppers?

I think it has been used once - but other C verification tools have been used far more. Which suggests that there might be problems that made them stop?
(I'm using KLEE because it uses LLVM-IR and so I can use it with rustc or even with rustc+clang. The tools normally used for the Linux kernel are C-only.)

Another thing I would like to have is Miri/KASAN/UBSAN etc. for the Rust side.

Yes, definitely!

In fact, maybe the right development sequence is:

1. Usermode tests that can be run directly.
2. Usermode tests that can be run in miri and with sanitizers
3. Usermode parameterized tests that can used with fuzzers and with KLEE.

("Parameterized tests" are also called "structure-aware fuzzing", "property-based testing" or "verification harnesses" - depending on which tool you are thinking about using - but they are more or less the same thing and writing tests that can be used for all is my preferred way of using KLEE.)

Having said that, I might jump straight from (1) to (3) because the KLEE part is the bit that worries me most and is my primary focus.

[ojeda]

My interpretation of Linus Torvalds' comment on your RFC was that any panics from the Rust code were unacceptable to him. He focused on the out-of-memory panic that can be fixed by using a panic-free memory API but I'd think that he would have the same reaction to the 'invisible' panics inserted by the compiler to implement runtime overflow checks and runtime array bounds checks because they also cause the kernel to halt with loss of service and potential loss of data.
Of course, I may be over-generalizing his statement and/or he may shift his position...

They are very different cases:

• The alloc-related panics are there simply because there is no fallible approach. It is an interface design problem. Having a allocation failure does not mean the kernel has a bug, just that it does not have memory to serve a particular request.
• On the other hand, OOB accesses, unexpected overflows, etc. are severe bugs with potentially unbounded consequences. In general, one cannot trust the environment after they occur. They are not a design problem in the sense of the alloc ones.

That is why latter ones should panic the same way dereferencing the start of the address space does (whether such a panic should kill only the current thread or the entire kernel is another question).

Furthermore, in general, potential loss of data is way better than silent corruption of data. Similarly, denial of service is way better than getting a server compromised.

Having said that, I might jump straight from (1) to (3) because the KLEE part is the bit that worries me most and is my primary focus.

Of course, feel free to take the approach that you prefer, the one that you think will net more benefits or the one that is best for your research.

My points in the previous message were only about what I think we can handle/merge at the present time (e.g. refactoring many things before we have managed to get Rust in mainline may be problematic).

[ojeda]

("Parameterized tests" are also called "structure-aware fuzzing", "property-based testing" or "verification harnesses" - depending on which tool you are thinking about using - but they are more or less the same thing and writing tests that can be used for all is my preferred way of using KLEE.)

I am sure you have seen it already, but just in case: Rust has a nice library for property-based testing called proptest inspired in hypothesis for Python.

[alastairreid]

Yes, I like proptest a lot!
More or less the first thing I did after figuring out how to use KLEE with Rust was reimplement the proptest API for KLEE (and any similar tool). I wrote this blog article about it.

btw In case you're interested, here is the status of getting KLEE to work with Rust (another blog article).

[ojeda]

Closing -- this was a very "general" issue. Nowadays we have several ways of testing Rust code (and docs about it). If there is something that needs to be done here, then we should probably create more focused/targeted issues.