Make casts of pointers to trait objects stricter

[WaffleLapkin]

This is an attempt to fix #120222 and #120217.

This is done by adding restrictions on casting pointers to trait objects.

Before this PR the rules were as follows:

When casting *const X<dyn A> -> *const Y<dyn B>, principal traits in A and B must refer to the same trait definition (or no trait).

With this PR the rules are changed to

When casting *const X<dyn Src> -> *const Y<dyn Dst>

• if Dst has a principal trait DstP,
  • Src must have a principal trait SrcP
  • dyn SrcP and dyn DstP must be the same type (modulo the trait object lifetime, dyn T+'a -> dyn T+'b is allowed)
  • Auto traits in Dst must be a subset of auto traits in Src
    • Not adhering to this is currently a FCW (warn-by-default + FutureReleaseErrorReportInDeps), instead of an error
• if Src has a principal trait Dst must as well
  • this restriction will be removed in a follow up PR

This ensures that

1. Principal trait's generic arguments match (no *const dyn Tr<A> -> *const dyn Tr<B> casts, which are a problem for #120222)
2. Principal trait's lifetime arguments match (no *const dyn Tr<'a> -> *const dyn Tr<'b> casts, which are a problem for #120217)
3. No auto traits can be added (this is a problem for arbitrary self types, see this comment)

Some notes:

• We only care about the metadata/last field, so you can still cast *const dyn T to *const WithHeader<dyn T>, etc
• The lifetime of the trait object itself (dyn A + 'lt) is not checked, so you can still cast *mut FnOnce() + '_ to *mut FnOnce() + 'static, etc
  • This feels fishy, but I couldn't come up with a reason it must be checked

The diagnostics are currently not great, to say the least, but as far as I can tell this correctly fixes the issues.

cc @oli-obk @compiler-errors @lcnr

[rustbot]

r? @TaKO8Ki

(rustbot has picked a reviewer for you, use r? to override)

[compiler-errors]

👍 Ah, ok, so this is fine because we erase regions in check_ptr_ptr_cast.

[compiler-errors]

@bors try

[bors]

⌛ Trying commit 3c3cf17 with merge 2c99da5...

[lukas-code]

This will only work for pointers that can be coerced modulo regions, it will still let something like this pass:

trait Trait<'a> {}

fn cast<'a, 'b>(x: *mut dyn Trait<'a>) -> *mut (dyn Trait<'b> + Send) {
    x as _
}

I think the correct fix would be to actually borrow check the principals for equality around here:

rust/compiler/rustc_borrowck/src/type_check/mod.rs

Line 2294 in d5fd099

CastKind::PtrToPtr => {

Also, extending the trait object lifetime itself (e.g. *mut dyn Trait<'a> + 'a -> *mut dyn Trait<'a> + 'static) is harmless and should be allowed, just like adding auto-traits is.

---

Edit: borrowck version here: c471b9b

[WaffleLapkin]

something like this pass:

It does indeed let it pass, but why? These are not coercible?..

I think the correct fix would be to actually borrow check the principals for equality around here:

I also thought of that, but it seemed somewhat painful to do. And since we handle upcasts with CastKind::PointerCoercion, this seemed like a good solution. Borrowck change might be the right solution after all though.

Edit: borrowck version here: c471b9b

This won't help with *mut (u8, dyn Trait<'a>) -> *mut (u8, dyn Trait<'b>), but that should be an easy fix with struct_tail...

Also, extending the trait object lifetime itself (e.g. *mut dyn Trait<'a> + 'a -> *mut dyn Trait<'a> + 'static) is harmless and should be allowed, [...]

Hm, I can't find an example why it isn't harmless, but I also can't really convince myself that it is. Is this something to do with the fact that you can given dyn Trait<'a> + 'b you can use 'a in Trait's method's signatures, but not 'b?

[...], harmless and should be allowed, just like adding auto-traits is.

Adding auto-traits is not harmless. Here is an example that segfaults in safe code with arbitraty self receivers and adding send:

example

#![feature(arbitrary_self_types)]

trait Trait {
    fn f(self: *const Self)
    where
        Self: Send;
}

impl Trait for *const () {
    fn f(self: *const Self) {
        unreachable!()
    }
}

fn main() {
    let unsend: *const dyn Trait = &(&() as *const ());
    let send_bad: *const (dyn Trait + Send) = unsend as _;
    send_bad.f();
}

fish: Job 1, './t' terminated by signal SIGSEGV (Address boundary error)

(play)

Notably, even though we had to implement f, it's not actually in a vtable, since you should not be able to call it (because of unsatisfied Send bound):

vtable layout

error: vtable entries for `<*const () as Trait>`: [
           MetadataDropInPlace,
           MetadataSize,
           MetadataAlign,
           Vacant,
       ]
  --> ./t.rs:24:1
   |
24 | trait Trait {
   | ^^^^^^^^^^^

[lukas-code]

Adding auto-traits is not harmless. Here is an example that segfaults in safe code with arbitraty self receivers and adding send:

Interesting. I was somehow convinced these functions were still codegened and thought it'd be harmless because you can't do anything bad with with a sendable raw pointer alone.

This also means that borrow check alone won't be enough and we need to treat these casts more like coercions.

Is this something to do with the fact that you can given dyn Trait<'a> + 'b you can use 'a in Trait's method's signatures, but not 'b?

Yeah that was my reasoning as well. You can add bounds like Self: 'static, but since the maybe-not-'static thing is behind a raw pointer you shouldn't be able to do anything bad with it. But that same reasoning was already wrong for auto traits so I'm not sure anymore.

Also, extending the lifetime of trait objects is often used in unsafe code (like the one in std), so forbidding this will probably cause widespread breakage.

something like this pass:

It does indeed let it pass, but why? These are not coercible?..

What I'm trying to say is that because these are not coercible, the try_coercion_cast above will fail, which means that the cast will be a CastKind::PtrToPtr and not a coercion. And CastKind::PtrToPtr is absolutely not borrow checked so any lifetime can be cast to any other lifetime.

[WaffleLapkin]

This also means that borrow check alone won't be enough and we need to treat these casts more like coercions.

I think that we can check that in hir_typeck, the same way we check

let x: &dyn Trait + Send = &();
let y: &dyn Trait = x;

(I think this is done in hir_typeck? will have to find out...)

What I'm trying to say is that because these are not coercible, the try_coercion_cast above will fail, which means that the cast will be a CastKind::PtrToPtr and not a coercion. And CastKind::PtrToPtr is absolutely not borrow checked so any lifetime can be cast to any other lifetime.

Ah, I see. Right.

Do you think moving the check out of try_coercion_cast's match would fix the issue (possibly always reporting an error if try_coercion_cast fails for *mut dyn ...)? Or should I move to your proposed solution with doing things for PtrToPtr in borrowck?

[lukas-code]

Do you think moving the check out of try_coercion_cast's match would fix the issue (possibly always reporting an error if try_coercion_cast fails for *mut dyn ...)?

Where exactly do you want to move it? If this is checked unconditionally even after do_check, then all ptr-ptr casts with vtables that can't be coerced will be just be forbidden, right? Like casting *mut Wrapper<dyn Trait> <-> *mut dyn Trait.

I think that we can check that in hir_typeck, the same way we check

let x: &dyn Trait + Send = &();
let y: &dyn Trait = x;

I think checking this here in check_ptr_ptr_cast where we also compare the principals might be fine. Just check that the source includes all auto traits from the target.

And I'm pretty sure we'd still need borrowck for the principal on top of that.

[lukas-code]

Checking this with the Unsize looks risky to me. That trait is also implemented for trait upcasts, e.g. dyn Subtrait: Unsize<dyn Supertrait>, because trait upcasting is essentially treated as a "unsizing" coercion before codegen, except that the source type is already unsized.

Because of that, with your current implementation this will compile:

trait Super {}
trait Sub: Super {}

struct Wrapper<T: ?Sized>(T);

fn cast(ptr: *const dyn Sub) -> *const Wrapper<dyn Super> {
    // this is a "reinterpret cast" and not an upcast, the vtable is just copied
    ptr as _
}

This could theoretically be fine, because it looks like there is currently no (safe) way to cast from *const Wrapper<dyn Super> to *const dyn Super. This happens, because try_coercion_cast will (via Coerce::coerce_unsized) eagerly evaluate the CoerceUnsized and Unsize obligations, but not their nested obligations, which means that casting *const Wrapper<dyn Super> to *const dyn Super is treated as a coercion and fails. But this also leads to weird inconsistencies whether a cast is considered a coercion cast or pointer-pointer cast.

example

trait Trait {}

struct Wrapper<T: ?Sized>(T);

fn cast1(x: *const dyn Send) -> *const (dyn Send + Sync) {
    x as _ // is a ptr-ptr cast, compiles
}

fn cast2(x: *const Wrapper<dyn Send>) -> *const (dyn Send + Sync) {
    x as _ // is an unsizing coercion cast, error
}

So I'd argue the current behavior is a bug and it should always be allowed to cast *const Wrapper<dyn Trait> to *const dyn Trait, which would make your implementation incorrect.

[WaffleLapkin]

*const dyn Sub -> *const Wrapper<dyn Super> being a "reinterpret cast" is very bad, for the same reasons we have the issues this PR attempts to fix — having an incorrect vtable is not sound with other features. I'll add back the check that the principal traits are the same. While I don't see a safe way to go *const W<dyn Trait> -> *const dyn Trait, this transformation looks more than fine for me, so I really don't want to depend on it being impossible. (tbh I assumed it was possible, considering you can do *const dyn Trait -> *const W<dyn Trait>, it's very weird we don't allow the opposite...)

So I'd argue the current behavior is a bug and it should always be allowed to cast *const Wrapper<dyn Trait> to *const dyn Trait, which would make your implementation incorrect.

In other words: I agree with this.

Although I'm not sure how you can add this in a non-confusing way.

• *const () -> *const Send must be unsizing
• *const W<dyn Trait> -> *const dyn Trait should be ptr-ptr
  • But what if W<dyn Trait>: Trait? Currently it's unsizing, do we keep that?

[lukas-code]

• But what if W<dyn Trait>: Trait? Currently it's unsizing, do we keep that?

*const W<dyn Trait> -> *const dyn Trait can't be an unsizing cast, because W<dyn Trait> is not sized and not a trait object. We can't keep the vtable metadata that is already in the fat pointer *const W<dyn Trait> and add the metadata of an impl Trait for W<dyn Trait> on top of that.

Casting *const T to *const U should be an unsizing coercion if and only if T: Unsize<U>, but W<dyn Trait> can never implement Unsize<dyn Trait>.

The problem with the current implementation is that it doesn't fully check W<dyn Trait>: Unsize<dyn Trait>. It only sees that W<dyn Trait>: Unsize<dyn Trait> holds if the nested obligations W<dyn Trait>: Sized and W<dyn Trait>: Trait hold but then directly assumes the cast must be an unsizing coercion and only checks the nested obligations too late.

[WaffleLapkin]

Oooh, right.

Yeah, then *const W<dyn Trait> -> *const dyn Trait should definitely be supported, but maybe as a follow up to this :)

[RustyYato]

It is possible to do *const W<dyn Trait> -> *const dyn Trait right now in safe code. You just need to write a small helper function

impl<T: ?Sized> Wrapper<T> {
    fn get(this: *const Self) -> *const T {
        this as _
    }
}

fn cast(ptr: *const Wrapper<dyn Super>) -> *const dyn Super {
    Wrapper::get(ptr)
}

[bors]

☀️ Try build successful - checks-actions
Build commit: 2c99da5 (2c99da556cc12f27b8bb6597475ebd722872cb69)

[TaKO8Ki]

@rustbot author

[lcnr]

not following this pr too closely for now 🙇 afaict there are some challenges in getting this to work correctly.

wanted to quickly chime in that i believe at some point #120222 (comment) to be preferable simply because it's easier to confirm that it is actually correct.

I believe this to be fairly straightforward by computing the vtable layout once for a fully generic princial, e.g. reusing the same vtable layout for all instances of for<T, U> dyn Foo<T, U>, regardless of the actual arguments.

for #120217 we could similarly simply forbid calling raw-ptr methods on trait objects/not making them object safe.

[WaffleLapkin]

@bors try

[bors]

⌛ Trying commit 5f27951 with merge 13cae60...

[bors]

☀️ Try build successful - checks-actions
Build commit: 13cae60 (13cae6066db2d5e6c9bb508caceec8d34ca49980)

[bors]

☀️ Try build successful - checks-actions
Build commit: 13cae60 (13cae6066db2d5e6c9bb508caceec8d34ca49980)

[lcnr]

@craterbot check

[craterbot]

👌 Experiment pr-120248 created and queued.
🤖 Automatically detected try build 13cae60
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[traviscross]

This was clearly a ton of work. Huge thanks to @WaffleLapkin for making this happen.

[Kobzol]

@rust-timer build 68e716e

[rust-timer]

Finished benchmarking commit (68e716e): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results (secondary 2.6%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.6%	[2.5%, 2.6%]	3
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 701.941s -> 700.809s (-0.16%)
Artifact size: 328.61 MiB -> 328.59 MiB (-0.00%)

[lvella]

So, this broke my code when updating from 1.80 to 1.81. Shouldn't such change come with deprecation/warning?

Out of curiosity, this is the code snippet:

    let p3: Box<dyn Backend<FInner>> = p3;
    let p3 = Box::into_raw(p3);

    // This is safe because we know that FInner == FOuter.
    Some(unsafe { Box::from_raw(p3 as *mut dyn Backend<FOuter>) })

[WaffleLapkin]

@lvella IIRC we decided that because this is a very rare pattern of code, we can go straight to a hard error. (note that before code was unsound pretty much always (unless, like in your case, there is some kind of proof invisible to the compiler that types are actually the same)).

[oxalica]

• We only care about the metadata/last field, so you can still cast *const dyn T to *const WithHeader<dyn T>, etc

This check makes the pointer metadata of a struct observable by as on stable rustc. Is this intended? So this means changing the ptr metadata of a struct in a library is a breaking change.

playground

struct S<T: ?Sized> {
    // s: Box<T>, // error: thin to fat pointer cast
    // _marker: PhantomData<T>, s: [u8], // error: ptr metadata mismatch
    s: T, // OK
}

fn cast_dyn<T: ?Sized>(t: *mut S<T>) -> *mut T {
    t as _
}

It's reasonable that changing between thin and fat pointer is breaking. But also exposing the unsized representation (T v.s. PhantamData<T>, [u8]) sounds a bit unintuitive to me.

[lukas-code]

This check makes the pointer metadata of a struct observable by as on stable rustc. Is this intended? So this means changing the ptr metadata of a struct in a library is a breaking change.

This behavior was not introduced by this PR, it's been like that since Rust 1.2.0.