[slice] Document slice DSTs, including size guarantees

[joshlf]

Makes progress on rust-lang/unsafe-code-guidelines#465

If you're looking for breadcrumbs, see also: #121965

[rustbot]

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

[joshlf]

I'd like to add an example to document what is meant by "the instance of that type with 0 elements", although I'm not sure how to do that in a way that doesn't rely either on subtle pointer behavior (such as creating a *const [()] and then casting it to a *const T, which preserves slice element count) or on unstable features (such as size_of_val_raw).

cc @RalfJung

[asquared31415]

assert_eq!(2 * pointer_size, std::mem::size_of::<&SliceDst>());

This is an additional new guarantee. Right now everywhere that says that a fat pointer is 2xusize explicitly says that it's non-normative.

[joshlf]

assert_eq!(2 * pointer_size, std::mem::size_of::<&SliceDst>());

This is an additional new guarantee. Right now everywhere that says that a fat pointer is 2xusize explicitly says that it's non-normative.

I copied that from the preceding text in the same doc comment, which does not disclaim it as non-normative. I'm happy to add a disclaimer or use a different example, but if the intention is to be non-normative, we should update the preceding text.

[asquared31415]

Oh interesting, the Reference is not consistent on whether this is a guarantee.

The section on DSTs says

Pointer types to DSTs are sized but have twice the size of pointers to sized types

The section on general type layout of pointers says

Though you should not rely on this, all pointers to DSTs are currently twice the size of the size of usize and have the same alignment.

[RalfJung]

Sorry, I don't have a good idea how to express that.

[joshlf]

Oh interesting, the Reference is not consistent on whether this is a guarantee.

The section on DSTs says

Pointer types to DSTs are sized but have twice the size of pointers to sized types

The section on general type layout of pointers says

Though you should not rely on this, all pointers to DSTs are currently twice the size of the size of usize and have the same alignment.

Given that this is just a copy-paste within the same module doc comment (and so this isn't a regression in terms of implying a guarantee that doesn't exist), maybe we can go with the existing example for now, and address all of these examples together if/when we do that?

[joshlf]

Sorry, I don't have a good idea how to express that.

Maybe we can just leave it prose-only. The vast majority of users won't need to reason about or rely on this property, and for the few that do, they already know what they need - they just need a guarantee somewhere that they can anchor on for proofs of soundness in e.g. safety comments.

[RalfJung]

Cc @rust-lang/opsem @chorman0773 for more input, maybe someone has an idea. This is not a t-opsem question but still, someone might have good idea for how to best phrase this.

[chorman0773]

If we're guaranteeing the size of slice, perhaps:

The layout of a pointer to a slice of type T (&[T], &mut [T], *const [T], or Box<[T]>) is the same as a single pointer to T pointing to the first element of the slice and a usize length (in units of T). The order of the fields is not specified, but there is no padding between them, or following them.

CHERI might be something that's fun to keep in mind, but this will guarantee size/alignment of a slice (this could be written in a note, like "As a result, a pointer to a slice is twice the size as a pointer to T"), but not fully guarantee the layout. If we're guaranteeing the full layout I might simply say

The layout of a pointer to a slice of type T is the same as the struct definition #[repr(C)] struct SlicePtr<T>(*const T, usize);

[joshlf]

I'd like to add an example to document what is meant by "the instance of that type with 0 elements", although I'm not sure how to do that in a way that doesn't rely either on subtle pointer behavior (such as creating a *const [()] and then casting it to a *const T, which preserves slice element count) or on unstable features (such as size_of_val_raw).

cc @RalfJung

@chorman0773 IIUC, @RalfJung was asking about this, not about whether/how to document fat pointer layout.

If we're guaranteeing the size of slice, perhaps:

The layout of a pointer to a slice of type T (&[T], &mut [T], *const [T], or Box<[T]>) is the same as a single pointer to T pointing to the first element of the slice and a usize length (in units of T). The order of the fields is not specified, but there is no padding between them, or following them.

CHERI might be something that's fun to keep in mind, but this will guarantee size/alignment of a slice (this could be written in a note, like "As a result, a pointer to a slice is twice the size as a pointer to T"), but not fully guarantee the layout. If we're guaranteeing the full layout I might simply say

The layout of a pointer to a slice of type T is the same as the struct definition #[repr(C)] struct SlicePtr<T>(*const T, usize);

That said, it's worth recording these ideas. I wonder if there's a good place to track questions about what we guarantee with regards to fat pointer layout?

[joshlf]

Is there any objection to merging this as-is? As I mentioned here, the current version of this PR has what I need to be able to write safety proofs. We can always follow up with more clarity or examples later if desired.

[RalfJung]

What does "a slice DST compiles successfully" mean? Is there precedent for a statement like this somewhere else that we can follow?

[joshlf]

It's just meant to account for the fact that you could in principle write down a type which, were it to compile, would violate this guarantee (i.e., a type whose fixed part itself overflows isize). I can just remove that parenthetical, though, since it's implicit - obviously you only care about code that compiles.

[RalfJung]

Okay, I see. Do we usually phrase this like that? "Rust guarantees that all types are no larger than isize." That's not what I would have said, I would have said that Ruts requires that all types fit in an isize, and halts compilation otherwise. But I don't know which wording we are using elsewhere.

Do we have a test for this? We have tests for large types but it might be worth having one specifically for this clause.

[RalfJung]

TBH I am not very happy making such a completely random guarantee, it feels way too specific. Shouldn't this be a general principle? We already have that sized types fit in an isize. (I assume this is documented somewhere?) Then furthermore, for unsized types we have the notion of their static prefix size (or some such term), which is basically the size that the type has if the dynamic portion has size 0 (and minimal alignment, in case of dyn Trait). We also need that that fits in an isize.

If I were to look for such a thing I would never look at the docs for slices. It's not even really a property of slices, it's a property of structs/tuples with unsized tails.

[joshlf]

TBH I am not very happy making such a completely random guarantee, it feels way too specific. Shouldn't this be a general principle? We already have that sized types fit in an isize. (I assume this is documented somewhere?) Then furthermore, for unsized types we have the notion of their static prefix size (or some such term), which is basically the size that the type has if the dynamic portion has size 0 (and minimal alignment, in case of dyn Trait). We also need that that fits in an isize.

IIUC, this isn't a nicely compositional property. In particular, the padding in a DST can come after the trailing slice field, so it could be the case that the offset of the trailing slice fits in an isize but, once the post-slice padding is added, it no longer fits in an isize. In other words, the "static prefix" is not necessarily a valid Rust type on its own.

In this example, the offset of the trailing slice field is 3 bytes despite the type's alignment being 2. That would not constitute a legal Rust type.

If I were to look for such a thing I would never look at the docs for slices. It's not even really a property of slices, it's a property of structs/tuples with unsized tails.

I'd be happy to move this somewhere else.

[RalfJung]

In this example, the offset of the trailing slice field is 3 bytes despite the type's alignment being 2. That would not constitute a legal Rust type.

That's a u8 slice, alignment 1. I don't see how this is not legal?

IIUC, this isn't a nicely compositional property. In particular, the padding in a DST can come after the trailing slice field, so it could be the case that the offset of the trailing slice fits in an isize but, once the post-slice padding is added, it no longer fits in an isize. In other words, the "static prefix" is not necessarily a valid Rust type on its own.

Yeah, it's not a type. Also "static prefix" is probably a bad term, in your case that prefix has size 4 -- it's the smallest size the type can have. We don't have a lot of good terminology for these kinds of unsized types.

But it is fully compositional, it is computed by rustc in a compositional way after all.

I'd be happy to move this somewhere else.

Where do we discuss layout of structs / tuples? It might fit there.

Where do we say that a sized type is never bigger than isize? I know we say it for allocations now but that's not the same statement (though ofc they are related by soundness).

But I see now that I originally suggested the slice type. I clearly don't know where it should go, sorry. It's such an oddly specific thing to ask about I can't fit it into any category.^^ (Did I ask why you want this particular guarantee?^^)

[joshlf]

In this example, the offset of the trailing slice field is 3 bytes despite the type's alignment being 2. That would not constitute a legal Rust type.

That's a u8 slice, alignment 1. I don't see how this is not legal?

I meant that if we treat a DST as a fixed-size, valid Rust type followed by a slice, then this is a counter-argument: the fixed sized prefix is 3 bytes, but it contains a u16 field, and thus has alignment 2. Thus, we can't treat the fixed-sized prefix as a valid type on its own.

IIUC, this isn't a nicely compositional property. In particular, the padding in a DST can come after the trailing slice field, so it could be the case that the offset of the trailing slice fits in an isize but, once the post-slice padding is added, it no longer fits in an isize. In other words, the "static prefix" is not necessarily a valid Rust type on its own.

Yeah, it's not a type. Also "static prefix" is probably a bad term, in your case that prefix has size 4 -- it's the smallest size the type can have. We don't have a lot of good terminology for these kinds of unsized types.

I would say that the "smallest instance of the type" has size 4, but the prefix (ie, the bytes that precede the trailing slice field) has size 3. That's why I'm arguing that this does not just fall naturally out of our other existing rules.

But it is fully compositional, it is computed by rustc in a compositional way after all.

I'd be happy to move this somewhere else.

Where do we discuss layout of structs / tuples? It might fit there.

Where do we say that a sized type is never bigger than isize? I know we say it for allocations now but that's not the same statement (though ofc they are related by soundness).

My understanding is that it's true of allocations, and then by guaranteeing that &T always points to an allocation, we ensure it must be true by implication (I suppose unless there are types you aren't allowed to take a reference to, but I'm assuming that's not a thing).

But I see now that I originally suggested the slice type. I clearly don't know where it should go, sorry. It's such an oddly specific thing to ask about I can't fit it into any category.^^ (Did I ask why you want this particular guarantee?^^)

I'm on my phone and I can't easily look it up right now, but my recollection is that it has to do proving that certain synthesized references never address more than isize bytes. We're adding support to zerocopy to synthesize DST references.

[joshlf]

But I see now that I originally suggested the slice type. I clearly don't know where it should go, sorry. It's such an oddly specific thing to ask about I can't fit it into any category.^^ (Did I ask why you want this particular guarantee?^^)

I'm on my phone and I can't easily look it up right now, but my recollection is that it has to do proving that certain synthesized references never address more than isize bytes. We're adding support to zerocopy to synthesize DST references.

Nvm, turns out the reason is actually to address this concern: #69835 (comment)

[RalfJung]

I meant that if we treat a DST as a fixed-size, valid Rust type followed by a slice, then this is a counter-argument: the fixed sized prefix is 3 bytes, but it contains a u16 field, and thus has alignment 2. Thus, we can't treat the fixed-sized prefix as a valid type on its own.

The prefix isn't "what's before the DST field". That doesn't even make sense for dyn Trait DST fields as their offsets are dynamic so "before the field" depends on the dynamic type of the field.

What I mean by "prefix" is the layout of the type if the DST field has minimal size and alignment (for slices: size 0 and alignment as given by the element type; for dyn Trait: size 0 and alignment 1). This concept already exists in the Rust compiler, it's what you get when you ask for the layout of a DST and then ask for its size. "Prefix" might be a bad name; this thing doesn't have a name inside the compiler. But this is exactly the thing that rustc uses to check for "too big", so this is ultimately exactly what you want to capture here.

I would say that the "smallest instance of the type" has size 4, but the prefix (ie, the bytes that precede the trailing slice field) has size 3. That's why I'm arguing that this does not just fall naturally out of our other existing rules.

I think we're talking about the same thing, I just picked a bad name.^^

My understanding is that it's true of allocations, and then by guaranteeing that &T always points to an allocation, we ensure it must be true by implication (I suppose unless there are types you aren't allowed to take a reference to, but I'm assuming that's not a thing).

Okay so strictly speaking you can only conclude this for types to which you hold a shared reference.

We should probably add that here; that seems like a good place to say that there's a maximum size. (There's a maximum alignment, too, but I don't how how stably we guarantee that. It is currently 2^29.)

Do you think there is a good way to add this info about dynamically sized types there as well?

[joshlf]

We should probably add that here; that seems like a good place to say that there's a maximum size. (There's a maximum alignment, too, but I don't how how stably we guarantee that. It is currently 2^29.)

That sounds reasonable.

Do you think there is a good way to add this info about dynamically sized types there as well?

I'm not sure how to specify it for dyn Trait types, and I'm not sure it's all that important. #69835 (comment) is only concerned with Slice DSTs.

For Slice DSTs, I think that it would be sufficient to say that the instance with 0 trailing slice elements has a size which fits in isize. One prerequisite would be defining the term "slice DST". The Reference page on Dynamically Sized Types alludes to slice DSTs, but does not explicitly define them. It might also be worth expanding that page in the same PR to define slice DSTs and then add a paragraph to the Size and Alignment section which references that page and provides this "fits in isize" guarantee.

[RalfJung]

I don't view this as a property of "slice DST". I think this can be defined fully compositionally.

• Every type, including unsized types, has a minimal size and a minimal alignment
• For sized types, the minimal size and alignment match their regular size and alignment
• For slices, the minimal size is 0 and the minimal alignment is the alignment of the element type
• For dyn Trait, the minimal size is 0 and the minimal alignment is 1
• For struct types with an unsized field, the minimal size and alignment is computed using the minimal size and alignment of that field
• The minimal size of all types fits in isize

[joshlf]

I don't view this as a property of "slice DST". I think this can be defined fully compositionally.

• Every type, including unsized types, has a minimal size and a minimal alignment
• For sized types, the minimal size and alignment match their regular size and alignment
• For slices, the minimal size is 0 and the minimal alignment is the alignment of the element type
• For dyn Trait, the minimal size is 0 and the minimal alignment is 1
• For struct types with an unsized field, the minimal size and alignment is computed using the minimal size and alignment of that field
• The minimal size of all types fits in isize

Sounds perfect. I've put up a PR (happy to take any suggested edits): rust-lang/reference#1482

[RalfJung]

Marking as blocked on rust-lang/reference#1482.

Or does that PR entirely replace this one?

[joshlf]

Marking as blocked on rust-lang/reference#1482.

Or does that PR entirely replace this one?

For my purposes, I only need rust-lang/reference#1482; this is redundant. That said, I'd be happy to still land it if folks think it's useful to have this in a location that's more discoverable to users.

[RalfJung]

Okay, thanks! Given that we're still struggling with the wording for the reference, and that this PR is describing a special case what should IMO be explained as a general principle, let's close this then and get this documented somewhere before worrying about spreading docs in more places for discoverability.