From aa5c8e137e378f3e7e85093a7e1f6e58487ec35d Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Sat, 1 Aug 2020 10:46:20 +0200 Subject: [PATCH 01/22] Start procedural vtables RFC --- text/0000-procedural-vtables.md | 95 +++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 text/0000-procedural-vtables.md diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md new file mode 100644 index 00000000000..8e48d0048d2 --- /dev/null +++ b/text/0000-procedural-vtables.md @@ -0,0 +1,95 @@ +- Feature Name: (fill me in with a unique ident, `my_awesome_feature`) +- Start Date: (fill me in with today's date, YYYY-MM-DD) +- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) + +# Summary +[summary]: #summary + +One paragraph explanation of the feature. + +# Motivation +[motivation]: #motivation + +Why are we doing this? What use cases does it support? What is the expected outcome? + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +Explain the proposal as if it was already included in the language and you were teaching it to another Rust programmer. That generally means: + +- Introducing new named concepts. +- Explaining the feature largely in terms of examples. +- Explaining how Rust programmers should *think* about the feature, and how it should impact the way they use Rust. It should explain the impact as concretely as possible. +- If applicable, provide sample error messages, deprecation warnings, or migration guidance. +- If applicable, describe the differences between teaching this to existing Rust programmers and new Rust programmers. + +For implementation-oriented RFCs (e.g. for compiler internals), this section should focus on how compiler contributors should think about the change, and give examples of its concrete impact. For policy RFCs, this section should provide an example-driven introduction to the policy, and explain its impact in concrete terms. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +This is the technical portion of the RFC. Explain the design in sufficient detail that: + +- Its interaction with other features is clear. +- It is reasonably clear how the feature would be implemented. +- Corner cases are dissected by example. + +The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +- Why is this design the best in the space of possible designs? +- What other designs have been considered and what is the rationale for not choosing them? +- What is the impact of not doing this? + +# Prior art +[prior-art]: #prior-art + +Discuss prior art, both the good and the bad, in relation to this proposal. +A few examples of what this can include are: + +- For language, library, cargo, tools, and compiler proposals: Does this feature exist in other programming languages and what experience have their community had? +- For community proposals: Is this done by some other community and what were their experiences with it? +- For other teams: What lessons can we learn from what other communities have done here? +- Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. + +This section is intended to encourage you as an author to think about the lessons from other languages, provide readers of your RFC with a fuller picture. +If there is no prior art, that is fine - your ideas are interesting to us whether they are brand new or if it is an adaptation from other languages. + +Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. +Please also take into consideration that rust sometimes intentionally diverges from common language features. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to resolve through the RFC process before this gets merged? +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Future possibilities +[future-possibilities]: #future-possibilities + +Think about what the natural extension and evolution of your proposal would +be and how it would affect the language and project as a whole in a holistic +way. Try to use this section as a tool to more fully consider all possible +interactions with the project and language in your proposal. +Also consider how the this all fits into the roadmap for the project +and of the relevant sub-team. + +This is also a good place to "dump ideas", if they are out of scope for the +RFC you are writing but otherwise related. + +If you have tried and cannot think of any future possibilities, +you may simply state that you cannot think of anything. + +Note that having something written down in the future-possibilities section +is not a reason to accept the current or a future RFC; such notes should be +in the section on motivation or rationale in this or subsequent RFCs. +The section merely provides additional information. From da9f3fc535da05c8202c9af36e4fbfdd0412d2c7 Mon Sep 17 00:00:00 2001 From: HackMD Date: Sat, 1 Aug 2020 15:20:54 +0000 Subject: [PATCH 02/22] First draft --- text/0000-procedural-vtables.md | 235 ++++++++++++++++++++++++-------- 1 file changed, 181 insertions(+), 54 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 8e48d0048d2..0153eb66b38 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -1,95 +1,222 @@ -- Feature Name: (fill me in with a unique ident, `my_awesome_feature`) -- Start Date: (fill me in with today's date, YYYY-MM-DD) +- Feature Name: `procedural-vtables` +- Start Date: 2020-08-01 - RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) - Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) # Summary [summary]: #summary -One paragraph explanation of the feature. +All vtable generation happens outside the compiler by invoking a `const fn` that generates said vtable from a generic description of a trait impl. By default, if no vtable generator function is specified for a specific trait, `std::vtable::default` is invoked. # Motivation [motivation]: #motivation -Why are we doing this? What use cases does it support? What is the expected outcome? +The only way we're going to satisfy all users' use cases is by allowing users complete freedom in how their wide pointers' metadata is built. Instead of hardcoding certain vtable layouts in the language (https://github.com/rust-lang/rfcs/pull/2955) we can give users the capability to invent their own layouts at their leisure. This should also help with the work on custom DSTs, as this scheme doesn't specify the size of the wide pointer metadata field. # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -Explain the proposal as if it was already included in the language and you were teaching it to another Rust programmer. That generally means: - -- Introducing new named concepts. -- Explaining the feature largely in terms of examples. -- Explaining how Rust programmers should *think* about the feature, and how it should impact the way they use Rust. It should explain the impact as concretely as possible. -- If applicable, provide sample error messages, deprecation warnings, or migration guidance. -- If applicable, describe the differences between teaching this to existing Rust programmers and new Rust programmers. - -For implementation-oriented RFCs (e.g. for compiler internals), this section should focus on how compiler contributors should think about the change, and give examples of its concrete impact. For policy RFCs, this section should provide an example-driven introduction to the policy, and explain its impact in concrete terms. +In order to mark a trait as using a custom vtable layout, you apply the +`#[unsafe_custom_vtable = "foo"]` attribute to the trait declaration. +This is unsafe, because the `foo` function supplies functionality for accessing +`foo` denotes a `const fn` with the signature +`const fn() -> std::vtable::DstInfo`. + +All `impl`s of this trait will now use `foo` for generating the wide pointer +metadata (which contains the vtable). The `WidePointerMetadata` struct that +describes the metadata is a `#[nonexhaustive]` struct. You create instances of it by invoking its `new` method, which gives you a `WidePointerMetadata` that essentially is a `()`. This means you do not have any metadata, similar to `extern type`s. Since the `wide_ptr_metadata` field of the `DstInfo` struct is public, you can now modify it to whatever layout you desire. +You are actually generating the pointer metadata, not a description of it. Since your `const fn` is being interpreted in the target's environment, all target specific information will match up. +Now, you need some information about the `impl` in order to generate your metadata (and your vtable). You get this information partially from the type directly (the `T` parameter), and all the `impl` block specific information is encoded in `ImplDescription`, which you get as a const generic parameter. + +As an example, consider the `std::vtable::default` function which is what normally generates your metadata: + +```rust +/// DISCLAIMER: this uses a `Vtable` struct which is just a part of the +/// default trait objects. Your own trait objects can use any metadata and +/// thus "vtable" layout that they want. +pub const fn default< + const IMPL: &'static std::vtable::ImplDescription, + T, +>() -> std::vtable::DstInfo { + let mut info = DstInfo::new(); + // We generate the metadata and put a pointer to the metadata into + // the field. This looks like it's passing a reference to a temporary + // value, but this uses promotion + // (https://doc.rust-lang.org/stable/reference/destructors.html?highlight=promotion#constant-promotion), + // so the value lives long enough. + info.unsize(|ptr| { + ( + ptr, + &default_meta::() as *const _ as *const (), + ) + }); + // We supply a function for invoking trait methods + info.method_id_to_fn_ptr(|idx, parents, meta| unsafe { + + }); + info.size_of(|meta| unsafe { + let meta = *(meta as *&'static Vtable<0>); + meta.size + }); + info.align_of(|meta| unsafe { + let meta = *(meta as *&'static Vtable<0>); + meta.align + }); + info.drop(|meta| unsafe { + let meta = *(meta as *&'static Vtable<0>); + meta.drop + }); + info +} + +// Compute the total number of methods, including super-traits +const fn num_methods< + const IMPL: &'static std::vtable::ImplDescription, +>() -> usize { + let mut n = IMPL.methods.len(); + let mut current = IMPL; + while let Some(next) = current.parent { + n += next.methods.len(); + current = next.parent; + } + n +} + +const fn default_meta< + const IMPL: &'static std::vtable::ImplDescription, + T, +>() -> &'static VTable<{num_methods::()}> { + // The metadata of a wide pointer for trait objects is a reference + // to the vtable. + &default_vtable::() +} + +const fn default_vtable< + const IMPL: &'static std::vtable::ImplDescription, + T, +>() -> VTable<{num_methods::()}> { + // `VTable` is a `#[repr(C)]` type with fields at the appropriate + // places. + let mut vtable = VTable { + size: std::mem::size_of::(), + align: std::mem::size_of::(), + drop: transmute::(std::ptr::drop_in_place::), + methods: [std::ptr::null(); num_methods::()], + }; + let mut i = 0; + while i < num_methods::() { + if let Some(method) = IMPL.methods[i] { + // The `method` variable is a function pointer, but + // cast to `*const ()`. + vtable.methods[i] = method; + } + } + vtable +} +``` # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -This is the technical portion of the RFC. Explain the design in sufficient detail that: - -- Its interaction with other features is clear. -- It is reasonably clear how the feature would be implemented. -- Corner cases are dissected by example. - -The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. +The two types `ImplDescription` and `DstInfo` are `#[nonexhaustive]` in order to allow arbitrary extension in the future. +Instances of the `ImplDescription` struct are created by rustc, basically replacing today's vtable generation, and then outsourcing the actual vtable generation to the const evaluator. The wide pointer metadata can be copied verbatim from the `DstInfo`'s `meta` field. When obtaining function pointers from vtables, instead of computing an offset, the `method_id_to_fn_ptr` function is invoked at runtime and computes a function pointer after being given a method index, the indices of all parents and a pointer to a metadata field. Through the use of MIR optimizations (e.g. inlining), the final LLVM assembly is tuned to be exactly the same as today. + +These types' declarations are provided below: + +```rust +#[nonexhaustive] +struct ImplDescription { + pub methods: &'static [*const ()], + pub parent: &'static ImplDescription, +} +#[nonexhaustive] +struct DstInfo { + unsize: *const (), + method_id_to_fn_ptr: fn(usize, &'static [usize], *const ()) -> *const (), + size_of: fn(*const()) -> usize, + align_of: fn(*const()) -> usize, + drop: fn drop(*mut ()), +} +impl DstInfo { + fn new() -> Self { + Self { + meta: &(), + method_id_to_fn_ptr: |idx, parents, meta| { + panic!("method called on trait object with custom vtable without method_id_to_fn_ptr") + }, + } + } + unsafe fn unsize(f: fn(*const T)) -> WIDE_PTR) { + self.drop = transmute(f); + } + /// The given function returns a function pointer to the method that + /// is being requested. + /// * The first argument is the method index, + /// * the second argument is a list of indices used to traverse the + /// super-trait tree to find the trait whose method is being invoked, and + /// * the thrid argument is a pointer to the metadata (so in case of trait objects, usually it would be `&'static &'static Vtable`). + /// This indirection is necessary, because we don't know the size of the metadata. + unsafe fn method_id_to_fn_ptr(f: fn(usize, &'static [usize], *const ()) -> *const ()) { + self.method_id_to_fn_ptr = f; + } + unsafe fn size_of(f: fn(*const ()) -> usize) { + self.size_of = f; + } + unsafe fn align_of(f: fn(*const ()) -> usize) { + self.align_of = f; + } + unsafe fn drop(f: fn(*const ()) -> fn(*mut ())) { + self.drop = f; + } +} +``` # Drawbacks [drawbacks]: #drawbacks -Why should we *not* do this? +* This may be a serious case of overengineering. We're basically taking vtables out of the language and making dynamic dispatch on trait objects a user definable thing. +* This may slow down compilation, likely entirely preventable by keeping a special case in the compiler for regular trait objects. # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives -- Why is this design the best in the space of possible designs? -- What other designs have been considered and what is the rationale for not choosing them? -- What is the impact of not doing this? +## Don't use const generics + +If we wait until we have a heap in const eval, we can use `Vec`s and `Box`es, which would allow us to avoid the const generics scheme, likely making all the code less roundabout. + +We can still expose the current scheme and once we get heap in const eval, we can actually implement a convenience layer in user code. So this is basically like procedural macros, where a stringy API is exposed and user code (`syn`) gives a better API. # Prior art [prior-art]: #prior-art -Discuss prior art, both the good and the bad, in relation to this proposal. -A few examples of what this can include are: +I don't know of any prior art where a compile-time language has procedural vtable generation. You can see lots of similar tricks being employed in dynamic languages like ruby and python, where "types" are built by changing functions in objects at runtime. If this is just done at startup and not during actual program execution, it is essentially the same concept here, except that our "startup phase" is at compile-time. + +## Other Custom DST RFCs -- For language, library, cargo, tools, and compiler proposals: Does this feature exist in other programming languages and what experience have their community had? -- For community proposals: Is this done by some other community and what were their experiences with it? -- For other teams: What lessons can we learn from what other communities have done here? -- Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. +This list is shamelessly taken from [strega-nil's Custom DST RFC](https://github.com/rust-lang/rfcs/pull/2594): -This section is intended to encourage you as an author to think about the lessons from other languages, provide readers of your RFC with a fuller picture. -If there is no prior art, that is fine - your ideas are interesting to us whether they are brand new or if it is an adaptation from other languages. +- [mzabaluev's Version](https://github.com/rust-lang/rfcs/pull/709) +- [strega-nil's new version](https://github.com/rust-lang/rfcs/pull/2594) +- [strega-nil's Old Version](https://github.com/rust-lang/rfcs/pull/1524) +- [japaric's Pre-RFC](https://github.com/japaric/rfcs/blob/unsized2/text/0000-unsized-types.md) +- [mikeyhew's Pre-RFC](https://internals.rust-lang.org/t/pre-erfc-lets-fix-dsts/6663) +- [MicahChalmer's RFC](https://github.com/rust-lang/rfcs/pull/9) +- [nrc's Virtual Structs](https://github.com/rust-lang/rfcs/pull/5) +- [Pointer Metadata and VTable](https://github.com/rust-lang/rfcs/pull/2580) +- [Syntax of ?Sized](https://github.com/rust-lang/rfcs/pull/490) -Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. -Please also take into consideration that rust sometimes intentionally diverges from common language features. +This RFC differentiates itself from all the other RFCs in that it provides a procedural way to generate vtables, thus also permitting arbitrary user-defined compile-time conditions by aborting via `panic!`. # Unresolved questions [unresolved-questions]: #unresolved-questions -- What parts of the design do you expect to resolve through the RFC process before this gets merged? -- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? -- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? +- Do we want this kind of flexibility? With power comes responsibility... +- I believe we can do multiple super traits, including downcasts with this scheme, make sure that's true. # Future possibilities [future-possibilities]: #future-possibilities -Think about what the natural extension and evolution of your proposal would -be and how it would affect the language and project as a whole in a holistic -way. Try to use this section as a tool to more fully consider all possible -interactions with the project and language in your proposal. -Also consider how the this all fits into the roadmap for the project -and of the relevant sub-team. - -This is also a good place to "dump ideas", if they are out of scope for the -RFC you are writing but otherwise related. - -If you have tried and cannot think of any future possibilities, -you may simply state that you cannot think of anything. - -Note that having something written down in the future-possibilities section -is not a reason to accept the current or a future RFC; such notes should be -in the section on motivation or rationale in this or subsequent RFCs. -The section merely provides additional information. +* We can change slice (`[T]`) types to be backed by a `trait Slice` which uses this scheme to generate just a single `usize` for the metadata +* We can use this scheme and remove `extern type`s from the language, as they just become a trait with a custom metadata generator that uses `()` for the metadata. So `CStr` doesn't become an extern type, instead it becomes a trait. +* We can handle traits with parents which are created by a different metadata generator function, we just need to figure out how to communicate this to such a function so it can special case this situation. +* We can give the vtable pointer of the super traits to the constructor function, so it doesn't have to recompute anything and just grab the info off there. Basically `ImplDescription::parent` would not be a pointer to the parent, but a struct which contains at least the pointer to the parent and the pointer to the `DstInfo` \ No newline at end of file From 446d1ad3e384938fd5b432dea308df033eefdbae Mon Sep 17 00:00:00 2001 From: HackMD Date: Sat, 1 Aug 2020 15:24:42 +0000 Subject: [PATCH 03/22] Add PR id --- text/0000-procedural-vtables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 0153eb66b38..67a7f961f15 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -1,6 +1,6 @@ - Feature Name: `procedural-vtables` - Start Date: 2020-08-01 -- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- RFC PR: [rust-lang/rfcs#2967](https://github.com/rust-lang/rfcs/pull/2967) - Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) # Summary From 5d1b767db73a96f2040b9bbb44714314934a139c Mon Sep 17 00:00:00 2001 From: HackMD Date: Sat, 1 Aug 2020 15:44:15 +0000 Subject: [PATCH 04/22] Add future possibilty for C++-ish vtables --- text/0000-procedural-vtables.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 67a7f961f15..2930066bede 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -216,7 +216,9 @@ This RFC differentiates itself from all the other RFCs in that it provides a pro # Future possibilities [future-possibilities]: #future-possibilities +* This scheme can be used to generate C++-like vtables where the data and vtable are in the same allocation by making the `unsize` function create a heap allocation and write the value and the vtable into this new allocation. It's not clear yet how to handle this kind of "ownership takeover", since all unsizing in Rust currently happens either in a borrowed manner or in `Box`, which is special anyway. * We can change slice (`[T]`) types to be backed by a `trait Slice` which uses this scheme to generate just a single `usize` for the metadata * We can use this scheme and remove `extern type`s from the language, as they just become a trait with a custom metadata generator that uses `()` for the metadata. So `CStr` doesn't become an extern type, instead it becomes a trait. * We can handle traits with parents which are created by a different metadata generator function, we just need to figure out how to communicate this to such a function so it can special case this situation. -* We can give the vtable pointer of the super traits to the constructor function, so it doesn't have to recompute anything and just grab the info off there. Basically `ImplDescription::parent` would not be a pointer to the parent, but a struct which contains at least the pointer to the parent and the pointer to the `DstInfo` \ No newline at end of file +* We can give the vtable pointer of the super traits to the constructor function, so it doesn't have to recompute anything and just grab the info off there. Basically `ImplDescription::parent` would not be a pointer to the parent, but a struct which contains at least the pointer to the parent and the pointer to the `DstInfo` +* Totally off-topic, but a similar scheme can be used to generate type declarations procedurally with const eval. \ No newline at end of file From 49384540052226696c3c06563f713a545632ed60 Mon Sep 17 00:00:00 2001 From: HackMD Date: Sat, 1 Aug 2020 17:05:19 +0000 Subject: [PATCH 05/22] Add lots of full examples (and fix up the existing one, oops) --- text/0000-procedural-vtables.md | 199 +++++++++++++++++++++++++++----- 1 file changed, 170 insertions(+), 29 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 2930066bede..979ed0aafab 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -44,27 +44,43 @@ pub const fn default< // value, but this uses promotion // (https://doc.rust-lang.org/stable/reference/destructors.html?highlight=promotion#constant-promotion), // so the value lives long enough. - info.unsize(|ptr| { + info.unsize(|ptr, owned| { ( ptr, &default_meta::() as *const _ as *const (), ) }); - // We supply a function for invoking trait methods - info.method_id_to_fn_ptr(|idx, parents, meta| unsafe { - + // We supply a function for invoking trait methods. + // This is always inlined and will thus get optimized to a single + // deref and offset (strong handwaving happening here). + info.method_id_to_fn_ptr(|mut idx, parents, meta| unsafe { + let meta = *(meta as *const (*const(), &'static Vtable)); + let mut table = IMPL; + for parent in parents { + // we don't support multi-parents yet + assert_eq!(parent, 0); + idx += table.methods.len(); + // Never panics, there are always fewer or equal number of + // parents given as the argument as there are in reality. + table = table.parent.unwrap(); + } + meta.1.methods[idx]; }); info.size_of(|meta| unsafe { - let meta = *(meta as *&'static Vtable<0>); - meta.size + let meta = *(meta as *const (*const(), &'static Vtable)); + meta.1.size }); info.align_of(|meta| unsafe { - let meta = *(meta as *&'static Vtable<0>); - meta.align + let meta = *(meta as *const (*const(), &'static Vtable)); + meta.1.align }); info.drop(|meta| unsafe { - let meta = *(meta as *&'static Vtable<0>); - meta.drop + let meta = *(meta as *const (*const(), &'static Vtable)); + meta.1.drop + }); + info.self_ptr(|meta| unsafe { + let meta = *(meta as *const (*const(), &'static Vtable)); + meta.0 }); info } @@ -104,17 +120,116 @@ const fn default_vtable< methods: [std::ptr::null(); num_methods::()], }; let mut i = 0; - while i < num_methods::() { - if let Some(method) = IMPL.methods[i] { - // The `method` variable is a function pointer, but - // cast to `*const ()`. - vtable.methods[i] = method; + let mut current = IMPL; + loop { + match IMPL.methods.get(i) { + Some(Some(method)) => { + // The `method` variable is a function pointer, but + // cast to `*const ()`. + vtable.methods[i] = method; + i += 1; + }, + Some(None) => { + // Method that cannot be called on this vtable + i += 1; + } + None => match current.parent { + Some(next) => { + current = next; + i = 0; + }, + None => break, + } } } vtable } ``` +Now, if you want to implement a fancier vtable, this RFC enables that. + +## Null terminated strings (std::ffi::CStr) + +This is how I see all extern types being handled + +```rust +pub const fn default< + const IMPL: &'static std::vtable::ImplDescription, + T, +>() -> std::vtable::DstInfo { + let mut info = DstInfo::new(); + info.unsize(|ptr, owned| ptr); + info.method_id_to_fn_ptr(|idx, parents, meta| unsafe { + panic!("CStr has no trait methods, it's all inherent methods acting on the pointer") + }); + info.size_of(|meta| unsafe { + let ptr = *(meta as *const *const u8); + strlen(ptr) + }); + info.align_of(|meta| 1); + // Nothing to drop (just `u8`s) and we are not in charge of dealloc + info.drop(|meta| None); + info.self_ptr(|ptr| { + let ptr = *(meta as *const *const u8); + ptr + }); + info +} +``` + +## C++ like vtables + +Most of the boilerplate is the same as with regular vtables. + +```rust +pub const fn default< + const IMPL: &'static std::vtable::ImplDescription, + T, +>() -> std::vtable::DstInfo { + let mut info = DstInfo::new(); + info.unsize(|ptr, owned| { + assert!(owned); + let size = std::mem::size_of::(); + let vtable_size = std::mem::size_of::>(); + let layout = Layout::from_size_align(size + vtable_size, std::mem::align_of::()).unwrap(); + let new_ptr = std::alloc::alloc(layout); + // Move the value to the new allocation + std::ptr::copy_nonoverlapping(ptr, new_ptr.offset(vtable_size)); + std::alloc::dealloc(ptr); + + // Copy the vtable into the shared allocation. + // Note that we are reusing the same vtable generation as in + // the regular Rust case. + std::ptr::write(new_ptr, default_meta::()); + new_ptr + }); + info.method_id_to_fn_ptr(|idx, parents, meta| unsafe { + let meta = *(meta as *const &'static Vtable); + // The rest of the function body is the same as with regular + // vtables. + }); + info.size_of(|meta| unsafe { + let meta = *(meta as *const &'static Vtable); + meta.size + }); + info.align_of(|meta| unsafe { + let meta = *(meta as *const &'static Vtable); + meta.align + }); + info.drop(|meta| unsafe { + let meta = *(meta as *const &'static Vtable); + meta.drop + }); + info.self_ptr(|meta| unsafe { + let meta = *(meta as *const *const Vtable); + let vtable_size = std::mem::size_of::>(); + meta.offset(vtable_size) + }); + info +} +``` + + # Reference-level explanation [reference-level-explanation]: #reference-level-explanation @@ -135,18 +250,31 @@ struct DstInfo { method_id_to_fn_ptr: fn(usize, &'static [usize], *const ()) -> *const (), size_of: fn(*const()) -> usize, align_of: fn(*const()) -> usize, - drop: fn drop(*mut ()), + drop: *const (), + self_ptr: fn(*const()) -> *const (), } + impl DstInfo { - fn new() -> Self { + const fn new() -> Self { Self { - meta: &(), - method_id_to_fn_ptr: |idx, parents, meta| { - panic!("method called on trait object with custom vtable without method_id_to_fn_ptr") - }, + unsize: std::ptr::null(), + method_id_to_fn_ptr: None, + size_of: None, + align_of: None, + drop: None, + self_ptr: None, } } - unsafe fn unsize(f: fn(*const T)) -> WIDE_PTR) { + /// If the `bool` flag is `true`, this is an owned conversion like + /// in `Box as Box`. This distinction is important, as + /// unsizing that creates a vtable in the same allocation as the object + /// (like C++ does), cannot work on non-owned conversions. You can't just + /// move away the owned object. So you need to just move a pointer into + /// the shared vtable+object allocation instead of the entire object. + unsafe fn unsize(f: fn(*const T, bool)) -> WIDE_PTR) { + // Since we don't know the return type size, the `drop` field + // cannot be named. Thus we just use an opaque `*const ()` for the + // function pointer. self.drop = transmute(f); } /// The given function returns a function pointer to the method that @@ -157,16 +285,27 @@ impl DstInfo { /// * the thrid argument is a pointer to the metadata (so in case of trait objects, usually it would be `&'static &'static Vtable`). /// This indirection is necessary, because we don't know the size of the metadata. unsafe fn method_id_to_fn_ptr(f: fn(usize, &'static [usize], *const ()) -> *const ()) { - self.method_id_to_fn_ptr = f; + self.method_id_to_fn_ptr = Some(f); } + /// Set the function that extracts the dynamic size unsafe fn size_of(f: fn(*const ()) -> usize) { - self.size_of = f; + self.size_of = Some(f); } + /// Set the function that extracts the dynamic alignment unsafe fn align_of(f: fn(*const ()) -> usize) { - self.align_of = f; + self.align_of = Some(f); + } + /// Set the function that extracts the drop code. + unsafe fn drop(f: fn(*const ()) -> Option) { + self.drop = Some(f); } - unsafe fn drop(f: fn(*const ()) -> fn(*mut ())) { - self.drop = f; + /// Set the function that extracts the `&self` pointer + /// from the wide pointer + /// for calling trait methods. This needs a method as + /// wide pointer layouts may place their `self` pointer + /// anywhere they desire. + unsafe fn self_ptr(f: fn(*const ()) -> *const ()) { + self.self_ptr = Some(f); } } ``` @@ -216,9 +355,11 @@ This RFC differentiates itself from all the other RFCs in that it provides a pro # Future possibilities [future-possibilities]: #future-possibilities -* This scheme can be used to generate C++-like vtables where the data and vtable are in the same allocation by making the `unsize` function create a heap allocation and write the value and the vtable into this new allocation. It's not clear yet how to handle this kind of "ownership takeover", since all unsizing in Rust currently happens either in a borrowed manner or in `Box`, which is special anyway. +* This scheme can be used to generate C++-like vtables where the data and vtable are in the same allocation by making the `unsize` function create a heap allocation and write the value and the vtable into this new allocation. * We can change slice (`[T]`) types to be backed by a `trait Slice` which uses this scheme to generate just a single `usize` for the metadata * We can use this scheme and remove `extern type`s from the language, as they just become a trait with a custom metadata generator that uses `()` for the metadata. So `CStr` doesn't become an extern type, instead it becomes a trait. * We can handle traits with parents which are created by a different metadata generator function, we just need to figure out how to communicate this to such a function so it can special case this situation. * We can give the vtable pointer of the super traits to the constructor function, so it doesn't have to recompute anything and just grab the info off there. Basically `ImplDescription::parent` would not be a pointer to the parent, but a struct which contains at least the pointer to the parent and the pointer to the `DstInfo` -* Totally off-topic, but a similar scheme can be used to generate type declarations procedurally with const eval. \ No newline at end of file +* Totally off-topic, but a similar scheme (via const eval) can be used to procedurally generate type declarations. +* We can likely access associated consts and types of the trait directly without causing cycle errors, this should be investigated +* This scheme is forward compatible to adding associated fields later. \ No newline at end of file From 0dbcf30c987f40af47950c09aa52a207db5440cd Mon Sep 17 00:00:00 2001 From: HackMD Date: Sat, 1 Aug 2020 17:28:08 +0000 Subject: [PATCH 06/22] Correctly use `unsafe` in the examples --- text/0000-procedural-vtables.md | 198 ++++++++++++++++---------------- 1 file changed, 102 insertions(+), 96 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 979ed0aafab..0ea32aa159e 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -39,49 +39,51 @@ pub const fn default< T, >() -> std::vtable::DstInfo { let mut info = DstInfo::new(); - // We generate the metadata and put a pointer to the metadata into - // the field. This looks like it's passing a reference to a temporary - // value, but this uses promotion - // (https://doc.rust-lang.org/stable/reference/destructors.html?highlight=promotion#constant-promotion), - // so the value lives long enough. - info.unsize(|ptr, owned| { - ( - ptr, - &default_meta::() as *const _ as *const (), - ) - }); - // We supply a function for invoking trait methods. - // This is always inlined and will thus get optimized to a single - // deref and offset (strong handwaving happening here). - info.method_id_to_fn_ptr(|mut idx, parents, meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable)); - let mut table = IMPL; - for parent in parents { - // we don't support multi-parents yet - assert_eq!(parent, 0); - idx += table.methods.len(); - // Never panics, there are always fewer or equal number of - // parents given as the argument as there are in reality. - table = table.parent.unwrap(); - } - meta.1.methods[idx]; - }); - info.size_of(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable)); - meta.1.size - }); - info.align_of(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable)); - meta.1.align - }); - info.drop(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable)); - meta.1.drop - }); - info.self_ptr(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable)); - meta.0 - }); + unsafe { + // We generate the metadata and put a pointer to the metadata into + // the field. This looks like it's passing a reference to a temporary + // value, but this uses promotion + // (https://doc.rust-lang.org/stable/reference/destructors.html?highlight=promotion#constant-promotion), + // so the value lives long enough. + info.unsize(|ptr, owned| { + ( + ptr, + &default_meta::() as *const _ as *const (), + ) + }); + // We supply a function for invoking trait methods. + // This is always inlined and will thus get optimized to a single + // deref and offset (strong handwaving happening here). + info.method_id_to_fn_ptr(|mut idx, parents, meta| unsafe { + let meta = *(meta as *const (*const(), &'static Vtable)); + let mut table = IMPL; + for parent in parents { + // we don't support multi-parents yet + assert_eq!(parent, 0); + idx += table.methods.len(); + // Never panics, there are always fewer or equal number of + // parents given as the argument as there are in reality. + table = table.parent.unwrap(); + } + meta.1.methods[idx]; + }); + info.size_of(|meta| unsafe { + let meta = *(meta as *const (*const(), &'static Vtable)); + meta.1.size + }); + info.align_of(|meta| unsafe { + let meta = *(meta as *const (*const(), &'static Vtable)); + meta.1.align + }); + info.drop(|meta| unsafe { + let meta = *(meta as *const (*const(), &'static Vtable)); + meta.1.drop + }); + info.self_ptr(|meta| unsafe { + let meta = *(meta as *const (*const(), &'static Vtable)); + meta.0 + }); + } info } @@ -158,21 +160,23 @@ pub const fn default< T, >() -> std::vtable::DstInfo { let mut info = DstInfo::new(); - info.unsize(|ptr, owned| ptr); - info.method_id_to_fn_ptr(|idx, parents, meta| unsafe { - panic!("CStr has no trait methods, it's all inherent methods acting on the pointer") - }); - info.size_of(|meta| unsafe { - let ptr = *(meta as *const *const u8); - strlen(ptr) - }); - info.align_of(|meta| 1); - // Nothing to drop (just `u8`s) and we are not in charge of dealloc - info.drop(|meta| None); - info.self_ptr(|ptr| { - let ptr = *(meta as *const *const u8); - ptr - }); + unsafe { + info.unsize(|ptr, owned| ptr); + info.method_id_to_fn_ptr(|idx, parents, meta| { + panic!("CStr has no trait methods, it's all inherent methods acting on the pointer") + }); + info.size_of(|meta| unsafe { + let ptr = *(meta as *const *const u8); + strlen(ptr) + }); + info.align_of(|meta| 1); + // Nothing to drop (just `u8`s) and we are not in charge of dealloc + info.drop(|meta| None); + info.self_ptr(|ptr| { + let ptr = *(meta as *const *const u8); + ptr + }); + } info } ``` @@ -187,44 +191,46 @@ pub const fn default< T, >() -> std::vtable::DstInfo { let mut info = DstInfo::new(); - info.unsize(|ptr, owned| { - assert!(owned); - let size = std::mem::size_of::(); - let vtable_size = std::mem::size_of::>(); - let layout = Layout::from_size_align(size + vtable_size, std::mem::align_of::()).unwrap(); - let new_ptr = std::alloc::alloc(layout); - // Move the value to the new allocation - std::ptr::copy_nonoverlapping(ptr, new_ptr.offset(vtable_size)); - std::alloc::dealloc(ptr); - - // Copy the vtable into the shared allocation. - // Note that we are reusing the same vtable generation as in - // the regular Rust case. - std::ptr::write(new_ptr, default_meta::()); - new_ptr - }); - info.method_id_to_fn_ptr(|idx, parents, meta| unsafe { - let meta = *(meta as *const &'static Vtable); - // The rest of the function body is the same as with regular - // vtables. - }); - info.size_of(|meta| unsafe { - let meta = *(meta as *const &'static Vtable); - meta.size - }); - info.align_of(|meta| unsafe { - let meta = *(meta as *const &'static Vtable); - meta.align - }); - info.drop(|meta| unsafe { - let meta = *(meta as *const &'static Vtable); - meta.drop - }); - info.self_ptr(|meta| unsafe { - let meta = *(meta as *const *const Vtable); - let vtable_size = std::mem::size_of::>(); - meta.offset(vtable_size) - }); + unsafe { + info.unsize(|ptr, owned| unsafe { + assert!(owned); + let size = std::mem::size_of::(); + let vtable_size = std::mem::size_of::>(); + let layout = Layout::from_size_align(size + vtable_size, std::mem::align_of::()).unwrap(); + let new_ptr = std::alloc::alloc(layout); + // Move the value to the new allocation + std::ptr::copy_nonoverlapping(ptr, new_ptr.offset(vtable_size)); + std::alloc::dealloc(ptr); + + // Copy the vtable into the shared allocation. + // Note that we are reusing the same vtable generation as in + // the regular Rust case. + std::ptr::write(new_ptr, default_meta::()); + new_ptr + }); + info.method_id_to_fn_ptr(|idx, parents, meta| unsafe { + let meta = *(meta as *const &'static Vtable); + // The rest of the function body is the same as with regular + // vtables. + }); + info.size_of(|meta| unsafe { + let meta = *(meta as *const &'static Vtable); + meta.size + }); + info.align_of(|meta| unsafe { + let meta = *(meta as *const &'static Vtable); + meta.align + }); + info.drop(|meta| unsafe { + let meta = *(meta as *const &'static Vtable); + meta.drop + }); + info.self_ptr(|meta| unsafe { + let meta = *(meta as *const *const Vtable); + let vtable_size = std::mem::size_of::>(); + meta.offset(vtable_size) + }); + } info } ``` From 61570409a24e44e1de846fc631921341c8dbe534 Mon Sep 17 00:00:00 2001 From: HackMD Date: Sat, 1 Aug 2020 18:06:55 +0000 Subject: [PATCH 07/22] Discuss `dyn A + B` --- text/0000-procedural-vtables.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 0ea32aa159e..d80afe6a672 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -357,10 +357,13 @@ This RFC differentiates itself from all the other RFCs in that it provides a pro - Do we want this kind of flexibility? With power comes responsibility... - I believe we can do multiple super traits, including downcasts with this scheme, make sure that's true. +- This scheme could support downcasting `dyn A` to `dyn B` if `trait A: B` if we make `T: ?Sized` (`T` is the `impl` block type). But that will not allow the sized use-cases anymore. If we have something like `MaybeSized` that has `size_of` and `align_of` methods returning `Option`, then maybe we could do this. # Future possibilities [future-possibilities]: #future-possibilities +* Add a scheme that allows super traits to have different vtable generators and permit a vtable generator to process them. So `trait A: B + C`, where `B` and `C` have different vtable generators and `A` unites them in some manner. This requires the information about the vtable generators to be part of the `ImplDescription` type. We can likely even put a function pointer to the vtable generator into the `ImplDescription`. +* Add a scheme allowing `dyn A + B`. I have no idea how, but maybe we just need to add a method to `DstInfo` that allows chaining vtable generators. So we first generate `dyn A`, then out of that we generate `dyn A + B` * This scheme can be used to generate C++-like vtables where the data and vtable are in the same allocation by making the `unsize` function create a heap allocation and write the value and the vtable into this new allocation. * We can change slice (`[T]`) types to be backed by a `trait Slice` which uses this scheme to generate just a single `usize` for the metadata * We can use this scheme and remove `extern type`s from the language, as they just become a trait with a custom metadata generator that uses `()` for the metadata. So `CStr` doesn't become an extern type, instead it becomes a trait. From 4e73560d01c17c4f92e75a464a9fd5ca187b6c1f Mon Sep 17 00:00:00 2001 From: HackMD Date: Sun, 2 Aug 2020 08:50:57 +0000 Subject: [PATCH 08/22] Split unsizing and other ops into separate functions --- text/0000-procedural-vtables.md | 195 ++++++++++++++++++-------------- 1 file changed, 111 insertions(+), 84 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index d80afe6a672..5096716fe5f 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -18,39 +18,56 @@ The only way we're going to satisfy all users' use cases is by allowing users co In order to mark a trait as using a custom vtable layout, you apply the `#[unsafe_custom_vtable = "foo"]` attribute to the trait declaration. -This is unsafe, because the `foo` function supplies functionality for accessing +This is unsafe, because the `foo` function supplies functionality for accessing the `self` pointers. `foo` denotes a `const fn` with the signature -`const fn() -> std::vtable::DstInfo`. +`const fn() -> std::vtable::DstInfo`. + +Additionally, if your trait supports automatically unsizing from the types it's implemented for (unlike `CStr`, `str` and `[T]`, which require type-specific logic), you can supply your trait with an `unsizing` function by +specifying `#[unsafe_custom_unsize = "bar"]`. `bar` denotes a `const fn` +with the signature +`const fn() -> std::vtable::UnsizeInfo`, where `T` is your concrete type. All `impl`s of this trait will now use `foo` for generating the wide pointer metadata (which contains the vtable). The `WidePointerMetadata` struct that describes the metadata is a `#[nonexhaustive]` struct. You create instances of it by invoking its `new` method, which gives you a `WidePointerMetadata` that essentially is a `()`. This means you do not have any metadata, similar to `extern type`s. Since the `wide_ptr_metadata` field of the `DstInfo` struct is public, you can now modify it to whatever layout you desire. You are actually generating the pointer metadata, not a description of it. Since your `const fn` is being interpreted in the target's environment, all target specific information will match up. -Now, you need some information about the `impl` in order to generate your metadata (and your vtable). You get this information partially from the type directly (the `T` parameter), and all the `impl` block specific information is encoded in `ImplDescription`, which you get as a const generic parameter. +Now, you need some information about the `impl` in order to generate your metadata (and your vtable). You get this information partially from the type directly (the `T` parameter), and all the `impl` block specific information is encoded in `TraitDescription`, which you get as a const generic parameter. -As an example, consider the `std::vtable::default` function which is what normally generates your metadata: +As an example, consider the function which is what normally generates your metadata. ```rust +/// If the `owned` flag is `true`, this is an owned conversion like +/// in `Box as Box`. This distinction is important, as +/// unsizing that creates a vtable in the same allocation as the object +/// (like C++ does), cannot work on non-owned conversions. You can't just +/// move away the owned object. The flag allows you to forbid such +/// unsizings by triggering a compile-time `panic` with an explanation +/// for the user. +pub const fn custom_unsize< + T, + const IMPL: &'static std::vtable::TraitDescription, + const _owned: bool, +>(*const T) -> (*const T, &'static VTable<{num_methods::()}>) { + // We generate the metadata and put a pointer to the metadata into + // the field. This looks like it's passing a reference to a temporary + // value, but this uses promotion + // (https://doc.rust-lang.org/stable/reference/destructors.html?highlight=promotion#constant-promotion), + // so the value lives long enough. + ( + ptr, + &default_vtable::() as *const _ as *const (), + ) +} + /// DISCLAIMER: this uses a `Vtable` struct which is just a part of the /// default trait objects. Your own trait objects can use any metadata and /// thus "vtable" layout that they want. -pub const fn default< - const IMPL: &'static std::vtable::ImplDescription, +pub const fn custom_vtable< + const IMPL: &'static std::vtable::TraitDescription, T, >() -> std::vtable::DstInfo { let mut info = DstInfo::new(); unsafe { - // We generate the metadata and put a pointer to the metadata into - // the field. This looks like it's passing a reference to a temporary - // value, but this uses promotion - // (https://doc.rust-lang.org/stable/reference/destructors.html?highlight=promotion#constant-promotion), - // so the value lives long enough. - info.unsize(|ptr, owned| { - ( - ptr, - &default_meta::() as *const _ as *const (), - ) - }); // We supply a function for invoking trait methods. // This is always inlined and will thus get optimized to a single // deref and offset (strong handwaving happening here). @@ -89,7 +106,7 @@ pub const fn default< // Compute the total number of methods, including super-traits const fn num_methods< - const IMPL: &'static std::vtable::ImplDescription, + const IMPL: &'static std::vtable::TraitDescription, >() -> usize { let mut n = IMPL.methods.len(); let mut current = IMPL; @@ -100,17 +117,8 @@ const fn num_methods< n } -const fn default_meta< - const IMPL: &'static std::vtable::ImplDescription, - T, ->() -> &'static VTable<{num_methods::()}> { - // The metadata of a wide pointer for trait objects is a reference - // to the vtable. - &default_vtable::() -} - const fn default_vtable< - const IMPL: &'static std::vtable::ImplDescription, + const IMPL: &'static std::vtable::TraitDescription, T, >() -> VTable<{num_methods::()}> { // `VTable` is a `#[repr(C)]` type with fields at the appropriate @@ -152,16 +160,23 @@ Now, if you want to implement a fancier vtable, this RFC enables that. ## Null terminated strings (std::ffi::CStr) -This is how I see all extern types being handled +This is how I see all extern types being handled. +There can be no impls of `CStr` for any type, because the `unsize` +function is missing. The `CStr` ```rust -pub const fn default< - const IMPL: &'static std::vtable::ImplDescription, - T, + +// Not setting the `unsafe_custom_unsize` function, there's no sized +// equivalent like with normal traits. See the future +// extensions section for more details on unsizing. +#[unsafe_custom_vtable = "c_str"] +pub trait CStr {} + +pub const fn c_str< + const IMPL: &'static std::vtable::TraitDescription, >() -> std::vtable::DstInfo { let mut info = DstInfo::new(); unsafe { - info.unsize(|ptr, owned| ptr); info.method_id_to_fn_ptr(|idx, parents, meta| { panic!("CStr has no trait methods, it's all inherent methods acting on the pointer") }); @@ -186,49 +201,58 @@ pub const fn default< Most of the boilerplate is the same as with regular vtables. ```rust -pub const fn default< - const IMPL: &'static std::vtable::ImplDescription, + +pub const fn cpp_unsize< T, + const IMPL: &'static std::vtable::TraitDescription, + const owned: bool, +>( + ptr: *const T, +) -> *const (VTable<{num_methods::()}>, T) +where { + assert!(owned, "cannot unsize borrowed object for C++ like trait") +}, +{ + let size = std::mem::size_of::(); + let vtable_size = std::mem::size_of::>(); + let layout = Layout::from_size_align(size + vtable_size, std::mem::align_of::()).unwrap(); + let new_ptr = std::alloc::alloc(layout); + // Move the value to the new allocation + std::ptr::copy_nonoverlapping(ptr, new_ptr.offset(vtable_size), 1); + std::alloc::dealloc(ptr); + + // Copy the vtable into the shared allocation. + // Note that we are reusing the same vtable generation as in + // the regular Rust case. + std::ptr::write(new_ptr, default_vtable::()); + new_ptr +} + +pub const fn cpp< + const IMPL: &'static std::vtable::TraitDescription, >() -> std::vtable::DstInfo { let mut info = DstInfo::new(); unsafe { - info.unsize(|ptr, owned| unsafe { - assert!(owned); - let size = std::mem::size_of::(); - let vtable_size = std::mem::size_of::>(); - let layout = Layout::from_size_align(size + vtable_size, std::mem::align_of::()).unwrap(); - let new_ptr = std::alloc::alloc(layout); - // Move the value to the new allocation - std::ptr::copy_nonoverlapping(ptr, new_ptr.offset(vtable_size)); - std::alloc::dealloc(ptr); - - // Copy the vtable into the shared allocation. - // Note that we are reusing the same vtable generation as in - // the regular Rust case. - std::ptr::write(new_ptr, default_meta::()); - new_ptr - }); info.method_id_to_fn_ptr(|idx, parents, meta| unsafe { - let meta = *(meta as *const &'static Vtable); + let meta = *(meta as *const *const Vtable); // The rest of the function body is the same as with regular // vtables. }); info.size_of(|meta| unsafe { - let meta = *(meta as *const &'static Vtable); - meta.size + let meta = *(meta as *const *const Vtable); + (*meta).size }); info.align_of(|meta| unsafe { - let meta = *(meta as *const &'static Vtable); - meta.align + let meta = *(meta as *const *const Vtable); + (*meta).align }); info.drop(|meta| unsafe { - let meta = *(meta as *const &'static Vtable); - meta.drop + let meta = *(meta as *const *const Vtable); + (*meta).drop }); info.self_ptr(|meta| unsafe { - let meta = *(meta as *const *const Vtable); - let vtable_size = std::mem::size_of::>(); - meta.offset(vtable_size) + let ptr = *(meta as *const *const (Vtable, ())); + &raw const (*ptr).1; }); } info @@ -239,20 +263,31 @@ pub const fn default< # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -The two types `ImplDescription` and `DstInfo` are `#[nonexhaustive]` in order to allow arbitrary extension in the future. -Instances of the `ImplDescription` struct are created by rustc, basically replacing today's vtable generation, and then outsourcing the actual vtable generation to the const evaluator. The wide pointer metadata can be copied verbatim from the `DstInfo`'s `meta` field. When obtaining function pointers from vtables, instead of computing an offset, the `method_id_to_fn_ptr` function is invoked at runtime and computes a function pointer after being given a method index, the indices of all parents and a pointer to a metadata field. Through the use of MIR optimizations (e.g. inlining), the final LLVM assembly is tuned to be exactly the same as today. +The two types `TraitDescription` and `DstInfo` are `#[nonexhaustive]` in order to allow arbitrary extension in the future. +Instances of the `TraitDescription` struct are created by rustc, basically replacing today's vtable generation, and then outsourcing the actual vtable generation to the const evaluator. + +When unsizing, the `const fn` specified +via `unsafe_custom_unsize` is invoked. The only reason that function is +`const fn` is to restrict what kind of things you can do in there. We +can lift this restriction in the future. + +For all other operations, the `unsafe_custom_vtable` function is invoked. +This one must be `const fn`, as it is evaluated at compile-time and the +compiler then inspects the resulting `DstInfo` at compile-time. + +When obtaining function pointers from vtables, instead of computing an offset, the `method_id_to_fn_ptr` function is invoked at runtime and computes a function pointer after being given a method index, the indices of all parents and a pointer to a metadata field. Through the use of MIR optimizations (e.g. inlining), the final LLVM assembly is tuned to be exactly the same as today. These types' declarations are provided below: ```rust #[nonexhaustive] -struct ImplDescription { +struct TraitDescription { pub methods: &'static [*const ()], - pub parent: &'static ImplDescription, + pub parent: &'static TraitDescription, } + #[nonexhaustive] struct DstInfo { - unsize: *const (), method_id_to_fn_ptr: fn(usize, &'static [usize], *const ()) -> *const (), size_of: fn(*const()) -> usize, align_of: fn(*const()) -> usize, @@ -263,7 +298,6 @@ struct DstInfo { impl DstInfo { const fn new() -> Self { Self { - unsize: std::ptr::null(), method_id_to_fn_ptr: None, size_of: None, align_of: None, @@ -271,25 +305,13 @@ impl DstInfo { self_ptr: None, } } - /// If the `bool` flag is `true`, this is an owned conversion like - /// in `Box as Box`. This distinction is important, as - /// unsizing that creates a vtable in the same allocation as the object - /// (like C++ does), cannot work on non-owned conversions. You can't just - /// move away the owned object. So you need to just move a pointer into - /// the shared vtable+object allocation instead of the entire object. - unsafe fn unsize(f: fn(*const T, bool)) -> WIDE_PTR) { - // Since we don't know the return type size, the `drop` field - // cannot be named. Thus we just use an opaque `*const ()` for the - // function pointer. - self.drop = transmute(f); - } /// The given function returns a function pointer to the method that /// is being requested. /// * The first argument is the method index, /// * the second argument is a list of indices used to traverse the /// super-trait tree to find the trait whose method is being invoked, and - /// * the thrid argument is a pointer to the metadata (so in case of trait objects, usually it would be `&'static &'static Vtable`). - /// This indirection is necessary, because we don't know the size of the metadata. + /// * the third argument is a pointer to the wide pointer (so in case of trait objects, usually it would be `*const (*const T, &'static Vtable)`). + /// This indirection is necessary, because we don't know the size of the wide pointer. unsafe fn method_id_to_fn_ptr(f: fn(usize, &'static [usize], *const ()) -> *const ()) { self.method_id_to_fn_ptr = Some(f); } @@ -358,17 +380,22 @@ This RFC differentiates itself from all the other RFCs in that it provides a pro - Do we want this kind of flexibility? With power comes responsibility... - I believe we can do multiple super traits, including downcasts with this scheme, make sure that's true. - This scheme could support downcasting `dyn A` to `dyn B` if `trait A: B` if we make `T: ?Sized` (`T` is the `impl` block type). But that will not allow the sized use-cases anymore. If we have something like `MaybeSized` that has `size_of` and `align_of` methods returning `Option`, then maybe we could do this. +* Do we want the generic parameters on generic traits to influence vtable generation? +* Need to be generic over the allocator, too. This would allow us to remove the `owned` field by just passing dummy allocators (which panic) for un-owned unsizings. +* how does this interact with `Pin`? +* Should we make `DstInfo` generic over the `TraitDescription` in order to use fewer untyped pointers? # Future possibilities [future-possibilities]: #future-possibilities -* Add a scheme that allows super traits to have different vtable generators and permit a vtable generator to process them. So `trait A: B + C`, where `B` and `C` have different vtable generators and `A` unites them in some manner. This requires the information about the vtable generators to be part of the `ImplDescription` type. We can likely even put a function pointer to the vtable generator into the `ImplDescription`. +* Add a scheme that allows super traits to have different vtable generators and permit a vtable generator to process them. So `trait A: B + C`, where `B` and `C` have different vtable generators and `A` unites them in some manner. This requires the information about the vtable generators to be part of the `TraitDescription` type. We can likely even put a function pointer to the vtable generator into the `TraitDescription`. * Add a scheme allowing `dyn A + B`. I have no idea how, but maybe we just need to add a method to `DstInfo` that allows chaining vtable generators. So we first generate `dyn A`, then out of that we generate `dyn A + B` * This scheme can be used to generate C++-like vtables where the data and vtable are in the same allocation by making the `unsize` function create a heap allocation and write the value and the vtable into this new allocation. * We can change slice (`[T]`) types to be backed by a `trait Slice` which uses this scheme to generate just a single `usize` for the metadata * We can use this scheme and remove `extern type`s from the language, as they just become a trait with a custom metadata generator that uses `()` for the metadata. So `CStr` doesn't become an extern type, instead it becomes a trait. * We can handle traits with parents which are created by a different metadata generator function, we just need to figure out how to communicate this to such a function so it can special case this situation. -* We can give the vtable pointer of the super traits to the constructor function, so it doesn't have to recompute anything and just grab the info off there. Basically `ImplDescription::parent` would not be a pointer to the parent, but a struct which contains at least the pointer to the parent and the pointer to the `DstInfo` +* We can give the vtable pointer of the super traits to the constructor function, so it doesn't have to recompute anything and just grab the info off there. Basically `TraitDescription::parent` would not be a pointer to the parent, but a struct which contains at least the pointer to the parent and the pointer to the `DstInfo` * Totally off-topic, but a similar scheme (via const eval) can be used to procedurally generate type declarations. * We can likely access associated consts and types of the trait directly without causing cycle errors, this should be investigated -* This scheme is forward compatible to adding associated fields later. \ No newline at end of file +* This scheme is forward compatible to adding associated fields later. +* If/once we expose the `Unsize` traits on stable, we could consider adding a method to the `Unsize` trait that performs the conversion. This way more complex unsizings like `String` -> `str` could be performed without going through the `Deref` trait which does the conversion. This would allow us to essentially write `impl str for String` if we make `str` a `trait`. \ No newline at end of file From 66c74387d65b1d4351706538c025933c9993072e Mon Sep 17 00:00:00 2001 From: HackMD Date: Sun, 2 Aug 2020 09:23:36 +0000 Subject: [PATCH 09/22] Show how to use this for `[T]` --- text/0000-procedural-vtables.md | 65 ++++++++++++++++++++++++++------- 1 file changed, 52 insertions(+), 13 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 5096716fe5f..713edb17623 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -33,7 +33,8 @@ describes the metadata is a `#[nonexhaustive]` struct. You create instances of i You are actually generating the pointer metadata, not a description of it. Since your `const fn` is being interpreted in the target's environment, all target specific information will match up. Now, you need some information about the `impl` in order to generate your metadata (and your vtable). You get this information partially from the type directly (the `T` parameter), and all the `impl` block specific information is encoded in `TraitDescription`, which you get as a const generic parameter. -As an example, consider the function which is what normally generates your metadata. +As an example, consider the function which is what normally generates your metadata. Note that if these methods are used for generic traits, the method needs additional generic parameters, one for each parameter of the trait. +See the `[T]` demo further down for an example. ```rust /// If the `owned` flag is `true`, this is an owned conversion like @@ -55,7 +56,7 @@ pub const fn custom_unsize< // so the value lives long enough. ( ptr, - &default_vtable::() as *const _ as *const (), + &default_vtable::() as *const _ as *const (), ) } @@ -64,7 +65,6 @@ pub const fn custom_unsize< /// thus "vtable" layout that they want. pub const fn custom_vtable< const IMPL: &'static std::vtable::TraitDescription, - T, >() -> std::vtable::DstInfo { let mut info = DstInfo::new(); unsafe { @@ -118,8 +118,8 @@ const fn num_methods< } const fn default_vtable< - const IMPL: &'static std::vtable::TraitDescription, T, + const IMPL: &'static std::vtable::TraitDescription, >() -> VTable<{num_methods::()}> { // `VTable` is a `#[repr(C)]` type with fields at the appropriate // places. @@ -196,6 +196,43 @@ pub const fn c_str< } ``` +## `[T]` as sugar for a `Slice` trait + +We could remove `[T]` from the language and just make it desugar to +a `std::slice::Slice` trait. + +```rust +#[unsafe_custom_vtable = "slice"] +pub trait Slice {} + +pub const fn slice< + T, + const IMPL: &'static std::vtable::TraitDescription, +>() -> std::vtable::DstInfo { + let mut info = DstInfo::new(); + unsafe { + info.method_id_to_fn_ptr(|idx, parents, meta| { + panic!("CStr has no trait methods, it's all inherent methods acting on the pointer") + }); + info.size_of(|meta| unsafe { + let ptr = *(meta as *const (*const T, usize); + ptr.1 + }); + info.align_of(|meta| std::mem::align_of::()); + info.drop(|meta| { + let ptr = *(meta as *const (*const T, usize); + let mut data_ptr = ptr.0; + for i in 0..ptr.1 { + std::ptr::drop_in_place(data_ptr); + data_ptr = data_ptr.offset(1); + } + }); + info.self_ptr(|ptr| ptr); + } + info +} +``` + ## C++ like vtables Most of the boilerplate is the same as with regular vtables. @@ -224,7 +261,7 @@ where { // Copy the vtable into the shared allocation. // Note that we are reusing the same vtable generation as in // the regular Rust case. - std::ptr::write(new_ptr, default_vtable::()); + std::ptr::write(new_ptr, default_vtable::()); new_ptr } @@ -280,9 +317,15 @@ When obtaining function pointers from vtables, instead of computing an offset, t These types' declarations are provided below: ```rust +#[nonexhaustive] +struct TraitMethod { + fn_ptr: *const (), + // may get more fields like `name` or even `body` (the latter being a string of the body to be used with `syn`). +} + #[nonexhaustive] struct TraitDescription { - pub methods: &'static [*const ()], + pub methods: &'static [TraitMethod], pub parent: &'static TraitDescription, } @@ -379,20 +422,16 @@ This RFC differentiates itself from all the other RFCs in that it provides a pro - Do we want this kind of flexibility? With power comes responsibility... - I believe we can do multiple super traits, including downcasts with this scheme, make sure that's true. -- This scheme could support downcasting `dyn A` to `dyn B` if `trait A: B` if we make `T: ?Sized` (`T` is the `impl` block type). But that will not allow the sized use-cases anymore. If we have something like `MaybeSized` that has `size_of` and `align_of` methods returning `Option`, then maybe we could do this. -* Do we want the generic parameters on generic traits to influence vtable generation? -* Need to be generic over the allocator, too. This would allow us to remove the `owned` field by just passing dummy allocators (which panic) for un-owned unsizings. +- This scheme could support downcasting `dyn A` to `dyn B` if `trait A: B` if we make `T: ?Sized` (`T` is the `impl` block type). But that will not allow the sized use-cases anymore (since `size_of::` will fail). If we have something like `MaybeSized` that has `size_of` and `align_of` methods returning `Option`, then maybe we could do this. +* Need to be generic over the allocator, too, so that reallocs are actually sound. * how does this interact with `Pin`? -* Should we make `DstInfo` generic over the `TraitDescription` in order to use fewer untyped pointers? # Future possibilities [future-possibilities]: #future-possibilities * Add a scheme that allows super traits to have different vtable generators and permit a vtable generator to process them. So `trait A: B + C`, where `B` and `C` have different vtable generators and `A` unites them in some manner. This requires the information about the vtable generators to be part of the `TraitDescription` type. We can likely even put a function pointer to the vtable generator into the `TraitDescription`. * Add a scheme allowing `dyn A + B`. I have no idea how, but maybe we just need to add a method to `DstInfo` that allows chaining vtable generators. So we first generate `dyn A`, then out of that we generate `dyn A + B` -* This scheme can be used to generate C++-like vtables where the data and vtable are in the same allocation by making the `unsize` function create a heap allocation and write the value and the vtable into this new allocation. -* We can change slice (`[T]`) types to be backed by a `trait Slice` which uses this scheme to generate just a single `usize` for the metadata -* We can use this scheme and remove `extern type`s from the language, as they just become a trait with a custom metadata generator that uses `()` for the metadata. So `CStr` doesn't become an extern type, instead it becomes a trait. +* We can change string slice (`str`) types to be backed by a `trait StrSlice` which uses this scheme to generate just a single `usize` for the metadata (see also the `[T]` demo). * We can handle traits with parents which are created by a different metadata generator function, we just need to figure out how to communicate this to such a function so it can special case this situation. * We can give the vtable pointer of the super traits to the constructor function, so it doesn't have to recompute anything and just grab the info off there. Basically `TraitDescription::parent` would not be a pointer to the parent, but a struct which contains at least the pointer to the parent and the pointer to the `DstInfo` * Totally off-topic, but a similar scheme (via const eval) can be used to procedurally generate type declarations. From 24f8f5a9905177888aba96ccae0a6425c7efa0b4 Mon Sep 17 00:00:00 2001 From: HackMD Date: Sun, 2 Aug 2020 10:02:31 +0000 Subject: [PATCH 10/22] Vtables have a const generic param for the length, not the trait tree --- text/0000-procedural-vtables.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 713edb17623..4faa6b68d73 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -72,7 +72,7 @@ pub const fn custom_vtable< // This is always inlined and will thus get optimized to a single // deref and offset (strong handwaving happening here). info.method_id_to_fn_ptr(|mut idx, parents, meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable)); + let meta = *(meta as *const (*const(), &'static Vtable<{num_methods::()}>)); let mut table = IMPL; for parent in parents { // we don't support multi-parents yet @@ -85,19 +85,19 @@ pub const fn custom_vtable< meta.1.methods[idx]; }); info.size_of(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable)); + let meta = *(meta as *const (*const(), &'static Vtable<{num_methods::()}>)); meta.1.size }); info.align_of(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable)); + let meta = *(meta as *const (*const(), &'static Vtable<{num_methods::()}>)); meta.1.align }); info.drop(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable)); + let meta = *(meta as *const (*const(), &'static Vtable<{num_methods::()}>)); meta.1.drop }); info.self_ptr(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable)); + let meta = *(meta as *const (*const(), &'static Vtable<{num_methods::()}>)); meta.0 }); } @@ -251,7 +251,7 @@ where { }, { let size = std::mem::size_of::(); - let vtable_size = std::mem::size_of::>(); + let vtable_size = std::mem::size_of::()}>>(); let layout = Layout::from_size_align(size + vtable_size, std::mem::align_of::()).unwrap(); let new_ptr = std::alloc::alloc(layout); // Move the value to the new allocation @@ -271,24 +271,24 @@ pub const fn cpp< let mut info = DstInfo::new(); unsafe { info.method_id_to_fn_ptr(|idx, parents, meta| unsafe { - let meta = *(meta as *const *const Vtable); + let meta = *(meta as *const *const Vtable<{num_methods::()}>); // The rest of the function body is the same as with regular // vtables. }); info.size_of(|meta| unsafe { - let meta = *(meta as *const *const Vtable); + let meta = *(meta as *const *const Vtable<{num_methods::()}>); (*meta).size }); info.align_of(|meta| unsafe { - let meta = *(meta as *const *const Vtable); + let meta = *(meta as *const *const Vtable<{num_methods::()}>); (*meta).align }); info.drop(|meta| unsafe { - let meta = *(meta as *const *const Vtable); + let meta = *(meta as *const *const Vtable<{num_methods::()}>); (*meta).drop }); info.self_ptr(|meta| unsafe { - let ptr = *(meta as *const *const (Vtable, ())); + let ptr = *(meta as *const *const (Vtable<{num_methods::()}>, ())); &raw const (*ptr).1; }); } From 49f2b6852a8c8d4ce2ee999299bd3018ea3ea7de Mon Sep 17 00:00:00 2001 From: HackMD Date: Sun, 2 Aug 2020 11:05:33 +0000 Subject: [PATCH 11/22] Rewrite the RFC to be more type safe --- text/0000-procedural-vtables.md | 417 ++++++++++++++++---------------- 1 file changed, 203 insertions(+), 214 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 4faa6b68d73..7bb450d0027 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -17,26 +17,27 @@ The only way we're going to satisfy all users' use cases is by allowing users co [guide-level-explanation]: #guide-level-explanation In order to mark a trait as using a custom vtable layout, you apply the -`#[unsafe_custom_vtable = "foo"]` attribute to the trait declaration. -This is unsafe, because the `foo` function supplies functionality for accessing the `self` pointers. -`foo` denotes a `const fn` with the signature -`const fn() -> std::vtable::DstInfo`. - -Additionally, if your trait supports automatically unsizing from the types it's implemented for (unlike `CStr`, `str` and `[T]`, which require type-specific logic), you can supply your trait with an `unsizing` function by -specifying `#[unsafe_custom_unsize = "bar"]`. `bar` denotes a `const fn` -with the signature -`const fn() -> std::vtable::UnsizeInfo`, where `T` is your concrete type. - -All `impl`s of this trait will now use `foo` for generating the wide pointer -metadata (which contains the vtable). The `WidePointerMetadata` struct that -describes the metadata is a `#[nonexhaustive]` struct. You create instances of it by invoking its `new` method, which gives you a `WidePointerMetadata` that essentially is a `()`. This means you do not have any metadata, similar to `extern type`s. Since the `wide_ptr_metadata` field of the `DstInfo` struct is public, you can now modify it to whatever layout you desire. -You are actually generating the pointer metadata, not a description of it. Since your `const fn` is being interpreted in the target's environment, all target specific information will match up. +`#[custom_wide_pointer = "Foo"]` attribute to the trait declaration. +`Foo` is a type which implements the `CustomUnsized` trait and optionally the `CustomUnsize` trait. + +All `impl`s of this trait will now use `Foo` for generating the wide pointer +metadata (which contains the vtable). The `TraitDescription` struct that +describes the metadata is a `#[nonexhaustive]` struct. +You are actually generating the pointer metadata, not a description of it. +Since your `const impl`'s `from` function is being interpreted in the target's environment, all target specific information will match up. Now, you need some information about the `impl` in order to generate your metadata (and your vtable). You get this information partially from the type directly (the `T` parameter), and all the `impl` block specific information is encoded in `TraitDescription`, which you get as a const generic parameter. -As an example, consider the function which is what normally generates your metadata. Note that if these methods are used for generic traits, the method needs additional generic parameters, one for each parameter of the trait. +As an example, consider the function which is what normally generates your metadata. Note that if you have a generic trait, +the `CustomUnsize` and `CustomUnsized` impls need additional generic parameters, one for each parameter of the trait. See the `[T]` demo further down for an example. ```rust +#[repr(C)] +struct Pointer { + ptr: *const (), + vtable: &'static VTable<{num_methods::()}>, +} + /// If the `owned` flag is `true`, this is an owned conversion like /// in `Box as Box`. This distinction is important, as /// unsizing that creates a vtable in the same allocation as the object @@ -44,64 +45,55 @@ See the `[T]` demo further down for an example. /// move away the owned object. The flag allows you to forbid such /// unsizings by triggering a compile-time `panic` with an explanation /// for the user. -pub const fn custom_unsize< - T, - const IMPL: &'static std::vtable::TraitDescription, - const _owned: bool, ->(*const T) -> (*const T, &'static VTable<{num_methods::()}>) { - // We generate the metadata and put a pointer to the metadata into - // the field. This looks like it's passing a reference to a temporary - // value, but this uses promotion - // (https://doc.rust-lang.org/stable/reference/destructors.html?highlight=promotion#constant-promotion), - // so the value lives long enough. - ( - ptr, - &default_vtable::() as *const _ as *const (), - ) +unsafe impl const CustomUnsize for Pointer { + fn from(ptr: *const T) -> Self { + // We generate the metadata and put a pointer to the metadata into + // the field. This looks like it's passing a reference to a temporary + // value, but this uses promotion + // (https://doc.rust-lang.org/stable/reference/destructors.html?highlight=promotion#constant-promotion), + // so the value lives long enough. + Pointer { + ptr, + vtable: &default_vtable::(), + } + } } /// DISCLAIMER: this uses a `Vtable` struct which is just a part of the /// default trait objects. Your own trait objects can use any metadata and /// thus "vtable" layout that they want. -pub const fn custom_vtable< - const IMPL: &'static std::vtable::TraitDescription, ->() -> std::vtable::DstInfo { - let mut info = DstInfo::new(); - unsafe { - // We supply a function for invoking trait methods. - // This is always inlined and will thus get optimized to a single - // deref and offset (strong handwaving happening here). - info.method_id_to_fn_ptr(|mut idx, parents, meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable<{num_methods::()}>)); - let mut table = IMPL; - for parent in parents { - // we don't support multi-parents yet - assert_eq!(parent, 0); - idx += table.methods.len(); - // Never panics, there are always fewer or equal number of - // parents given as the argument as there are in reality. - table = table.parent.unwrap(); - } - meta.1.methods[idx]; - }); - info.size_of(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable<{num_methods::()}>)); - meta.1.size - }); - info.align_of(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable<{num_methods::()}>)); - meta.1.align - }); - info.drop(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable<{num_methods::()}>)); - meta.1.drop - }); - info.self_ptr(|meta| unsafe { - let meta = *(meta as *const (*const(), &'static Vtable<{num_methods::()}>)); - meta.0 - }); +unsafe impl CustomUnsized for Pointer { + fn method_id_to_fn_ptr( + self, + method_index: usize, + super_tree_path: usize, + ) -> *const () { + let mut table = IMPL; + for parent in parents { + // we don't support multi-parents yet + assert_eq!(parent, 0); + idx += table.methods.len(); + // Never panics, there are always fewer or equal number of + // parents given as the argument as there are in reality. + table = table.parent.unwrap(); + } + self.vtable.methods[idx] + } + fn size_of(self) -> usize { + self.vtable.size + } + fn align_of(self) -> usize { + self.vtable.align + } + fn drop(self) { + unsafe { + let drop = self.vtable.drop; + drop(&raw mut self.ptr) + } + } + fn self_ptr(self) -> *const () { + self.ptr } - info } // Compute the total number of methods, including super-traits @@ -156,7 +148,7 @@ const fn default_vtable< } ``` -Now, if you want to implement a fancier vtable, this RFC enables that. +Now, if you want to implement a fancier vtable, this RFC enables you to do that. ## Null terminated strings (std::ffi::CStr) @@ -165,34 +157,38 @@ There can be no impls of `CStr` for any type, because the `unsize` function is missing. The `CStr` ```rust - -// Not setting the `unsafe_custom_unsize` function, there's no sized -// equivalent like with normal traits. See the future -// extensions section for more details on unsizing. -#[unsafe_custom_vtable = "c_str"] +#[custom_wide_pointer = "CStrPtr"] pub trait CStr {} -pub const fn c_str< + +#[repr(C)] +struct CStrPtr { + ptr: *const u8, +} + +impl< + T, const IMPL: &'static std::vtable::TraitDescription, ->() -> std::vtable::DstInfo { - let mut info = DstInfo::new(); - unsafe { - info.method_id_to_fn_ptr(|idx, parents, meta| { - panic!("CStr has no trait methods, it's all inherent methods acting on the pointer") - }); - info.size_of(|meta| unsafe { - let ptr = *(meta as *const *const u8); - strlen(ptr) - }); - info.align_of(|meta| 1); +> CustomUnsized for CStrPtr { + fn method_id_to_fn_ptr( + self, + method_index: usize, + super_tree_path: usize, + ) -> *const () { + panic!("CStr has no trait methods, it's all inherent methods acting on the pointer") + } + fn size_of(self) -> usize { + unsafe { strlen(self.ptr) } + } + fn align_of(self) -> usize { + 1 + } + fn drop(self) { // Nothing to drop (just `u8`s) and we are not in charge of dealloc - info.drop(|meta| None); - info.self_ptr(|ptr| { - let ptr = *(meta as *const *const u8); - ptr - }); } - info + fn self_ptr(self) -> *const () { + self.ptr + } } ``` @@ -202,34 +198,44 @@ We could remove `[T]` from the language and just make it desugar to a `std::slice::Slice` trait. ```rust -#[unsafe_custom_vtable = "slice"] +#[custom_wide_pointer = "SlicePtr"] pub trait Slice {} -pub const fn slice< +#[repr(C)] +struct SlicePtr { + ptr: *const T, + len: usize, +} + +impl< T, const IMPL: &'static std::vtable::TraitDescription, ->() -> std::vtable::DstInfo { - let mut info = DstInfo::new(); - unsafe { - info.method_id_to_fn_ptr(|idx, parents, meta| { - panic!("CStr has no trait methods, it's all inherent methods acting on the pointer") - }); - info.size_of(|meta| unsafe { - let ptr = *(meta as *const (*const T, usize); - ptr.1 - }); - info.align_of(|meta| std::mem::align_of::()); - info.drop(|meta| { - let ptr = *(meta as *const (*const T, usize); - let mut data_ptr = ptr.0; - for i in 0..ptr.1 { - std::ptr::drop_in_place(data_ptr); - data_ptr = data_ptr.offset(1); - } - }); - info.self_ptr(|ptr| ptr); +> CustomUnsized for Slice { + fn method_id_to_fn_ptr( + self, + method_index: usize, + super_tree_path: usize, + ) -> *const () { + panic!("CStr has no trait methods, it's all inherent methods acting on the pointer") + } + fn size_of(self) -> usize { + self.len + } + fn align_of(self) -> usize { + st::mem::align_of::() + } + fn drop(self) { + let mut data_ptr = self.ptr; + for i in 0..self.len { + std::ptr::drop_in_place(data_ptr); + data_ptr = data_ptr.offset(1); + } + } + fn self_ptr(self) -> *const () { + // Not quite sure what to do here, need to think on this. + // Technically I want to return `Self`, but other schemes do not. + self.ptr } - info } ``` @@ -238,61 +244,63 @@ pub const fn slice< Most of the boilerplate is the same as with regular vtables. ```rust +#[repr(C)] +struct Pointer { + ptr: *const (VTable<{num_methods::()}>, ()) +} -pub const fn cpp_unsize< - T, - const IMPL: &'static std::vtable::TraitDescription, - const owned: bool, ->( - ptr: *const T, -) -> *const (VTable<{num_methods::()}>, T) -where { - assert!(owned, "cannot unsize borrowed object for C++ like trait") -}, -{ - let size = std::mem::size_of::(); - let vtable_size = std::mem::size_of::()}>>(); - let layout = Layout::from_size_align(size + vtable_size, std::mem::align_of::()).unwrap(); - let new_ptr = std::alloc::alloc(layout); - // Move the value to the new allocation - std::ptr::copy_nonoverlapping(ptr, new_ptr.offset(vtable_size), 1); - std::alloc::dealloc(ptr); - - // Copy the vtable into the shared allocation. - // Note that we are reusing the same vtable generation as in - // the regular Rust case. - std::ptr::write(new_ptr, default_vtable::()); - new_ptr +unsafe impl const CustomUnsize for Pointer { + fn from(ptr: *const T) -> Self { + unsafe { + let size = std::mem::size_of::(); + let vtable_size = std::mem::size_of::()}>>(); + let layout = Layout::from_size_align(size + vtable_size, std::mem::align_of::()).unwrap(); + let new_ptr = std::alloc::alloc(layout); + // Move the value to the new allocation + std::ptr::copy_nonoverlapping(ptr, new_ptr.offset(vtable_size), 1); + std::alloc::dealloc(ptr); + + // Copy the vtable into the shared allocation. + // Note that we are reusing the same vtable generation as in + // the regular Rust case. + std::ptr::write(new_ptr, default_vtable::()); + Self { ptr: new_ptr } + } + } } -pub const fn cpp< - const IMPL: &'static std::vtable::TraitDescription, ->() -> std::vtable::DstInfo { - let mut info = DstInfo::new(); - unsafe { - info.method_id_to_fn_ptr(|idx, parents, meta| unsafe { - let meta = *(meta as *const *const Vtable<{num_methods::()}>); - // The rest of the function body is the same as with regular - // vtables. - }); - info.size_of(|meta| unsafe { - let meta = *(meta as *const *const Vtable<{num_methods::()}>); - (*meta).size - }); - info.align_of(|meta| unsafe { - let meta = *(meta as *const *const Vtable<{num_methods::()}>); - (*meta).align - }); - info.drop(|meta| unsafe { - let meta = *(meta as *const *const Vtable<{num_methods::()}>); - (*meta).drop - }); - info.self_ptr(|meta| unsafe { - let ptr = *(meta as *const *const (Vtable<{num_methods::()}>, ())); - &raw const (*ptr).1; - }); + +unsafe impl CustomUnsized for Pointer { + fn method_id_to_fn_ptr( + self, + method_index: usize, + super_tree_path: usize, + ) -> *const () { + let meta = unsafe { &*self.ptr }; + // The rest of the function body is the same as with regular + // vtables. + } + fn size_of(self) -> usize { + unsafe { + (*self.ptr).0.size + } + } + fn align_of(self) -> usize { + unsafe { + (*self.ptr).0.align + } + } + fn drop(self) { + unsafe { + let drop = (*self.ptr).0.drop; + drop(&raw mut (*self.ptr).1) + } + } + fn self_ptr(self) -> *const () { + unsafe { + &raw const (*self.ptr).1 + } } - info } ``` @@ -300,17 +308,16 @@ pub const fn cpp< # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -The two types `TraitDescription` and `DstInfo` are `#[nonexhaustive]` in order to allow arbitrary extension in the future. +The type `TraitDescription` is `#[nonexhaustive]` in order to allow arbitrary extension in the future. Instances of the `TraitDescription` struct are created by rustc, basically replacing today's vtable generation, and then outsourcing the actual vtable generation to the const evaluator. When unsizing, the `const fn` specified -via `unsafe_custom_unsize` is invoked. The only reason that function is -`const fn` is to restrict what kind of things you can do in there. We -can lift this restriction in the future. +via `::from` is invoked. `WidePtrType` refers to the argument of the +`#[custom_wide_ptr = "WidePtrType"]` attribute. The only reason that trait must be +`impl const WidePtrType` is to restrict what kind of things you can do in there, it's not +strictly necessary. -For all other operations, the `unsafe_custom_vtable` function is invoked. -This one must be `const fn`, as it is evaluated at compile-time and the -compiler then inspects the resulting `DstInfo` at compile-time. +For all other operations, the methods on `` is invoked. When obtaining function pointers from vtables, instead of computing an offset, the `method_id_to_fn_ptr` function is invoked at runtime and computes a function pointer after being given a method index, the indices of all parents and a pointer to a metadata field. Through the use of MIR optimizations (e.g. inlining), the final LLVM assembly is tuned to be exactly the same as today. @@ -329,55 +336,35 @@ struct TraitDescription { pub parent: &'static TraitDescription, } -#[nonexhaustive] -struct DstInfo { - method_id_to_fn_ptr: fn(usize, &'static [usize], *const ()) -> *const (), - size_of: fn(*const()) -> usize, - align_of: fn(*const()) -> usize, - drop: *const (), - self_ptr: fn(*const()) -> *const (), -} - -impl DstInfo { - const fn new() -> Self { - Self { - method_id_to_fn_ptr: None, - size_of: None, - align_of: None, - drop: None, - self_ptr: None, - } - } - /// The given function returns a function pointer to the method that +unsafe trait CustomUnsized: Copy { + /// Returns a function pointer to the method that /// is being requested. /// * The first argument is the method index, /// * the second argument is a list of indices used to traverse the /// super-trait tree to find the trait whose method is being invoked, and /// * the third argument is a pointer to the wide pointer (so in case of trait objects, usually it would be `*const (*const T, &'static Vtable)`). /// This indirection is necessary, because we don't know the size of the wide pointer. - unsafe fn method_id_to_fn_ptr(f: fn(usize, &'static [usize], *const ()) -> *const ()) { - self.method_id_to_fn_ptr = Some(f); - } - /// Set the function that extracts the dynamic size - unsafe fn size_of(f: fn(*const ()) -> usize) { - self.size_of = Some(f); - } - /// Set the function that extracts the dynamic alignment - unsafe fn align_of(f: fn(*const ()) -> usize) { - self.align_of = Some(f); - } - /// Set the function that extracts the drop code. - unsafe fn drop(f: fn(*const ()) -> Option) { - self.drop = Some(f); - } - /// Set the function that extracts the `&self` pointer + fn method_id_to_fn_ptr( + self, + method_index: usize, + super_tree_path: usize, + ) -> *const (); + fn size_of(self) -> usize; + fn align_of(self) -> usize; + fn drop(self); + /// Extracts the `&self` pointer /// from the wide pointer /// for calling trait methods. This needs a method as /// wide pointer layouts may place their `self` pointer /// anywhere they desire. - unsafe fn self_ptr(f: fn(*const ()) -> *const ()) { - self.self_ptr = Some(f); - } + fn self_ptr(self) -> *const (); +} + +unsafe trait CustomUnsize: CustomUnsized { + fn from< + T + const owned: bool, + >(t: *const T) -> Self; } ``` @@ -390,6 +377,8 @@ impl DstInfo { # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives +This is a frequently requested feature and as a side effect obsoletes `extern type`s which have all kinds of problems (like the inability to invoke `size_of_val` on pointers to them). + ## Don't use const generics If we wait until we have a heap in const eval, we can use `Vec`s and `Box`es, which would allow us to avoid the const generics scheme, likely making all the code less roundabout. @@ -421,10 +410,10 @@ This RFC differentiates itself from all the other RFCs in that it provides a pro [unresolved-questions]: #unresolved-questions - Do we want this kind of flexibility? With power comes responsibility... -- I believe we can do multiple super traits, including downcasts with this scheme, make sure that's true. +- I believe we can do multiple super traits, including downcasts with this scheme and no additional extensions, but I need to make sure that's true. - This scheme could support downcasting `dyn A` to `dyn B` if `trait A: B` if we make `T: ?Sized` (`T` is the `impl` block type). But that will not allow the sized use-cases anymore (since `size_of::` will fail). If we have something like `MaybeSized` that has `size_of` and `align_of` methods returning `Option`, then maybe we could do this. * Need to be generic over the allocator, too, so that reallocs are actually sound. -* how does this interact with `Pin`? +* how does this RFC (especially the `owned` flag) interact with `Pin`? # Future possibilities [future-possibilities]: #future-possibilities @@ -437,4 +426,4 @@ This RFC differentiates itself from all the other RFCs in that it provides a pro * Totally off-topic, but a similar scheme (via const eval) can be used to procedurally generate type declarations. * We can likely access associated consts and types of the trait directly without causing cycle errors, this should be investigated * This scheme is forward compatible to adding associated fields later. -* If/once we expose the `Unsize` traits on stable, we could consider adding a method to the `Unsize` trait that performs the conversion. This way more complex unsizings like `String` -> `str` could be performed without going through the `Deref` trait which does the conversion. This would allow us to essentially write `impl str for String` if we make `str` a `trait`. \ No newline at end of file +* If/once we expose the `Unsize` (do not confuse with `CustomUnsize`) traits on stable, we could consider adding a method to the `Unsize` trait that performs the conversion. This way more complex unsizings like `String` -> `str` could be performed without going through the `Deref` trait which does the conversion. This would allow us to essentially write `impl str for String` if we make `str` a `trait`. We could also move the `CustomUnsize` trait's `from` method's `T` parameter onto the trait, thus allowing users to manually `impl CustomUnsize for CStrPtr`. \ No newline at end of file From b7e9b923ad034ff36ac8f3d68e4966b4ae36961e Mon Sep 17 00:00:00 2001 From: HackMD Date: Sun, 2 Aug 2020 13:06:24 +0000 Subject: [PATCH 12/22] Clarify some things and write a more extensive intro --- text/0000-procedural-vtables.md | 97 +++++++++++++++++++++++++-------- 1 file changed, 73 insertions(+), 24 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 7bb450d0027..48b2b455166 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -6,26 +6,56 @@ # Summary [summary]: #summary -All vtable generation happens outside the compiler by invoking a `const fn` that generates said vtable from a generic description of a trait impl. By default, if no vtable generator function is specified for a specific trait, `std::vtable::default` is invoked. +Building a wide pointer from a concrete pointer (and thus vtable generation) +can be controlled by choosing a custom wide pointer type for the trait. +The custom wide pointer must implement a trait that generates said wide pointer +by taking a concrete pointer and a generic description of a trait impl. +By default, if no vtable generator function is specified for a specific trait, +the unspecified scheme used today keeps getting used. # Motivation [motivation]: #motivation -The only way we're going to satisfy all users' use cases is by allowing users complete freedom in how their wide pointers' metadata is built. Instead of hardcoding certain vtable layouts in the language (https://github.com/rust-lang/rfcs/pull/2955) we can give users the capability to invent their own layouts at their leisure. This should also help with the work on custom DSTs, as this scheme doesn't specify the size of the wide pointer metadata field. +The only way we're going to satisfy all users' use cases is by allowing users +complete freedom in how their wide pointers' metadata is built. +Instead of hardcoding certain vtable layouts in the language +(https://github.com/rust-lang/rfcs/pull/2955) we can give users the capability +to invent their own wide pointers (and thus custom dynamically sized types). # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -In order to mark a trait as using a custom vtable layout, you apply the -`#[custom_wide_pointer = "Foo"]` attribute to the trait declaration. -`Foo` is a type which implements the `CustomUnsized` trait and optionally the `CustomUnsize` trait. +In order to mark a trait as using a custom vtable layout, you apply an attribute to +your trait declaration. -All `impl`s of this trait will now use `Foo` for generating the wide pointer -metadata (which contains the vtable). The `TraitDescription` struct that -describes the metadata is a `#[nonexhaustive]` struct. -You are actually generating the pointer metadata, not a description of it. +```rust +#[custom_wide_pointer = "MyWidePointer"] +trait MyTrait {} + +``` + +`MyWidePointer` is a type which implements the `CustomUnsized` trait and optionally the `CustomUnsize` trait. +`MyWidePointer` is the type that is going to be used for wide pointers to the given trait, so + +* `&dyn MyTrait` +* `Box` +* `Arc` + +and any other container types that can use wide pointers. Normally when unsizing from a concrete +pointer like `&MyStruct` to `&dyn MyTrait` a wide pointer (that essentially is +`(&MyStruct, &'static Vtable)`) is produced. This is currently done via compiler magic, but +further down in this section you can see how it theoretically could be done in user code. +Likely it will stay compiler magic out of compile-time performance reasons. + +All `impl`s of `MyTrait` will now use `MyWidePointer` for generating the wide pointer. +The `TraitDescription` struct describes the metadata like the list of methods and a tree +of super traits. + +You are actually generating the wide pointer, not a description of it. Since your `const impl`'s `from` function is being interpreted in the target's environment, all target specific information will match up. -Now, you need some information about the `impl` in order to generate your metadata (and your vtable). You get this information partially from the type directly (the `T` parameter), and all the `impl` block specific information is encoded in `TraitDescription`, which you get as a const generic parameter. +Now, you need some information about the `impl` in order to generate your metadata (and your vtable). +You get this information partially from the type directly (the `T` parameter), +and all the `impl` block specific information is encoded in `TraitDescription`, which you get as a const generic parameter. As an example, consider the function which is what normally generates your metadata. Note that if you have a generic trait, the `CustomUnsize` and `CustomUnsized` impls need additional generic parameters, one for each parameter of the trait. @@ -127,7 +157,7 @@ const fn default_vtable< match IMPL.methods.get(i) { Some(Some(method)) => { // The `method` variable is a function pointer, but - // cast to `*const ()`. + // cast to `*const ()` in order to support null pointers. vtable.methods[i] = method; i += 1; }, @@ -153,8 +183,10 @@ Now, if you want to implement a fancier vtable, this RFC enables you to do that. ## Null terminated strings (std::ffi::CStr) This is how I see all extern types being handled. -There can be no impls of `CStr` for any type, because the `unsize` -function is missing. The `CStr` +There can be no impls of `CStr` for any type, because the `Unsize` +trait impl is missing. See the future extension section at the end of this RFC for +ideas that could support `CString` -> `CStr` unsizing by allowing `CString` to implement +`CStr` instead of having a `Deref` impl that converts. ```rust #[custom_wide_pointer = "CStrPtr"] @@ -194,8 +226,8 @@ impl< ## `[T]` as sugar for a `Slice` trait -We could remove `[T]` from the language and just make it desugar to -a `std::slice::Slice` trait. +We could remove `[T]` (and even `str`) from the language and just make it desugar to +a `std::slice::Slice` (or `StrSlice`) trait. ```rust #[custom_wide_pointer = "SlicePtr"] @@ -312,16 +344,23 @@ The type `TraitDescription` is `#[nonexhaustive]` in order to allow arbitrary ex Instances of the `TraitDescription` struct are created by rustc, basically replacing today's vtable generation, and then outsourcing the actual vtable generation to the const evaluator. When unsizing, the `const fn` specified -via `::from` is invoked. `WidePtrType` refers to the argument of the +via `::from` is invoked. +The `TraitDescription` parameter of `CustomUnsize` contains function pointers to the +concrete functions of the `impl`. The `TraitDescription` parameter of `CustomUnsized` +contains function pointers to default impls (if any), or mostly just null pointers. +The same datastructure is shared here to make implementations easier and because they +contain mostly the same data anyway. +`WidePtrType` refers to the argument of the `#[custom_wide_ptr = "WidePtrType"]` attribute. The only reason that trait must be `impl const WidePtrType` is to restrict what kind of things you can do in there, it's not strictly necessary. For all other operations, the methods on `` is invoked. -When obtaining function pointers from vtables, instead of computing an offset, the `method_id_to_fn_ptr` function is invoked at runtime and computes a function pointer after being given a method index, the indices of all parents and a pointer to a metadata field. Through the use of MIR optimizations (e.g. inlining), the final LLVM assembly is tuned to be exactly the same as today. +When obtaining function pointers from vtables, instead of computing an offset, the `method_id_to_fn_ptr` function is invoked at runtime and computes a function pointer after being given a method index, the indices of all parents, and a pointer to a metadata field. +Through the use of MIR optimizations (e.g. inlining), the final LLVM assembly is tuned to be exactly the same as today. -These types' declarations are provided below: +These types' and trait's declarations are provided below: ```rust #[nonexhaustive] @@ -404,7 +443,8 @@ This list is shamelessly taken from [strega-nil's Custom DST RFC](https://github - [Pointer Metadata and VTable](https://github.com/rust-lang/rfcs/pull/2580) - [Syntax of ?Sized](https://github.com/rust-lang/rfcs/pull/490) -This RFC differentiates itself from all the other RFCs in that it provides a procedural way to generate vtables, thus also permitting arbitrary user-defined compile-time conditions by aborting via `panic!`. +This RFC differs from all the other RFCs in that it provides a procedural way to generate vtables, +thus also permitting arbitrary user-defined compile-time conditions by aborting via `panic!`. # Unresolved questions [unresolved-questions]: #unresolved-questions @@ -418,11 +458,20 @@ This RFC differentiates itself from all the other RFCs in that it provides a pro # Future possibilities [future-possibilities]: #future-possibilities -* Add a scheme that allows super traits to have different vtable generators and permit a vtable generator to process them. So `trait A: B + C`, where `B` and `C` have different vtable generators and `A` unites them in some manner. This requires the information about the vtable generators to be part of the `TraitDescription` type. We can likely even put a function pointer to the vtable generator into the `TraitDescription`. -* Add a scheme allowing `dyn A + B`. I have no idea how, but maybe we just need to add a method to `DstInfo` that allows chaining vtable generators. So we first generate `dyn A`, then out of that we generate `dyn A + B` -* We can change string slice (`str`) types to be backed by a `trait StrSlice` which uses this scheme to generate just a single `usize` for the metadata (see also the `[T]` demo). -* We can handle traits with parents which are created by a different metadata generator function, we just need to figure out how to communicate this to such a function so it can special case this situation. -* We can give the vtable pointer of the super traits to the constructor function, so it doesn't have to recompute anything and just grab the info off there. Basically `TraitDescription::parent` would not be a pointer to the parent, but a struct which contains at least the pointer to the parent and the pointer to the `DstInfo` +* Add a scheme that allows upcasting to super traits that have different vtable generators. + So `trait A: B + C`, where `B` and `C` have different vtable generators and `A` unites them in some manner. + This requires the information about the vtable generators to be part of the `TraitDescription` type. + We can likely even put a function pointer to the vtable generator into the `TraitDescription`. +* Add a scheme allowing `dyn A + B`. I have no idea how, but maybe we just need to add a method to `CustomUnsize` + that allows chaining vtable generators. So we first generate `dyn A`, then out of that we generate `dyn A + B` +* We can change string slice (`str`) types to be backed by a `trait StrSlice` which uses this scheme + to generate just a single `usize` for the metadata (see also the `[T]` demo). +* We can handle traits with parents which are created by a different metadata generator function, + we just need to figure out how to communicate this to such a function so it can special case this situation. +* We can give the vtable pointer of the super traits to the constructor function, + so it doesn't have to recompute anything and just grab the info off there. + Basically `TraitDescription::parent` would not be a pointer to the parent, + but a struct which contains at least the pointer to the parent and the pointer to the `CustomUnsize::from` function. * Totally off-topic, but a similar scheme (via const eval) can be used to procedurally generate type declarations. * We can likely access associated consts and types of the trait directly without causing cycle errors, this should be investigated * This scheme is forward compatible to adding associated fields later. From 41599c4ce761e3dc5472c19856fa89c820c456c3 Mon Sep 17 00:00:00 2001 From: HackMD Date: Thu, 6 Aug 2020 07:41:03 +0000 Subject: [PATCH 13/22] Fix up variable names --- text/0000-procedural-vtables.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 48b2b455166..734b736e806 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -95,8 +95,8 @@ unsafe impl const CustomUnsi unsafe impl CustomUnsized for Pointer { fn method_id_to_fn_ptr( self, - method_index: usize, - super_tree_path: usize, + mut idx: usize, + parents: &[usize], ) -> *const () { let mut table = IMPL; for parent in parents { From b4ecf4731ca8009eafdf79e1c6eb48a3b4e763d3 Mon Sep 17 00:00:00 2001 From: HackMD Date: Thu, 6 Aug 2020 08:20:49 +0000 Subject: [PATCH 14/22] Copy paste mistakes --- text/0000-procedural-vtables.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 734b736e806..4a27d4cb664 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -248,7 +248,7 @@ impl< method_index: usize, super_tree_path: usize, ) -> *const () { - panic!("CStr has no trait methods, it's all inherent methods acting on the pointer") + panic!("Slices have no trait methods, it's all inherent methods acting on the pointer") } fn size_of(self) -> usize { self.len @@ -266,7 +266,7 @@ impl< fn self_ptr(self) -> *const () { // Not quite sure what to do here, need to think on this. // Technically I want to return `Self`, but other schemes do not. - self.ptr + panic!("it only makes sense to call this method in order to call a method") } } ``` From 45b844bad2cfbc0df4bbc0d4b6e1f1f334c80b44 Mon Sep 17 00:00:00 2001 From: HackMD Date: Sat, 8 Aug 2020 12:02:20 +0000 Subject: [PATCH 15/22] Generalize the scheme to allow unsizing from arbitrary types --- text/0000-procedural-vtables.md | 339 ++++++++++---------------------- 1 file changed, 103 insertions(+), 236 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 4a27d4cb664..a1fc6b3e34a 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -25,17 +25,21 @@ to invent their own wide pointers (and thus custom dynamically sized types). # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -In order to mark a trait as using a custom vtable layout, you apply an attribute to -your trait declaration. +In order to mark a trait (`MyTrait`) as using a custom vtable layout, you implement the `CustomUnsized` trait for `dyn MyTrait`. ```rust -#[custom_wide_pointer = "MyWidePointer"] -trait MyTrait {} +trait MyTrait: SomeSuperTrait { + fn some_fn(&mut self, x: u32) -> i32; +} + +impl CustomUnsized for dyn MyTrait { + type WidePointer = MyWidePointer; + // details later in this RFC +} ``` -`MyWidePointer` is a type which implements the `CustomUnsized` trait and optionally the `CustomUnsize` trait. -`MyWidePointer` is the type that is going to be used for wide pointers to the given trait, so +`MyWidePointer` is the backing type that is going to be used for wide pointers to the given trait, so * `&dyn MyTrait` * `Box` @@ -52,20 +56,17 @@ The `TraitDescription` struct describes the metadata like the list of methods an of super traits. You are actually generating the wide pointer, not a description of it. -Since your `const impl`'s `from` function is being interpreted in the target's environment, all target specific information will match up. +Since your `impl`'s `from` function is being interpreted in the target's environment, all target specific information will match up. Now, you need some information about the `impl` in order to generate your metadata (and your vtable). -You get this information partially from the type directly (the `T` parameter), -and all the `impl` block specific information is encoded in `TraitDescription`, which you get as a const generic parameter. +You get this information directly from the type (the `T` parameter) by adding trait bounds on it. -As an example, consider the function which is what normally generates your metadata. Note that if you have a generic trait, -the `CustomUnsize` and `CustomUnsized` impls need additional generic parameters, one for each parameter of the trait. -See the `[T]` demo further down for an example. +As an example, consider the function which is what normally generates your metadata. ```rust #[repr(C)] -struct Pointer { - ptr: *const (), - vtable: &'static VTable<{num_methods::()}>, +struct Pointer { + ptr: *mut T, + vtable: &'static VTable, } /// If the `owned` flag is `true`, this is an owned conversion like @@ -75,8 +76,8 @@ struct Pointer { /// move away the owned object. The flag allows you to forbid such /// unsizings by triggering a compile-time `panic` with an explanation /// for the user. -unsafe impl const CustomUnsize for Pointer { - fn from(ptr: *const T) -> Self { +unsafe impl const CustomUnsize for T { + fn unsize(ptr: *mut T) -> Pointer { // We generate the metadata and put a pointer to the metadata into // the field. This looks like it's passing a reference to a temporary // value, but this uses promotion @@ -84,7 +85,7 @@ unsafe impl const CustomUnsi // so the value lives long enough. Pointer { ptr, - vtable: &default_vtable::(), + vtable: &default_vtable::(), } } } @@ -92,89 +93,35 @@ unsafe impl const CustomUnsi /// DISCLAIMER: this uses a `Vtable` struct which is just a part of the /// default trait objects. Your own trait objects can use any metadata and /// thus "vtable" layout that they want. -unsafe impl CustomUnsized for Pointer { - fn method_id_to_fn_ptr( - self, - mut idx: usize, - parents: &[usize], - ) -> *const () { - let mut table = IMPL; - for parent in parents { - // we don't support multi-parents yet - assert_eq!(parent, 0); - idx += table.methods.len(); - // Never panics, there are always fewer or equal number of - // parents given as the argument as there are in reality. - table = table.parent.unwrap(); - } - self.vtable.methods[idx] - } +unsafe impl Unsized for dyn MyTrait { fn size_of(self) -> usize { self.vtable.size } fn align_of(self) -> usize { self.vtable.align } - fn drop(self) { - unsafe { - let drop = self.vtable.drop; - drop(&raw mut self.ptr) - } - } - fn self_ptr(self) -> *const () { - self.ptr - } } -// Compute the total number of methods, including super-traits -const fn num_methods< - const IMPL: &'static std::vtable::TraitDescription, ->() -> usize { - let mut n = IMPL.methods.len(); - let mut current = IMPL; - while let Some(next) = current.parent { - n += next.methods.len(); - current = next.parent; +impl Drop for dyn MyTrait { + fn drop(&mut self) { + // using a dummy concrete type for `Pointer` + let ptr = transmute::<&mut dyn Trait, Pointer<()>>(self); + unsafe { + let drop = ptr.vtable.drop; + drop(&raw mut ptr.ptr) + } } - n } -const fn default_vtable< - T, - const IMPL: &'static std::vtable::TraitDescription, ->() -> VTable<{num_methods::()}> { +const fn default_vtable() -> VTable { // `VTable` is a `#[repr(C)]` type with fields at the appropriate // places. - let mut vtable = VTable { + VTable { size: std::mem::size_of::(), align: std::mem::size_of::(), - drop: transmute::(std::ptr::drop_in_place::), - methods: [std::ptr::null(); num_methods::()], - }; - let mut i = 0; - let mut current = IMPL; - loop { - match IMPL.methods.get(i) { - Some(Some(method)) => { - // The `method` variable is a function pointer, but - // cast to `*const ()` in order to support null pointers. - vtable.methods[i] = method; - i += 1; - }, - Some(None) => { - // Method that cannot be called on this vtable - i += 1; - } - None => match current.parent { - Some(next) => { - current = next; - i = 0; - }, - None => break, - } - } + drop: ::drop, + some_fn: fn (&mut T, u32) -> i32, } - vtable } ``` @@ -189,38 +136,21 @@ ideas that could support `CString` -> `CStr` unsizing by allowing `CString` to i `CStr` instead of having a `Deref` impl that converts. ```rust -#[custom_wide_pointer = "CStrPtr"] pub trait CStr {} #[repr(C)] struct CStrPtr { - ptr: *const u8, + ptr: *mut u8, } -impl< - T, - const IMPL: &'static std::vtable::TraitDescription, -> CustomUnsized for CStrPtr { - fn method_id_to_fn_ptr( - self, - method_index: usize, - super_tree_path: usize, - ) -> *const () { - panic!("CStr has no trait methods, it's all inherent methods acting on the pointer") - } +impl CustomUnsized for dyn CStr { fn size_of(self) -> usize { unsafe { strlen(self.ptr) } } fn align_of(self) -> usize { 1 } - fn drop(self) { - // Nothing to drop (just `u8`s) and we are not in charge of dealloc - } - fn self_ptr(self) -> *const () { - self.ptr - } } ``` @@ -230,44 +160,43 @@ We could remove `[T]` (and even `str`) from the language and just make it desuga a `std::slice::Slice` (or `StrSlice`) trait. ```rust -#[custom_wide_pointer = "SlicePtr"] pub trait Slice {} #[repr(C)] struct SlicePtr { - ptr: *const T, + ptr: *mut T, len: usize, } -impl< - T, - const IMPL: &'static std::vtable::TraitDescription, -> CustomUnsized for Slice { - fn method_id_to_fn_ptr( - self, - method_index: usize, - super_tree_path: usize, - ) -> *const () { - panic!("Slices have no trait methods, it's all inherent methods acting on the pointer") +// This impl must be in the `vec` module, to give it access to the `vec` +// internals instead of going through `&mut Vec` or `&Vec`. +impl CustomUnsize> for Vec { + fn unsize(ptr: *mut Vec) -> SlicePtr { + SlicePtr { + ptr: vec.data, + len: vec.len, + } } +} + +impl CustomUnsized for dyn Slice { fn size_of(self) -> usize { self.len } fn align_of(self) -> usize { st::mem::align_of::() } - fn drop(self) { - let mut data_ptr = self.ptr; - for i in 0..self.len { +} + +impl Drop for dyn Slice { + fn drop(&mut self) { + let wide_ptr = transmute::<&mut dyn Slice, SlicePtr>(self); + let mut data_ptr = wide_ptr.ptr; + for i in 0..wide_ptr.len { std::ptr::drop_in_place(data_ptr); data_ptr = data_ptr.offset(1); } } - fn self_ptr(self) -> *const () { - // Not quite sure what to do here, need to think on this. - // Technically I want to return `Self`, but other schemes do not. - panic!("it only makes sense to call this method in order to call a method") - } } ``` @@ -277,41 +206,23 @@ Most of the boilerplate is the same as with regular vtables. ```rust #[repr(C)] -struct Pointer { - ptr: *const (VTable<{num_methods::()}>, ()) +struct Pointer { + ptr: *mut (VTable, T), } -unsafe impl const CustomUnsize for Pointer { - fn from(ptr: *const T) -> Self { +unsafe impl const CustomUnsize> for T { + fn unsize(ptr: *mut T) -> Pointer { unsafe { - let size = std::mem::size_of::(); - let vtable_size = std::mem::size_of::()}>>(); - let layout = Layout::from_size_align(size + vtable_size, std::mem::align_of::()).unwrap(); - let new_ptr = std::alloc::alloc(layout); - // Move the value to the new allocation - std::ptr::copy_nonoverlapping(ptr, new_ptr.offset(vtable_size), 1); + let new = Box::new((default_vtable::(), std::ptr::read(ptr))); std::alloc::dealloc(ptr); - // Copy the vtable into the shared allocation. - // Note that we are reusing the same vtable generation as in - // the regular Rust case. - std::ptr::write(new_ptr, default_vtable::()); - Self { ptr: new_ptr } + Pointer { ptr: Box::into_ptr(new) } } } } -unsafe impl CustomUnsized for Pointer { - fn method_id_to_fn_ptr( - self, - method_index: usize, - super_tree_path: usize, - ) -> *const () { - let meta = unsafe { &*self.ptr }; - // The rest of the function body is the same as with regular - // vtables. - } +unsafe impl CustomUnsized for dyn MyTrait { fn size_of(self) -> usize { unsafe { (*self.ptr).0.size @@ -322,88 +233,64 @@ unsafe impl CustomUnsized fo (*self.ptr).0.align } } - fn drop(self) { - unsafe { - let drop = (*self.ptr).0.drop; - drop(&raw mut (*self.ptr).1) - } +} +``` + + +## zero sized references to MMIO + +Instead of having one type per MMIO register bank, we could have one +trait per bank and use a zero sized wide pointer format. + +```rust +struct NoPointer; + +trait MyRegisterBank { + fn flip_important_bit(&mut self); +} + +unsafe impl CustomUnsized for dyn MyRegisterBank { + fn size_of(self) -> usize { + 4 } - fn self_ptr(self) -> *const () { - unsafe { - &raw const (*self.ptr).1 - } + fn align_of(self) -> usize { + 4 } } -``` +impl MyRegisterBank for dyn MyRegisterBank { + fn flip_important_bit(&mut self) { + std::mem::volatile_write::(42, !std::mem::volatile_read::(42)) + } +} +``` # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -The type `TraitDescription` is `#[nonexhaustive]` in order to allow arbitrary extension in the future. -Instances of the `TraitDescription` struct are created by rustc, basically replacing today's vtable generation, and then outsourcing the actual vtable generation to the const evaluator. - -When unsizing, the `const fn` specified -via `::from` is invoked. -The `TraitDescription` parameter of `CustomUnsize` contains function pointers to the -concrete functions of the `impl`. The `TraitDescription` parameter of `CustomUnsized` -contains function pointers to default impls (if any), or mostly just null pointers. -The same datastructure is shared here to make implementations easier and because they -contain mostly the same data anyway. -`WidePtrType` refers to the argument of the -`#[custom_wide_ptr = "WidePtrType"]` attribute. The only reason that trait must be -`impl const WidePtrType` is to restrict what kind of things you can do in there, it's not -strictly necessary. +When unsizing, the `::unsize` is invoked. +The only reason that trait must be +`impl const CustomUnsize` is to restrict what kind of things you can do in there, it's not +strictly necessary. This restriction may be lifted in the future. -For all other operations, the methods on `` is invoked. +For all other operations, the methods on `` is invoked. -When obtaining function pointers from vtables, instead of computing an offset, the `method_id_to_fn_ptr` function is invoked at runtime and computes a function pointer after being given a method index, the indices of all parents, and a pointer to a metadata field. +When obtaining function pointers from vtables, instead of computing an offset, the `MyTrait for dyn MyTrait` impl's +methods are invoked, allowing users to insert their own logic for obtaining the runtime function pointer. Through the use of MIR optimizations (e.g. inlining), the final LLVM assembly is tuned to be exactly the same as today. +The above statement contains significant hand-waving, but I propose we block the stabilization of this RFC on the +relevant optimizations existing, which will then allow users to reproduce the performance of the built-in unsizing. These types' and trait's declarations are provided below: ```rust -#[nonexhaustive] -struct TraitMethod { - fn_ptr: *const (), - // may get more fields like `name` or even `body` (the latter being a string of the body to be used with `syn`). -} - -#[nonexhaustive] -struct TraitDescription { - pub methods: &'static [TraitMethod], - pub parent: &'static TraitDescription, -} - -unsafe trait CustomUnsized: Copy { - /// Returns a function pointer to the method that - /// is being requested. - /// * The first argument is the method index, - /// * the second argument is a list of indices used to traverse the - /// super-trait tree to find the trait whose method is being invoked, and - /// * the third argument is a pointer to the wide pointer (so in case of trait objects, usually it would be `*const (*const T, &'static Vtable)`). - /// This indirection is necessary, because we don't know the size of the wide pointer. - fn method_id_to_fn_ptr( - self, - method_index: usize, - super_tree_path: usize, - ) -> *const (); +unsafe trait CustomUnsized { fn size_of(self) -> usize; fn align_of(self) -> usize; - fn drop(self); - /// Extracts the `&self` pointer - /// from the wide pointer - /// for calling trait methods. This needs a method as - /// wide pointer layouts may place their `self` pointer - /// anywhere they desire. - fn self_ptr(self) -> *const (); } -unsafe trait CustomUnsize: CustomUnsized { - fn from< - T - const owned: bool, - >(t: *const T) -> Self; +unsafe trait CustomUnsize where WidePtrType: CustomUnsized { + fn unsize(t: *mut Self) -> WidePtrType; } ``` @@ -418,12 +305,6 @@ unsafe trait CustomUnsize: CustomUnsized { This is a frequently requested feature and as a side effect obsoletes `extern type`s which have all kinds of problems (like the inability to invoke `size_of_val` on pointers to them). -## Don't use const generics - -If we wait until we have a heap in const eval, we can use `Vec`s and `Box`es, which would allow us to avoid the const generics scheme, likely making all the code less roundabout. - -We can still expose the current scheme and once we get heap in const eval, we can actually implement a convenience layer in user code. So this is basically like procedural macros, where a stringy API is exposed and user code (`syn`) gives a better API. - # Prior art [prior-art]: #prior-art @@ -443,7 +324,7 @@ This list is shamelessly taken from [strega-nil's Custom DST RFC](https://github - [Pointer Metadata and VTable](https://github.com/rust-lang/rfcs/pull/2580) - [Syntax of ?Sized](https://github.com/rust-lang/rfcs/pull/490) -This RFC differs from all the other RFCs in that it provides a procedural way to generate vtables, +This RFC differs from all the other RFCs in that it focusses on a procedural way to generate vtables, thus also permitting arbitrary user-defined compile-time conditions by aborting via `panic!`. # Unresolved questions @@ -451,28 +332,14 @@ thus also permitting arbitrary user-defined compile-time conditions by aborting - Do we want this kind of flexibility? With power comes responsibility... - I believe we can do multiple super traits, including downcasts with this scheme and no additional extensions, but I need to make sure that's true. -- This scheme could support downcasting `dyn A` to `dyn B` if `trait A: B` if we make `T: ?Sized` (`T` is the `impl` block type). But that will not allow the sized use-cases anymore (since `size_of::` will fail). If we have something like `MaybeSized` that has `size_of` and `align_of` methods returning `Option`, then maybe we could do this. +- This scheme support downcasting `dyn A` to `dyn B` if `trait A: B` if you `impl CustomUnsize for dyn B` +* this scheme allows `dyn A + B`. By implemting `CustomUnsized` for `dyn A + B` * Need to be generic over the allocator, too, so that reallocs are actually sound. * how does this RFC (especially the `owned` flag) interact with `Pin`? # Future possibilities [future-possibilities]: #future-possibilities -* Add a scheme that allows upcasting to super traits that have different vtable generators. - So `trait A: B + C`, where `B` and `C` have different vtable generators and `A` unites them in some manner. - This requires the information about the vtable generators to be part of the `TraitDescription` type. - We can likely even put a function pointer to the vtable generator into the `TraitDescription`. -* Add a scheme allowing `dyn A + B`. I have no idea how, but maybe we just need to add a method to `CustomUnsize` - that allows chaining vtable generators. So we first generate `dyn A`, then out of that we generate `dyn A + B` * We can change string slice (`str`) types to be backed by a `trait StrSlice` which uses this scheme to generate just a single `usize` for the metadata (see also the `[T]` demo). -* We can handle traits with parents which are created by a different metadata generator function, - we just need to figure out how to communicate this to such a function so it can special case this situation. -* We can give the vtable pointer of the super traits to the constructor function, - so it doesn't have to recompute anything and just grab the info off there. - Basically `TraitDescription::parent` would not be a pointer to the parent, - but a struct which contains at least the pointer to the parent and the pointer to the `CustomUnsize::from` function. -* Totally off-topic, but a similar scheme (via const eval) can be used to procedurally generate type declarations. -* We can likely access associated consts and types of the trait directly without causing cycle errors, this should be investigated -* This scheme is forward compatible to adding associated fields later. -* If/once we expose the `Unsize` (do not confuse with `CustomUnsize`) traits on stable, we could consider adding a method to the `Unsize` trait that performs the conversion. This way more complex unsizings like `String` -> `str` could be performed without going through the `Deref` trait which does the conversion. This would allow us to essentially write `impl str for String` if we make `str` a `trait`. We could also move the `CustomUnsize` trait's `from` method's `T` parameter onto the trait, thus allowing users to manually `impl CustomUnsize for CStrPtr`. \ No newline at end of file +* This scheme is forward compatible to adding associated fields later, but it is a breaking change to add such fields to an existing trait. \ No newline at end of file From e7a0ef7801fb96200d292ef576b0538c9b74ee37 Mon Sep 17 00:00:00 2001 From: HackMD Date: Sat, 8 Aug 2020 12:14:57 +0000 Subject: [PATCH 16/22] Fix trait method argument types --- text/0000-procedural-vtables.md | 67 +++++++++++++++------------------ 1 file changed, 30 insertions(+), 37 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index a1fc6b3e34a..54d20fd6071 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -94,11 +94,12 @@ unsafe impl const CustomUnsize for T { /// default trait objects. Your own trait objects can use any metadata and /// thus "vtable" layout that they want. unsafe impl Unsized for dyn MyTrait { - fn size_of(self) -> usize { - self.vtable.size + type WidePointer = Pointer; + fn size_of(ptr: Pointer) -> usize { + ptr.vtable.size } - fn align_of(self) -> usize { - self.vtable.align + fn align_of(ptr: Pointer) -> usize { + ptr.vtable.align } } @@ -138,17 +139,12 @@ ideas that could support `CString` -> `CStr` unsizing by allowing `CString` to i ```rust pub trait CStr {} - -#[repr(C)] -struct CStrPtr { - ptr: *mut u8, -} - impl CustomUnsized for dyn CStr { - fn size_of(self) -> usize { - unsafe { strlen(self.ptr) } + type WidePointer = *mut u8; + fn size_of(ptr: *mut u8) -> usize { + unsafe { strlen(ptr) } } - fn align_of(self) -> usize { + fn align_of(_: *mut u8) -> usize { 1 } } @@ -180,11 +176,12 @@ impl CustomUnsize> for Vec { } impl CustomUnsized for dyn Slice { - fn size_of(self) -> usize { - self.len + type WidePointer = SlicePtr; + fn size_of(ptr: SlicePtr) -> usize { + ptr.len } - fn align_of(self) -> usize { - st::mem::align_of::() + fn align_of(_: SlicePtr) -> usize { + std::mem::align_of::() } } @@ -205,32 +202,28 @@ impl Drop for dyn Slice { Most of the boilerplate is the same as with regular vtables. ```rust -#[repr(C)] -struct Pointer { - ptr: *mut (VTable, T), -} - -unsafe impl const CustomUnsize> for T { - fn unsize(ptr: *mut T) -> Pointer { +unsafe impl const CustomUnsize for T { + fn unsize(ptr: *mut T) -> *mut (Vtable, ()) { unsafe { let new = Box::new((default_vtable::(), std::ptr::read(ptr))); std::alloc::dealloc(ptr); - Pointer { ptr: Box::into_ptr(new) } + Box::into_ptr(new) as *mut _ } } } unsafe impl CustomUnsized for dyn MyTrait { - fn size_of(self) -> usize { + type WidePointer = *mut (VTable, ()); + fn size_of(ptr: Self::WidePointer) -> usize { unsafe { - (*self.ptr).0.size + (*ptr).0.size } } - fn align_of(self) -> usize { + fn align_of(self: Self::WidePointer) -> usize { unsafe { - (*self.ptr).0.align + (*ptr).0.align } } } @@ -243,17 +236,16 @@ Instead of having one type per MMIO register bank, we could have one trait per bank and use a zero sized wide pointer format. ```rust -struct NoPointer; - trait MyRegisterBank { fn flip_important_bit(&mut self); } unsafe impl CustomUnsized for dyn MyRegisterBank { - fn size_of(self) -> usize { + type WidePointer = (); + fn size_of(():()) -> usize { 4 } - fn align_of(self) -> usize { + fn align_of(():()) -> usize { 4 } } @@ -285,12 +277,13 @@ These types' and trait's declarations are provided below: ```rust unsafe trait CustomUnsized { - fn size_of(self) -> usize; - fn align_of(self) -> usize; + type WidePointer: Copy; + fn size_of(ptr: WidePointer) -> usize; + fn align_of(ptr: WidePointer) -> usize; } -unsafe trait CustomUnsize where WidePtrType: CustomUnsized { - fn unsize(t: *mut Self) -> WidePtrType; +unsafe trait CustomUnsize where DynTrait: CustomUnsized { + fn unsize(t: *mut Self) -> DynTrait::WidePointer; } ``` From 70a224c43370be1981707f20ddafebe8bd285590 Mon Sep 17 00:00:00 2001 From: HackMD Date: Sat, 8 Aug 2020 23:12:47 +0000 Subject: [PATCH 17/22] custom index operation results --- text/0000-procedural-vtables.md | 41 ++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 54d20fd6071..05ee51377b0 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -229,8 +229,44 @@ unsafe impl CustomUnsized for dyn MyTrait { } ``` +## Slicing into matrices via the `Index` trait + +The `Index` trait's `index` method returns references. Thus, +when indexing an `ndarray::Array2` in order to obtain a slice +into part of that array, we cannot use the `Index` trait and +have to invoke a function. Right now this means `array.index(s![5..8, 3..])` +in order to obtain a slice that takes indices 5-7 in the first +dimension and all indices after the 3rd dimension. Instead of +having our own `ArrayView` type like what is returned by `Array2::index` +we can create a trait with a custom vtable and reuse the `Index` trait. +We keep using the `ArrayView` type as the type of the wide pointer. -## zero sized references to MMIO +```rust +trait Slice2 {} + +unsafe impl CustomUnsized for dyn Slice2 { + type WidePointer = ndarray::ArrayView; + fn size_of(ptr: WidePointer) -> usize { + ptr.len() + } + fn align_of(ptr: WidePointer) -> usize { + std::mem::align_of::() + } +} + +impl<'a, T, U, V> Index<&'a SliceInfo> for Array2 { + type Output = dyn Slice2; + fn index(&self, idx: &'a SliceInfo) -> &dyn Slice2 { + unsafe { + // This can get a better impl, but for simplicity we reuse + // the existing function. + transmute(Array2::index(idx)) + } + } +} +``` + +## Zero sized references to MMIO Instead of having one type per MMIO register bank, we could have one trait per bank and use a zero sized wide pointer format. @@ -287,11 +323,14 @@ unsafe trait CustomUnsize where DynTrait: CustomUnsized { } ``` +The + # Drawbacks [drawbacks]: #drawbacks * This may be a serious case of overengineering. We're basically taking vtables out of the language and making dynamic dispatch on trait objects a user definable thing. * This may slow down compilation, likely entirely preventable by keeping a special case in the compiler for regular trait objects. +* This completely locks us into never adding multiple vtable formats for a single trait. So you can't use a trait both as a C++ like vtable layout in some situations and a Rust wide pointer layout in others. # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives From 85b9d3605f841fa1298dd9790785d9267fa14bcb Mon Sep 17 00:00:00 2001 From: HackMD Date: Sat, 8 Aug 2020 23:13:59 +0000 Subject: [PATCH 18/22] Remove reference to old version of RFC --- text/0000-procedural-vtables.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 05ee51377b0..897b352a1c6 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -52,8 +52,6 @@ further down in this section you can see how it theoretically could be done in u Likely it will stay compiler magic out of compile-time performance reasons. All `impl`s of `MyTrait` will now use `MyWidePointer` for generating the wide pointer. -The `TraitDescription` struct describes the metadata like the list of methods and a tree -of super traits. You are actually generating the wide pointer, not a description of it. Since your `impl`'s `from` function is being interpreted in the target's environment, all target specific information will match up. From 5f63195a279e1a8fe516f4b829d4ec8deb23a93a Mon Sep 17 00:00:00 2001 From: HackMD Date: Sat, 8 Aug 2020 23:20:44 +0000 Subject: [PATCH 19/22] Various smaller review changes --- text/0000-procedural-vtables.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 897b352a1c6..3e0eda047cf 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -103,9 +103,9 @@ unsafe impl Unsized for dyn MyTrait { impl Drop for dyn MyTrait { fn drop(&mut self) { - // using a dummy concrete type for `Pointer` - let ptr = transmute::<&mut dyn Trait, Pointer<()>>(self); unsafe { + // using a dummy concrete type for `Pointer` + let ptr = transmute::<&mut dyn Trait, Pointer<()>>(self); let drop = ptr.vtable.drop; drop(&raw mut ptr.ptr) } @@ -117,7 +117,7 @@ const fn default_vtable() -> VTable { // places. VTable { size: std::mem::size_of::(), - align: std::mem::size_of::(), + align: std::mem::align_of::(), drop: ::drop, some_fn: fn (&mut T, u32) -> i32, } @@ -176,7 +176,7 @@ impl CustomUnsize> for Vec { impl CustomUnsized for dyn Slice { type WidePointer = SlicePtr; fn size_of(ptr: SlicePtr) -> usize { - ptr.len + ptr.len * std::mem::size_of::() } fn align_of(_: SlicePtr) -> usize { std::mem::align_of::() @@ -185,11 +185,13 @@ impl CustomUnsized for dyn Slice { impl Drop for dyn Slice { fn drop(&mut self) { - let wide_ptr = transmute::<&mut dyn Slice, SlicePtr>(self); - let mut data_ptr = wide_ptr.ptr; - for i in 0..wide_ptr.len { - std::ptr::drop_in_place(data_ptr); - data_ptr = data_ptr.offset(1); + unsafe { + let wide_ptr = transmute::<&mut dyn Slice, SlicePtr>(self); + let mut data_ptr = wide_ptr.ptr; + for i in 0..wide_ptr.len { + std::ptr::drop_in_place(data_ptr); + data_ptr = data_ptr.offset(1); + } } } } @@ -245,7 +247,7 @@ trait Slice2 {} unsafe impl CustomUnsized for dyn Slice2 { type WidePointer = ndarray::ArrayView; fn size_of(ptr: WidePointer) -> usize { - ptr.len() + ptr.len() * std::mem::size_of::() } fn align_of(ptr: WidePointer) -> usize { std::mem::align_of::() From c8c9a8718be09157a428a366e1951115deb0c670 Mon Sep 17 00:00:00 2001 From: HackMD Date: Wed, 12 Aug 2020 16:21:54 +0000 Subject: [PATCH 20/22] Explain how unsized structs work --- text/0000-procedural-vtables.md | 34 +++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 3e0eda047cf..3464db5ed5e 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -293,6 +293,40 @@ impl MyRegisterBank for dyn MyRegisterBank { } ``` +## Unsized structs + +This RFC does not actually affect unsized structs (or tuples for that matter), because the unsizing of aggregates +with trailing unsized types delegates to the unsizing of the built-in unsized types. + +If you have a + +```rust +struct Foo { + a: i32, + b: T, +} +``` + +and you're using it to convert from a pointer to the sized version to an unsized version (Side note: I'm assuming you are talking about sized vs unsized, even though you mentioned "owned". `Box` is also owned, just owning an unsized thing). + +```rust +let x = Foo { a: 42, b: SomeStruct }; +let y: &Foo = &x; +let z: &Foo = y; +``` + +Then two steps happen, the first one is trivial, you get a pointer to `x` and store it in `y`. This is a thin pointer, whose value is the address of `x`. +Then you do the unsizing, which invokes ` as CustomUnsize::>::unsize::(y)`, giving you essentially + +```rust +Pointer { + ptr: &x, + vtable, +} +``` + +where `vtable` is the same vtable you'd get for `&SomeStruct as &dyn MyTrait`. Since you can't invoke `MyTrait` methods on `Foo`, there are no pointer indirection problems or anything. This is also how it works without this RFC. If you want to invoke methods on the `b` field, you have to do `z.b.foo()`, which will give you a `&dyn MyTrait` reference via `&z.b` and then everything works out just like with direct references (well, because now you have a direct reference). + # Reference-level explanation [reference-level-explanation]: #reference-level-explanation From 6c89ffe3afd8aed91f86fdcfe104e88f18e8f2e5 Mon Sep 17 00:00:00 2001 From: HackMD Date: Thu, 10 Sep 2020 16:13:01 +0000 Subject: [PATCH 21/22] Intermediate push (no projection into unsized fields for C++-vtables) --- text/0000-procedural-vtables.md | 148 ++++++++++++++++++++++++-------- 1 file changed, 114 insertions(+), 34 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 3464db5ed5e..8eb32941a35 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -62,9 +62,26 @@ As an example, consider the function which is what normally generates your metad ```rust #[repr(C)] -struct Pointer { - ptr: *mut T, - vtable: &'static VTable, +struct Pointer { + ptr: *mut u8, + vtable: &'static VTable, +} + +/// For going from a `&SomeStruct` to a field of `SomeStruct`. +unsafe impl CustomProjection for Pointer { + unsafe fn project(ptr: Pointer, offset: usize) -> *mut u8 { + ptr.ptr.offset(offset), + } +} + +unsafe impl CustomProjectionToUnsized for Pointer { + /// For going from `&SomeStruct` to the unsized field + unsafe fn project_unsized(ptr: Pointer, offset: usize) -> Pointer { + Pointer { + ptr: ptr.ptr, + vtable: ptr.vtable, + } + } } /// If the `owned` flag is `true`, this is an owned conversion like @@ -75,14 +92,14 @@ struct Pointer { /// unsizings by triggering a compile-time `panic` with an explanation /// for the user. unsafe impl const CustomUnsize for T { - fn unsize(ptr: *mut T) -> Pointer { + fn unsize(ptr: *mut T) -> Pointer { // We generate the metadata and put a pointer to the metadata into // the field. This looks like it's passing a reference to a temporary // value, but this uses promotion // (https://doc.rust-lang.org/stable/reference/destructors.html?highlight=promotion#constant-promotion), // so the value lives long enough. Pointer { - ptr, + ptr: ptr as *mut u8, vtable: &default_vtable::(), } } @@ -92,6 +109,7 @@ unsafe impl const CustomUnsize for T { /// default trait objects. Your own trait objects can use any metadata and /// thus "vtable" layout that they want. unsafe impl Unsized for dyn MyTrait { + // Using a dummy type for the vtable type WidePointer = Pointer; fn size_of(ptr: Pointer) -> usize { ptr.vtable.size @@ -112,7 +130,7 @@ impl Drop for dyn MyTrait { } } -const fn default_vtable() -> VTable { +const fn default_vtable() -> VTable { // `VTable` is a `#[repr(C)]` type with fields at the appropriate // places. VTable { @@ -157,28 +175,43 @@ a `std::slice::Slice` (or `StrSlice`) trait. pub trait Slice {} #[repr(C)] -struct SlicePtr { - ptr: *mut T, +struct SlicePtr { + ptr: *mut u8, len: usize, } +unsafe impl CustomProjection for SlicePtr { + unsafe fn project(ptr: Pointer, offset: usize) -> *mut u8 { + ptr.ptr.offset(offset) + } +} + +unsafe impl CustomProjectionToUnsized for SlicePtr { + unsafe fn project_unsized(ptr: Pointer, offset: usize) -> Pointer { + Pointer { + ptr.ptr.offset(offset), + ptr.len, + } + } +} + // This impl must be in the `vec` module, to give it access to the `vec` // internals instead of going through `&mut Vec` or `&Vec`. -impl CustomUnsize> for Vec { - fn unsize(ptr: *mut Vec) -> SlicePtr { +impl CustomUnsize for Vec { + fn unsize(ptr: *mut Vec) -> SlicePtr { SlicePtr { - ptr: vec.data, + ptr: vec.data as *mut _, len: vec.len, } } } impl CustomUnsized for dyn Slice { - type WidePointer = SlicePtr; - fn size_of(ptr: SlicePtr) -> usize { + type WidePointer = SlicePtr; + fn size_of(ptr: SlicePtr) -> usize { ptr.len * std::mem::size_of::() } - fn align_of(_: SlicePtr) -> usize { + fn align_of(_: SlicePtr) -> usize { std::mem::align_of::() } } @@ -203,27 +236,40 @@ Most of the boilerplate is the same as with regular vtables. ```rust unsafe impl const CustomUnsize for T { - fn unsize(ptr: *mut T) -> *mut (Vtable, ()) { + fn unsize(ptr: *mut T) -> CppPtr { unsafe { let new = Box::new((default_vtable::(), std::ptr::read(ptr))); std::alloc::dealloc(ptr); - Box::into_ptr(new) as *mut _ + CppPtr(Box::into_ptr(new) as *mut _) } } } +struct CppPtr(*mut (Vtable, ())); + +unsafe impl CustomProjection for CppPtr { + unsafe fn project(ptr: CppPtr, offset: usize) -> *mut u8 { + (&raw mut (*ptr.0).1).offset(offset) + } +} + +// No CustomProjectionToUnsized as you can't have +// `struct Foo(i32, dyn MyTrait);` as that would require +// us to rewrite the vtable on unsizing. Rust puts the +// unsized field at the end, while C++ puts the in the front of +// the class. unsafe impl CustomUnsized for dyn MyTrait { - type WidePointer = *mut (VTable, ()); - fn size_of(ptr: Self::WidePointer) -> usize { + type WidePointer = CppPtr; + fn size_of(ptr: CppPtr) -> usize { unsafe { - (*ptr).0.size + (*ptr.0).0.size } } - fn align_of(self: Self::WidePointer) -> usize { + fn align_of(self: CppPtr) -> usize { unsafe { - (*ptr).0.align + (*ptr.0).0.align } } } @@ -269,30 +315,46 @@ impl<'a, T, U, V> Index<&'a SliceInfo> for Array2 { ## Zero sized references to MMIO Instead of having one type per MMIO register bank, we could have one -trait per bank and use a zero sized wide pointer format. +trait per bank and use a zero sized wide pointer format. There's no `Unsize` +impl as you can't create these pointers except by transmuting a zst to them. ```rust +const REG_ADDR: usize = 42; + trait MyRegisterBank { fn flip_important_bit(&mut self); } +struct NoPointer; + +impl CustomProjection for NoPointer { + fn project(NoPointer: NoPointer, offset: usize) -> *mut u8 { + (REG_ADDR + offset) as *mut u8 + } +} + +// No CustomProjectionToUnsized as there's nothing there to access + unsafe impl CustomUnsized for dyn MyRegisterBank { - type WidePointer = (); - fn size_of(():()) -> usize { - 4 + type WidePointer = NoPointer; + fn size_of(NoPointer:NoPointer) -> usize { + 4 // MMIO registers on our hypothetical systems are 32 bit } - fn align_of(():()) -> usize { + fn align_of(NoPointer:NoPointer) -> usize { 4 } } impl MyRegisterBank for dyn MyRegisterBank { fn flip_important_bit(&mut self) { - std::mem::volatile_write::(42, !std::mem::volatile_read::(42)) + std::mem::volatile_write::(REG_ADDR, !std::mem::volatile_read::(REG_ADDR)) } } ``` +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + ## Unsized structs This RFC does not actually affect unsized structs (or tuples for that matter), because the unsizing of aggregates @@ -307,7 +369,7 @@ struct Foo { } ``` -and you're using it to convert from a pointer to the sized version to an unsized version (Side note: I'm assuming you are talking about sized vs unsized, even though you mentioned "owned". `Box` is also owned, just owning an unsized thing). +and you're using it to convert from a pointer to the sized version to an unsized version ```rust let x = Foo { a: 42, b: SomeStruct }; @@ -325,17 +387,22 @@ Pointer { } ``` -where `vtable` is the same vtable you'd get for `&SomeStruct as &dyn MyTrait`. Since you can't invoke `MyTrait` methods on `Foo`, there are no pointer indirection problems or anything. This is also how it works without this RFC. If you want to invoke methods on the `b` field, you have to do `z.b.foo()`, which will give you a `&dyn MyTrait` reference via `&z.b` and then everything works out just like with direct references (well, because now you have a direct reference). +where `vtable` is the same vtable you'd get for `&SomeStruct as &dyn MyTrait`. Since you can't invoke `MyTrait` methods on `Foo`, there are no pointer indirection problems or anything. This is also how it works without this RFC. -# Reference-level explanation -[reference-level-explanation]: #reference-level-explanation +If you want to invoke methods on the `b` field, you have to do `z.b.foo()`, which will works by +invoking `CustomProjectionToUnsized::project_unsized(z, offset!(SomeStruct::b))`. The resulting pointer +is again a wide pointer `&dyn MyTrait`, but with an adjusted data pointer to allow any trait methods to properly +work on the type. This data pointer adjustment is wide pointer specific and overridable via the `CustomProjectionToUnsized` trait. +For regular fields the `CustomProjection` trait handles the extraction of the sized pointer to the field. + +## Traits managing the unsizing and projecting When unsizing, the `::unsize` is invoked. The only reason that trait must be `impl const CustomUnsize` is to restrict what kind of things you can do in there, it's not strictly necessary. This restriction may be lifted in the future. -For all other operations, the methods on `` is invoked. +For all other operations, the methods on `::WidePointer` are invoked. When obtaining function pointers from vtables, instead of computing an offset, the `MyTrait for dyn MyTrait` impl's methods are invoked, allowing users to insert their own logic for obtaining the runtime function pointer. @@ -355,6 +422,16 @@ unsafe trait CustomUnsized { unsafe trait CustomUnsize where DynTrait: CustomUnsized { fn unsize(t: *mut Self) -> DynTrait::WidePointer; } + +unsafe trait CustomProjection: CustomUnsized { + /// The offset is in bytes. + unsafe fn project(ptr: ::WidePointer, offset: usize) -> *mut u8; +} + +unsafe trait CustomProjectionToUnsized: CustomUnsized { + /// The offset is in bytes and must be the exact offset from the start of the unsized struct to its unsized field. + unsafe fn project_unsized(ptr: ::WidePointer, offset: usize) -> ::WidePointer; +} ``` The @@ -391,7 +468,9 @@ This list is shamelessly taken from [strega-nil's Custom DST RFC](https://github - [Syntax of ?Sized](https://github.com/rust-lang/rfcs/pull/490) This RFC differs from all the other RFCs in that it focusses on a procedural way to generate vtables, -thus also permitting arbitrary user-defined compile-time conditions by aborting via `panic!`. +thus also permitting arbitrary user-defined compile-time conditions by aborting via `panic!`. Another +difference is that this RFC allows arbitrary layouts of the wide pointer instead of just allowing custom +metadata fields of wide pointers. # Unresolved questions [unresolved-questions]: #unresolved-questions @@ -408,4 +487,5 @@ thus also permitting arbitrary user-defined compile-time conditions by aborting * We can change string slice (`str`) types to be backed by a `trait StrSlice` which uses this scheme to generate just a single `usize` for the metadata (see also the `[T]` demo). -* This scheme is forward compatible to adding associated fields later, but it is a breaking change to add such fields to an existing trait. \ No newline at end of file +* This scheme is forward compatible to adding associated fields later, but it is a breaking change to add such fields to an existing trait. +* We can add a scheme for safely converting from a wide pointer to its representation struct. \ No newline at end of file From e0c8ca73b30a483bc8d247a1f82b33c60b86bc88 Mon Sep 17 00:00:00 2001 From: HackMD Date: Thu, 10 Sep 2020 18:10:10 +0000 Subject: [PATCH 22/22] Address some review nits --- text/0000-procedural-vtables.md | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/text/0000-procedural-vtables.md b/text/0000-procedural-vtables.md index 8eb32941a35..e5d7e822b4d 100644 --- a/text/0000-procedural-vtables.md +++ b/text/0000-procedural-vtables.md @@ -78,7 +78,7 @@ unsafe impl CustomProjectionToUnsized for Pointer { /// For going from `&SomeStruct` to the unsized field unsafe fn project_unsized(ptr: Pointer, offset: usize) -> Pointer { Pointer { - ptr: ptr.ptr, + ptr: ptr.ptr.offset(offset), vtable: ptr.vtable, } } @@ -122,7 +122,7 @@ unsafe impl Unsized for dyn MyTrait { impl Drop for dyn MyTrait { fn drop(&mut self) { unsafe { - // using a dummy concrete type for `Pointer` + // using a dummy concrete type for `Pointer` let ptr = transmute::<&mut dyn Trait, Pointer<()>>(self); let drop = ptr.vtable.drop; drop(&raw mut ptr.ptr) @@ -142,7 +142,7 @@ const fn default_vtable() -> VTable { } ``` -Now, if you want to implement a fancier vtable, this RFC enables you to do that. +If you want to implement a fancier vtable, this RFC enables you to do that. ## Null terminated strings (std::ffi::CStr) @@ -277,15 +277,14 @@ unsafe impl CustomUnsized for dyn MyTrait { ## Slicing into matrices via the `Index` trait -The `Index` trait's `index` method returns references. Thus, -when indexing an `ndarray::Array2` in order to obtain a slice -into part of that array, we cannot use the `Index` trait and -have to invoke a function. Right now this means `array.index(s![5..8, 3..])` -in order to obtain a slice that takes indices 5-7 in the first -dimension and all indices after the 3rd dimension. Instead of -having our own `ArrayView` type like what is returned by `Array2::index` -we can create a trait with a custom vtable and reuse the `Index` trait. -We keep using the `ArrayView` type as the type of the wide pointer. +We cannot use the `Index` trait to obtain a slice of an `ndarray::Array2` +because the `Index` trait's `index` method returns references. As a workaround, +we have to invoke a function, such as `array.index(s![5..8, 3..])` to obtain +a slice that takes indices 5-7 in the first dimension and all indices after +the 3rd in the second dimension. +Instead of having our own `ArrayView` type, we can create a trait with a +custom vtable and use the `Index` trait. We reuse the `ArrayView` type +as the type of the wide pointer. ```rust trait Slice2 {} @@ -397,7 +396,7 @@ For regular fields the `CustomProjection` trait handles the extraction of the si ## Traits managing the unsizing and projecting -When unsizing, the `::unsize` is invoked. +When unsizing, the `::unsize` method is invoked. The only reason that trait must be `impl const CustomUnsize` is to restrict what kind of things you can do in there, it's not strictly necessary. This restriction may be lifted in the future.