Auto merge of #129313 - RalfJung:coroutine-niches, r=compiler-errors

Supress niches in coroutines to avoid aliasing violations

As mentioned [here](rust-lang/rust#63818 (comment)), using niches in fields of coroutines that are referenced by other fields is unsound: the discriminant accesses violate the aliasing requirements of the reference pointing to the relevant field. This issue causes [Miri errors in practice](rust-lang#3780).

The "obvious" fix for this is to suppress niches in coroutines. That's what this PR does. However, we have several tests explicitly ensuring that we *do* use niches in coroutines. So I see two options:
- We guard this behavior behind a `-Z` flag (that Miri will set by default). There is no known case of these aliasing violations causing miscompilations. But absence of evidence is not evidence of absence...
- (What this PR does right now.) We temporarily adjust the coroutine layout logic and the associated tests until the proper fix lands. The "proper fix" here is to wrap fields that other fields can point to in [`UnsafePinned`](rust-lang/rust#125735) and make `UnsafePinned` suppress niches; that would then still permit using niches of *other* fields (those that never get borrowed). However, I know that coroutine sizes are already a problem, so I am not sure if this temporary size regression is acceptable.

`@compiler-errors` any opinion? Also who else should be Cc'd here?

tests/pass/async-niche-aliasing.rs

@@ -0,0 +1,66 @@
+//@revisions: stack tree
+//@[tree]compile-flags: -Zmiri-tree-borrows
+
+use std::{
+    future::Future,
+    pin::Pin,
+    sync::Arc,
+    task::{Context, Poll, Wake},
+    mem::MaybeUninit,
+};
+
+struct ThingAdder<'a> {
+    // Using `MaybeUninit` to ensure there are no niches here.
+    thing: MaybeUninit<&'a mut String>,
+}
+
+impl Future for ThingAdder<'_> {
+    type Output = ();
+
+    fn poll(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<Self::Output> {
+        unsafe {
+            **self.get_unchecked_mut().thing.assume_init_mut() += ", world";
+        }
+        Poll::Pending
+    }
+}
+
+fn main() {
+    let mut thing = "hello".to_owned();
+    // This future has (at least) two fields, a String (`thing`) and a ThingAdder pointing to that string.
+    let fut = async move { ThingAdder { thing: MaybeUninit::new(&mut thing) }.await };
+
+    let mut fut = MaybeDone::Future(fut);
+    let mut fut = unsafe { Pin::new_unchecked(&mut fut) };
+
+    let waker = Arc::new(DummyWaker).into();
+    let mut ctx = Context::from_waker(&waker);
+    // This ends up reading the discriminant of the `MaybeDone`. If that is stored inside the
+    // `thing: String` as a niche optimization, that causes aliasing conflicts with the reference
+    // stored in `ThingAdder`.
+    assert_eq!(fut.as_mut().poll(&mut ctx), Poll::Pending);
+    assert_eq!(fut.as_mut().poll(&mut ctx), Poll::Pending);
+}
+
+struct DummyWaker;
+
+impl Wake for DummyWaker {
+    fn wake(self: Arc<Self>) {}
+}
+
+pub enum MaybeDone<F: Future> {
+    Future(F),
+    Done,
+}
+impl<F: Future<Output = ()>> Future for MaybeDone<F> {
+    type Output = ();
+
+    fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
+        unsafe {
+            match *self.as_mut().get_unchecked_mut() {
+                MaybeDone::Future(ref mut f) => Pin::new_unchecked(f).poll(cx),
+                MaybeDone::Done => unreachable!(),
+            }
+        }
+    }
+}