impl-trait return type is bounded by all input type parameters, even when unnecessary

[Arnavion]

rustc 1.20.0-nightly (f590a44ce 2017-06-27)
binary: rustc
commit-hash: f590a44ce61888c78b9044817d8b798db5cd2ffd
commit-date: 2017-06-27
host: x86_64-pc-windows-msvc
release: 1.20.0-nightly
LLVM version: 4.0

trait Future {}

impl<F> Future for Box<F> where F: Future + ?Sized {}

struct SomeFuture<'a>(&'a Client);
impl<'a> Future for SomeFuture<'a> {}

struct Client;
impl Client {
    fn post<'a, B>(&'a self, _body: &B) -> impl Future + 'a /* (1) */ {
        SomeFuture(self)
    }
}

fn login<'a>(client: &'a Client, username: &str) -> impl Future + 'a {
    client.post(&[username])
}

fn main() {
    let client = Client;
    let _f = {
        let username = "foo".to_string();
        login(&client, &username)
    };
}

Since SomeFuture borrows 'a Client, I'd expect impl Future + 'a to be the correct return type, but it gives this error:

error[E0700]: hidden type for `impl Future + 'a` captures lifetime that does not appear in bounds
  --> src/main.rs:16:5
   |
15 | fn login<'a>(client: &'a Client, username: &str) -> impl Future + 'a {
   |                                            ----     ---------------- opaque type defined here
   |                                            |
   |                                            hidden type `impl Future + 'a` captures the anonymous lifetime defined here
16 |     client.post(&[username])
   |     ^^^^^^^^^^^^^^^^^^^^^^^^
   |
help: add a `use<...>` bound to explicitly capture `'_`
   |
15 | fn login<'a>(client: &'a Client, username: &str) -> impl Future + 'a + use<'a, '_> {
   |                                                                      +++++++++++++

• Changing _body to have an explicit lifetime like _body: &'b B where 'b is independent of 'a or where 'a: 'b does not change the error. This and the original error make it seem that returning an impl trait is somehow causing the _body parameter to get the 'a lifetime, even though it's clearly unused, let alone used in a way that it would require the 'a lifetime.

• Changing (1) from impl Future + 'a to SomeFuture<'a> fixes it.

• Changing (1) from impl Future + 'a to Box<Future + 'a> and returning Box::new(SomeFuture(self)) fixes it.

[Arnavion]

From some experimentation, it seems to be because fn foo<T>(_: T) -> impl Bar compiles to something like fn foo<T>(_: T) -> impl Bar + 't, where 't is the lifetime of T. (I don't think this relationship can be expressed in regular Rust code, though Ralith on IRC suggested one could consider the anonymous type to contain PhantomData<T>)

Thus in the original code fn post<'a, B>(&'a self, _body: &B) -> impl Future + 'a effectively forces the return type to be bounded by B's lifetime in addition to 'a. This is why changing _body: &B to _body: B does not change anything, nor does annotating the parameter as _body: &'b B for an unconstrained 'b

---

In light of that, here's a smaller repro:

#![feature(conservative_impl_trait)]

trait Tr { }

struct S;
impl Tr for S { }

fn foo<T>(_t: T) -> impl Tr {
    S
}

struct S2;

fn main() {
    let _bar = {
        let s2 = S2;
        foo(&s2)
    };
}

which complains that s2 is dropped while still borrowed because the compiler thinks it needs to live as long as _bar.

[cramertj]

This is an intentional restriction. See RFC 1951 for the reasoning.

[Arnavion]

Sure. So can there be a way that I can convince the compiler that Client::post's and foo's results don't depend on all the inputs? Maybe it can be made to not override explicit lifetimes like it does in the Client::post example? (I recognize this won't help for the foo example since there is no way to ascribe a lifetime to T.)

The reasoning in the RFC is that eventually there will be syntax that provides control over which lifetime are and are not part of the returned existential ("Assumption 1"). But for the OP example where there already is an explicit lifetime and bound so it could be made to work today.

[cramertj]

Oh, I understand what you're saying now. Yes, if the return type contains an explicit lifetime bound, the compiler should be able to understand that the returned type outlives that lifetime bound. Currently, it cannot do that if there are type parameters involved. This should be fixed. Thanks for the report!

[Arnavion]

This is fixed by existential_type

• existential type PostFuture<'a>: Future + 'a; fn post<'a, B>(&'a self, _body: &B) -> PostFuture<'a> {

• existential type SomeTr: Tr; fn foo<T>(_t: T) -> SomeTr {

The fact that existential types intentionally don't have the "generic lifetime inheriting" behavior that impl trait has is documented here.

---

Updated 2023-10-09 for current syntax:

• type PostFuture<'a> = impl std::future::Future<Output = ()> + 'a; fn post<'a, B>(&'a self, _body: &B) -> PostFuture<'a> { Note that the let f = client.post(); async { f.await } construction is required. Just client.post() or let f = client.post(); f fail with "opaque type's hidden type cannot be another opaque type from the same scope"

• type SomeTr = impl Tr; fn foo<T>(_t: T) -> SomeTr {

[alexreg]

@cramertj @pnkfelix This should probably be closed for the same reason as #53450, if I'm not mistaken?

[cramertj]

No, this is still a bug. The fact that existential type provides a workaround does not mean that the code here shouldn't work-- if you say that your return type outlives some lifetime, then it shouldn't also be bound by the lifetime of a type that cannot be contained in the return type.

e.g. this doesn't compile today and shouldn't:

trait X {}
impl<T> X for T {}
fn foo<'a, T>(x: &'a u8, t: T) -> impl X + 'a {
    (x, t)
}

You have to explicitly add T: 'a in order for the return type to satisfy the impl X + 'a bound. If T didn't already outlive 'a, it couldn't appear in the return type.

[alexreg]

@cramertj Okay, so this is an issue with a lifetime inference, right? (Not actual bounds checking.) Would you mind writing up mentoring instructions so someone can tackle this? (Maybe even me.)

[alexreg]

@nikomatsakis Can we do something about this soon you think? :-) Seems kind of urgent to me, though perhaps this affects me more than most users.

[aidanhs]

Just to check, this is the same issue right?

use std::path::Path;

trait RR {}
impl RR for () {}

fn foo<'a>(path: &'a Path) -> impl RR + 'static {
    bar(path)
}
fn bar<P: AsRef<Path>>(path: P) -> impl RR + 'static {
    ()
}

https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=0a65678ab18f34888291c9c514ec2834

gives

error: cannot infer an appropriate lifetime
 --> src/lib.rs:7:9
  |
6 | fn foo<'a>(path: &'a Path) -> impl RR + 'static {
  |                               ----------------- this return type evaluates to the `'static` lifetime...
7 |     bar(path)
  |         ^^^^ ...but this borrow...
  |
note: ...can't outlive the lifetime 'a as defined on the function body at 6:8
 --> src/lib.rs:6:8
  |
6 | fn foo<'a>(path: &'a Path) -> impl RR + 'static {
  |        ^^
help: you can add a constraint to the return type to make it last less than `'static` and match the lifetime 'a as defined on the function body at 6:8
  |
6 | fn foo<'a>(path: &'a Path) -> impl RR + 'static + 'a {
  |                               ^^^^^^^^^^^^^^^^^^^^^^

(I also think the suggestion is misleading - won't adding a 'a after 'static' be a no-op since it will choose the longer of the two?)

[cramertj]

@aidanhs Yes to both (well, the answer to the second question is a bit more complicated, but "it won't work" is correct ;) ).

[Marwes]

Also encountered this issue with the following (minimised) code. Also found a workaround by specifying a dummy trait and moving the (static) lifetime there.

Errors

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=56e0e8506c1b32ee44d471d5cb6ed215

trait Parser2 {
    type Input;
    type PartialState;
}

struct Test<I>(::std::marker::PhantomData<fn(I)>);

impl<I> Parser2 for Test<I> {
    type Input = I;
    type PartialState = ();
}

fn line<'a, I>() -> impl Parser2<Input = I, PartialState = impl Send + 'static> {
    Test(::std::marker::PhantomData)
}

fn status<'a, I>() -> impl Parser2<Input = I, PartialState = impl Send + 'static> {
    line()
}

fn main() {
}

Works

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=74f4abad271f12df1d79a133007ee07f

trait Parser2 {
    type Input;
    type PartialState;
}

struct Test<I>(::std::marker::PhantomData<fn(I)>);

impl<I> Parser2 for Test<I> {
    type Input = I;
    type PartialState = ();
}

trait Static: Send + 'static {}
impl<T> Static for T where T: Send + 'static {}

fn line<'a, I>() -> impl Parser2<Input = I, PartialState = impl Static> {
    Test(::std::marker::PhantomData)
}

fn status<'a, I>() -> impl Parser2<Input = I, PartialState = impl Static> {
    line()
}

[alexreg]

@nikomatsakis Would be curious to get your thoughts on this along with the other not-too-dissimilar covariant lifetimes issue.

[nikomatsakis]

I've often found it pretty useful to model impl Trait in terms of associated types. If we include the effects of the default keyword, in fact, we get a very close modeling (without the inference), except for "auto trait propagation". I think this modeling is relevant here.

the model

Basically an impl Trait instance like

fn post<'a, B>(&'a self, _body: &B) -> impl Future + 'a { .. }

can be modeled as if there were a "one-off" trait with a single impl:

trait Post<'a, B> {
    //     ^^^^ these are the "captured" parameters, per RFC 1951
    type Output: Future + 'a;
}

impl<'a, B> Post for () {
    default type Output = /* the hidden type that compiler infers */;
}

and the function were then declared like so:

fn post<'a, B>(&'a self, _body: &B) -> <() as Post>::Output { .. }

Why does this matter?

Look at the next function, login:

fn login<'a>(client: &'a Client, username: &str) -> impl Future + 'a { .. }

this function winds up inferring that the hidden type X is <() as Post<'0, B>>::Output for some fresh inference variable '0, and hence we must infer that X: Future and X: 'a. Since the Output is declared as default in the impl, we can't fully normalize it, and must instead rely on the declared bounds (type Output: Future + 'a, where the 'a here will be the variable '0).

The bug report here seems correct: we should be able to prove that X: 'a by adding the requirement that '0: 'a and not requiring B: 'a. And if you try to write out the model I wrote above, you'll find that the compiler does so (at least I expect you will).

How does this work for associated types?

The compiler handles similar problems already for associated types. The rules were outlined in RFC 1214, along with some of the challenges. I think we should probably be able to apply the compiler's existing heuristics to this problem, but it might take a bit of work.

The relevant function is here:

rust/src/librustc/infer/outlives/obligations.rs

Lines 357 to 362 in daa53a5

	fn projection_must_outlive(
	&mut self,
	origin: infer::SubregionOrigin<'tcx>,
	region: ty::Region<'tcx>,
	projection_ty: ty::ProjectionTy<'tcx>,
	) {

In particular, I believe this heuristic is the one that will help:

rust/src/librustc/infer/outlives/obligations.rs

Lines 445 to 459 in daa53a5

	// If we found a unique bound `'b` from the trait, and we
	// found nothing else from the environment, then the best
	// action is to require that `'b: 'r`, so do that.
	//
	// This is best no matter what rule we use:
	//
	// - OutlivesProjectionEnv: these would translate to the requirement that `'b:'r`
	// - OutlivesProjectionTraitDef: these would translate to the requirement that `'b:'r`
	// - OutlivesProjectionComponent: this would require `'b:'r`
	// in addition to other conditions
	if !trait_bounds.is_empty()
	&& trait_bounds[1..]
	.iter()
	.chain(approx_env_bounds.iter().map(|b| &b.1))
	.all(|b| *b == trait_bounds[0])

[chpio]

this stackoverflow question seems related + a funny error message from the compiler about the lifetime, that could be changed?!

[pnkfelix]

tagging with wg-async with hopes that it will be enqueued as a polish issue for a volunteer from that group to work on.

@rustbot label: +wg-async

[traviscross]

The important thing to understand about this issue is that it doesn't have anything to do with the lifetime bounds. The lifetime bounds work exactly as you would expect that they would. Consider this minimized version of the issue:

fn capture<'o, T>(_t: T) -> impl Send + 'o {}
fn outlives<'o, T: 'o>(_t: T) {}

fn test<'o>(x: ()) -> impl Send + 'o {
    outlives::<'o, _>(capture::<'o, &'_ ()>(&x)); // OK.
    capture::<'o, &'_ ()>(&x)
//                        ^^
// ERROR: Borrowed value does not live long enough.
}

Playground link

The line labeled "OK" compiles fine. This shows that the opaque type returned by capture does in fact outlive 'o.

However, Rust won't let a value of this type escape up the stack (without type erasure) because there's no way to give the captured lifetime a name that's valid outside of the function.

A full analysis of this subtle issue is available here.

[Rua]

How can I tell Rust that no, the return type does not, in fact, capture any input lifetimes, and is entirely static?

[Arnavion]

How can I tell Rust that no, the return type does not, in fact, capture any input lifetimes, and is entirely static?

#42940 (comment)

[Rua]

How can I tell Rust that no, the return type does not, in fact, capture any input lifetimes, and is entirely static?

#42940 (comment)

Sadly that doesn't work when I try it, it says the feature is unstable.

[Arnavion]

If it was possible to use TAIT in stable this issue wouldn't still be open.

[davidzeng0]

video.webm

Wild that removing mut gets rid of the error
Same behavior on nightly

Maybe someone can look into this, I'm relatively new to rust and I don't think mut should have any affect on lifetime checks. Probably a compiler bug

Edit: on second thought maybe the pointer cast is causing it, not entirely sure

Either way, it should work as all lifetimes are bounded to the impl trait, yet it doesn't