compiler diverges without error message when generic function has infinite instantiations

[carl-eastlund]

The program below causes the compiler to diverge because instantiating generic requires an unbounded number of instantiations. Unfortunately, divergence provides no information to the user about what went wrong. Hopefully there is some way for the compiler to detect this scenario and report it.

// If the definition of Data is simpler, the compiler attempts to inline
// generic<T>, fails, and produces an error.
struct Data(~Option<Data>);

fn generic<T>( _ : ~[(Data,T)] ) {
    // The important detail is substituting (bool,T) for T on each
    // recursive call so the number of representations necessary to compile
    // this function is infinite.
    let rec : ~[(Data,(bool,T))] = ~[];
    generic( rec );
}

fn main () {
    // Use generic<T> at least once to trigger instantiation.
    let input : ~[(Data,())] = ~[];
    generic(input);
}

[huonw]

This looks like polymorphic recursion, and so is covered by #4287.

Closing, feel free to reopen if I've misinterpreted.

[nikomatsakis]

If there is no error message, this is a bug.

[nikomatsakis]

That is, a separate bug.

[alexcrichton]

This now emits a reasonable error:

foo.rs:5:1: 11:2 error: reached the recursion limit during monomorphization

Flagging as needstest.

[thestinger]

26398b4 fixed this and included a test for it, although I don't know if it covers enough.

[ghost]

The original example from this issue is also already included in the test suite: https://github.com/rust-lang/rust/blob/776c17f476c4be92f6cfe4dab528886973ea8c03/src/test/compile-fail/issue-8727.rs so I think this can be closed.