ICE with "-Z ast-json": Attempted to encode LazyTokenStream

[dgrunwald]

Code

https://github.com/dgrunwald/rust-cpython
Master branch; run make PY=3 test

Meta

rust-cpython has automated tests using rustc -Z ast-json to check that all declared symbols in the python3-sys library really exist in the .so file. These tests recently started failing with an ICE in rustc.

According to cargo-bisect-rustc, this is a regression in 22e6b9c (#77250)

Error output

   Compiling python3-sys v0.5.1 (/mnt/c/work/rust-cpython/python3-sys)
thread 'rustc' panicked at 'Attempted to encode LazyTokenStream', /rustc/ffa2e7ae8fbf9badc035740db949b9dae271c29f/compiler/rustc_ast/src/tokenstream.rs:178:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.49.0-nightly (ffa2e7ae8 2020-10-24) running on x86_64-unknown-linux-gnu

note: compiler flags: -Z ast-json -C embed-bitcode=no -C debuginfo=2 -C incremental --crate-type lib

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack

Backtrace

stack backtrace:
   0: std::panicking::begin_panic
   1: <rustc_ast::tokenstream::LazyTokenStreamInner as rustc_serialize::serialize::Encodable<S>>::encode
   2: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_struct
   3: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_seq
   4: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_struct
   5: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_enum
   6: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_struct
   7: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_seq
   8: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_struct
   9: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_enum
  10: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_struct
  11: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_seq
  12: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_struct
  13: <rustc_serialize::json::Encoder as rustc_serialize::serialize::Encoder>::emit_struct
  14: <rustc_serialize::json::AsJson<T> as core::fmt::Display>::fmt
  15: core::fmt::write
             at /rustc/ffa2e7ae8fbf9badc035740db949b9dae271c29f/library/core/src/fmt/mod.rs:1076:17
  16: std::io::Write::write_fmt
             at /rustc/ffa2e7ae8fbf9badc035740db949b9dae271c29f/library/std/src/io/mod.rs:1516:15
  17: <&std::io::stdio::Stdout as std::io::Write>::write_fmt
             at /rustc/ffa2e7ae8fbf9badc035740db949b9dae271c29f/library/std/src/io/stdio.rs:655:9
  18: <std::io::stdio::Stdout as std::io::Write>::write_fmt
             at /rustc/ffa2e7ae8fbf9badc035740db949b9dae271c29f/library/std/src/io/stdio.rs:629:9
  19: std::io::stdio::print_to::{{closure}}
             at /rustc/ffa2e7ae8fbf9badc035740db949b9dae271c29f/library/std/src/io/stdio.rs:990:28
  20: core::option::Option<T>::unwrap_or_else
             at /rustc/ffa2e7ae8fbf9badc035740db949b9dae271c29f/library/core/src/option.rs:427:21
  21: std::io::stdio::print_to
             at /rustc/ffa2e7ae8fbf9badc035740db949b9dae271c29f/library/std/src/io/stdio.rs:971:18
  22: std::io::stdio::_print
             at /rustc/ffa2e7ae8fbf9badc035740db949b9dae271c29f/library/std/src/io/stdio.rs:1005:5
  23: rustc_interface::passes::configure_and_expand_inner
  24: rustc_interface::passes::configure_and_expand::{{closure}}
  25: rustc_data_structures::box_region::PinnedGenerator<I,A,R>::new
  26: rustc_interface::passes::configure_and_expand
  27: rustc_interface::queries::Queries::expansion
  28: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  29: rustc_span::with_source_map
  30: rustc_interface::interface::create_compiler_and_run
  31: scoped_tls::ScopedKey<T>::set
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[matthiaskrgr]

No idea if this is the underlying issue but I found that for example this snippet crashes with -Zast-json

fn main() {}

macro_rules! mac_extern {
    ($i:item) => {
        extern "C" { $i }
    }
}

mac_extern! {
    fn foo();
}

[Aaron1011]

@matthiaskrgr Thanks for the minimal reproduction!

I've opened #78408 with a fix, which also fixes @dgrunwald's issue