Panicked at 'attempt to subtract with overflow', src/parsing/parser.rs

[robinst]

I got a panic when parsing a Java file, here's a minimal reproduction case:

$ echo '// TODO "%>"' > Minimal.java
$ RUST_BACKTRACE=1 cargo run --example synstats Minimal.java

Stack trace:

thread 'main' panicked at 'attempt to subtract with overflow', src/parsing/parser.rs:147:41
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:68
             at src/libstd/sys_common/backtrace.rs:57
             at src/libstd/panicking.rs:381
   2: std::panicking::default_hook
             at src/libstd/panicking.rs:397
   3: std::panicking::begin_panic
             at src/libstd/panicking.rs:577
   4: std::panicking::begin_panic
             at src/libstd/panicking.rs:538
   5: std::panicking::try::do_call
             at src/libstd/panicking.rs:522
   6: std::panicking::try::do_call
             at src/libstd/panicking.rs:498
   7: <core::ops::range::Range<Idx> as core::fmt::Debug>::fmt
             at src/libcore/panicking.rs:71
   8: <core::ops::range::Range<Idx> as core::fmt::Debug>::fmt
             at src/libcore/panicking.rs:51
   9: syntect::parsing::parser::ParseState::parse_next_token
             at src/parsing/parser.rs:147
  10: <F as core::str::pattern::Pattern<'a>>::into_searcher
             at src/parsing/parser.rs:122
  11: synstats::count
             at examples/synstats.rs:138
  12: synstats::main
             at examples/synstats.rs:163
  13: std::rt::lang_start::{{closure}}
             at /Users/travis/build/rust-lang/rust/src/libstd/rt.rs:74
  14: std::panicking::try::do_call
             at src/libstd/rt.rs:59
             at src/libstd/panicking.rs:480
  15: panic_unwind::dwarf::eh::read_encoded_pointer
             at src/libpanic_unwind/lib.rs:101
  16: std::sys_common::bytestring::debug_fmt_bytestring
             at src/libstd/panicking.rs:459
             at src/libstd/panic.rs:365
             at src/libstd/rt.rs:58
  17: std::rt::lang_start
             at /Users/travis/build/rust-lang/rust/src/libstd/rt.rs:74
  18: <synstats::Stats as core::default::Default>::default

In a release build the error looks like this:

thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 18446744073709551615'

It looks like self.stack is empty here: https://github.com/trishume/syntect/blob/v2.0.0/src/parsing/parser.rs#L147

(CC @keith-hall)

[keith-hall]

I'm guessing the main context gets pushed off the stack due to https://github.com/sublimehq/Packages/blob/1c1aa090b6e892620faf4c05e29dc4797deff59e/Java/Java.sublime-syntax#L26-L28. I thought we re-push it like ST does when it is the root syntax, but I guess it's not working properly in this case due to the lookahead not consuming anything. Nice find @robinst.
EDIT: #127 could be related

[robinst]

Turns out #127 is not related, but @keith-hall's comment helped me figure out what was wrong. See fix in #148.