syntect handles push & immediately pop non-consuming patterns differently to ST

[keith-hall]

In ST, the following syntax:

%YAML 1.2
---
# See http://www.sublimetext.com/docs/3/syntax.html
name: push_pop_test
scope: source.push_pop_test
contexts:
  main:
    - match: (?=hello)
      push: test
  test:
    - match: (?!world)
      pop: true
    - match: \w+
      scope: comment.matched

accompanied by the following test passes:

# SYNTAX TEST "push_pop_test.sublime-syntax"
hello
# ^^^ comment.matched

in syntect, it doesn't:

loading syntax definitions from testdata/push_pop_test.sublime-syntax
Testing file testdata/syntax_test_push_pop_test.txt
The test file references syntax definition file: push_pop_test.sublime-syntax
-- debugging line 1 -- scope stack: ScopeStack { clear_stack: [], scopes: [] }
# SYNTAX TEST "push_pop_test.sublime-syntax"
^ +source.push_pop_test
-- debugging line 2 -- scope stack: ScopeStack { clear_stack: [], scopes: [<source.push_pop_test>] }
no operations for this line...
  Assertion selector "comment.matched" from line 3 failed against line 2, column range 2-5 (with text "llo") has scope [<source.push_pop_test>]
Ok(FailedAssertions(3, 3))
exiting with code 1

I believe this is due to how ST handles a match that:

• consumes no characters and
• pushes one context
• where that context has a match which immediately pops without consuming any characters

In ST, it ignores the pop until at least one character has been consumed. Currently, syntect seems to push, then pop, then ignore the push again until at least one char has been consumed.

Related reading: sublimehq/sublime_text#1906 (comment)

This is the cause for most of the C# syntax test failures in syntect alluded to in #124 (comment) since sublimehq/Packages@e29d674#diff-9b5e53d4e14d87efc8f78017464d6b94.

[keith-hall]

I've made a PR to the Packages repo that fixes the C# syntax so that all syntax tests pass in syntect: sublimehq/Packages#1359

[trishume]

I think I also know how to fix this. I don't remember exactly, I'll have to look at the code, but syntect does something like detecting cycles, whereas it seems that Sublime just stops you from reversing direction without advancing. It should be simple to switch to the way Sublime does it.

[robinst]

Looking into this. While reading the code and learning how it works, I've created #144 to make the main method a bit smaller and hopefully easier to understand.

[robinst]

Fix released in 2.0.1!