Syntax highlighting variables

[stevecopley]

Can't find any prior mention of this in the discussions, so...

To aid understanding the concept of user-created variables, it would be helpful to subtly highlight variables within print statements, so that they are visually different to the surrounding text. Students could then see that their variables are placed within the text.

[Felienne]

Hi @stevecopley!

This is a lovely idea (that we have also thought of) but it is technically not really viable, since we use the Ace syntax highlighter which operates entirely separate from our parser. We can't really change the highlighter dynamically after each parse.

If you want to take a stab at it, go ahead, but I think it will be very tricky.

[stevecopley]

I don't really have the skills to have a useful stab at this! But I was having a poke around and thinking about it...

As an aside... Ace doesn't correctly identify variables in Python code, and many other languages - it marks any non-keyword with a class of 'ace_identifier', a catch-all for any unrecognised words. In languages where variables are clearly identifiable, like PHP (prefixed with '$'), Ace correctly marks them with a class of 'ace_variable'.

If I manually add a <span class='ace_variable'> to variables in Hedy code, the appropriate CSS kicks in and they are highlighted nicely:

I'm guessing your parser runs on the back-end when the code is submitted for execution, so the front-end (i.e. the editor) has no knowledge of what's a variable (as you said).

But there is a possible solution kludge: You would need to modify the HTML generated by Ace as the code is edited.

To do this, you could run some JS triggered on each update in the editor to crudely parse the code and generate a variable list. For English it could be a pretty simple regex: /^(\w+)\sis\s.*$/gm or something like that. You'd need a bunch of different regexs, one for every language, but they should all be similar (since, for the lower-level Hedy code, variable are always created the same way, e.g. with the 'is' keyword).

Then you could loop through the variable list and update the DOM to apply the appropriate <span> to any matching text in the editor.

All of this would add some additional load to the front-end, but given that mode Hedy code will be relatively short snippets, it might be ok... Maybe!

[Felienne]

Wow thanks for diving in @stevecopley!! Someone like @ToniSkulj might feel like diving into this!

[Felienne]

We might not even need the (not so pretty and not-localized) regex because we already send a list of variable to the front-end to put into the watch list:

So we just also have to add them to the html page in between an ace variable and that should work right? That is very doable, I stand corrected!

[ToniSkulj]

Wow @stevecopley that's a nice idea! Combining your idea (wrapping the variables in a span with the ace_variable class) and the fact that the back-end returns the variables names (as pointed out be @Felienne) makes this a doable and great solution. I will take a stab at it this weekend.

The only thing is that the variables will only be highlighted after running the program. I don't know if this is a deal breaker?

[stevecopley]

Just FYI.... The existing code in the editor is already full of spans (anything that is not a Hedy keyword sits in a classless span)

If you wrap any found variables in another span withe the variable class, you now have nested spans. I wasn't sure that this was valid HTML, but it apparently is.

The only thing is that the variables will only be highlighted after running the program. I don't know if this is a deal breaker?

Worth trying anyway!

[ToniSkulj]

I made a draft PR that implements the before mentioned solution - #3988, the only difference is that I changed the green color to yellow. Since we already use green for numbers.

Before:

After:

Let me know what you guys think!

[stevecopley]

Nice! Looks great, and the yellow works well.

(Presume I'd have to setup the dev. environment to run this version?)

[ToniSkulj]

Great! Yeah you can find some info on how to do that in CONTRIBUTING.md.
Let me know if you need some help with that.

[jpelay]

Sadly we didn't manage to quite get over Ace limitation when it comes to highlighting variables. Luckily, we don't use Ace anymore!! Now we use CodeMirror which has support for a parser called Lezer. This allows us to generate full fledged abstract syntax trees on the fly while the user writes their program.

This could allow us to highlight variables, although it would need a bit of work. Lezer uses a LR parser while Lark, the tool we use to actually transpile Hedy to Python, uses Eearley. This means that Lezer doesn't like the ambiguities we use, specially in the first few levels, therefore we can't have variable information directly at parsing time, since the trees we generate look like this:

name is ask 'what is your name'
print name? Pretty!

Program(
  Command(
    Ask(ask,Text,Text,Text,Text)
  ),
  Command(
    Print(print, Text, Text, Text)
 )
)

As you can see, variables are actually Text nodes, because we don't have a clear way of distinguishing between the two. But I think we can go through the tree once we have it, and check if a Text node is a variable by iterating through is and ask commands, and then color them.

[Felienne]

Ah so cool we are moving this forward @jpelay!! Shall we migrate this into an issue?