Fix the Markdown language

[DmitrySharabin]

• Add the missing inside token and fix the existing one
• Fix the regex that is too greedy now: if we have more than one code block next to each other, all of them starting from the second one will be interpreted as a code block of the first one
• Ignore spaces before the language code

However, to fully support it, we need to support functional tokens first. For now, we don't, and we work with it as it is a regex, or an object with the pattern property. In both cases, matchPattern() (in tokenize/match.ts) throws.

[github-actions]

No JS Changes

Generated by 🚫 dangerJS against aae926d

[LeaVerou]

I thought I added code in tokenize() to support functional tokens?

[DmitrySharabin]

I thought I added code in tokenize() to support functional tokens?

You did. It covers functional inside (without arguments) and cases where every part of a string we tokenize belongs to a named group (inside the corresponding pattern).

If, OTOH, we have parts that are not in a group (like ”```“ in a code block in Markdown), we tokenize it by passing the same grammar (with functional tokens):

prism/src/core/tokenize/match.ts

Line 174 in b86c4e0

content = tokenize.call(prism, content, localInsideGrammar);

While tokenizing, we iterate over the grammar’s tokens:

prism/src/core/tokenize/match.ts

Line 27 in b86c4e0

for (const token in grammar) {

These functional tokens are not regular expressions, and when we call exec() on them (inside matchPattern()), our code throws:

prism/src/core/tokenize/match.ts

Line 131 in b86c4e0

match = matchPattern(pattern, 0, str, lookbehind);

prism/src/core/tokenize/match.ts

Line 224 in b86c4e0

const match = pattern.exec(text);