When a math fence immediately follows text, the resulting tokens are multiplied

[schanzer]

I would expect that the following markdown:

---
# AAA

BBB
$$$ math
1
$$$

To have an HR token, a heading containing the "AAA" text, a paragraph containing the "BBB" text, and a generated_image with the $$$ markup and 1 content.

Instead, I'm seeing the following tokens:

[
    {
        "type": "hr",
        "tag": "hr",
        "attrs": null,
        "map": [
            1,
            2
        ],
        "nesting": 0,
        "level": 0,
        "children": null,
        "content": "",
        "markup": "---",
        "info": "",
        "meta": null,
        "block": true,
        "hidden": false
    },
    {
        "type": "heading_open",
        "tag": "h1",
        "attrs": null,
        "map": [
            2,
            3
        ],
        "nesting": 1,
        "level": 0,
        "children": null,
        "content": "",
        "markup": "#",
        "info": "",
        "meta": null,
        "block": true,
        "hidden": false
    },
    {
        "type": "inline",
        "tag": "",
        "attrs": null,
        "map": [
            2,
            3
        ],
        "nesting": 0,
        "level": 1,
        "children": [
            {
                "type": "text",
                "tag": "",
                "attrs": null,
                "map": null,
                "nesting": 0,
                "level": 0,
                "children": null,
                "content": "AAA",
                "markup": "",
                "info": "",
                "meta": null,
                "block": false,
                "hidden": false
            }
        ],
        "content": "AAA",
        "markup": "",
        "info": "",
        "meta": null,
        "block": true,
        "hidden": false
    },
    {
        "type": "heading_close",
        "tag": "h1",
        "attrs": null,
        "map": null,
        "nesting": -1,
        "level": 0,
        "children": null,
        "content": "",
        "markup": "#",
        "info": "",
        "meta": null,
        "block": true,
        "hidden": false
    },
    {
        "type": "generated_image",
        "tag": "div",
        "attrs": null,
        "map": [
            5,
            8
        ],
        "nesting": 0,
        "level": 0,
        "children": null,
        "content": "1\n",
        "markup": "$$$",
        "info": " math",
        "meta": null,
        "block": true,
        "hidden": false
    },
    {
        "type": "generated_image",
        "tag": "div",
        "attrs": null,
        "map": [
            5,
            8
        ],
        "nesting": 0,
        "level": 0,
        "children": null,
        "content": "1\n",
        "markup": "$$$",
        "info": " math",
        "meta": null,
        "block": true,
        "hidden": false
    },
    {
        "type": "paragraph_open",
        "tag": "p",
        "attrs": null,
        "map": [
            4,
            5
        ],
        "nesting": 1,
        "level": 0,
        "children": null,
        "content": "",
        "markup": "",
        "info": "",
        "meta": null,
        "block": true,
        "hidden": false
    },
    {
        "type": "inline",
        "tag": "",
        "attrs": null,
        "map": [
            4,
            5
        ],
        "nesting": 0,
        "level": 1,
        "children": [
            {
                "type": "text",
                "tag": "",
                "attrs": null,
                "map": null,
                "nesting": 0,
                "level": 0,
                "children": null,
                "content": "BBB",
                "markup": "",
                "info": "",
                "meta": null,
                "block": false,
                "hidden": false
            }
        ],
        "content": "BBB",
        "markup": "",
        "info": "",
        "meta": null,
        "block": true,
        "hidden": false
    },
    {
        "type": "paragraph_close",
        "tag": "p",
        "attrs": null,
        "map": null,
        "nesting": -1,
        "level": 0,
        "children": null,
        "content": "",
        "markup": "",
        "info": "",
        "meta": null,
        "block": true,
        "hidden": false
    },
    {
        "type": "generated_image",
        "tag": "div",
        "attrs": null,
        "map": [
            5,
            8
        ],
        "nesting": 0,
        "level": 0,
        "children": null,
        "content": "1\n",
        "markup": "$$$",
        "info": " math",
        "meta": null,
        "block": true,
        "hidden": false
    }
]

I ran it through Dingus as well, to make sure the reference implementation wasn't also generating multiple images!

[rlidwka]

I ran it through Dingus as well, to make sure the reference implementation wasn't also generating multiple images!

I also just did that:

https://spec.commonmark.org/dingus/?text=%0A---%0A%23%20AAA%0A%0ABBB%0A%24%24%24%20math%20%0A1%0A%24%24%24

https://markdown-it.github.io/#md3=%7B%22source%22%3A%22%5Cn---%5Cn%23%20AAA%5Cn%5CnBBB%5Cn%24%24%24%20math%20%5Cn1%5Cn%24%24%24%22%2C%22defaults%22%3A%7B%22html%22%3Afalse%2C%22xhtmlOut%22%3Afalse%2C%22breaks%22%3Afalse%2C%22langPrefix%22%3A%22language-%22%2C%22linkify%22%3Atrue%2C%22typographer%22%3Atrue%2C%22_highlight%22%3Atrue%2C%22_strict%22%3Afalse%2C%22_view%22%3A%22html%22%7D%7D

The result is identical for both reference parser and this one.

So... where's the bug exactly?

[schanzer]

Sorry, Alex, this may just be my own ignorance about how the markdown-it parser is used. Here's the code that's generating the token stream:

const parser = markdownIt(mdOptions)
  .use(attrs)
  .use(lazyHeaders)
  .use(emoji, {shortcuts: {}})
  .use(expandTabs, {tabWidth: 4})
  .use(generatedImage)
  .use(video, {youtube: {width: 640, height: 390}});

function parseMarkdown(markdown: string): Token[] {
  const parseTree = parser.parse(markdown, {});
  console.log(JSON.stringify(parseTree, null, 4))
  return parseTree;
}

The link you shared certainly produces the right html after tokenizing - but I've got a custom renderer consuming the token stream and I'm trying to make sense of the tokens I'm seeing. So I guess this boils down to two related questions:

1. If you examine the output of your tokenizer on this input, do you expect see three different image tokens? If so, then this is clearly not a bug!
2. If it's not a bug, and three different image tokens are expected for a single fence, how are you collapsing those tokens back to a single fence in your rendered HTML output, so that I can do the same in my renderer?

[schanzer]

Aha! I see now that it's one of the plugins that seems to be wreaking havoc (probably the generatedImage one!). My apologies, Alex.

[rlidwka]

No problem.

Yes, generated_image token is generated by one of the plugins (markdown-it does not support mathjax by itself), so you should look into that plugin first.