Inline tokens

[UziTech]

Description

Right now the Lexer returns the block tokens and the Parser sends the text to the InlineLexer which sends the information to the Renderer. There is no way to get the inline tokens before sending them to the Renderer.

This PR changes the Lexer to return all tokens with block tokens that have tokens inside of them in a tokens property on the token and changes the Parser to act on all tokens instead of the text from the block tokens.

Each token that has inline or block tokens will have a tokens property that is an array of tokens.

The list token will have an items property which will be an array of list items. Each list item will have a tokens property with the tokens inside the item.

This PR also adds a raw property to the tokens so a user can get the markdown that created that token.

fixes #1626
fixes #361

TODO:

• Write unit tests for Lexer, and Parser
• Make sure demo works
  • You can see the new tokens here
• Update docs
• Refactor to make benchmarks faster
  • About 5-10% faster than master
  • It looks like this sped the compiled es5 version (lib/marked.js) up even more.

Contributor

• Test(s) exist to ensure functionality and minimize regression.
• If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

• Draft GitHub release notes have been updated.
• CI is green (no forced merge required).
• Merge PR

[vercel]

This pull request is being automatically deployed with ZEIT Now (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://zeit.co/markedjs/markedjs/qba5ws1f2
✅ Preview: https://markedjs-git-fork-uzitech-inline-tokens.markedjs.now.sh

[UziTech]

Each token that has inline tokens will have a tokens property that is an array of inline tokens.

I wasn't sure about how to add the inline tokens to the tokens array. Some tokens (like lists and blockquotes) have other tokens inside of them already so they have start and end tokens (e.g. list_start, list_end) before and after the tokens inside of them. I figured we could go one of three ways:

1. Change all tokens that have inline tokens to have start and end tokens (e.g. change header tokens to be header_start and header_end with the inline tokens between them.)
2. Give all tokens that have inline tokens a tokens property that contains their inline tokens.
3. Keep block tokens that can have other block tokens inside of them with start and end tokens and have all inline tokens added to a tokens property.

This PR currently does #3 but I am leaning toward #2. I think #2 is the cleanest but it would create major breaking changes for any dependents currently using the tokens from marked.lexer.

[UziTech]

One thing I noticed while writing tests for the InlineLexer is that the mangle options mangles autolinks (e.g. <test@example.com>) but not plain email in gfm (e.g. test@example.com) demo

So I fixed it. demo

[UziTech]

I'm not sure why travis ci isn't reporting but the tests are passing. travis ci log

[styfle]

I didn't review all the tests thoroughly but looks good! 🥇

[UziTech]

@styfle what do you think about #1627 (comment)?

[styfle]

Can you give an example of what the token output would look like for the 3 examples?

Also, our PR demo doesn't seem to work anymore

[UziTech]

I have rebased and implemented option 2 from #1627 (comment)

It didn't make sense to keep the InlineLexer so I just merged it into the Lexer

[UziTech]

I renamed the lexer functions and reordered the parameters for the lexer so they are consistent.

@styfle I think this is ready to merge if you approve.

[styfle]

Great work!

[experimentalPrivateRepos]

Hey @UziTech, thank you for the great work.
I would like to use the updated Lexer in a personal project of mine.
Do you have any idea when this new version will be available on npm?

[UziTech]

@experimentalPrivateRepos We have a few more changes to get into the next release. I would estimate it will be about a month before the next release.

[experimentalPrivateRepos]

@UziTech Alright, thank you for the quick response!

[upupming]

I have rebased and implemented option 2 from #1627 (comment)

It didn't make sense to keep the InlineLexer so I just merged it into the Lexer

Hi, since the InlineLexer class has been deleted, can you delete the type definitions in @types/marked too, thanks!