add walkTokens and fix highlight async

[UziTech]

Marked version: 1.0.0

Description

Using an async highlighting function was broken in v1.0.0 when we moved block tokens in blockquote and lists from the top level of the tokens array to a tokens property on the blockquote or listitem token.

This PR adds a public method to easily iterate over all tokens and send each token to a function specified by the user (marked.walkTokens(tokens, callback)) and uses that function to highlight code blocks asynchronously.

This also adds a walkTokens option so extensions can use it to run a function for each token.

Example: Overriding heading tokens to start at h2.

const marked = require('marked');

// Override function
const walkTokens = (token) => {
  if (token.type === 'heading') {
    token.depth += 1;
  }
};

marked.use({ walkTokens });

// Run marked
console.log(marked('# heading 2\n\n## heading 3'));

Output:

<h2 id="heading-2">heading 2</h2>
<h3 id="heading-3">heading 3</h3>

• Fixes highlight callback is undefined #1656

Contributor

• Test(s) exist to ensure functionality and minimize regression
• If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

• Draft GitHub release notes have been updated.
• CI is green (no forced merge required).
• Merge PR

[vercel]

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/markedjs/markedjs/1zxn2ys77
✅ Preview: https://markedjs-git-fork-uzitech-fix-highlight-async.markedjs.now.sh

[styfle]

walkedOnce is 2? Maybe it should be renamed?

[UziTech]

That variable counts how many tokens were walked at least once

[davisjam]

Perhaps nWalkedOnce and nWalkedTwice?

[styfle]

Thanks!