New placeholder feature & hast refactoring

[Jule-]

Hi,

This pull request will add the ability to put the table of content in place of a string placeholder.
For that I have refactored a little bit your plugin with hast types instead of unist since rehype is based on it.

Concretely in your .mdx file you can put something like that:

Hello

Some stuff here...

![Some image](https://www.example.com/img.png)

{{TOC}}

Other stuff here...

## Title 1

...

## Title 2

...

Declare this plugin with placeholder option, but not necessarily since {{TOC}} is the default one (if the placeholder node is not found, it fallbacks to the previous default behaviour).

const {createCompiler} = require('@mdx-js/mdx')
const vfile = require('vfile')
const slug = require("rehype-slug");
const autolinkHeadings = require("rehype-autolink-headings");
const toc = require("@jsdevtools/rehype-toc");

const file = vfile(`
Hello

Some stuff here...

![Some image](https://www.example.com/img.png)

TABLE_OF_CONTENTS

Other stuff here...

## Title 1

...

## Title 2

...
`)

mdxCompiler = createCompiler({
  rehypePlugins: [
    slug,
    autolinkHeadings,
    [toc, { placeholder: "TABLE_OF_CONTENTS" }]
    // if you want to stick with default "{{TOC}}" you can use toc without option
  ]
})

mdxCompiler.process(file, function done(err, file) {
  // TABLE_OF_CONTENTS is replaced by the table of contents
})

I don't know if this PR can fits the need of #2 but maybe it is, at least partially... I have not tested with jsx type element but the function that is seeking for the placeholder could evolve in order to find it inside them and maybe render the toc inside it... I am laking of knowledge or time to investigate more, sorry. 😉

Hope you will enjoy the feature.

Cheers!

[Jule-]

Well good news! For what I have tested it could really fix #2! 😃

This kind of stuff is working:

Hello

Some stuff here...

![Some image](https://www.example.com/img.png)

<div className="my-super-sticky-class" style={{textAlign: "center"}}>

{{TOC}}

</div>

Other stuff here...

## Title 1

...

## Title 2

...

Here you go!

[Jule-]

Well I changed my mind on the fallback behaviour: I feel like it is useful to have generic processing of mdx files and optional TOCs thus if a placeholder is given in the configuration and not found in the mdx then no TOC generation at all.
At least I benefit from this feature ! 😄

[Jule-]

Ok last update had broken the generation, v3.1.2 is working correctly now. For the ones who wants to try it out until @JamesMessinger comes back:
npm i @atomictech/rehype-toc

[hipstersmoothie]

I can confirm this work in my project, but I did have to edit the optional chaining to work with node 12

[Jule-]

@hipstersmoothie ho yeah right... My bad, I will patch it asap! 😉

[karlhorky]

@JamesMessinger, aside from the optional chaining, what do you think about this PR?

@Jule- what's your plan for the optional chaining? Just reverting it to something like boolean operators?

[Jule-]

@karlhorky ho yeah thank you for the ping, I have totally forgotten about this. Yup I will revert to something more vanilla! 😉 I think I will have some time to publish in few hours.

[Jule-]

@hipstersmoothie, @karlhorky Here it is: v3.1.3! Please check it out and tell me if I missed some optional chaining stuff! 😉

[digitalsadhu]

Any progress on this? Would be handy!

[pavelloz]

Hmm, it would be great to merge it.

Having said that, i cant make it work yet :)

  .use(rehypeToc, {
    headings: ["h1", "h2", "h3"],
    placeholder: "TABLE_OF_CONTENTS"
  })

<aside class="TABLE-OF-CONTENTS">
TABLE_OF_CONTENTS
</aside>

# Hello

## Section One
### Lorem ipsum
## Section Two
### Lorem ipsum
## Section Three
### Lorem ipsum
## Section Four

Result:

Im using rehypeRaw to preserve html used in there. Ideally i would like to define <aside> tag and TABLE_OF_CONTENTS outside (in [slug].svelte), but those files are not processed by rehype, so im trying this trick to see if anything can be done in that regards, but no luck yet.

[pavelloz]

I did a crazy experiment and did that:

<aside> TABLE_OF_CONTENTS </aside>
<aside> TABLE_OF_CONTENTS </aside>
<aside> TABLE_OF_CONTENTS </aside>
<aside> TABLE_OF_CONTENTS </aside>

And it rendered one toc and 3x this string, so it must see it, and render, but ignores html around it.

[Jule-]

@pavelloz I do not know your env but for markdown to be parsed correctly normally you have to let 1 blank line before and after in order to not being processed as "text in HTML", so try it like that:

<aside class="TABLE-OF-CONTENTS">

TABLE_OF_CONTENTS

</aside>

# Hello

[pavelloz]

^ I tried that already before i created issue. If i do that no TOC is rendered at all - it is ignored by the plugin.

<aside>

TABLE_OF_CONTENTS

</aside>

Result:

---

Is the plugins order important in rehype chain?
I experimented with it i didnt notice any changes.
This is how i currently test:

let runner = unified()
  .use(remark2rehype, { allowDangerousHtml: true })
  .use(rehypeSlug)
  .use(rehypeAutolinkHeadings)
  .use(rehypeRaw)
  .use(rehypeFormat)
  .use(rehypeHighlight)
  .use(rehypeToc, {
    headings: ["h1", "h2", "h3"],
    placeholder: "TABLE_OF_CONTENTS"
  })
  .use(rehypeStringify);

[Jule-]

I am not really familiar with all these plugins, but order is definitely important!

My guess is to try to put rehypeToc just after rehypeAutolinkHeadings. This is how I use it but I use it with MDX and thus do not require all other plugins.

Maybe rehypeFormat is collapsing useful whitespaces/blank lines? Maybe it is something else... Try to see the impact of each plugin, removing one by one, to understand what could mess things up.

[pavelloz]

Gotcha. I will disable everything and try adding one by one from the most important to the least and shuffling order.

[joe307bad]

I was able to get this to work with my Next.js + mdx setup here: https://github.com/joe307bad/rehype-toc

The main things I had to do to get it to work for me:

• Changes to the type-guards.ts file. I needed to also allow node.type === "mdxJsxFlowElement" for isElement. I am not sure what this element type means. Also had to change the logic for isText to disregard node.type.
• Using visit instead of custom recursive functions. I am not sure why the other logic wasn't working for me, but this seems to work okay.

@Jule- I am curious if you have feedback or can clarify why these changes may have been necessary for my instance? Could it be because I am using Next.js? Thanks for all the work! It really helped me.

[Jule-]

@joe307bad you're welcome! 🙂
It's been a long time since I didn't put my eyes on this code, but I think you have used visit for unist which is not designed for hast which I think is the root cause for your type issues. Like I said in the introduction of my PR:

For that I have refactored a little bit your plugin with hast types instead of unist since rehype is based on it.

Then for the fact that you didn't succeed in using Next.js + mdx first, I cannot say without knowing which versions you use for you project and how you use them. From my side, I actually succeed to use this plugin with [email protected] and @next/[email protected]. Maybe your version is newer/older and can explain that you had to use node.type === "mdxJsxFlowElement" or maybe you try to use this plugin with unist tree. I don't know but that could lead your investigations. 😉

[essential-randomness]

Is there a plan to eventually merge this change? From the discussion, I'm unsure what's left for it to be ready, and whether there's anything that can be done to help get it there.

[JagritGumber]

It has been really long, And I really could use this feature, I really want to get me TOC in an aside and not on the top

[JagritGumber]

@Jule- The issue stated over here persists with me
The issue is because when an element is added in MDX it is marked as mdxJsxFlowElement so the isElement test case fails for it.

^ I tried that already before i created issue. If i do that no TOC is rendered at all - it is ignored by the plugin.

<aside>

TABLE_OF_CONTENTS

</aside>

Result:

Is the plugins order important in rehype chain? I experimented with it i didnt notice any changes. This is how i currently test:

let runner = unified()
  .use(remark2rehype, { allowDangerousHtml: true })
  .use(rehypeSlug)
  .use(rehypeAutolinkHeadings)
  .use(rehypeRaw)
  .use(rehypeFormat)
  .use(rehypeHighlight)
  .use(rehypeToc, {
    headings: ["h1", "h2", "h3"],
    placeholder: "TABLE_OF_CONTENTS"
  })
  .use(rehypeStringify);

This one on the other hand seems like I should try because they faced the same problem as me.

I was able to get this to work with my Next.js + mdx setup here: https://github.com/joe307bad/rehype-toc

The main things I had to do to get it to work for me:

• Changes to the type-guards.ts file. I needed to also allow node.type === "mdxJsxFlowElement" for isElement. I am not sure what this element type means. Also had to change the logic for isText to disregard node.type.
• Using visit instead of custom recursive functions. I am not sure why the other logic wasn't working for me, but this seems to work okay.

@Jule- I am curious if you have feedback or can clarify why these changes may have been necessary for my instance? Could it be because I am using Next.js? Thanks for all the work! It really helped me.

I know it's an old thread but I wanted to use this feature

[Jule-]

@JagritGumber Hi, without more information, it is pretty hard to help you.

Did you use the version I published?

npm i @atomictech/rehype-toc

Did you add extra blank lines like in this example?

<aside>                  // JSX node
                         // <-- here, to reset parser and allow new node definition
TABLE_OF_CONTENTS        // Markdown Text node
                         // <-- and here, to reset parser and allow new node definition
</aside>                 // JSX node

(because if not, then the node could rightly be considered as `mdxJsxFlowElement`)

<aside>                  // JSX node start
TABLE_OF_CONTENTS        // JSX Child Text node => Not a markdown text node
</aside>                 // JSX node end

What is your plugins setup?

To clarify, I am not the maintainer of this plugin. I just made a fork and a PR to try to have something working merged.

The only thing I can tell is that this is working on my setup with these plugins declared in this order:

import { GetStaticProps } from 'next';
import { serialize } from 'next-mdx-remote/serialize';
import autolinkHeadings from 'rehype-autolink-headings';
import slug from 'rehype-slug';

import toc from '@atomictech/rehype-toc';

// ...

export const getStaticProps: GetStaticProps = async (context) => {
  const content = /* get your content */;

  const mdxSource = await serialize(content, {
    mdxOptions: {
      rehypePlugins: [slug, autolinkHeadings, [toc, { placeholder: 'TABLE_OF_CONTENTS' }]],
    },
  });

  return { props: { mdxSource } };
};

Hope this can help.

[JagritGumber]

@Jule-
I am using the same library @atomictech/rehype-toc The thing is Whenever I setup my hierarchy like this

<aside>                  // JSX node
                         // here, to reset parser and allow new node definition
TABLE_OF_CONTENTS        // Markdown Text node
                         //  and here, to reset parser and allow new node definition 
</aside>                 // JSX node

I get this in my hierarch

<aside></aside>
<p>TABLE_OF_CONTENTS</p>
<aside></aside>

I am using AstrojsMDX btw and I found out it is due to how it handles mdx files and just had to switch up with some other way thanks for help btw