"Functored" AST

[Ericson2314]

Instead of doing e.g.

data Block = ...Block...

what if we did

data BlockF block = ...block...
   deriving Functor
newtype Block = MkBlock (BlockF Block)

This trick allows ASTs to be extensible, among other benefits.

I generally like this approach, but I had a specific problem in mind that would benefit which I would like to share. https://github.com/obsidiansystems/dombuilder-pandoc/blob/master/src/Reflex/Dom/Builder/Pandoc.hs is some code to translate the Pandoc AST to reflex-dom "dom builder action" order to use within a website built with reflex dom.

I would like to have my own custom handling of pandoc the pandoc AST --- e.g. parsing relative URLs in links into a Route AST --- without having to copy and paste that function. But if we do the above "functored AST" approach, I can replace

block :: DomBuilder t m => Block -> m ()

with

blockF :: DomBuilder t m => BlockF a -> m ()

or

blockF :: DomBuilder t m => BlockF (m ()) -> m ()

These can be thought of as specialized versions of traverse_ or sequence_, and they are very nice to work with!

[jgm]

I'd like to understand this better. Maybe you could give a small example, with a very abbreviated version of the pandoc types (e.g. with just Str, Emph, Para, and BlockQuote).

As for the application, it's hard for me to see what you're trying to accomplish just from the types you give. I assume you're familiar with our existing Walk module?

[Ericson2314]

Yeah sure, I would be happy to make the example.

[Ericson2314]

OK see #99

I frequent concern when this sort of thing comes up is breaking downstream code. If that is a concern for you, I am happy to rename the variants and create a bunch of compatibility patterns

[Ericson2314]

obsidiansystems/dombuilder-pandoc#1 shows how I would change the DOM builder to use this. The "prime" functions are not recursive, and thus can be reused to render something like

data MyInline 
  = Regular (InlineF MyInline) -- fixed typo was `Inline` before
  | ReatliveLink Attr [MyInline] MyRoute Text

which would accurately represent a document with checked relative URLs.

[jgm]

Should that be Regular (InlineF MyInline)?

One question is how many changes would be needed to the pandoc code base if pandoc-types were to change in this way.
A lot, it seems to me -- but maybe pattern synonyms could be defined that would allow current code to compile?

[Ericson2314]

Should that be Regular (InlineF MyInline)?

Yes, thanks!

One question is how many changes would be needed to the pandoc code base if pandoc-types were to change in this way.
A lot, it seems to me -- but maybe pattern synonyms could be defined that would allow current code to compile?

Indeed. I wanted to wait to hear your thoughts first, but I am happy to write those pattern synonyms and then make the (far more minimal) changes in pandoc too.

[madeline-os]

Here is a report from another project that saw great utility out of structuring their AST in this way: http://newartisans.com/2018/04/win-for-recursion-schemes/ I thought it would be useful to note here.

[Ericson2314]

Thanks @madeline-os, I was completely blanking on where the existing literature was on this technique.

[tarleb]

The one downside I can think off is that the change would make pandoc less accessible to newcomers. However, I believe that this is not as big a problem anymore. Pandoc's code base reflects more and more of the complexity that's inherent to the problems it solves, and this would be just another step in that direction.

I note that the "functorization" could potentially be used to solve jgm/pandoc#684.

[Ericson2314]

OK I created pattern synonyms such that that the PR should be almost breakage free. (Exception is cannot associate data constructors to type synonyms for aliases, which are used for tables.)

[jgm]

Have you tried compiling pandoc against the altered pandoc-types, to see where breakage might happen?

[Ericson2314]

@jgm trying that now.

[Ericson2314]

pandoc/pandoc-lua-marshal#5 jgm/pandoc#8084 are the changes. As you can see are quite minimal and just making some imports uglier because of the GHC deficiency.

[jgm]

Wow, that's quite impressive. I will try to have a closer look at this before long, but it might be a couple weeks.

[Ericson2314]

Sounds good. Just let me know about any and all questions once you do :).

[Ericson2314]

@jgm thanks for giving this another look and repeatedly enabling CI for me.

All 3 PRs:

• Recreate the AST with functors #99
• Change imports to work with functored AST pandoc/pandoc-lua-marshal#5
• Change imports to work with functored AST pandoc#8084

are now passing, anything more you need from me?

[jgm]

Many thanks! I was just waiting for CI to pass before taking a closer look. I'll do that next, but it may be a little while before I can find the time. Don't worry, I haven't forgotten about this.

[Ericson2314]

Sure, sounds good. Take the time you need and thanks for reviewing. Just want to make sure I've gotten everything done on my end, to avoid any "I am waiting on you, you wait on me" deadlock. :)