Description Lists Extension #86

ayosec · 2018-08-19T04:48:33Z

This patch adds support for description lists, as described in #52.

Overview

A description-list is written as a group of terms, each one with a description (details). It is required to put a blank line between terms and descriptions.

For example, the following source:

Term 0

: Description 0

Term 1

: Description 1

    More info

Is rendered to:

<dl>
  <dt>
    <p>Term 0</p>
  </dt>
  <dd>
    <p>Description 0</p>
  </dd>
  <dt>
    <p>Term 1</p>
  </dt>
  <dd>
    <p>Description 1</p>
    <p>More info</p>
  </dd>
</dl>

It also supports nested nodes. The following source:

Is rendered to:

<p>0</p>
<dl>
  <dt>
    <p>1</p>
  </dt>
  <dd>
    <p>2</p>
  </dd>
  <dt>
    <p>3</p>
  </dt>
  <dd>
    <p>4</p>
    <ol>
      <li>
        <p>5</p>
      </li>
      <li>
        <p>6</p>
        <blockquote>
          <dl>
            <dt>
              <p>7</p>
            </dt>
            <dd>
              <p>8</p>
            </dd>
            <dt>
              <p>9</p>
            </dt>
            <dd>
              <p>10</p>
            </dd>
          </dl>
          <p>11</p>
        </blockquote>
      </li>
    </ol>
  </dd>
</dl>

Implementation Details

I tried to imitate the support for regular lists in some parts of the implementation, mostly to support nested nodes.

There are four new node variants in NodeValue:

DescriptionList
DescriptionItem
DescriptionTerm
DescriptionDetails

DescriptionItem is used only for the parser. It is not emitted in the HTML.

To detect a description list:

The parser checks for a : symbol as the first non-space character.
When it is found, it checks if the previous node in the same container is a paragraph .
If it is paragraph:
- Detach it
- Check if the (new) last child is a DescriptionList.
  - If not, add a new DescriptionList node.
- Creates a tree like DescriptionItem / DescriptionTerm / $old-paragraph, and append it to the DescriptionList node.
- Finally, create a DescriptionDetails node under DescriptionItem, and use it as the new container.

A tricky part of the implementation is that I have to change the open field of an existing DescriptionList. This is required because every new term is created as a paragraph (before detect the : line with the description), so the previous list is closed. I don't know if this is the proper way to do it.

To visualize this:

If we have this source:

A

: B

C

: D

The behaviour of the parser is something like:

Found A:
- Create a Paragraph node, and append it to the top container.
- The document is:
```
Document
    Paragraph "A"
```
Found ::
- Detects that last child is a Paragraph, so it is detached.
- There is no DescriptionList at the end of the container, so it creates a new one.
- Append the detached paragraph to the new list.
- The document is:
```
Document
    DescriptionList
        DescriptionItem
            DescriptionTerm
                Paragraph "A"
            DescriptionDetails
```
- DescriptionDetails is new current container to add nodes.
Found B after ::
- Creates a Paragraph and add it to the DescriptionDetails node.

Found C:

Closes the current DescriptionList and add the text as a new Paragraph.

The document is:

Document
    DescriptionList
        DescriptionItem
            DescriptionTerm
                Paragraph "A"
            DescriptionDetails
                Paragraph "B"
    Paragraph "C"

Found ::

Detects that last child is a Paragraph, so it is detached.
After detach, found a DescriptionList as the last child.
- Mark its nodes as open.
Append the DescriptionItem node to the current list.

The document is:

Document
    DescriptionList
        DescriptionItem
            DescriptionTerm
                Paragraph "A"
            DescriptionDetails
                Paragraph "B"
        DescriptionItem
            DescriptionTerm
                Paragraph "C"
            DescriptionDetails

The requirement of the blank line between terms and descriptions is to keep the implementation simpler. Many parsers supports a syntax like:

term 1
: description 1
term 2
: description 2

But it was not clear to me how to support this. I guess that this is something that can be added later.

Changes in the CLI

The CLI supports the flag description-lists to enable the extension.

S-Expressions in the examples

An example program in examples/s-expr.rs can be used to print the AST as S-expressions.

$ cargo run --example s-expr file1.md file2.md ...
$ cat file.md | cargo run --example s-expr

For example, the following source:

Lorem ipsum dolor sit amet:

* Consectetur adipiscing elit
* Sed do **eiusmod** tempor incididunt
    * Ut labore et [dolore](magna-aliqua).
    * Ut enim ad minim veniam.

Generates this output:

$ cargo run --example s-expr path/to/file.md
(Document
    (Paragraph "Lorem ipsum dolor sit amet.")
    (List(NodeList { list_type: Bullet, marker_offset: 0, padding: 2, start: 1, delimiter: Period, bullet_char: 42, tight: true })
        (Item(NodeList { list_type: Bullet, marker_offset: 0, padding: 2, start: 1, delimiter: Period, bullet_char: 42, tight: false })
            (Paragraph "Consectetur adipiscing elit"))
        (Item(NodeList { list_type: Bullet, marker_offset: 0, padding: 2, start: 1, delimiter: Period, bullet_char: 42, tight: false })
            (Paragraph "Sed do " (Strong "eiusmod") " tempor incididunt")
            (List(NodeList { list_type: Bullet, marker_offset: 2, padding: 2, start: 1, delimiter: Period, bullet_char: 42, tight: true })
                (Item(NodeList { list_type: Bullet, marker_offset: 2, padding: 2, start: 1, delimiter: Period, bullet_char: 42, tight: false })
                    (Paragraph "Ut labore et " (Link(NodeLink { url: [109, 97, 103, 110, 97, 45, 97, 108, 105, 113, 117, 97], title: [] }) "dolore") "."))
                (Item(NodeList { list_type: Bullet, marker_offset: 2, padding: 2, start: 1, delimiter: Period, bullet_char: 42, tight: false })
                    (Paragraph "Ut enim ad minim veniam.")))))
)

I added it to the patch because it was useful to debug the AST, and it may be useful for other people. If you think that it should not be included, I will remove it.

Closes #52

…e details per item.

kivikakk · 2018-08-19T07:18:57Z

I’m incredibly impressed. I’ll take time to review this tomorrow, but I just wanted to leave a comment now to say how amazing this is!

kivikakk

Just one comment!

kivikakk · 2018-08-19T23:49:27Z

src/main.rs

@@ -64,6 +64,7 @@ fn main() {
                    "tasklist",
                    "superscript",
                    "footnotes",
+                    "description-lists",


We add description-lists as a possible -e value here …

kivikakk · 2018-08-19T23:49:41Z

src/main.rs

+            clap::Arg::with_name("description-lists")
+                .long("description-lists")
+                .help("Parse description lists"),
+        )


… and add --description-lists as a separate flag here ...

kivikakk · 2018-08-19T23:50:47Z

src/main.rs

@@ -125,6 +131,7 @@ fn main() {
        ext_superscript: exts.remove("superscript"),
        ext_header_ids: matches.value_of("header-ids").map(|s| s.to_string()),
        ext_footnotes: matches.is_present("footnotes"),
+        ext_description_lists: matches.is_present("description-lists"),


… and use the flag toggle here, not the exts set. As a result:

$ cargo run <(echo "a"; echo; echo ": b") --description-lists Finished dev [unoptimized + debuginfo] target(s) in 0.06s Running `target/debug/comrak /dev/fd/63 --description-lists` <dl><dt> <p>a</p> </dt> <dd> <p>b</p> </dd> </dl> $ cargo run <(echo "a"; echo; echo ": b") -e description-lists Finished dev [unoptimized + debuginfo] target(s) in 0.06s Running `target/debug/comrak /dev/fd/63 -e description-lists` thread 'main' panicked at 'assertion failed: exts.is_empty()', src/main.rs:137:5 note: Run with `RUST_BACKTRACE=1` for a backtrace. $

Ideally description lists should be an -e option. (And so should footnotes, frankly.)

Oops.. I didn't notice that =/.

Fixed in d7469e7.

kivikakk · 2018-08-20T02:07:55Z

Thank you! ❤️

brson · 2018-08-20T20:44:29Z

Awesome writeup and patch.

ayosec added 12 commits August 17, 2018 12:16

Option to enable description lists extension

7266e11

Example to show a XML from the AST

6a37354

Enable all bool extenions in cm2xml

cb9aa4e

Basic support for description lists

b47343f

Format for description lists in both HTML and CM

1dc7efc

Mark all nodes for description lists as blocks

2c2eb66

Example program to print the AST as S-expressions

092f519

Remove example for XML

98e5524

Include content for nodes like Code or HtmlInline in s-expr example

dd18474

Able to detect nested content in description details. Discard multipl…

3502dd9

…e details per item.

Update README with info for description lists

1542260

Remove references to the multiple details feature

80f6614

kivikakk approved these changes Aug 19, 2018

View reviewed changes

Remove --description-lists flag. Use only the item in the -e option.

d7469e7

kivikakk merged commit 03c6ad3 into kivikakk:master Aug 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Description Lists Extension #86

Description Lists Extension #86

ayosec commented Aug 19, 2018

kivikakk commented Aug 19, 2018

kivikakk left a comment

kivikakk Aug 19, 2018

kivikakk Aug 19, 2018

kivikakk Aug 19, 2018

ayosec Aug 20, 2018

kivikakk commented Aug 20, 2018

brson commented Aug 20, 2018

Description Lists Extension #86

Description Lists Extension #86

Conversation

ayosec commented Aug 19, 2018

Overview

Implementation Details

Changes in the CLI

S-Expressions in the examples

kivikakk commented Aug 19, 2018

kivikakk left a comment

Choose a reason for hiding this comment

kivikakk Aug 19, 2018

Choose a reason for hiding this comment

kivikakk Aug 19, 2018

Choose a reason for hiding this comment

kivikakk Aug 19, 2018

Choose a reason for hiding this comment

ayosec Aug 20, 2018

Choose a reason for hiding this comment

kivikakk commented Aug 20, 2018

brson commented Aug 20, 2018