Clap derive should parse markdown doc comment into normal text

[ducaale]

Maintainer's notes:

• Workaround: verbatim_doc_comment attribute

Blocked on finalizing details of StyledStr, including

• Allow good/warning/error/hint color to be customized #3234
• Allow styled text in template #1433
• The header for my examples are not colored like the others #3108

---

Please complete the following tasks

• I have searched the discussions
• I have searched the existing issues

Rust Version

rustc 1.48.0 (7eac88abb 2020-11-16)

Clap Version

3.0.0-beta.2

Minimal reproducible code

use clap::Clap;

#[derive(Clap, Debug)]
#[clap(name = "xh")]
struct Cli {
    /// Optional key-value pairs to be included in the request.
    ///
    /// - key:=value to add a complex JSON value (e.g. `numbers:=[1,2,3]`)
    /// - key@filename to upload a file from filename (with --form)
    /// - header:value to add a header
    /// - header: to unset a header
    /// - header; to add a header with an empty value
    ///
    /// A backslash can be used to escape special characters (e.g. weird\:key=value).
    #[clap(value_name = "REQUEST_ITEM")]
    raw_rest_args: Vec<String>
}

fn main() {
      let _ = Cli::parse();
}

Steps to reproduce the bug with the above code

cargo run -- --help

Actual Behaviour

ARGS:
    <REQUEST_ITEM>...
            Optional key-value pairs to be included in the request.

            * key==value to add a parameter to the URL \n * key=value to add a JSON field (--json)
            or form field (--form) * key:=value to add a complex JSON value (e.g.
            `numbers:=[1,2,3]`) * key@filename to upload a file from filename (with --form) *
            header:value to add a header * header: to unset a header * header; to add a header with
            an empty value

            A backslash can be used to escape special characters (e.g. weird\:key=value).

Expected Behaviour

ARGS:
    <REQUEST_ITEM>...
            Optional key-value pairs to be included in the request.

            * key==value to add a parameter to the URL
            * key=value to add a JSON field (--json) or form field (--form)
            * key:=value to add a complex JSON value (e.g. `numbers:=[1,2,3]`)
            * key@filename to upload a file from filename (with --form)
            * header:value to add a header
            * header: to unset a header
            * header; to add a header with an empty value

            A backslash can be used to escape special characters (e.g. weird\:key=value).

Additional Context

In StructOpt, it possible to use the long_help attribute to preserve newlines but I couldn't find something similar in clap-derive.

Just found that it is called long_about in clap-derive.

Debug Output

Will add if requested

[pksunkara]

There is quite a bit of discussion in #2401

[epage]

One of the challenges with this is we'd basically need to pull in pulldown-cmark which is quite a large dependency. This would need to be put behind a feature flag.

[ducaale]

My main goal was to preserve newlines in the doc comment and that should be achievable with the verbatim_doc_comment attribute.

Example

// clap = { version = "3.0.0-rc.4", features = ["derive"] }

use clap::Parser;

#[derive(Parser, Debug)]
#[clap(name = "xh")]
struct Cli {
    /// Optional key-value pairs to be included in the request.
    ///
    ///   • key:=value to add a complex JSON value (e.g. `numbers:=[1,2,3]`)
    ///   • key@filename to upload a file from filename (with --form)
    ///   • header:value to add a header
    ///   • header: to unset a header
    ///   • header; to add a header with an empty value
    ///
    /// A backslash can be used to escape special characters (e.g. weird\:key=value).
    #[clap(value_name = "REQUEST_ITEM", verbatim_doc_comment)]
    raw_rest_args: Vec<String>,
}

fn main() {
    let _ = Cli::parse();
}

Output

USAGE:
    temp-clap [REQUEST_ITEM]...

ARGS:
    <REQUEST_ITEM>...
            Optional key-value pairs to be included in the request.

              • key:=value to add a complex JSON value (e.g. `numbers:=[1,2,3]`)
              • key@filename to upload a file from filename (with --form)
              • header:value to add a header
              • header: to unset a header
              • header; to add a header with an empty value

            A backslash can be used to escape special characters (e.g. weird\:key=value).

OPTIONS:
    -h, --help
            Print help information

I wouldn't mind if this issue was to be closed.

[epage]

I think its still an interesting topic for us to weigh out. I did call out the workaround in the issue so its more discoverable.

[I60R]

We may need only a subset of markdown features in CLI parser:

• newline escaping (implemented in Allow to escape newline with \ like in markdown #3228)
• lists as suggested here (should be relatively easy to implement, even with nesting)
• bold, italic, bold-italic, inline code - if colors enabled (not sure whether it's worth effort, but why not?)
• fenced code blocks (without syntax highlighting, of course)

There also could be some mechanism to cut off help message from doc comment as @epage suggested here — IMO markdown headers are a perfect mechanism for that e.g.:

• without any header everything goes to -h --help output
  • that's the same as if everything was placed under # Help header
• if # Long help is specified that section will be appended to --help output (only)
• we may also have # Manpage section (if clap will provide some mechanism for manpage generation)
• text under all other headers will be ignored by clap

---

I think this subset should satisfy 99.9% of all use cases of clap, everyone should like it, and it doesn't seem to be so complicated to require pulldown-cmark or any other heavy dependency.

[epage]

My experience writing one off parsers has made me a bit cautious of doing so. I threw together some md parser benchmarks to see what might be small enough for being an optional dependency. minimad looks tempting.

[epage]

Another thought on markdown support, should we support it at runtime, compile time, or both?

The core of this Issue is our line breaks which is a problem exclusive to the derive API. For that, we could build markdown parsing directly into clap_derive.

Another step up is markdown formatting. For that, we'd want to expose it to both APIs. Say we generalized our Colorizer struct and accepted it for all our user-facing inputs (for #1790, #1433), users can do whatever styling they want, though it might be a bit arduous. We then provided a markdown-parsing macro that would code-generate Colorizer. clap_derive would then have this built-in for all doc comments.

The main benefit of this is that this would remove the binary-size overhead of the markdown parser. We'd still need to compile it (though it'd be optional). Though this would mean we wouldn't regress in --help and error performance by introducing markdown, the performance would have to be pretty bad for us to care.

This has the added benefit that we wouldn't need yet-another Setting in the API for doing controlling this at runtime.

[epage]

Another question for us to answer in this issue is how disruptive would markdown parsing be. While --helps output is not considered "stable", we shouldn't dramatically alter users crafting of their help output. To what degree of markdown parsing can we add in a patch, a feature release, or a breaking release?

[gibfahn]

Another thought on markdown support, should we support it at runtime, compile time, or both?

Compile time only sounds reasonable to me (but I only use the derive API, so I guess I would say that 😁) .

Another question for us to answer in this issue is how disruptive would markdown parsing be.

I think that (as a user) if markdown parsing were turned on for existing help text that would probably break a bunch of people, but if it was behind a flag, or minor updates to the markdown output were later made in minor/patch updates, that would probably be fine, as what was intended to be output would still be there, just maybe looking slightly prettier.

The core of this Issue is our line breaks which is a problem exclusive to the derive API. For that, we could build markdown parsing directly into clap_derive.

For what it's worth there are a bunch of other things I would use if we had markdown support, not least:

1. Italic and bold inline code
2. Subheadings and code blocks, e.g.:

   ## EXAMPLES: <!-- Should be highlighted yellow the same way the other sections in the generated help like USAGE: are today. -->
   
   <!-- Syntax highlighting in examples would be really neat. -->
   ```console
   # Run the subcommand with the arg specified.
   ❯ my-command my-subcommand --my-arg
   ```

3. docs.rs compatibility -> If you have a binary/library combo that takes a set of options, it's nice to be able to use the same documentation for docs.rs and for mycmd --help.

[epage]

<!-- Syntax highlighting in examples would be really neat. -->

I was hoping someone wasn't going to say that :)

syntect is the main go-to for syntax highlighting. It can be a bit slow to compile and big. It can be made optional. Looks like we can also disable the built-in language / theme set. I doubt we need all of them. The question is what languages do we need at minimum and how should we respond if someone asks for another language.

We'll need to do some analysis to decide how worth it this is vs generating help at build-time

[epage]

One design problem I've been running into is how to render markdown at compile time when the colors aren't known until runtime.

I think the way to do this is to allow a fn(fmt, palette) -> fmt::Result to be passed into StyledStr and the derive would generate this, parsing the markdown at compile time but rendering the styles at runtime.

[pronebird]

I'd only ask for italic and bold for basic formatting.

[nrdxp]

In my case I just want to add a space in a list without having to add a newline between the bullets and without having to use verbatim doc comments (so wrap_help is respected properly)