Add generalized support for specifying Beautiful Soup options

[chrispy-snps]

In #206, a new beautiful_soup_parser configuration option was added that lets Beautiful Soup's HTML parser be specified:

MarkdownConverter(beautiful_soup_parser="lxml")

The specified parser keyword is passed to the features parameter of the BeautifulSoup constructor, which is the second option after the HTML markup:

class BeautifulSoup(Tag):
    # ...
    def __init__(
        self,
        markup="",
        features=None,
        builder=None,
        parse_only=None,
        from_encoding=None,
        exclude_encodings=None,
        element_classes=None,
        **kwargs,
    ):
        # ...

But as shown above, the BeautifulSoup constructor provides other options that might be useful, such as providing hints to the text encoding detection for the HTML document.

Before #206 ships in a production release, perhaps we could extend its implementation to generally support all Beautiful Soup configuration options using a kwargs-based approach (including new options in the future). For example,

MarkdownConverter(beautiful_soup_options={"features": "lxml"})

MarkdownConverter(beautiful_soup_options={"exclude_encodings": ["iso-8859-7"]})

Or perhaps a bit shorter to make up for the extra kwargs length,

MarkdownConverter(bs4_options={"features": "lxml"})

MarkdownConverter(bs4_options={"exclude_encodings": ["iso-8859-7"]})

[chrispy-snps]

@AlexVonB - if you're open to this, I'll submit a pull request.

[AlexVonB]

I like this! Maybe the shorter option name is preferable. One other method could be handing a bs4 instance directly, e.g. if one already has one configured to their liking. But I prefer your approach.

[matthewwithanm]

• Any concern that this exposes implementation details/couples you to a specific version of BS?
• Another option would be to allow users to pass in their own soup instance (composition).

Don't get me wrong; I don't have a strong objection or anything. You two should do whatever you think best. (:

EDIT: Oops, just saw that my second point was also raised by @AlexVonB! Serves me right for popping my head in randomly. 😂

[AlexVonB]

:D no worries! Thanks for chiming in! That's a good point. Composition could solve this a bit.

[chrispy-snps]

I don't think passing a BeautifulSoup object would work, as the HTML to parse is passed as part of the constructor. And for users working directly with BeautifulSoup objects (as our application does), they can simply call convert_soup() as described here.

I started working on a pull request to implement a kwargs-based solution, but I ran into a problem - there is no standard way to pass a dict as an argument value in the shell wrapper script:

https://github.com/matthewwithanm/python-markdownify/blob/develop/markdownify/main.py

Perhaps this is a sign to keep the current parser-only option, and if users have more complex needs, they can create their own BeautifulSoup objects and call convert_soup().

[chrispy-snps]

@AlexVonB, @matthewwithanm - I think I came up with a solution that is the best of both worlds - see #224 and let me know what you think!