diff --git a/README.md b/README.md index 34e162d..ed04e34 100644 --- a/README.md +++ b/README.md @@ -102,5 +102,5 @@ Exceptions to this license include the fonts in `static/fonts`, which are licens The Bytemark name and logo (`static/images/bytemark.png`) are registered trademarks of Bytemark Limited. -[tag-previous]: https://github.com/HackSoc/hacksoc.org/tree/08694ad0fd706c4ff4580303a97031452d73772d +[tag-previous]: https://github.com/HackSoc/hacksoc.org/tree/node-last [tag-hackyll]: https://github.com/HackSoc/hacksoc.org/tree/hakyll-last diff --git a/docs/adding_features_python.md b/docs/adding_features_python.md index 80340ce..214b071 100644 --- a/docs/adding_features_python.md +++ b/docs/adding_features_python.md @@ -34,7 +34,7 @@ NB: The Jinja documentation uses the term "template" to refer to any of: [URL generators](https://pythonhosted.org/Frozen-Flask/#url-generators) should only be used for files that aren't intended to be part of the main site, or are intended to be "unlisted". If there's no chain of links from the main page to another page, then you can't really expect anyone to find that page organically. Implementing a URL generator just requires adding a Python function that calls `yield` with each of the routes you want to ensure are rendered, and decorating the function with `@freezer.register_generator`. ## Modifying the Markdown handler -All things Markdown are hidden away in `markdown.py` so the rest of the package doesn't have to worry about it. It exports a single function, `render_markdown` which takes a Markdown source string and returns a rendered HTML string. Even if this wraps a single function from a Markdown library, if that library ever changes in the future it means that the codebase only needs to be changed in one place. +All things Markdown are hidden away in `markdown/__init__.py` so the rest of the `hacksoc_org` package doesn't have to worry about it. It exports a single function, `render_markdown` which takes a Markdown source string and returns a rendered HTML string. Even if this wraps a single function from a Markdown library, if that library ever changes in the future it means that the codebase only needs to be changed in one place. Currently the website uses [`python-markdown2`][pymd2] and loads the following extras: - `fenced-code-blocks` allow blocks of code to be placed between triple-backticks (\`\`\`) @@ -46,20 +46,74 @@ Since the server READMEs use fenced code blocks and AGMs will often use many tab ### Choice of Markdown libraries Choosing a Markdown backend is not straightforward; implementations vary in their interpretation of the spec (Gruber's `markdown.pl` or the less ambiguous CommonMark standard) and their extra features (tables, code block highlighting, smart quotes). Currently [`markdown2`][pymd2] is used, although its non-conformance with CommonMark makes a replacement desireable. -To help test between Markdown backends, non-default backends can be selected with the `--markdown` command-line option. Only [`cmark`](https://github.com/commonmark/cmark) is available (provided through the [`cmarkgfm`](https://github.com/theacodes/cmarkgfm) Python bindings). +To help test between Markdown backends, non-default backends can be selected with the `--markdown` command-line option. ``` # Equivalent; markdown2 is the default backend hacksoc_org run -hacksoc_org run --markdown markdown2 +hacksoc_org run --markdown markdown2 -# Use cmark instead +# Use an alternative instead hacksoc_org run --markdown cmark +hacksoc_org run --markdown commonmark +hacksoc_org run --markdown mistletoe +hacksoc_org run --markdown markdown-it # this works with all subcommands -hacksoc_org freeze --markdown cmark +hacksoc_org freeze --markdown cmark + +# in any order +hacksoc_org --markdown cmark run +``` +#### Markdown2 (Current default) +[GitHub](https://github.com/trentm/python-markdown2/wiki) + +Provides syntax highlighting via Pygments. Aims to conform to Gruber's Markdown rather than CommonMark. + +#### Cmark +GitHub ([Python bindings](https://github.com/theacodes/cmarkgfm), [C implementation](https://github.com/github/cmark-gfm)) + +Fork of the CommonMark reference implementation, maintained by GitHub to provide [GFM](https://github.github.com/gfm/) features. Only provides syntax highlighing for the client-side, by adding a (eg.) `lang="py"` attribute to `
` blocks for a JavaScript library to parse and highlight. +#### CommonMark +[GitHub](https://github.com/readthedocs/commonmark.py) + +Python port of CommonMark.js, maintained by readthedocs. Provides no syntax highlighting out-of-the-box, but has been added with [`commonmark_pygments_renderer.py`](../hacksoc_org/markdown/commonmark_pygments_renderer.py). + +**Does not support tables**. +#### Mistletoe +[GitHub](https://github.com/miyuchina/mistletoe) + +Gives an example integration of Pygments for syntax highlighing, adapted into [`mistletoe_pygments_renderer.py`](../hacksoc_org/markdown/mistletoe_pygments_renderer.py) +#### Markdown-it +[GitHub](https://github.com/executablebooks/markdown-it-py) + +Complicated architecture. Syntax highlighing added with Pygments in [`markdownit_pygments_highlighter.py`](../hacksoc_org/markdown/markdownit_pygments_highlighter.py). + +```html + +++ +
++ ++ + +
++ ++ ``` +#### Others +There are many more CommonMark-comforming backends that aren't enabled (yet). They include: + + - [Mistune](https://github.com/lepture/mistune) + - Doesn't provide syntax highlighing, but could be added by overriding HTMLRenderer::block_code + - Not [strictly](https://github.com/miyuchina/mistletoe#performance) CommonMark compliant + ## Serving Flask in production Some of Flask's extra power (handling POST requests, HTTP redirects) require it to be run in production (as opposed to generating HTML files and serving those from a static web server). Currently the [configuration](../.flaskenv) of Flask puts it into debug mode. This is extremely unsafe to run in production. Secondly, `hacksoc_org run` or `app.run()` should not be used in production as it used Flask's built-in development server, which is not suitable for production use even when debug mode is disabled. Instead, consult [Flask's documentation](https://flask.palletsprojects.com/en/2.0.x/deploying/#self-hosted-options) on options for WSGI and CGI servers. diff --git a/hacksoc_org/__init__.py b/hacksoc_org/__init__.py index 5a93d57..d49f78b 100644 --- a/hacksoc_org/__init__.py +++ b/hacksoc_org/__init__.py @@ -14,6 +14,9 @@ app = flask.Flask(__name__, static_folder=None, template_folder=None) # these folders are defined in the Blueprint anyway app.config["TEMPLATES_AUTO_RELOAD"] = True +app.config["ENV"] = "development" +app.config["DEBUG"] = True + app.jinja_env.add_extension("jinja2.ext.do") # adds support for the jinja "do" statement diff --git a/hacksoc_org/cli.py b/hacksoc_org/cli.py index cd8bd13..74fd5c5 100644 --- a/hacksoc_org/cli.py +++ b/hacksoc_org/cli.py @@ -3,6 +3,7 @@ from hacksoc_org.consts import * from hacksoc_org.freeze import freeze from hacksoc_org.serve import serve +from hacksoc_org.markdown import implementations, get_backend_help import argparse @@ -18,13 +19,11 @@ def inner(fn): def main(args=None): - parser = argparse.ArgumentParser( - allow_abbrev=False, - epilog=""" + epilog = """ SUBCOMMANDS run - Starts a local development server on https://localhost:5000/. Automatically + Starts a local development server on http://localhost:5000/. Automatically reloads when templates or Python code is changed. Recommended while developing pages or features. @@ -36,11 +35,17 @@ def main(args=None): serve Calls `freeze` then starts a local HTTP server from `build/` on - https://localhost:5000/. Will not automatically rebuild the website on + http://localhost:5000/. Will not automatically rebuild the website on content change, you will need to re-run `serve`. Recommended to use this at least once to check that a) new content is part of the "frozen" site and b) no errors occur in freezing the site. -""".strip(), +""".strip() + + epilog += "\n\n" + get_backend_help() + + parser = argparse.ArgumentParser( + allow_abbrev=False, + epilog=epilog, formatter_class=argparse.RawDescriptionHelpFormatter, ) @@ -50,7 +55,7 @@ def main(args=None): parser.add_argument( "--markdown", - choices=["markdown2", "cmark"], + choices=list(implementations.keys()), default="markdown2", help="Markdown backend to use (default markdown2)", ) diff --git a/hacksoc_org/markdown.py b/hacksoc_org/markdown.py deleted file mode 100644 index e13d146..0000000 --- a/hacksoc_org/markdown.py +++ /dev/null @@ -1,71 +0,0 @@ -""" - Wrapper module to provide a consistent Markdown handler, regardless of the library - implementation used. -""" - -import abc -from hacksoc_org.consts import CFG_MARKDOWN_IMPL -from hacksoc_org import app - - -class AbstractMarkdown(abc.ABC): - @abc.abstractmethod - def __init__(self) -> None: - super().__init__() - - @abc.abstractmethod - def render_markdown(self, markdown_src: str) -> str: - pass - - -class Markdown2MD(AbstractMarkdown): - def __init__(self) -> None: - import markdown2 - - self.md = markdown2.Markdown( - extras=[ - "fenced-code-blocks", - "cuddled-lists", - "tables", - # Markdown2 has a `metadata` Extra to allow frontmatter parsing - # this is not loaded and python-frontmatter is used instead to allow the markdown parser to - # be changed easily if required. - ] - ) - - def render_markdown(self, markdown_src: str) -> str: - return self.md.convert(markdown_src) - - -class CmarkgfmMD(AbstractMarkdown): - def __init__(self) -> None: - import cmarkgfm - - self.cmarkgfm = cmarkgfm - - def render_markdown(self, markdown_src: str) -> str: - return self.cmarkgfm.github_flavored_markdown_to_html(markdown_src) - - -def get_markdown_cls(): - return {"markdown2": Markdown2MD, "cmark": CmarkgfmMD}[app.config[CFG_MARKDOWN_IMPL]] - - -_markdowner = None - - -def render_markdown(markdown_src: str) -> str: - """Renders the given markdown source into HTML - - Args: - markdown_src (str): Markdown source - - Returns: - str: HTML text - """ - global _markdowner - - if _markdowner is None: - _markdowner = get_markdown_cls()() - - return _markdowner.render_markdown(markdown_src) diff --git a/hacksoc_org/markdown/__init__.py b/hacksoc_org/markdown/__init__.py new file mode 100644 index 0000000..2871add --- /dev/null +++ b/hacksoc_org/markdown/__init__.py @@ -0,0 +1,176 @@ +""" + Wrapper module to provide a consistent Markdown handler, regardless of the library + implementation used. +""" + +import abc +from hacksoc_org.consts import CFG_MARKDOWN_IMPL +from hacksoc_org import app +import textwrap +from inspect import cleandoc + + +class AbstractMarkdown(abc.ABC): + @abc.abstractmethod + def __init__(self) -> None: + super().__init__() + + @abc.abstractmethod + def render_markdown(self, markdown_src: str) -> str: + pass + + @abc.abstractstaticmethod + def key() -> str: + pass + + +class Markdown2MD(AbstractMarkdown): + """ + Default + Not CommonMark compliant (aims to match Markdown.pl) + """ + + def __init__(self) -> None: + import markdown2 + + self.md = markdown2.Markdown( + extras=[ + "fenced-code-blocks", + "cuddled-lists", + "tables", + # Markdown2 has a `metadata` Extra to allow frontmatter parsing + # this is not loaded and python-frontmatter is used instead to allow the markdown parser to + # be changed easily if required. + ] + ) + + def render_markdown(self, markdown_src: str) -> str: + return self.md.convert(markdown_src) + + def key(): + return "markdown2" + + +class CmarkgfmMD(AbstractMarkdown): + """ + CommonMark compliant + Doesn't support syntax highlighting + """ + + def __init__(self) -> None: + import cmarkgfm + + self.cmarkgfm = cmarkgfm + self.options = cmarkgfm.Options.CMARK_OPT_UNSAFE + + def render_markdown(self, markdown_src: str) -> str: + return self.cmarkgfm.github_flavored_markdown_to_html(markdown_src, self.options) + + def key() -> str: + return "cmark" + + +class CommonMarkMD(AbstractMarkdown): + """ + CommonMark compliant + Doesn't support tables + """ + + def __init__(self) -> None: + import commonmark + from .commonmark_pygments_renderer import PygmentsRenderer + + self.parser = commonmark.Parser() + self.renderer = PygmentsRenderer() + + def render_markdown(self, markdown_src: str) -> str: + ast = self.parser.parse(markdown_src) + return self.renderer.render(ast) + + def key() -> str: + return "commonmark" + + +class MistletoeMD(AbstractMarkdown): + """ + CommonMark compliant + """ + + def __init__(self) -> None: + import mistletoe + from .mistletoe_pygments_renderer import PygmentsRenderer + + self.renderer = PygmentsRenderer() + self.Document = mistletoe.Document + + def render_markdown(self, markdown_src: str) -> str: + return self.renderer.render(self.Document(markdown_src)) + + def key() -> str: + return "mistletoe" + + +class MarkdownItMD(AbstractMarkdown): + """ + CommonMark compliant + """ + + def __init__(self) -> None: + import markdown_it + from .markdownit_pygments_highlighter import PygmentsHighlighter + + self.md = markdown_it.MarkdownIt().enable("table") + self.md.options["highlight"] = PygmentsHighlighter() + + def render_markdown(self, markdown_src: str) -> str: + return self.md.render(markdown_src) + + def key() -> str: + return "markdown-it" + + +implementations = { + c.key(): c + for c in [ + Markdown2MD, + CmarkgfmMD, + CommonMarkMD, + MistletoeMD, + MarkdownItMD, + ] +} + + +def get_markdown_cls(): + return implementations[app.config[CFG_MARKDOWN_IMPL]] + + +_markdowner = None + + +def render_markdown(markdown_src: str) -> str: + """Renders the given markdown source into HTML + + Args: + markdown_src (str): Markdown source + + Returns: + str: HTML text + """ + global _markdowner + + if _markdowner is None: + _markdowner = get_markdown_cls()() + + return _markdowner.render_markdown(markdown_src) + + +def get_backend_help() -> str: + s = "MARKDOWN BACKENDS\n\n" + + for impl in implementations.values(): + s += " " + impl.key() + "\n" + if impl.__doc__ is not None: + s += textwrap.indent(cleandoc(impl.__doc__), " ") + s += "\n\n" + return s.strip() diff --git a/hacksoc_org/markdown/commonmark_pygments_renderer.py b/hacksoc_org/markdown/commonmark_pygments_renderer.py new file mode 100644 index 0000000..f13b185 --- /dev/null +++ b/hacksoc_org/markdown/commonmark_pygments_renderer.py @@ -0,0 +1,26 @@ +import commonmark + +from pygments import highlight +from pygments.lexers import get_lexer_by_name, guess_lexer +from pygments.formatters.html import HtmlFormatter + + +class PygmentsRenderer(commonmark.HtmlRenderer): + def __init__(self, options={}) -> None: + super().__init__(options=options) + self.formatter = HtmlFormatter() + self.formatter.cssclass = "codehilite" + self.formatter.wrapcode = True + + def code_block(self, node, entering) -> None: + attrs = self.attrs(node) + + info_words = node.info.split() if node.info else [] + if len(info_words) > 0 and len(info_words[0]) > 0: + lexer = get_lexer_by_name(info_words[0]) + else: + lexer = guess_lexer(node.literal) + + self.cr() + self.lit(highlight(node.literal, lexer, self.formatter)) + self.cr() diff --git a/hacksoc_org/markdown/markdownit_pygments_highlighter.py b/hacksoc_org/markdown/markdownit_pygments_highlighter.py new file mode 100644 index 0000000..296718c --- /dev/null +++ b/hacksoc_org/markdown/markdownit_pygments_highlighter.py @@ -0,0 +1,18 @@ +from pygments import highlight +from pygments.lexers import get_lexer_by_name, guess_lexer +from pygments.formatters.html import HtmlFormatter + + +class PygmentsHighlighter: + def __init__(self) -> None: + self.formatter = HtmlFormatter() + self.formatter.cssclass = "codehilite" + self.formatter.wrapcode = True + + def __call__(self, content, lang, attrs): + if lang is None or len(lang) == 0: + lexer = guess_lexer(content) + else: + lexer = get_lexer_by_name(lang) + + return highlight(content, lexer, self.formatter) diff --git a/hacksoc_org/markdown/mistletoe_pygments_renderer.py b/hacksoc_org/markdown/mistletoe_pygments_renderer.py new file mode 100644 index 0000000..cc34190 --- /dev/null +++ b/hacksoc_org/markdown/mistletoe_pygments_renderer.py @@ -0,0 +1,46 @@ +""" +contrib/pygments_renderer.py from https://github.com/miyuchina/mistletoe + +The MIT License + +Copyright 2017 Mi Yu + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +""" +from mistletoe import HTMLRenderer +from pygments import highlight +from pygments.styles import get_style_by_name as get_style +from pygments.lexers import get_lexer_by_name as get_lexer, guess_lexer +from pygments.formatters.html import HtmlFormatter + + +class PygmentsRenderer(HTMLRenderer): + formatter = HtmlFormatter() + formatter.noclasses = False + formatter.cssclass = "codehilite" + formatter.wrapcode = True + + def __init__(self, *extras, style="default"): + super().__init__(*extras) + self.formatter.style = get_style(style) + + def render_block_code(self, token): + code = token.children[0].content + lexer = get_lexer(token.language) if token.language else guess_lexer(code) + return highlight(code, lexer, self.formatter) diff --git a/init-venv.sh b/init-venv.sh deleted file mode 100755 index 0ea8208..0000000 --- a/init-venv.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash - -set -e - -python3 -m venv venv/ -source venv/bin/activate -pip install --upgrade -r pip-requirements.txt diff --git a/setup.cfg b/setup.cfg index 7c0be00..2a5e30c 100644 --- a/setup.cfg +++ b/setup.cfg @@ -14,12 +14,10 @@ install_requires = black markdown2 cmarkgfm - # markdown-it-py - # mistletoe - # commonmark + commonmark + mistletoe + markdown-it-py [options.entry_points] console_scripts = hacksoc_org = hacksoc_org.cli:main - -# [options.extras_require]