Apply formatting to markdown code blocks

[amyreese]

Adds initial support for formatting Python code blocks inside Markdown files.

• Adds Markdown source types/kinds
• Maps .md file extension to Markdown by default
• Uses simple regex adapted from blacken-docs to find and format fenced python code blocks
• Dedents contents before formatting, and reapplies indent from fenced ```py header
• Selects Python vs Stub options based on language label on code block
• Silently skips formatting for any code block with syntax errors or that produce formatting errors.
• CLI tests formatting via both stdin and from filesystem
• Requires running with --preview, and otherwise emits formatting error when given a markdown file
• Requires a user to extend-include = ["**/*.md"] if they want to format markdown files by default

Limitations:

• Returns a formatting error if run with a range of any sort
• Ignores implicit code blocks (no code fence)
• Doesn't yet support ~~~ fences, arbitrary fence lengths, or code blocks inside blockquotes

Issue #3792

[astral-sh-bot]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[amyreese]

minimal working prototype using regex adapted from blacken-docs:

amethyst@lunatone ~/workspace/ruff amy/ruffen-docs » cat ~/scratch/test.md
Hello, this is a *markdown* document.

This is a rust code block:

```rust
fn main() {
    for x in 0..10 {
        println!("x = {x}");
    }
}
```

This is a poorly formatted python code block:

```py
def foo(arg1,
           arg2):
    print( "hello world")



foo(1 , 2)
```

This is another python code block, also poorly formatted, but now in a list:

1. List item 1
2. List item 2

    ```python
    dataset = [1, 2, 3,
        4, 5, 6]
    if 1+2==3:
        print('yes')
    ```

And here's an unlabeled code block that happens to have valid python code — what do we do?

```
print("hello")
```

amethyst@lunatone ~/workspace/ruff amy/ruffen-docs » cargo run -p ruff -- format --no-cache --diff ~/scratch/test.md
   Compiling ruff v0.14.11 (/Users/amethyst/workspace/ruff/crates/ruff)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.66s
     Running `target/debug/ruff format --no-cache --diff /Users/amethyst/scratch/test.md`
--- /Users/amethyst/scratch/test.md
+++ /Users/amethyst/scratch/test.md
@@ -13,13 +13,11 @@
 This is a poorly formatted python code block:

 ```py
-def foo(arg1,
-           arg2):
-    print( "hello world")
+def foo(arg1, arg2):
+    print("hello world")


-
-foo(1 , 2)
+foo(1, 2)
 ```

 This is another python code block, also poorly formatted, but now in a list:
@@ -28,10 +26,9 @@
 2. List item 2

     ```python
-    dataset = [1, 2, 3,
-        4, 5, 6]
-    if 1+2==3:
-        print('yes')
+    dataset = [1, 2, 3, 4, 5, 6]
+    if 1 + 2 == 3:
+        print("yes")
     ```

 And here's an unlabeled code block that happens to have valid python code — what do we do?

1 file would be reformatted
> [1]

[amyreese]

Some notes on the regex adapted from blacken-docs:

• it does not support ~~~ delimited code blocks
• it does not support code blocks delimited by more than three backticks/tildes
• it does verify that indentation and the end delimiter matches the starting delimiter

Prototype also silently discards any formatting error, and those should either be warnings or get tracked somehow to re-raise them outside of the closure.

[amyreese]

Also need to decide on if/how to gate this behind --preview

[amyreese]

Not familiar enough with ty to know why this changed what files ty looks at, or how to make ty ignore the md files when loading the project.

[ntBre]

Nice! I found a few nits, but this otherwise looks good to me!

[ntBre]

Does this need to be tempdir_filtered? I don't think it's a big deal either way, but I wouldn't think this is needed.

[amyreese]

Micha suggested moving this into the base helper in https://github.com/astral-sh/ruff/pull/22470/changes/BASE..80911e2bbe05b87b1f58c43356f26c804c8d292a#r2711704566

[ntBre]

Yep, I saw his comment. I just thought it might have applied only to the crate_root part. I don't think the crate root should be in a temporary directory, so I think we could just leave off the tempdir_filter part. Again, it's okay to leave it since it's a no-op if the filter doesn't match, but the tests should also fail loudly if it was necessary, so I don't think it hurts to try removing the tempdir_filter call.

[amyreese]

tempdir_filter just transforms the path into a regex string, not sure why "tempdir" is in the name other than the expected use case:

ruff/crates/ruff/tests/cli/main.rs

Line 26 in 80911e2

format!(r"{}[\\/]?", regex::escape(path.as_ref()))

[ntBre]

Ohhh, I see. I had always seen this used with actual temporary directories, so I thought it was doing something special with /tmp. Sorry for the noise!

[ntBre]

Is this the same as the comment on crates/ruff/resources/test/fixtures/unformatted.md? (Did we open a follow-up issue for that?)

[tvatter]

It would be great to also support quarto markdown (qmd) files, where executability of code chunks is triggered by the addition of brackets 3{} vs 3:

{python} # this is executable code

python # this isn't

Wondered if this comment had gone unnoticed? Is there a way to include this?

[ntBre]

Amy may have different thoughts, but I would expect Quarto formatting support to follow more general Quarto support tracked in #6140 rather than as part of this feature.

[tvatter]

Why would that be? It seems that quarto support follows immediately from this PR or am I missing something?

Is it because context is carried from one block to the next? Otherwise, wouldn't modifying this line be enough?

[ntBre]

Oh I may have misunderstood how Quarto notebooks are structured. They are just Markdown files where the python in the code block header is in brackets? Yes your updated comment makes sense to me in that case, sorry for the confusion.

For a demonstration of a line plot on a polar axis, see @fig-polar.

```{python}
#| label: fig-polar
#| fig-cap: "A line plot on a polar axis"

import numpy as np
import matplotlib.pyplot as plt

r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r
fig, ax = plt.subplots(
  subplot_kw = {'projection': 'polar'} 
)
ax.plot(theta, r)
ax.set_rticks([0.5, 1, 1.5, 2])
ax.grid(True)
plt.show()
```

From https://quarto.org/docs/computations/python.html#code-blocks

[tvatter]

@ntBre #22947 is what I had in mind. Forgive me if it's really basic, it's just to give an idea.

[amyreese]

I opened #22951 to track that feature request. I'll either include it as part of #22937 or adapt your PR at some point after that.