importlib.metadata.metadata("package") fails with email.errors.HeaderParseError when README contains valid markdown link

[hcording]

Bug report

Bug description:

Here is a minimal example project (tested on Python 3.10):

pyproject.toml

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "test"
readme = "README.md"
version = "0.1.0"

README.md

# Test

[MIT](<https://choosealicense.com/licenses/mit/>)

test package:

• __init__.py (empty)
• bug.py: import importlib.metadata; print(importlib.metadata.metadata("test"))

To reproduce:
pip install -e .
python test/bug.py

Traceback:

Traceback (most recent call last):
  File "bug.py", line 1, in <module>
    import importlib.metadata; print(importlib.metadata.metadata("test"))
  File "lib/python3.10/email/message.py", line 135, in __str__
    return self.as_string()
  File "lib/python3.10/email/message.py", line 158, in as_string
    g.flatten(self, unixfrom=unixfrom)
  File "/home/mint/miniforge3/lib/python3.10/email/generator.py", line 116, in flatten
    self._write(msg)
  File "lib/python3.10/email/generator.py", line 199, in _write
    self._write_headers(msg)
  File "lib/python3.10/email/generator.py", line 226, in _write_headers
    self.write(self.policy.fold(h, v))
  File "lib/python3.10/email/_policybase.py", line 326, in fold
    return self._fold(name, value, sanitize=True)
  File "lib/python3.10/email/_policybase.py", line 369, in _fold
    parts.append(h.encode(linesep=self.linesep, maxlinelen=maxlinelen))
  File "lib/python3.10/email/header.py", line 389, in encode
    raise HeaderParseError("header value appears to contain "
email.errors.HeaderParseError: header value appears to contain an embedded header: '# Test\n[MIT](<https://choosealicense.com/licenses/mit/>)'

It appears that [MIT] is seen as a header, as it would be in a TOML file. That's my best guess really. Any help is appreciated!

CPython versions tested on:

3.10

Operating systems tested on:

Linux

[jaraco]

Today I encountered this error as well on a file with RST in the description:

 ~ @ pip-run build -- -c "import build.util; print(build.util.project_wheel_metadata('~/code/jaraco/jaraco.packaging'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import build.util; print(build.util.project_wheel_metadata('~/code/jaraco/jaraco.packaging'))
                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-u6naqgo3/build/util.py", line 43, in project_wheel_metadata
    builder = ProjectBuilder.from_isolated_env(
        env,
        source_dir,
        runner=runner,
    )
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-u6naqgo3/build/_builder.py", line 179, in from_isolated_env
    return cls(
        source_dir=source_dir,
        python_executable=env.python_executable,
        runner=_wrap_subprocess_runner(runner, env),
    )
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-u6naqgo3/build/_builder.py", line 154, in __init__
    _validate_source_directory(source_dir)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-u6naqgo3/build/_builder.py", line 52, in _validate_source_directory
    raise BuildException(msg)
build._exceptions.BuildException: Source ~/code/jaraco/jaraco.packaging is not a directory
 ~ [1] @ pip-run build -- -c "import build.util; print(build.util.project_wheel_metadata('code/jaraco/jaraco.packaging'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import build.util; print(build.util.project_wheel_metadata('code/jaraco/jaraco.packaging'))
                       ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/email/message.py", line 165, in __str__
    return self.as_string()
           ~~~~~~~~~~~~~~^^
  File "/opt/python/lib/python3.13/email/message.py", line 188, in as_string
    g.flatten(self, unixfrom=unixfrom)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/email/generator.py", line 115, in flatten
    self._write(msg)
    ~~~~~~~~~~~^^^^^
  File "/opt/python/lib/python3.13/email/generator.py", line 198, in _write
    self._write_headers(msg)
    ~~~~~~~~~~~~~~~~~~~^^^^^
  File "/opt/python/lib/python3.13/email/generator.py", line 225, in _write_headers
    self.write(self.policy.fold(h, v))
               ~~~~~~~~~~~~~~~~^^^^^^
  File "/opt/python/lib/python3.13/email/_policybase.py", line 326, in fold
    return self._fold(name, value, sanitize=True)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/email/_policybase.py", line 369, in _fold
    parts.append(h.encode(linesep=self.linesep, maxlinelen=maxlinelen))
                 ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/email/header.py", line 385, in encode
    raise HeaderParseError("header value appears to contain "
        "an embedded header: {!r}".format(value))
email.errors.HeaderParseError: header value appears to contain an embedded header: ".. image:: https://img.shields.io/pypi/v/jaraco.packaging.svg\n   :target: https://pypi.org/project/jaraco.packaging\n.. image:: https://img.shields.io/pypi/pyversions/jaraco.packaging.svg\n.. image:: https://github.com/jaraco/jaraco.packaging/actions/workflows/main.yml/badge.svg\n   :target: https://github.com/jaraco/jaraco.packaging/actions?query=workflow%3A%22tests%22\n   :alt: tests\n.. image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json\n    :target: https://github.com/astral-sh/ruff\n    :alt: Ruff\n.. image:: https://readthedocs.org/projects/jaracopackaging/badge/?version=latest\n   :target: https://jaracopackaging.readthedocs.io/en/latest/?badge=latest\n.. image:: https://img.shields.io/badge/skeleton-2024-informational\n   :target: https://blog.jaraco.com/skeleton\nTools for packaging.\nsphinx\n======\nThis package provides a Sphinx extension that will inject into the config\nthe following values from the project's package metadata (as presented by\ndistutils):\n - project (from name)\n - author\n - copyright (same as author)\n - version\n - release (same as version)\n - package_url (from url)\nTo enable, include 'jaraco.packaging' in the requirements and add\n'jaraco.packaging.sphinx' to the list of extensions in a Sphinx config\nfile::\n    extensions=['jaraco.packaging.sphinx']\nThe extension by default builds the project in an isolated environment in\norder to extract the metadata. For offline builds, set\n``BUILD_ENVIRONMENT=current`` and ensure the build dependencies are\nmet in the current environment.\nDeprecated: To build the documentation offline,\nprovide an already built wheel by setting the environment variable\n``JARACO_PACKAGING_SPHINX_WHEEL`` to the path of an existing wheel.\nmake-tree\n=========\nA utility for taking output from ``pipdeptree --json`` and producing a tree\nrooted at a given package.\nUsage::\n    pipdeptree --json | python -m jaraco.packaging.make-tree mypkg\nmetadata\n=========\nA wrapper around ``build.util.project_wheel_metadata`` to enable dowstream\npackagers to indicate that they need an isolated build. Set the environment\nvariable ``BUILD_ENVIRONMENT=current`` to bypass build isolation and use the\ncurrent isolation for loading metadata from a project."

I believe this is a bug in the email module incorrectly inferring an error in the (multi-line) description.

A workaround is to not print the metadata, but instead print each of the keys:

md = importlib.metadata.metadata('test')
for key in md:
    print(key, md[key], sep=": ")

Edit: on further consideration, that approach will fail to emit unique values for headers of the same key. A better workaround is needed.

[jaraco]

The error is easily reproduced without involving importlib.metadata:

>>> import email.message
>>> msg = email.message.Message()
>>> msg.add_header('Foo', 'Test\n[bar](https://example.com)\n')
>>> print(msg)
Traceback (most recent call last):
...
email.errors.HeaderParseError: header value appears to contain an embedded header: 'Test\n[bar](https://example.com)'

[jaraco]

The regex that's rejecting the header value is:

cpython/Lib/email/header.py

Lines 50 to 52 in 7595e67

	# Find a header embedded in a putative header value. Used to check for
	# header injection attack.
	_embedded_header = re.compile(r'\n[^ \t]+:')

It's detecting the [bar](https as a header key and //example.com) as the value.

[jaraco]

Aha. So maybe importlib.metadata is implicated, since it's repairing the headers thus:

cpython/Lib/importlib/metadata/_adapters.py

Lines 63 to 73 in 6f1d448

	def _repair_headers(self):
	def redent(value):
	"Correct for RFC822 indentation"
	if not value or '\n' not in value:
	return value
	return textwrap.dedent(' ' * 8 + value)
	headers = [(key, redent(value)) for key, value in vars(self)['_headers']]
	if self._payload:
	headers.append(('Description', self.get_payload()))
	return headers

The email module is assuming the headers are internally represented in their messy, escaped format, but importlib.metadata adapts that to store friendly values internally.

Probably that means that importlib.metadata should also override the rendering behavior.

[henryiii]

I get this with iminuit, too. There's it's actually in a header - the license.

assuming the headers are internally represented in their messy, escaped format

The looks like it's just adding indent to already-indented headers. I don't think they could be broken that way. But are the headers actually escaped? The email module doesn't handle unicode (at least for the body), and the specification says it's RFC822 + unicode. And headers aren't allowed to have a new newline in the writers. So most writers customize the writing, and I don't think they escape anything.

[defnull]

Just for context: In emails, a header values cannot contain \n. Long headers can be folded into multiple lines with a line break and the next line starting with white-space, but that's only for transport. Semantically, a folded header value is still just a single line of text. Folding is reversed by removing the white-space and the line-break and stitching the value back together with a single white-space. The line-breaks are lost in the process. So, using the email module for key/value pairs with multi-line values may be tricky. The current approach sounds like a very fragile way to handle this type of data.

[henryiii]

I would love if Python package metadata was JSON instead of not-really-email format. :) There's a way to express metadata (starting with 2.1 or 2.2, forget which) as JSON, but no standard filename, etc. for it.

The original "long-description" section, before it was moved to the payload, used | like so:

To support empty lines and lines with indentation with respect to the RFC 822 format, any > CRLF character has to be suffixed by 7 spaces followed by a pipe (“|”) char. As a result, > the Description field is encoded into a folded field that can be interpreted by RFC822 parser.

Example:

Description: This project provides powerful math functions
     |For example, you can use `sum()` to sum numbers:
     |
     |Example::
     |
     |    >>> sum(1, 2)
     |    3
     |

This encoding implies that any occurrences of a CRLF followed by 7 spaces and a pipe char > have to be replaced by a single CRLF when the field is unfolded using a RFC822 reader.

Alternatively, the distribution’s description may instead be provided in the message body > (i.e., after a completely blank line following the headers, with no indentation or other special formatting necessary).

(TBH, payloads aren't unicode, so this requires more hacks.)

But this was never done for the license field, such as in this example:

License: This software may only be obtained by sending the
      author a postcard, and then the user promises not
      to redistribute it.

(See https://packaging.python.org/en/latest/specifications/core-metadata/#license)

When I've tried to validate single line for the one-line summary, I had to go back because people were loading paragraphs into the summary and were mad that it broke. 🤷

---

The issue here, though, is that there seems to be some escaping needed for things like markdown and restructure text code in a field (License for example), and I'm trying to find out if there's some escaping we can do to make the parser happy.

[henryiii]

Currently, I can't reproduce the error. The same version of iminuit I was running is not reporting an error even when I try it on a range of Pythons.

# /// script
# dependencies = ["nox"]
# ///

import nox
import nox.__main__


@nox.session(python=["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"], venv_backend="uv")
def check(session):
    session.install("iminuit")
    session.run("python", "-c", "import importlib.metadata; importlib.metadata.metadata('iminuit')['License']")

if __name__ == "__main__":
    nox.__main__.main()

[zyzzyxdonta]

This problem also occurs with license identifiers in HTML comments in README.md, e.g.

<!--
SPDX-FileCopyrightText: 2025 David (Me)

SPDX-License-Identifier: GPL-3.0-or-later
-->

contents here ...

with error

email.errors.HeaderParseError: header value appears to contain an embedded header: "(complete content of README.md here)"

[jaraco]

Here's a simpler reproducer:

 🐚 py -c "import importlib.metadata as md; print(md.metadata('pip'))"

Essentially, attempting to render the metadata for any package with a multiline description is likely to fail.

I would love if Python package metadata was JSON instead of not-really-email format. :)

I do observe that the importlib.metadata._adapters.Message has a .json property. Perhaps that's the best way to work around this issue until native rendering support can be added:

 🐚 py -c "import importlib.metadata as md; print(md.metadata('pip').json)"

[jaraco]

I've rolled out a fix in importlib_metadata 8.6.0. Unfortunately, it doesn't seem to be having the desired effect:

 🐚 pip-run importlib_metadata==8.6.0 -- -c "import importlib_metadata as md; print(md.metadata('pip'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import importlib_metadata as md; print(md.metadata('pip'))
                                     ~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.14t/email/message.py", line 165, in __str__
    return self.as_string()
           ~~~~~~~~~~~~~~^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-ays8in74/importlib_metadata/_adapters.py", line 113, in as_string
    return super().as_string(policy=RawPolicy())
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.14t/email/message.py", line 188, in as_string
    g.flatten(self, unixfrom=unixfrom)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.14t/email/generator.py", line 117, in flatten
    self._write(msg)
    ~~~~~~~~~~~^^^^^
  File "/opt/python/lib/python3.14t/email/generator.py", line 200, in _write
    self._write_headers(msg)
    ~~~~~~~~~~~~~~~~~~~^^^^^
  File "/opt/python/lib/python3.14t/email/generator.py", line 234, in _write_headers
    raise HeaderWriteError(
        f'folded header contains newline: {folded!r}')
email.errors.HeaderWriteError: folded header contains newline: "Description: pip - The Python Package Installer\n        ==================================\n\n        .. |pypi-version| image:: https://img.shields.io/pypi/v/pip.svg\n           :target: https://pypi.org/project/pip/\n           :alt: PyPI\n\n        .. |python-versions| image:: https://img.shields.io/pypi/pyversions/pip\n           :target: https://pypi.org/project/pip\n           :alt: PyPI - Python Version\n\n        .. |docs-badge| image:: https://readthedocs.org/projects/pip/badge/?version=latest\n           :target: https://pip.pypa.io/en/latest\n           :alt: Documentation\n\n        |pypi-version| |python-versions| |docs-badge|\n\n        pip is the `package installer`_ for Python. You can use pip to install packages from the `Python Package Index`_ and other indexes.\n\n        Please take a look at our documentation for how to install and use pip:\n\n        * `Installation`_\n        * `Usage`_\n\n        We release updates regularly, with a new version every 3 months. Find more details in our documentation:\n\n        * `Release notes`_\n        * `Release process`_\n\n        If you find bugs, need help, or want to talk to the developers, please use our mailing lists or chat rooms:\n\n        * `Issue tracking`_\n        * `Discourse channel`_\n        * `User IRC`_\n\n        If you want to get involved head over to GitHub to get the source code, look at our development documentation and feel free to jump on the developer mailing lists and chat rooms:\n\n        * `GitHub page`_\n        * `Development documentation`_\n        * `Development IRC`_\n\n        Code of Conduct\n        ---------------\n\n        Everyone interacting in the pip project's codebases, issue trackers, chat\n        rooms, and mailing lists is expected to follow the `PSF Code of Conduct`_.\n\n        .. _package installer: https://packaging.python.org/guides/tool-recommendations/\n        .. _Python Package Index: https://pypi.org\n        .. _Installation: https://pip.pypa.io/en/stable/installation/\n        .. _Usage: https://pip.pypa.io/en/stable/\n        .. _Release notes: https://pip.pypa.io/en/stable/news.html\n        .. _Release process: https://pip.pypa.io/en/latest/development/release-process/\n        .. _GitHub page: https://github.com/pypa/pip\n        .. _Development documentation: https://pip.pypa.io/en/latest/development\n        .. _Issue tracking: https://github.com/pypa/pip/issues\n        .. _Discourse channel: https://discuss.python.org/c/packaging\n        .. _User IRC: https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:+6697/pypa\n        .. _Development IRC: https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:+6697/pypa-dev\n        .. _PSF Code of Conduct: https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md\n"

It seems that although I added some support for rendering content with newlines (by indenting it), there are still some blank lines that are not being indented.

Aha. It seems that textwrap.indent does not indent blank lines by default.

textwrap.indent('foo\n\nbar', prefix='!!!')
'!!!foo\n\n!!!bar'

[jaraco]

In importlib_metadata 8.6.1, it now honors blank lines as well. Please try it out and confirm it addresses your concerns; it should get merged into Python 3.14.