Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BODYSTRUCTURE response missing extended parameters #217

Closed
danieleggert opened this issue Sep 28, 2024 · 5 comments
Closed

BODYSTRUCTURE response missing extended parameters #217

danieleggert opened this issue Sep 28, 2024 · 5 comments

Comments

@danieleggert
Copy link

The BODYSTRUCTURE response for MIME messages is missing the extension parameters (body parameters), and most notably thus missing the MIME part boundaries of multipart messages.

Example:

Given a simple multipart message:

From: <[email protected]>
Content-type: multipart/alternative;
    boundary="47969A3F-1F13-4D6C-83F5-695D4B116B14"
MIME-version: 1.0
Subject: B
Message-id: <[email protected]>
Date: Thu, 26 Sep 2024 10:03:22 +0200
To: [email protected]

--47969A3F-1F13-4D6C-83F5-695D4B116B14
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=us-ascii

b

--47969A3F-1F13-4D6C-83F5-695D4B116B14
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
charset=us-ascii

<html><body>b</body></html>
--47969A3F-1F13-4D6C-83F5-695D4B116B14--

the server should return this BODYSTRUCTURE

4 FETCH (UID 6 BODYSTRUCTURE (("TEXT" "PLAIN" ("CHARSET" "us-ascii") NIL NIL "7BIT" 9 2 NIL NIL NIL NIL)("TEXT" "HTML" ("CHARSET" "us-ascii") NIL NIL "7BIT" 228 0 NIL NIL NIL NIL) "ALTERNATIVE" ("BOUNDARY" "47969A3F-1F13-4D6C-83F5-695D4B116B14") NIL NIL NIL))

Note the ("BOUNDARY" "47969A3F-1F13-4D6C-83F5-695D4B116B14") part.

But mox currently returns this BODYSTRUCTURE:

* 4 FETCH (UID 6 BODYSTRUCTURE (("TEXT" "PLAIN" ("CHARSET" "us-ascii") NIL NIL "7BIT" 9 2)("TEXT" "HTML" ("CHARSET" "us-ascii") NIL NIL "7BIT" 228 1) "ALTERNATIVE"))

Note: This boundary information should be included for all multi-part MIME parts, not just the main part’s boundary. If another multipart is nested inside the main message’s multipart, that part’s boundary information must also be present in the BODYSTRUCTURE.

@mjl-
Copy link
Owner

mjl- commented Oct 3, 2024

Thanks for report! Interesting, I looked at RFCs 9051 and 3501, and the multipart content-type parameters seem optional.

The fetch response item BODYSTRUCTURE is described at https://www.xmox.nl/xr/dev/rfc/9051.html#L5958. It has an example, it doesn't have multipart content-type header parameters:

      For example, a two-part message consisting of a text and a
      base64-encoded text attachment can have a body structure of:

         (("TEXT" "PLAIN" ("CHARSET" "US-ASCII") NIL NIL "7BIT" 1152 23)
          ("TEXT" "PLAIN" ("CHARSET" "US-ASCII" "NAME" "cc.diff")
          "<[email protected]>" "Compiler diff"
          "BASE64" 4554 73) "MIXED")

Just below the example:

      Extension data follows the multipart subtype.  Extension data is
      never returned with the BODY fetch but can be returned with a
      BODYSTRUCTURE fetch.  Extension data, if present, MUST be in the
      defined order.  The extension data of a multipart body part are in
      the following order:

"can be returned in a BODYSTRUCTURE fetch" seems optional too.

I believe the ABNF for the bodystructure starts at "body" at https://www.xmox.nl/xr/dev/rfc/9051.html#L6355. This is the example:

body            = "(" (body-type-1part / body-type-mpart) ")"

body-type-mpart = 1*body SP media-subtype
                     [SP body-ext-mpart]
                       ; MULTIPART body part

The SP body-ext-mpart is optional.

As counterpoints, the ABNF rule is called "body", and the syntax is mostly shared between "BODY" and "BODYSTRUCTURE", with some mention of BODY being a non-extensible form of BODYSTRUCTURE. Perhaps the "can" in "extension data [...] can be returned with a BODYSTRUCTURE fetch" was intended to mean that for BODY it won't be included and for BODYSTRUCTURE it will be. And perhaps the ABNF was written a bit sloppily.

RFC 3501 has pretty much the same text, at https://www.xmox.nl/xr/dev/rfc/3501.html#L4125. ABNF is the same too.

Are you aware of clients that expect/require it and can't parse the response otherwise?

I'm not opposed to adding the multipart content-type parameters to the response. We have the parsed information available. We don't have the all the other fields of "body-ext-mpart" available (that are NIL in your example, the disposition/language/location/...), but those are optional so I would leave them out (assuming clients don't expect need those too).

mjl- added a commit that referenced this issue Oct 4, 2024
not generating it yet from imapserver because we don't have content-md5
available. we could send "nil" instead of any actual content-md5 header (and
probably no contemporary messages include a content-md5 header), but it would
not be correct. if no known clients have problems in practice with absent
extensible data, it's better to just leave the bodystructure as is, with
extensible data.

for issue #217 by danieleggert
@mjl-
Copy link
Owner

mjl- commented Oct 4, 2024

While adding the extensible form to the bodystructure fetch attribute responses, I realized the body-ext-1part (for non-multiparts) would then also be expected by clients. That requires the contents of any Content-MD5 header, which isn't readily available when making the response. It would be a bit much to parse the whole message just to fetch that Content-MD5 field... The alternative is just returning NIL for the content-md5. No one is probably using the field, and messages probably never include that header, but it would not be compliant. So I'm holding off that change until it's clear if this is really needed for interoperability.

@danieleggert
Copy link
Author

danieleggert commented Nov 1, 2024

AFAIK, most servers cache BODYSTRUCTURE per message once it’s been generated. Not sure if that’s an option for mox. But they are quite expensive to generate because they require parsing the full message.

The critical pieces of info that’s currently missing are any attributes of the content-type.

mjl- added a commit that referenced this issue Nov 1, 2024
…ent-type parameters for multiparts so clients will get the mime boundary without having to parse the message themselves

"bodystructure" is like "body", but bodystructure allows returning more
information. we chose not to do that, initially because it was easier to
implement, and more recently because we can't easily return the additional
content-md5 field for leaf parts (since we don't have it in parsed form). but
now we just return the extended form for multiparts, and non-extended form for
leaf parts. likely no one would be looking for any content-md5-value for leaf
parts anyway. knowing the boundary is much more likely to be useful.

for issue #217 by danieleggert, thanks for reporting!
@mjl-
Copy link
Owner

mjl- commented Nov 1, 2024

@danieleggert The commit above changes the bodystructure fetch response item to return the content-type parameters for multiparts with the extended form. It is indeed useful for clients to get the multipart boundary, and we do have it ready in parsed form. The bodystructure response item for leaf parts (non-multiparts) don't get the extended form. This asymmetry is not great, but will hopefully not cause trouble (with parsers expecting extended fields for leaf parts too). Hopefully it won't cause trouble for you? And are you using the content-md5 value for leaf parts at all?

Thanks for reporting!

@mjl-
Copy link
Owner

mjl- commented Jan 23, 2025

Closing this issue, seems to be resolved. Please reopen if there's more work to be done. Thanks again for reporting!

@mjl- mjl- closed this as completed Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants