idn-email format does not validate RFC 5321 address-literals; email rejects leading-zero Snum octets

[mwadams]

Summary

The idn-email format does not validate RFC 5321 §4.1.3 address-literals at all, and the email format validates IPv4 address-literals against the wrong grammar (RFC 3986 dec-octet instead of RFC 5321 Snum).

RFC 6531 (SMTPUTF8) extends only the local part of a mailbox with UTF8-non-ascii; the address-literal production is unchanged from RFC 5321, so idn-email should accept exactly the same literals that email does.

Reproduction

Schema:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "format": "idn-email"
}

Verified with corvusjson validateDocument (V5 engine, current main):

Data	Expected	Actual
"δοκιμή@[192.0.2.1]"	valid	invalid
"δοκιμή@[IPv6:2001:db8::1]"	valid	valid (by accident, see below)
"user@[192.0.2.300]"	invalid	invalid (wrong reason, see below)
"user@[01.0.0.1]"	valid	invalid

Additional probes showing the passes are coincidental:

Data	Expected	Actual
"user@[IPv6:zzz]" (idn-email)	invalid	valid
"user@[]" (idn-email)	invalid	valid
"user@[01.0.0.1]" (email)	valid	invalid

Root causes

Both are in src/Corvus.Text.Json/Corvus/Text/Json/JsonSchema/Internal/JsonSchemaEvaluation.String.cs, the single implementation shared by the CLI, the runtime Corvus.Text.Json.Validator, and generated standalone evaluators.

1. MatchIdnEmail has no address-literal branch. MatchEmail handles [...] domains explicitly (IPv4 / IPv6: literal detection), but MatchIdnEmail sends the domain straight to MatchIdnHostname. The observed behaviour falls out of MatchDecodedHostname quirks: it requires a letter after every dot, so every dotted-decimal literal is rejected; and it does not reject [, :, ] runes, so the dot-free [IPv6:...] string passes with no validation of the address whatsoever (hence user@[IPv6:zzz] and user@[] validate).

2. The IPv4 literal check implements the wrong RFC. MatchEmail validates bracket contents with IPAddressParser.IsValidIPV4 in canonical mode, i.e. RFC 3986 dec-octet — leading zeros forbidden. RFC 5321 Snum is 1*3DIGIT valued 0–255 — decimal only, leading zeros permitted. The parser's non-canonical mode is not a substitute: it treats a leading 0 as octal and accepts 0x hex and shortened 1–3-part forms, all wrong for Snum.

Proposed fix

1. Add a dedicated RFC 5321 Snum IPv4 matcher in JsonSchemaEvaluation.String.cs: exactly 4 dot-separated groups of 1–3 ASCII digits, each ≤ 255. Allocation-free span loop; keep it out of the dotnet/runtime-derived IPAddressParser/IPv4AddressHelper files tracked for upstream review.
2. Add the address-literal branch to MatchIdnEmail, mirroring MatchEmail but hardened: a domain starting with [ must be a well-formed literal (ends with ], contents IPv6: + MatchIPV6 or the Snum matcher) and must never fall through to MatchIdnHostname.
3. Switch MatchEmail's IPv4 literal branch from MatchIPV4 to the Snum matcher. Existing JSON-Schema-Test-Suite expectations are preserved (joe.bloggs@[127.0.0.1] valid, joe.bloggs@[127.0.0.300] invalid, joe.bloggs@[IPv6:::1] valid).
4. Failing-first unit tests in tests/Corvus.Text.Json.Tests/JsonSchemaMatchingStringTests.cs (MatchEmail_ValidatesEmail / MatchIdnEmail_ValidatesIdnEmail data rows) covering the four cases above plus the negative probes.

Out of scope (adjacent findings)

• The idn-hostname format itself accepts [, :, ] (e.g. "[]" validates) via the same MatchDecodedHostname permissiveness — separate latent bug.
• V4 (src-v4/Corvus.Json.ExtendedTypes/Corvus.Json/Validate.cs, regex + IdnMapping) is an independent implementation and needs its own assessment if equivalent tests land in the test-suite submodule.

[mwadams]

The two adjacent findings originally listed as out of scope have been brought into scope on feature/904-idn-email-address-literals.

1. idn-hostname accepted ASCII symbols (V5)

MatchDecodedHostname only rejected characters via the disallowed-IDN rune list and the Unicode contextual rules, so ASCII symbols such as [, :, _ and ! fell through and were accepted. This made "[]" and "[IPv6:2001:db8::1]" validate as idn-hostname, and also leaked through the punycode path of the plain hostname format (e.g. xn--4gbwdl.foo_bar validated even though foo_bar alone did not).

Fix: any ASCII rune that is not a letter, digit, . or - is now rejected in decoded hostname labels. Failing-first rows (7 failures before, all passing after) cover [], a bracketed IPv6 literal, :, _, !, and the two punycode fall-through cases.

2. V4 email/idn-email address-literals

Assessment of the V4 engine (Corvus.Json.ExtendedTypes) against the same cases, via new data-driven tests in CoverageGap/FormatValidationTests.cs (6 failures before the fix):

Case	V4 before	Expected
δοκιμή@[192.0.2.1] (idn-email)	invalid	valid
δοκιμή@[IPv6:2001:db8::1] (idn-email)	invalid	valid
user@[01.0.0.1] (idn-email)	invalid	valid
user@[010.0.0.1] (email and idn-email)	invalid	valid
user@EXAMPLE.COM (email, net8.0+ only)	invalid	valid

Root causes and fixes:

• TypeIdnEmail rejected all address-literals: the domain is normalized with IdnMapping.GetAscii under UseStd3AsciiRules = true, which throws on [/]; the exception is caught and reported as invalid. Fixed by branching to a new Validate.IsIdnEmailAddressLiteralMatch helper (shared by Validate and ValidateWithoutCoreType; the Warning variants delegate and need no change) that validates IPv4 literals with an RFC 5321 Snum pattern and IPv6 literals the same way V4's own ipv6 format does (IPAddress.TryParse + InterNetworkV6 + no zone id).
• EmailPattern used RFC 3986 dec-octets: the IPv4 octet subpattern rejected 3-digit leading-zero octets (010) that RFC 5321 Snum permits. Replaced all 12 octet occurrences (6 per pattern copy, including the IPv6-embedded IPv4 forms) with a Snum-based subpattern.
• Bonus divergence found during assessment: the [GeneratedRegex] copy of EmailPattern ended the TLD with [a-z0-9] where the runtime fallback (net481) has [a-zA-Z0-9], so user@EXAMPLE.COM was rejected on net8.0+ but accepted on net481. Aligned the generated copy with the fallback.

V4 generated core types call Corvus.Json.Validate.TypeIdnEmail, and the V4 code generator emits calls into these same methods, so the fixes flow through without regeneration.