bolt09: clarify that writers SHOULD/MUST set even/odd/no bits for assumed bits?

[ekzyis]

DISCLAIMER

I'm not sure if this is the change we want, but I think there is something inconsistent between bolt11 and bolt09 with respect to assumed bits. I'm only opening this PR to start the discussion about whether there is something that should change.¹

---

I'm currently implementing bolt11 myself here. These are the features for the first test vector:

bolts/11-payment-encoding.md

Lines 388 to 390 in 0cf2151

	* `9`: features
	* `qr`: `data_length` (`q` = 0, `r` = 3; 0 * 32 + 3 == 3)
	* `sgq`: b100000100000000

Afaict, it has the 14th bit (payment_secret) and 8th bit (var_onion_optin) set. Since both bits are even, the payee indicates that both features are required, so the payer MUST use them, as mentioned in bolt09.

bolt09 also mentions that the features that can be used in bolt11 are marked with 9:

bolts/09-features.md

Line 30 in 0cf2151

* `9`: presented in [BOLT 11](11-payment-encoding.md) invoices.

However, unlike for payment_secret, even though it's also marked as assumed, var_onion_optin does not have 9 in its context column:

bolts/09-features.md

Lines 39 to 42 in 0cf2151

	| 8/9 | `var_onion_optin` | ASSUMED | | | |
	| 10/11 | `gossip_queries_ex` | Gossip queries can include additional information | IN | | [BOLT #7][bolt07-query] |
	| 12/13 | `option_static_remotekey` | ASSUMED | | | |
	| 14/15 | `payment_secret` | ASSUMED | IN9 | | [Routing Onion Specification][bolt04] |

Additionally, bolt09 also mentions this:

bolts/09-features.md

Lines 72 to 73 in 0cf2151

	* MUST support:
	* `var_onion_optin`

So to me, either the test vector is wrong for two possible reasons—it shouldn't set the bit at all, because var_onion_opint does not have 9 in its context column OR it should set the odd bit, because bolt09 specifically says it MUST support (not require!) var_onion_optin (as if there's something special about it)—OR bolt09 is inconsistent with bolt11.

The explanation about assumed bits in bolt09 also confused me:

bolts/09-features.md

Line 9 in 0cf2151

Some features were introduced and became so widespread they are `ASSUMED` to be present by all nodes, and can be safely ignored (and the semantics are only defined in prior revisions of this spec).

I thought "can be safely ignored" not only means that a reader can just assume the bit is set, but also that a writer doesn't have to set it. So maybe even payment_secret shouldn't be set? Or it just doesn't matter what I set? Now I was really confused.

For more context, since there seems to be special treatment wrt var_onion_optin, I looked into the history of it and found two relevant commits: 60cfb59 added "[origin node] MUST support var_onion_optin" for the first time in 2022, and e042c61 then marked it as assumed in 2024.

So maybe there's nothing special about var_onion_optin anymore actually, and it should also have 9 in the context column, and it doesn't need to specifically be mentioned wrt what the origin node should set?

Anyway, I hope I was able to clarify why I am confused.

The changes I propose match what I think how writers should handle assumed bits, but they are based on my limited understanding of what any of these features even actually mean.

My changes would also mean that ALL bolt11 test vectors need to be updated to include the even bits for assumed invoice features if they don't already, and I'm not in a position yet to do that, since my implementation doesn't even pass any of the existing test vectors yet. Or maybe the test vectors don't need to be updated, because the change only says writers SHOULD, not MUST? Or would that still be confusing then, if the test vectors don't even do what a writer SHOULD do?

So. Many. Questions. 😅

Footnotes

1. I also keep catching myself how I don't want to suggest updates to the spec before I even read all of it, because maybe it will be more clear then? But so far, reading more of the spec (bolt11+bolt09) just confused me more, so maybe this PR does make sense. 😅 ↩

[t-bast]

I'm not sure I followed all of your logical path towards this PR, but this change is definitely not what we want 😅

The reason for this inconsistency in the spec is simply backwards-compatibility and the fact that it's really annoying to update all of the existing test vectors (updating them isn't that hard, but getting all implementations to review the spec PR and update their test suites is painful).

We are currently in a weird state where we assume that everyone supports some features that all implementations have enabled by default for a while and don't allow node operators to disable (the ones we marked as ASSUMED). But that actually isn't true in real life: some nodes on the network are currently running forks or old versions don't support them, but we want to isolate them by rejecting connections and forcing them to either upgrade or eventually be cut off from the network. This way, we'll eventually end up in a state where we can really assume those features and maybe even remove references to the feature bits they used (but maybe not 🤷). Meanwhile, we don't want to completely remove from the spec references to the fact that these features were at some point no assumed, and may or may not be supported. We also want to minimize impact on existing customers, and be lenient in what we accept to avoid blocking payment flows just for the sake of a cleaner spec, so we allow setting those feature bits explicitly, or not setting them and assuming that they're set.

Does that clarify where we're at?

[ekzyis]

Ahh, yes, this makes a lot more sense now, thank you!!

But is this also why var_onion_optin does not have 9 in its context column, even though payment_secret does have 9? Both are assumed bits.

The spec reads to me as if some assumed bits are treated differently than others, but I’m not sure whether I’m missing something (in the spec or the real-world context), or whether there’s actually an inconsistency in the spec regarding assumed bits. I could imagine they’re treated differently because we have a different future planned for these features, as you explained.

Anyway, I don't see this as very important, since I understand that this doesn't really matter in real life.

I was mostly just curious!

[t-bast]

But is this also why var_onion_optin does not have 9 in its context column, even though payment_secret does have 9? Both are assumed bits.

It's simply that when making payment_secret assumed in #1242, the context column wasn't cleared (while it was cleared for other features when they were marked assumed). We can clear the context column for payment_secret to make it more consistent with other assumed features, that would avoid confusion.

[ekzyis]

Ok, thanks, I can do that in this PR then!

[ekzyis]

I decided to open a new PR instead. Closing.