Peer forces super high fees on a mutual close with only local balance

[viaj3ro]

peer forces crazy tx fee on my own to self transaction. Seems very unnecessary. Also happens during low fee periods. Can this be prevented?

https://blockstream.info/tx/42b6333c6a15feeda04549c0eb842ab80858e51c31ecbc3a2b583b718d5c1c19

2020-10-28 21:48:05,631 INFO  f.a.e.i.Peer n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - OUT msg=Shutdown(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,ByteVector(22 bytes, 0x001474d56abcf11c099eff8eb10a4d78af1aeaa4a8cc))
2020-10-28 21:48:05,632 INFO  f.a.eclair.Diagnostics n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - OUT msg=Shutdown(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,ByteVector(22 bytes, 0x001474d56abcf11c099eff8eb10a4d78af1aeaa4a8cc))
2020-10-28 21:48:06,104 INFO  f.a.eclair.Diagnostics n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - IN msg=Shutdown(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,ByteVector(22 bytes, 0x001430c7f666a92cc928a7898eec37667801f0006811))
2020-10-28 21:48:06,104 INFO  f.a.e.i.Peer n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - IN msg=Shutdown(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,ByteVector(22 bytes, 0x001430c7f666a92cc928a7898eec37667801f0006811))
2020-10-28 21:48:06,104 INFO  f.a.e.channel.Channel n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - using feeratePerKw=FeeratePerKw(Satoshi(253)) for initial closing tx
2020-10-28 21:48:06,105 INFO  f.a.e.channel.Channel n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - signed closing txid=aa17f16267cf982f8c85b95ea0a4486dadd35e021dcbb3821d1a3927d281a133 with closingFeeSatoshis=Satoshi(170)
2020-10-28 21:48:06,122 INFO  f.a.e.i.Peer n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - OUT msg=ClosingSigned(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,Satoshi(170),ec600bcdafc0fe211e26e89435d9a24f5f8c1e98e20f982524aacc14ffeae0fd604b834686eb1a09041f6402726423dd426cc4221bd43b94e389357cb3f832ad)
2020-10-28 21:48:06,123 INFO  f.a.eclair.Diagnostics n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - OUT msg=ClosingSigned(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,Satoshi(170),ec600bcdafc0fe211e26e89435d9a24f5f8c1e98e20f982524aacc14ffeae0fd604b834686eb1a09041f6402726423dd426cc4221bd43b94e389357cb3f832ad)
2020-10-28 21:48:06,433 INFO  f.a.eclair.Diagnostics n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - IN msg=ClosingSigned(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,Satoshi(42524),725460a809ed1dee7209c0d6bd3a694dd98ab2591bfec4ea5178c9d3ca30faad68c1f8b3f60bf55f8f3f82abab63a8e3e98f13860f11ee314792bbbb7856aeec)
2020-10-28 21:48:06,433 INFO  f.a.e.i.Peer n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - IN msg=ClosingSigned(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,Satoshi(42524),725460a809ed1dee7209c0d6bd3a694dd98ab2591bfec4ea5178c9d3ca30faad68c1f8b3f60bf55f8f3f82abab63a8e3e98f13860f11ee314792bbbb7856aeec)
2020-10-28 21:48:06,434 INFO  f.a.e.channel.Channel n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - received closingFeeSatoshis=Satoshi(42524)
2020-10-28 21:48:06,434 INFO  f.a.e.channel.Channel n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - signed closing txid=fe08176c00a2f719739da13b83834b99cc2a8a56cfe5c78f3f6130cd38d6bf37 with closingFeeSatoshis=Satoshi(42524)
2020-10-28 21:48:06,435 INFO  f.a.e.channel.Channel n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - signed closing txid=a405d5ba450bb13a598960f7689e75eb5565eb8b2fbcec29eed98919d6af00a7 with closingFeeSatoshis=Satoshi(21346)
2020-10-28 21:48:06,435 INFO  f.a.e.channel.Channel n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - proposing closingFeeSatoshis=Satoshi(21346)
2020-10-28 21:48:06,453 INFO  f.a.eclair.router.Router SYN n:032cc4541b25e86e39a7d450a979c1a9adbe2878df3a93fcb59c96c700bfe26aa3 - validating shortChannelId=562216x2271x1
2020-10-28 21:48:06,456 INFO  f.a.e.i.Peer n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - OUT msg=ClosingSigned(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,Satoshi(21346),d9c644ba2b7209436034b03e926c6eebc477b7eb4bfcbef207fea11b4c7ed3bf780a975fed396de6e80bd5bed0df38df774dc61bada0385a85e0ab511994b0a6)
2020-10-28 21:48:06,456 INFO  f.a.eclair.Diagnostics n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - OUT msg=ClosingSigned(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,Satoshi(21346),d9c644ba2b7209436034b03e926c6eebc477b7eb4bfcbef207fea11b4c7ed3bf780a975fed396de6e80bd5bed0df38df774dc61bada0385a85e0ab511994b0a6)
2020-10-28 21:48:06,568 INFO  f.a.eclair.router.Router SYN n:032cc4541b25e86e39a7d450a979c1a9adbe2878df3a93fcb59c96c700bfe26aa3 - got validation result for shortChannelId=562216x2271x1 (awaiting=1 stash.nodes=0 stash.updates=2)
2020-10-28 21:48:06,771 INFO  f.a.eclair.Diagnostics n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - IN msg=ClosingSigned(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,Satoshi(38272),fc962a9132899c3eb589efea15f565f9e0b21a42f99163fbb72dff89471c3ec56437a81ea942596a6a39d0e9569745c1c175199a4de94ff1c6182509b97e279c)
2020-10-28 21:48:06,771 INFO  f.a.e.i.Peer n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - IN msg=ClosingSigned(8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9,Satoshi(38272),fc962a9132899c3eb589efea15f565f9e0b21a42f99163fbb72dff89471c3ec56437a81ea942596a6a39d0e9569745c1c175199a4de94ff1c6182509b97e279c)
2020-10-28 21:48:06,771 INFO  f.a.e.channel.Channel n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - received closingFeeSatoshis=Satoshi(38272)
2020-10-28 21:48:06,772 INFO  f.a.e.channel.Channel n:03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e c:8706970005bf94fd9237adf4e83f8dd34e1deb621af9d6fb3d7eab8385f22cc9 - signed closing txid=e8fdb7c65b15e63a9f42bc0d046d3fc3dc4a57ea3341468567d54af3ec6ad271 with closingFeeSatoshis=Satoshi(38272)

[t-bast]

Unfortunately this is the design of mutual close, you and your peer have to agree on a fee.
The only other option is to force-close, but the fee will be even higher.
That's something that could be fixed in the long run by using CPFP like what we do for anchor outputs, but this isn't in the works for closing txs for now.

[viaj3ro]

I assumed anchor outputs are specifically used for closing txs. Seems like I'm mistaken, What else are they good for? Force close?

[t-bast]

Yes exactly, they are for force-close

[viaj3ro]

I see. Force close should hopefully be a rarity in the not so distant future (already got much, much better actually). Regular closing TX will always be quite common. IMHO it would be a big improvement for everyone, if anchor outputs could also be used for those.

Does it make sense to push for this? If yes, who should I talk to?

[t-bast]

I don't think it makes sense to have anchor outputs on mutual close.
Mutual close should by design result in a fee that both nodes are happy with (usually in the middle between your ideal fee and theirs). But it's true we could be more aggressive with aiming for a low fee, at the cost of potentially doing a CPFP later if it doesn't confirm soon enough.

Note that when the closings in your logs happened, the mempool was really full and the feerate was very high.

[viaj3ro]

well, lightning will become VERY expensive or might even fail if nothing is done to significantly reduce the fee burden. Routing nodes are losing money even at 1% fee (during a time of empty blocks and 1sat/b tx confirming). Now 100 or 200x that for a time with full blocks....

IMHO the only way lightning can stay competitive, is by making batch opening and closing standard (close n channels while simultaneously opening n channels all in one tx) plus lowballing the fee and making use of cheap weekend fees.

Otherwise full custodial solutions like PayPal, ETFs, Grayscale, BlueWallet & exchanges will take over even more and we all know what that means...

edit:
have you ever added up the network fees for the ACINQ node? They must be pretty enormous by now?!

[t-bast]

IMHO the only way lightning can stay competitive, is by making batch opening and closing standard (close n channels while simultaneously opening n channels all in one tx) plus lowballing the fee and making use of cheap weekend fees.

We are working on batch open thanks to PSBT, but closing cannot be batched right now.
The closing fee depends on both you and your partner, the high fees you saw are because your partners are much more conservative with their feerates. Were you the one who initiated the mutual close? If so, we may be able to add a desired_feerate to the command, and ensure the closing fee matches that; note however that if your peer refuses to sign for that fee, the only option will be a force-close, so what's important is to also notify your peers' implementations to work on that.

[viaj3ro]

Yeah, I know that it's not really up to me or eclair but mostly to my peers. That's why I initially raised this issue with LND lightningnetwork/lnd#4413 and only opened this one since it was a special case with balance only on my side. Was hoping there is a way for such a closing (where peer has no money at stake), fees could maybe be determined by my node alone. But apparently that's not possible.

What I really don't understand, is why being more fee efficient is such a low priority. LN promises to be more cost effective than on-chain. It's the sole reason for it's existence (yeah, ok, maybe not the only reason, but by far the biggest). But yet, even during almost 2 years of empty blocks, running a routing node is quite expensive even when you only count network fees and ignore everything else like hardware, electricity, opportunity cost, onlinewallet risk, etc. The only reason LN is somewhat operational still, is due to routing node operators donating their BTC satoshi for satoshi in order to keep their nodes running at a loss. That's not a viable long term strategy and it will only get MUCH, MUCH worse with the next hype cycle and long tx backlogs.

LND doesn't even have coin control and just creates a giant UTXO mess: https://twitter.com/matt_odell/status/1321806485937098757

And the biggest liquidity provider doesn't even seem to understand that his coins are unnecessarily wasted for overpaid network fees: https://twitter.com/lnbig_com/status/1299320526507311104

Unfortunately, the real balance at all nodes is slowly decreasing every day, despite the fees. It is difficult to understand the reason, because the funds are scattered in different places that are constantly changing: pending/opening/closing channels, wallets and etc.

I'm really at a loss here. Seems like I'm the only one who really cares even though it's most definitely hurting everyone. Probably most people don't actually realize how much they are losing. Will ask the RTL guys to ad a network fee info panel into their dashboard. Maybe this will help to highlight the issue.

I really have a hard time seeing how LN can survive in a high fee environment without serious improvements. If it doesn't happen soon, node operators will start throwing in the towel and it will be easy to discredit LN which in turn will hinder adoption or might even keep it in it's little economically meaningless niche where it is right now.

[t-bast]

What I really don't understand, is why being more fee efficient is such a low priority.

That's only because you're not following the technical discussions and spec meetings.
It is a priority, it's just that the "right" solutions are long term ones: they need package relay to be implemented in Bitcoin Core (no-one knows when that will land), better support for RBF/CPFP of PSBT in Bitcoin Core (probably around v0.22), and more work on anchor outputs (this one is making steady progress). Recent PSBT additions to Bitcoin Core will allow us to improve fees for funding txs and force-close scenarios: this is work in progress, be patient.

What you're also missing is that we're actively working on more important priorities: making sure funds are safe. This is the number 1 priority. Many cross-layer attacks have been discovered this year, and protecting against those has been a lot of work.

[viaj3ro]

Fair points. Any other good places besides the mailing list and bitcoinops to keep up with whats going on in the dev community?

I'm kind of barking up the wrong tree here anyway, since you guys actually did implement options to lower fees significantly. (Even though I think the standard setting for closing should be 144 blocks. Nobody dies from waiting a few hours for a closing tx to confirm and anyone unhappy can always change the setting.)

I'm mostly taking issue with LND nodes refusing to settle for anything reasonable on mutual close and raising the issue on their github hasn't really given me the impression that anyone cares. The mentioned UTXO mess and missing coin control kind of seemed to confirm this feeling.

[t-bast]

You can monitor the bi-weekly meeting issues we create on the spec repo, and skim through the spec meeting logs, see for example lightning/bolts#805

I'd like to finish some work on RBF/CPFP for eclair, once that's done I think we can indeed make the default target for closing 144 blocks (and it's always possible to do CPFP to speed up confirmation if needed). The RBF/CPFP work will be a bit of a rabbit hole, I'm not sure how long that will take but that's one of my priorities right now.

Regarding LND, it's tough for them because they really have a lot of users, which means a lot of noise. They're a bigger team, but it's probably not enough to compensate. I'm sure they're aware of these issues, but since they also have more features and projects to maintain, they can't do everything they would like to in a timely manner. We'll get there eventually though, don't hesitate to keep pushing!

[viaj3ro]

Thanks for having an open ear and taking the time to explain what is happening behind the curtains.

If so, we may be able to add a desired_feerate to the command, and ensure the closing fee matches that; note however that if your peer refuses to sign for that fee, the only option will be a force-close,

this would actually be really useful. Would love to try this and see how peers react. If it isn't too much work, I hope to see this implemented soon.

[hosiawak]

Similar issue in C-Lightning

@viaj3ro If you read the comments on the above C-Lightning issue you'll notice it has similar problems with fees being too high, especially in low fee periods.

[t-bast]

Fixed by #1768 (once implemented by other nodes in the network)