BOLT07: don't send historical updates for gossip_timestamp_filter

[cfromknecht]

This commit removes the suggestion for nodes to respond with all known
updates that satisfy the timestamps of a gossip_timestamp_filter
message. This behavior is identical to using the initial_routing_sync,
which has been deprecated due to the inefficient nature of sending the
entire graph on every connection. Now, the requirements simply say that
only new messages should be sent to peers.

Given the upcoming addition of extended_gossip_queries,
implementations should use this feature to locate missing graph
information instead of relying the on the remote peer to dump
everything.

[cfromknecht]

Looking to get some feedback, this requirement caused a lot of issues as the graph grew last year. Given that we already tried to deprecate initial_routing_sync, I think we should move to not sending historical updates on every connection. Thoughts?

[wpaulino]

SHOULD send all gossip messages whose timestamp is greater or
equal to first_timestamp, and less than first_timestamp plus
timestamp_range.

This still seems like something we'd want to keep. Perhaps it's better to note that implementations should be aware of the consequences of setting first_timestamp to a value too far in the past?

[cfromknecht]

This still seems like something we'd want to keep.

Why? The whole point of moving to gossip_queries was to avoid dumping the full channel graph on every connection, and you can't really make use of this without triggering it. Maybe it should be changed to MAY rather than SHOULD?

Perhaps it's better to note that implementations should be aware of the consequences of setting first_timestamp to a value too far in the past?

We want to be able to set timestamps in the past, that way we catch new updates arriving at tip that may be slow to propagate due to the staggered gossip. You could say that then we shouldn't set it too far back, but how do we arrive at that value.

It becomes harder to pick a value when some nodes send historical traffic and others don't. You have to choose a balance between DOSing the remote node and forcing yourself to filter out a mound of gossip data, or just accept the loss and miss out on slow-to-propagate updates.

[cfromknecht]

Also with extended_gossip_queries I think it makes this functionality redundant, and we should encourage people to get missing data through that rather dumping the graph

[rustyrussell]

If you really want a complete gossip blast, you can always reconnect. Generally asking for everything to be re-transmitted is a bit antisocial.

[rustyrussell]

We currently rely on this behavior for the case where we find that gossip is somehow missing (pick a random peer, ask for everything). We need to be more refined in our search, I think, but that was easy to implement.

[pm47]

We want to be able to set timestamps in the past, that way we catch new updates arriving at tip that may be slow to propagate due to the staggered gossip. You could say that then we shouldn't set it too far back, but how do we arrive at that value.

It's almost as if we needed a more fine grained mechanism, maybe like #557, which we have been working on for 6+ months. Then you could set first_timestamp to now and sync older messages easily.

The inefficient sync algorithm has caused us a lot of trouble with Eclair Mobile for the better part of the past 12 months. I think we should instead move forward with #557 which addresses the core issue.

[cfromknecht]

@pm47 this doesn't replace #557, it is complimentary. This is about reducing everyone's dependence on this insanely inefficient method of synchronization, now that we have the more efficient method outlined in #557.

It's honestly just not practical to respond to these queries, esp. on mobile if you have a channel partner that requests the whole graph every time the app is opened. I've been experimenting with ripping this out of lnd simply because it's heavy and easy to abuse.