Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a last-lsn header to the up-to-date control message #2215

Open
samwillis opened this issue Dec 30, 2024 · 0 comments
Open

Add a last-lsn header to the up-to-date control message #2215

samwillis opened this issue Dec 30, 2024 · 0 comments

Comments

@samwillis
Copy link
Contributor

samwillis commented Dec 30, 2024

When aligning / joining multiple streams it's important to know when the streams are up to date with each other. Currently this is not possible [1^].

It would be ideal if the up-to-date message had some form of LSN header that provided some sort of assurance of there this stream is up to date to.

There is likely to be some complexity in providing this due to the concurrent processing on shapes by a single server. Essentially it needs to be a message that says that this shape will not receive any messages <= a specific LSN, but can't just be calculated from the last LSN seen for a particular shape. It needs to be aware of the lower bound of LSNs that have been received and filtered.

[1]: We can sort of infer it is by aborting a long poll, reconnecting with live=false and as the request was made after the last message on another stream we can "guess" that we are up to date if we get an up-to-data message. However there is no guarantee (shapes are processed concurently), and we don't know for sure at what point that up-to-date was sent, so if any message came in in the interim it's impossible to know where it sits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant