Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cosmos: Harden and merge tip-isa-Batch branch #109

Closed
bartelink opened this issue Mar 8, 2019 · 2 comments · Fixed by #251
Closed

Cosmos: Harden and merge tip-isa-Batch branch #109

bartelink opened this issue Mar 8, 2019 · 2 comments · Fixed by #251

Comments

@bartelink
Copy link
Collaborator

There's a tip-isa-batch branch, which generalizes the storage scheme to provide even greater efficiency in terms of read and write costs.

This work was shelved for prioritisation purposes in order to avoid having to make projectors deal cleanly with seeing an event multiple times.

Given a projector that does some practical de-duplication of events, its pretty feasible to re-introduce this storage optimization; the performance and RU cost reductions can be inspected by using Equinox.Tool run and/or dotnet new eqxtestbed against said branch.

As noted in #108, this storage generalization also provides benefits wrt competing writer scenarios.

@bartelink bartelink added this to the 3.0 milestone Aug 1, 2019
@bartelink bartelink removed this from the 3.0 milestone Jul 22, 2020
@bartelink
Copy link
Collaborator Author

bartelink commented Jul 22, 2020

Removed milestone to reflect that, while this is unlikely to ever make it to a 2.x release (targeting Microsoft.Azure.DocumentDb V2), it's also not guaranteed to make a 3.x release as the Archival/Trimming facilities in #226 will likely yield a better overall solution. The other reason it's not a no-brainer to add back in is that the write amplification effects of having to read, write, and then have the CFPs read the redundant events are such that it requires a very specific combination of things (e.g. small total JSON length of events per stream) in order for this to be the best solution

@bartelink
Copy link
Collaborator Author

More reflections: Learnings and realizations from implementing Archival+Pruning actually make the case stronger for implementing this:

  • The archiver has the opportunity and incentive to spend more time packing (events in tip and/or compressing event bodies) when writing to a secondary - the write amplification considerations don't apply to the same extent as they do on primary

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant