Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider not writing the metadata for now #41

Open
ellisonbg opened this issue Nov 15, 2022 · 2 comments
Open

Consider not writing the metadata for now #41

ellisonbg opened this issue Nov 15, 2022 · 2 comments

Comments

@ellisonbg
Copy link

Right now we are storing configurable metadata along with each update in the YStore. The metadata is stored as a binary blob in the YStore. On the Jupyter side, we are passing a timestamp in the metadata, but we are not currently using that information. I can see a lot of usage cases for metadata, but in most cases, I believe we may want to do more complex queries of the YStore than the current approach allows. Examples include:

  1. Time rewinding capabilities would query for patches before or after a particular timestamp.
  2. "Suggest mode" would require us to tag updates with an identifier and then query for that identifier.
  3. Tracking changes back to users would require a user metadata and queries related to that.
  4. "Tagging" a version of a document would require tagging updates and querying based on those tags.

There is still considerable ambiguity about how all of this would work, and I am hesitant to commit to writing the metadata until we dive into all this and figure it out. It is the case that the current approach is flexible and allows us to add metadata fields without breaking the underlying database or file. It is unclear to me if we should (a) leave it as is until we figure these things out or (b) remove metadata until we do, but wanted to bring this up for discussion.

@davidbrochart
Copy link
Collaborator

Very interesting! I went for this solution of completely delegating the metadata handling (including (de)serialization) to the user, but now that you're mentioning how we could query the YStore using metadata, I can see that it would have a lot of value if it were part of the YStore API.

@davidbrochart
Copy link
Collaborator

Although if we start integrating database-like functionalities (like queries) in YStores, I'm afraid we will end up reimplementing a database engine for YStores that are not backed by databases, like file-based YStores.
Or we could just say that a YStore must be backed by a database, and drop the file-based YStore implementations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants