Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Take the DB plunge #422

Closed
tyarkoni opened this issue Mar 21, 2019 · 0 comments · Fixed by #432
Closed

Take the DB plunge #422

tyarkoni opened this issue Mar 21, 2019 · 0 comments · Fixed by #432
Assignees

Comments

@tyarkoni
Copy link
Collaborator

I've proposed this several times in the past (mostly on grabbit), but I think it's time to take the plunge and use a relational DB for all the indexing/querying operations. This will solve a bunch of our metadata-related problems (see e.g., #398, #397).

My plan is to use dataset, which wraps SQLAlchemy. The DB itself will be in-memory SQLite; I don't anticipate running into many BIDS datasets so large that memory becomes an issue. Switching to a DB-based approach will also make it much easier to save/load the entire index to avoid the re-indexing process.

If anyone has any objections/suggestions, now is a good time to raise them, otherwise I plan to work up something in the next few days.

@tyarkoni tyarkoni self-assigned this Mar 27, 2019
@tyarkoni tyarkoni mentioned this issue Apr 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant