-
Notifications
You must be signed in to change notification settings - Fork 320
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhance run table query performance with indices #1277
Conversation
I think this fails the tests because it's trying to add the index to the runs table with out a guid colloum in the upgrade 0 to 1 test. I think making this an upgrade too would solve the problem. @WilliamHPNielsen I don't know if that in this case should go in before the dependencies upgrade |
@jenshnielsen, I agree that this should be a distinct upgrade, and I think it makes sense that this PR gets database version number 2, since it is already done. @geoffroth should I make this into an upgrade or will you do that? |
Also: great PR, much appreciated! If anyone has time for benchmarking, it would be great to see if we do get the expected |
@WilliamHPNielsen Since it's just a few lines, would it make sense for you to perhaps simply incorporate this or similar into the db refactor you're currently working on? (I had already finished most of this before I knew about your work-in-progress). Otherwise, I'd certainly be happy to rewrite as upgrade 2. |
@geoffroth, my refactor is still one or two weeks out in the future, so I would prefer to have this be a separate upgrade that we can put in now. Thanks. |
Codecov Report
@@ Coverage Diff @@
## master #1277 +/- ##
==========================================
+ Coverage 72.51% 72.54% +0.02%
==========================================
Files 74 74
Lines 8438 8456 +18
==========================================
+ Hits 6119 6134 +15
- Misses 2319 2322 +3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Let get this in.
Merge: 1868289 756c89f Author: William H.P. Nielsen <[email protected]> Merge pull request #1277 from geoffroth/fix/run_indices
|
||
|
||
GIT_HASHES: Dict[int, str] = {0: '78d42620fc245a975b5a615ed5e33061baac7846', | ||
1: '056d59627e22fa3ca7aad4c265e9897c343f79cf'} | ||
|
||
DB_NAMES: Dict[int, List[str]] = {0: ['']} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this used?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nein. Should have been removed.
# Version 0: the original table schema, runs, experiments, layouts, | ||
# dependencies, result-tables | ||
# | ||
# Version 1: a GUID column is added to the runs table |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is description of version 2 missing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We will add that when we need it, i.e. when we have version 3 on the table.
Changes proposed in this pull request:
This should enhance performance (especially in large tables) for queries that retrieve data by either column. Indices are DESC based on the assumption that we will more likely be querying higher / more recent exp_ids and guids more often than older data.
@WilliamHPNielsen