WIP - encode labels into index #180
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updates shared to the version where `Report` has a `_labels_index` and makes use of them in processing.
giovanni-guidini
force-pushed
the
gio/label-index
branch
from
ee142cd
to
a3ff689
Compare
giovanni-guidini
added a commit
that referenced
this pull request
Nov 21, 2023
context: codecov/engineering-team #768 Creates a rollout for label compression. This is gonna help us to test and safely release the feature in the wild. Notice that currently the label compression does nothing. There are comments similar to `TODO: needs shared update`. The update in question is codecov/shared#79 So these changes mostly prep the terrain for future changes that will actually do something, and add the guardrails to avoid issues when deploying. In particular it adds some helper methods to the `ArchiveService`, creates a kinda-stubbed `LabelsIndexService`, passes more data to the `_adjust_sessions` portion of `raw_upload_processor` where most changes will occur. If you're curious as to what the end result will probably look like see #180
|
DON'T MERGE |
giovanni-guidini
added a commit
that referenced
this pull request
Nov 21, 2023
context: codecov/engineering-team #768 Creates a rollout for label compression. This is gonna help us to test and safely release the feature in the wild. Notice that currently the label compression does nothing. There are comments similar to `TODO: needs shared update`. The update in question is codecov/shared#79 So these changes mostly prep the terrain for future changes that will actually do something, and add the guardrails to avoid issues when deploying. In particular it adds some helper methods to the `ArchiveService`, creates a kinda-stubbed `LabelsIndexService`, passes more data to the `_adjust_sessions` portion of `raw_upload_processor` where most changes will occur. If you're curious as to what the end result will probably look like see #180
giovanni-guidini
added a commit
that referenced
this pull request
Nov 24, 2023
context: codecov/engineering-team #768 Creates a rollout for label compression. This is gonna help us to test and safely release the feature in the wild. Notice that currently the label compression does nothing. There are comments similar to `TODO: needs shared update`. The update in question is codecov/shared#79 So these changes mostly prep the terrain for future changes that will actually do something, and add the guardrails to avoid issues when deploying. In particular it adds some helper methods to the `ArchiveService`, creates a kinda-stubbed `LabelsIndexService`, passes more data to the `_adjust_sessions` portion of `raw_upload_processor` where most changes will occur. If you're curious as to what the end result will probably look like see #180
giovanni-guidini
added a commit
that referenced
this pull request
Nov 29, 2023
context: codecov/engineering-team #768 Creates a rollout for label compression. This is gonna help us to test and safely release the feature in the wild. Notice that currently the label compression does nothing. There are comments similar to `TODO: needs shared update`. The update in question is codecov/shared#79 So these changes mostly prep the terrain for future changes that will actually do something, and add the guardrails to avoid issues when deploying. In particular it adds some helper methods to the `ArchiveService`, creates a kinda-stubbed `LabelsIndexService`, passes more data to the `_adjust_sessions` portion of `raw_upload_processor` where most changes will occur. If you're curious as to what the end result will probably look like see #180
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a lot of code, and I'll probably break this PR in several others. I'm looking for early feedback here. It's a lot of code and a lot of changes.
This depends on the changes in codecov/shared#79. Check that PR for context.
New things
services/report/labels_index.pyhouses theLabelIndexService. This little bit of code controls read/write in the label index file (that is separate fromchunks.txt). It does the loading and unloading of theReport._labels_indexas needed.MinioEndpoints.label_indexand functions to write-to and read-from the file. There can be 1 such file for eachCommitReport.services/report/raw_upload_processor.py::make_sure_orginal_report_is_using_label_idsencodes the labels in aReportprior to merging it with another report (if needed)services/report/raw_upload_processor.py::make_sure_label_indexes_match"fixes" the index of the new report to be merged based on the original report's index so that we can be sure that both indexes point to the same labels.Important changes
pycoverage.pylanguage processor now spits out reports with a label lookup table andCoverageDatapointalready encodedlabel_analysistask works with label_ids and only translates them in the end.Things missing
label_analysishas to work if the report is not encoded with IDs as well.