Only write regressed crates as raw logs #659
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This avoids a whole bunch of time writing logs for every crate out there. We
could optimize the writing itself (e.g., doing it in parallel would likely be a
huge win), but in practice there's no real need for each individual crate to
have its own file in S3, and that has other costs that would be nice to avoid
(e.g., compression is much less practical).
We can add support at a later point for the HTML reports to link to some server
(or lambda, whatever) that processes the tarball and serves results from it. It
takes around ~3 seconds to cat an all-crate zstd-compressed tarball, which takes up 400 MB.
3 seconds is probably too long for point queries to be viable on a re-download
and decompress, but that kind of speed means that it's not infeasible for us
to query on demand with some lightweight improvements to that process.