-
Notifications
You must be signed in to change notification settings - Fork 365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot load a gzipped JSON trace with multiple blocks #872
Comments
We use zlib to implement decompression. If there's a way to configure zlib to read these gzip streams, happy to add support. Otherwise, we would not be able to fix this as it's just a bit too niche to justify adding to trace processor. We'd need someone externally to spend a bit of time to figure out tho how to configure zlib to read this though and doubly helpful if this can be contributed as it's a very self contained issue. |
Thanks @LalitMaganti |
|
In practice, the use of zlib happens in https://github.com/google/perfetto/blob/master/src/trace_processor/util/gzip_utils.cc |
This is what I learned today:
|
Therefore, I would change code inspired by node.js case Z_STREAM_END:
return Result{ResultCode::kEof, out_size - z_stream_->avail_out}; to case Z_STREAM_END:
if (next two bytes at offset out_size - z_stream_->avail_out are magic 0x1f 0x8b) {
// next stream detected
return Result{ResultCode::kOk, out_size - z_stream_->avail_out};
} else {
return Result{ResultCode::kEof, out_size - z_stream_->avail_out};
} |
Approach seems good to me in that case, patches adding support for this are welcome: please follow https://perfetto.dev/docs/contributing/getting-started#contributing |
https://r.android.com/3250057 should solve the edge case I point out in your change :) |
Perfetto trace loader doesn't support "FEXTRA" multi-block gzip files. How to reproduce:
https://github.com/vinlyx/mgzip
Why does this weird gzip format property matter to me? We, 100B-parameter base LLM trainers in PyTorch, deal with a few hundred megs of profile that require considerable time to compress every few minutes, so compressing them on 192 available CPU cores gives a considerable benefit.
The text was updated successfully, but these errors were encountered: