feat(rocprof-compute) Add gzip compression for large profile counter CSV artifacts (Phase B) by jamessiddeley-amd · Pull Request #9711 · ROCm/rocm-systems

jamessiddeley-amd · 2026-08-05T14:47:58Z

Motivation

Profile mode writes large counter CSVs (per-pass results_*.csv and per-process native counter output). Gzip compression cuts the size of these intermediate csv files by 97% with no change to peak memory, since both sides stream the compression.
Analyze accepts both compressed files and uncompressed csv files, so workloads from earlier releases still work.

Technical Details

Compression is applied at two write boundaries only:

src/utils/csv_compression.py: all profile/analyze CSV opens go through this module; .csv.gz is gzip, .csv is plain.
src/lib/utils/compression (C++): native tool counter output via GzipFileOutputStream.

Both sides share a format contract documented in docs/design/profile_interface_architecture.md. The previous OutputStream factory layer was removed in favor of direct gzip writers.

CI workflows add zlib dev packages. Truncated or corrupt gzip artifacts are reported as errors rather than partial data.

Issue Tracking

AIPROFCOMP-749

Test Plan

test_csv_compression, test_analysis_base, updated counters-writer and gzip-stream C++ tests
Updated profile/analyze integration tests (test_utils, test_torch_trace_analysis)
test-compression and test_counters_writer in C++ test suite

Test Result

pending

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

…tifacts

…o users/jamessiddeley-amd/gzip-csv-compression

Copilot

Pull request overview

Adds a gzip compression boundary for large profile/analyze counter CSV artifacts in rocprofiler-compute, so profile mode writes compressed outputs while analyze mode accepts both compressed and legacy uncompressed workloads.

Changes:

Introduces utils.csv_compression helpers and updates Python profile/analyze paths to read/write .csv.gz where appropriate.
Adds a small C++ compression utility library (GzipFileOutputStream) and updates the native counters writer to emit gzip-compressed CSV.
Updates tests, documentation, CI dependencies, and CMake wiring to cover/ship the new behavior.

Reviewed changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
projects/rocprofiler-compute/tests/test_utils.py	Updates tests to discover/read compressed `results_*.csv.gz` and validate empty/corrupt handling.
projects/rocprofiler-compute/tests/test_utils_profile_csv.py	Adds unit tests ensuring profile CSV helpers work with `.csv.gz`.
projects/rocprofiler-compute/tests/test_torch_trace_analysis.py	Adds regression test to ensure ML API trace counter copies preserve compression.
projects/rocprofiler-compute/tests/test_profile_general.py	Adjusts tests to locate compressed results artifacts via `find_csvs`.
projects/rocprofiler-compute/tests/test_csv_compression.py	New unit test suite for `utils.csv_compression`.
projects/rocprofiler-compute/tests/test_categories.yaml	Registers new Python and C++ compression-related tests in categories.
projects/rocprofiler-compute/tests/test_analysis_base.py	Adds analyze-base tests for concatenating compressed/mixed results and failing on truncation.
projects/rocprofiler-compute/tests/common.py	Routes CSV content scanning through `open_csv_read`; accepts `.csv.gz` for PMC artifacts.
projects/rocprofiler-compute/src/utils/utils_profile.py	Writes `results_*.csv.gz` and resolves native counter CSVs in either compressed/plain form.
projects/rocprofiler-compute/src/utils/utils_profile_csv.py	Opens CSVs through the compression boundary for consistent streaming behavior.
projects/rocprofiler-compute/src/utils/utils_analysis.py	Discovers compressed results and reports gzip truncation/corruption explicitly.
projects/rocprofiler-compute/src/utils/rocpd_data.py	Uses `open_csv_write` for CSV generation paths.
projects/rocprofiler-compute/src/utils/csv_compression.py	New module defining read/write/open and discovery helpers for `.csv` vs `.csv.gz`.
projects/rocprofiler-compute/src/rocprof_compute_analyze/analysis_base.py	Uses compression-aware open/discovery when merging `results_.csv` into `pmc_perf.csv`.
projects/rocprofiler-compute/src/lib/utils/compression/tests/test_gzip_output_stream.h	New gtest fixture declarations for gzip stream tests.
projects/rocprofiler-compute/src/lib/utils/compression/tests/test_gzip_output_stream.cpp	New gtests validating gzip writer correctness and robustness.
projects/rocprofiler-compute/src/lib/utils/compression/tests/main.cpp	Adds gtest main for the compression test binary.
projects/rocprofiler-compute/src/lib/utils/compression/tests/CMakeLists.txt	Builds/links the `test-compression` executable.
projects/rocprofiler-compute/src/lib/utils/compression/gzip_output_stream.h	Adds `GzipFileOutputStream` API used by the counters writer.
projects/rocprofiler-compute/src/lib/utils/compression/gzip_output_stream.cpp	Implements a thin gzFile-based streaming writer.
projects/rocprofiler-compute/src/lib/utils/compression/CMakeLists.txt	Adds `compression` static library target and zlib linkage.
projects/rocprofiler-compute/src/lib/utils/CMakeLists.txt	Wires the new compression subdirectory into the utils build.
projects/rocprofiler-compute/src/lib/rocprofiler_compute_tool/tests/test_rocprofiler_compute_tool.cpp	Updates expected default output filename suffix to `.csv.gz`.
projects/rocprofiler-compute/src/lib/rocprofiler_compute_tool/tests/test_counters_writer.h	New gtest fixture for counters writer unit tests.
projects/rocprofiler-compute/src/lib/rocprofiler_compute_tool/tests/test_counters_writer.cpp	New unit tests for batching/formatting and gzip output behavior.
projects/rocprofiler-compute/src/lib/rocprofiler_compute_tool/tests/CMakeLists.txt	Adds counters writer tests to the test target sources.
projects/rocprofiler-compute/src/lib/rocprofiler_compute_tool/rocprofiler_compute_tool.cpp	Changes default native counter artifact naming to `.csv.gz`.
projects/rocprofiler-compute/src/lib/rocprofiler_compute_tool/counters_writer.h	Exposes `format_counters_csv` and documents gzip output expectations.
projects/rocprofiler-compute/src/lib/rocprofiler_compute_tool/counters_writer.cpp	Switches writer to gzip stream output and adds batched formatting.
projects/rocprofiler-compute/src/lib/rocprofiler_compute_tool/CMakeLists.txt	Links the tool library against the new compression library.
projects/rocprofiler-compute/src/lib/CMakeLists.txt	Adds zlib pkg-config target wiring for the C++ side.
projects/rocprofiler-compute/docs/how-to/profile/mode.rst	Documents that profile writes `results_*.csv.gz` and native counter `.csv.gz`.
projects/rocprofiler-compute/docs/how-to/analyze/mode.rst	Documents analyze compatibility with both `.csv` and `.csv.gz`.
projects/rocprofiler-compute/docs/design/profile_interface_architecture.md	Updates Phase B status/description to reflect implementation.
projects/rocprofiler-compute/docs/design/lld-profiler-hub-native-tool-integration.md	Updates native tool output artifact naming to `.csv.gz`.
projects/rocprofiler-compute/CMakeLists.txt	Registers new Python and C++ tests with CTest and coverage lists.
projects/rocprofiler-compute/CHANGELOG.md	Notes gzip compression of large counter CSV artifacts.
.github/workflows/rocprofiler-compute-ubuntu-jammy.yml	Adds zlib dev package to CI dependencies.
.github/workflows/rocprofiler-compute-tarball.yml	Adds zlib dev package to CI dependencies.
.github/workflows/rocprofiler-compute-sanitizers.yml	Adds zlib dev package to CI dependencies.
.github/workflows/rocprofiler-compute-rhel.yml	Adds zlib dev package to CI dependencies.
.github/workflows/rocprofiler-compute-continuous-integration.yml	Adds zlib dev package to CI dependencies.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…md/gzip-csv-compression

…stems into users/jamessiddeley-amd/gzip-csv-compression

vedithal-amd

will review

- Write csv_compression for Python 3.8 directly instead of relying on `from __future__ import annotations`; document the rule in PYTHON_CODING_STYLE.md so it does not recur. - Qualify csv_compression calls at every call site. Bare find_csvs and open_csv_write gave a reader no hint that compression was involved. - Give CsvCountersWriter ownership of its artifact suffix, with a static_assert tying it to the gzip suffix. The filename and the format were decided in separate files with only a comment holding them together. - Narrow the corrupt-CSV handler in concat_result_csvs to the read; the pmc_perf.csv write is plain and cannot raise those errors. - Hold test-compression out of test_categories.yaml until TheRock registers it in the rocprofiler-compute post-install hook. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

vedithal-amd · 2026-08-13T14:44:49Z

Pushed the review fixes straight to the branch instead of round-tripping comments.

Python

csv_compression.py: dropped from __future__ import annotations and switched to typing.List, so the module is 3.8-parseable on its own terms. Added a "Python 3.8 Compatible Syntax" section to PYTHON_CODING_STYLE.md so it does not recur.
is_compressed is now _is_compressed, below the public functions. Only open_csv_read calls it in production.
Call sites import the module and qualify (csv_compression.find_csvs(...)). Bare find_csvs and open_csv_write gave a reader no hint that compression was involved. Applied across utils_analysis, analysis_base, utils_profile, utils_profile_csv, rocpd_data, and the tests.
save_ml_api_trace_inputs: the compressed-vs-plain conditional was dead, since the only caller passes results_csv, which is always .gz. Collapsed to compressed_name(...), which is what Chris asked for on that line.
concat_result_csvs: the try wrapped the pmc_perf.csv write, but that file is plain and cannot raise any member of CORRUPT_CSV_ERRORS. Narrowed it to the per-file read and reworded the message to lead with the source file.

C++ and CMake

CsvCountersWriter now owns kFileSuffix and create_tool_data uses it, with a static_assert tying it to compression::kGzipSuffix. The filename and the format were decided in separate files with only a comment holding them together.
kBatchBytes comment now says why batching exists and that the size is untuned. Left the value and the ostringstream alone: C++17 rules out std::move(batch).str(), and a raw std::string would change the double formatting your test pins. The per-flush batch.str() copy is still there if you want to revisit it.
Dropped the >=1.2.7 floor from pkg_check_modules(zlib ...) to match how libdw is declared right above it.

Tests

Removed test-compression from test_categories.yaml; the ctest registration stays. See the follow-up comment.
Folded test_concat_result_csvs_reads_compressed_results into the mixed test, it was a strict subset. Hoisted a function-local import gzip. Dropped a redundant assert in the header contract test.

Docs

Reverted the analyze-mode note to its previous wording. End users do not need the compressed-vs-plain compatibility matrix while both are supported.

Verified locally: unit tests and ruff clean, test-compression builds and passes standalone, and end to end with --set compute_thruput_util --no-roof over rocflop the native tool writes *_native_counter_collection.csv.gz, analyze builds pmc_perf.csv, and the reported utilization matches expectations. Re-running analyze with the results file gunzipped gives identical output.

vedithal-amd · 2026-08-13T14:44:51Z

test-compression needs two TheRock-side changes before it can go back into test_categories.yaml:

The rocprofiler-compute post-install hook enumerates gtest targets by name and does not include test-compression, so its RPATH origin resolves to bin/ instead of the tests directory.
The rocprofiler-compute subproject declares bundled elfutils, libdrm, and sqlite3 as runtime deps but no zlib, so pkg_check_modules(zlib REQUIRED ...) never sees the bundled zlib pkgconfig. It links system zlib while the rest of the stack loads the bundled one.

Both belong in the same TheRock PR. Once it merges, confirm the ref our workflows build against picks it up (the nightly tracks TheRock@main; anything pinning an explicit therock_ref needs a bump), then re-add test-compression to the categories.

vedithal-amd · 2026-08-13T14:44:51Z

Please drop plain-CSV support in a follow-up PR. Profile output is an internal contract, not something we owe compatibility on, and carrying both forms is where most of the complexity here lives: resolve_csv returning a compressed path when nothing exists, content-sniffing on read while dispatching on name for write, and the dedup in find_csvs. Removing it also means regenerating the cached tests/workloads/ fixtures.

vedithal-amd

LGTM

…CSV artifacts (Phase B) (#9711) Profile mode writes large counter CSVs (per-pass `results_*.csv` and per-process native counter output). Gzip compression cuts the size of these intermediate csv files by **97%** with no change to peak memory, since both sides stream the compression. Analyze accepts both compressed files and uncompressed csv files, so workloads from earlier releases still work. Compression is applied at two write boundaries only: - `src/utils/csv_compression.py`: all profile/analyze CSV opens go through this module; `.csv.gz` is gzip, `.csv` is plain. - src/lib/utils/compression (C++): native tool counter output via `GzipFileOutputStream`. Both sides share a format contract documented in `docs/design/profile_interface_architecture.md`. The previous `OutputStream` factory layer was removed in favor of direct gzip writers. CI workflows add `zlib` dev packages. Truncated or corrupt gzip artifacts are reported as errors rather than partial data.  AIPROFCOMP-749 - `test_csv_compression`, `test_analysis_base`, updated counters-writer and gzip-stream C++ tests - Updated profile/analyze integration tests (`test_utils`, `test_torch_trace_analysis`) - `test-compression` and `test_counters_writer` in C++ test suite pending - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Vignesh Edithal <Vignesh.Edithal@amd.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ode CI (#7421) ## Motivation Added a `test-compression` gtest in [#9711 of rocprofiler-compute,](ROCm/rocm-systems#9711), but TheRock CI can't run it in install mode yet: - The post-hook sets install RPATH for the other gtests under `libexec/rocprofiler-compute/tests/` but not `test-compression` - The subproject pulls bundled elfutils/libdrm/sqlite3 but not zlib, so `pkg_check_modules(zlib)` can link system `libz` instead of bundled sysdeps The follow-up in rocm-systems repo is to add `test-compression` to `test_categories.yaml` after this merges. ## Technical Details - Add `test-compression` to the RPATH post-hook foreach in `profiler/post_hook_rocprofiler-compute.cmake`. - Add `${THEROCK_BUNDLED_ZLIB}` to `rocprofiler-compute` `RUNTIME_DEPS` in `profiler/CMakeLists.txt`. ## Test Plan - Build with rocprofiler-compute + tests enabled. - Confirm `test-compression` installs under `libexec/rocprofiler-compute/tests/` with correct RPATH. - Run `ctest -R test-compression`. ## Test Result - Pending CI. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/CONTRIBUTING.md.

…ifacts (#10237) ## Motivation Counter CSVs are now written as `.csv.gz`, so the intermediate plain csv read paths and test fixtures are no longer needed and removing them simplifies the python compression boundary. Compressing the fixtures shrinks `tests/workloads` from 75M to 60M. ## Technical Details - Compress the marker artifacts: `out/pmc_1/<fbase>_marker_api_trace.csv` and its `ml_api_trace_*` copy come from the same rocpd databases as the counter CSV, so they are named through `compressed_name` too. Every artifact the profile CSV helpers handle is now `.csv.gz`. - Make `csv_compression` gzip-only: `open_csv_read`/`open_csv_write` become `open_gzip_csv_read`/`open_gzip_csv_write` with no suffix dispatch, and `resolve_csv` and content sniffing are gone. - Drop `find_csvs`, a one-line glob wrapper added with Phase B. `analysis_base` and `utils_analysis` glob for `GZIP_SUFFIX` directly, as they did before Phase B. - The two plain writes left, `sysinfo.csv` and the `pmc_perf.csv` analyze intermediate, use the builtin `open` at their call sites instead of routing through the compression module. - Update profile/analyze callers to use `compressed_name` directly (removes a redundant one line return helper function). - Compress the counter artifacts under `tests/workloads`: `results_*.csv`, plus the marker and `counter_collection` CSVs in the `ml_api_trace`, `torch_trace`, and `triton_trace` workloads. Drop the plain counter fixtures. - Update the Phase B and AD-2 sections of `docs/design/profile_interface_architecture.md`. - Related: Follow up to #9711 ## Issue Tracking Related: #9711 ## Test Plan - `pytest tests/unit` - `pytest tests/integration/test_analyze_workloads.py` - `pytest tests/integration/test_profile_path.py::test_path_rocpd tests/integration/test_profile_misc.py::test_analyze_rocpd tests/integration/test_profile_section.py::test_lds_section` ## Test Result All pass locally on gfx950. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. --------- Co-authored-by: Vignesh Edithal <Vignesh.Edithal@amd.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

jamessiddeley-amd force-pushed the users/jamessiddeley-amd/gzip-csv-compression branch from f26ab0b to 88a0a1f Compare August 5, 2026 14:50

github-actions Bot added documentation Improvements or additions to documentation project: rocprofiler-compute labels Aug 5, 2026

systems-assistant Bot added the organization: ROCm label Aug 5, 2026

Base automatically changed from users/jamessiddeley-amd/remove-join-type-option to rocprofiler-compute-develop August 6, 2026 17:44

github-actions Bot added the github actions Pull requests that update GitHub Actions code label Aug 7, 2026

jamessiddeley-amd changed the title ~~[rocprof-compute] compress intermediate profile artifacts with gzip streaming~~ feat(rocprof-compute) Add gzip compression for large profile counter CSV artifacts Aug 7, 2026

feat(rocprofiler-compute): gzip-compress large profile counter CSV ar…

efc115e

…tifacts

jamessiddeley-amd force-pushed the users/jamessiddeley-amd/gzip-csv-compression branch from a7e133b to a6a4117 Compare August 7, 2026 16:09

docs(rocprofiler-compute): document gzip counter CSV compression

cbe00fc

jamessiddeley-amd force-pushed the users/jamessiddeley-amd/gzip-csv-compression branch from a6a4117 to cbe00fc Compare August 7, 2026 16:14

jamessiddeley-amd changed the title ~~feat(rocprof-compute) Add gzip compression for large profile counter CSV artifacts~~ feat(rocprof-compute) Add gzip compression for large profile counter CSV artifacts (Phase B) Aug 7, 2026

jamessiddeley-amd added 2 commits August 7, 2026 16:54

gersemi format

84b2287

Merge remote-tracking branch 'origin/rocprofiler-compute-develop' int…

636d6bc

…o users/jamessiddeley-amd/gzip-csv-compression

jamessiddeley-amd marked this pull request as ready for review August 10, 2026 14:17

jamessiddeley-amd requested review from a team and prbasyal-amd as code owners August 10, 2026 14:17

vedithal-amd requested a lite review from Copilot August 10, 2026 15:33

Copilot started reviewing on behalf of vedithal-amd August 10, 2026 15:34

View session

Copilot AI reviewed Aug 10, 2026

View reviewed changes

Comment thread projects/rocprofiler-compute/src/lib/utils/compression/CMakeLists.txt Resolved

Comment thread projects/rocprofiler-compute/docs/how-to/analyze/mode.rst Resolved

jamessiddeley-amd and others added 4 commits August 10, 2026 12:16

fix comment grammar

4f6393c

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix gzip compression edge cases found in review

014287a

Merge branch 'rocprofiler-compute-develop' into users/jamessiddeley-a…

126f188

…md/gzip-csv-compression

compress counter_collection.csv

6d3710e

cfallows-amd reviewed Aug 12, 2026

View reviewed changes

cfallows-amd mentioned this pull request Aug 12, 2026

test(rocprofiler-compute): Move tests under tests/unit and tests/integration #9762

Merged

1 task done

abchoudh-amd suggested changes Aug 12, 2026

View reviewed changes

Comment thread projects/rocprofiler-compute/src/lib/utils/compression/tests/test_gzip_output_stream.cpp Resolved

Comment thread projects/rocprofiler-compute/src/lib/utils/compression/gzip_output_stream.cpp Resolved

Comment thread projects/rocprofiler-compute/src/utils/csv_compression.py Resolved

Merge branch 'rocprofiler-compute-develop' of github.com:ROCm/rocm-sy…

3325424

…stems into users/jamessiddeley-amd/gzip-csv-compression

address comments

1d58e77

jamessiddeley-amd requested review from abchoudh-amd and cfallows-amd August 12, 2026 17:18

vedithal-amd requested changes Aug 13, 2026

View reviewed changes

abchoudh-amd approved these changes Aug 13, 2026

View reviewed changes

vedithal-amd approved these changes Aug 13, 2026

View reviewed changes

clang format

c760fb9

vedithal-amd merged commit 4dff4bd into rocprofiler-compute-develop Aug 13, 2026
34 of 36 checks passed

vedithal-amd deleted the users/jamessiddeley-amd/gzip-csv-compression branch August 13, 2026 19:21

Navigation Menu

Repository navigation

feat(rocprof-compute) Add gzip compression for large profile counter CSV artifacts (Phase B) - #9711

feat(rocprof-compute) Add gzip compression for large profile counter CSV artifacts (Phase B)#9711
vedithal-amd merged 12 commits into
rocprofiler-compute-developfrom
users/jamessiddeley-amd/gzip-csv-compression

jamessiddeley-amd commented Aug 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vedithal-amd left a comment

Uh oh!

vedithal-amd commented Aug 13, 2026

Uh oh!

vedithal-amd commented Aug 13, 2026

Uh oh!

vedithal-amd commented Aug 13, 2026

Uh oh!

vedithal-amd left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jamessiddeley-amd commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Issue Tracking

Test Plan

Test Result

Submission Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vedithal-amd left a comment

Choose a reason for hiding this comment

Uh oh!

vedithal-amd commented Aug 13, 2026

Uh oh!

vedithal-amd commented Aug 13, 2026

Uh oh!

vedithal-amd commented Aug 13, 2026

Uh oh!

vedithal-amd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jamessiddeley-amd commented Aug 5, 2026 •

edited

Loading