Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# HLD: Memory chart layout in the analysis database

## System Context

- `rocprof-compute analyze` draws a memory chart (panel 3) in the terminal. The chart has
boxes, such as Compute Units, the caches, and HBM, with arrows between them. Each box
and arrow shows a panel 3 metric.
- A JSON layout file for each GPU family (`src/memory_chart/layouts/`) states everything
needed to draw the chart: where each box goes, which boxes sit inside others, the
metric each box and arrow shows, which way each arrow points, and how each part is
drawn. `src/memory_chart/README.md` documents the format.
- `analyze --output-format db` writes an SQLite analysis database. Each workload has its
own metric definitions. Data that a reader uses all at once is stored as JSON in
columns of the workload table, such as `sys_info_extdata` and
`profiling_config_extdata`.
- ROCm Optiq is a separate application that reads only the database. It draws the memory
chart from layouts built into Optiq, which find metrics by fixed `metric_id` values.

## Problem statement

- Optiq's built-in layouts use fixed `metric_id` values, such as `3.1.24`. These ids come
from each metric's position in the panel configuration, so they change between
rocprof-compute versions and GPU families. A database from a different
rocprof-compute version can show one metric's value under another metric's label, so
Optiq turned the feature off in its release.
- Optiq also has to keep its own copy of every layout in step with rocprof-compute.

## Requirements

### Functional

1. Store each workload's memory chart layout in the database, so a reader draws the
same boxes, nesting, order, and arrows as rocprof-compute from the database alone.
2. The stored layout is the layout file itself. There is one memory chart format, and
readers need no converter or parsing script.
3. Each metric in the layout is found by a reference that stays correct across
rocprof-compute versions.
4. Keep the full chart when `--block` leaves out some of its metrics. Those metrics have
no values.

### Non-functional

- Readers that don't know about the new column keep working; the change only adds a
column.
- Follow the existing conventions of the database.

## Design

The workload table gets one JSON column, `memory_chart_layout_extdata`. It holds the
layout file of the workload's GPU family, unchanged, or null when the family has no
layout.

- **Metrics are found by name within panel 3.** Each `metric` in the layout is the name
of a metric in the workload's panel 3 configuration. Names are unique within panel 3
on every GPU family, and a unit test keeps them unique. A reader finds the metric's
definition, values, and unit by that name among the workload's metrics whose
`metric_id` starts with `3.`.
- **Everything is stated in the file.** Each block has its `column`, its `order`, its
`position` (in the grid, above, or below), and its `parent` or `host`. Each arrow has
its `direction`. Drawing hints are also in the file: a block's `style` (`panel`, or
`list` for the compact compute side), whether a metric has a progress `bar`, and an
arrow's `style` (`inline` or `stacked`). The terminal chart reads the same fields, so
both charts always match.
- **No format version.** The database schema version covers the column.

### Decisions

| Decision | Why | Alternative considered |
|---|---|---|
| Store the layout file unchanged | One format for the terminal chart, the database, and Optiq. Nothing to convert, and nothing to keep in step. | Converting the layout into a separate format for Optiq, which is a second format to maintain and document. |
| Metric names, looked up within panel 3 | Names are unique within panel 3, and a name refers to the same metric across versions. Looking only in panel 3 avoids names that repeat in other panels, such as "LDS Utilization". | `metric_id`, which changes when metrics are added or removed. |
| Placement, direction, and drawing hints stated in the file | A reader draws the chart by reading fields, without working anything out from where an entry sits in the file, from units, or from categories. | Deriving them in each reader, which is what the terminal renderer did before. |
| One JSON column, not new tables | A reader uses the whole layout at once, which is what the JSON columns of the workload table are for. | Tables for blocks, block metrics, and arrows. These add table links the schema doesn't use anywhere else. |
| Named `memory_chart_layout_extdata` | Says what it holds, and follows the `*_extdata` convention. | `memory_chart_extdata`, the name Optiq's development branch reads in its older format, which an older Optiq build would misread. |
| Null when a GPU family has no layout | The column also covers a layout file that was skipped as broken. Readers then use their own fallback. | Failing the database output. |
| No format version | The database schema version already covers the column. | A `version` field in the layout. |
| Schema version 2.4.0 | The change only adds a column. | — |

### Out of scope

- Memory bandwidth stall notes: the database doesn't store memory bandwidth analysis
results. Only rocprof-compute reads the layout's `stall_level`.
- How to show values (units, decimal places, colors, box sizes, where an arrow sits
along its boxes): each reader decides.
- CSV output: the column isn't part of any view.

## Implementation phases

1. The layout format states placement, arrow direction, and drawing hints
(#12548).
2. Store the layout file in the database (this change).
3. Optiq draws the chart from the column, and finds metrics by name within panel 3.

## Validation, security and debuggability

- Unit tests check that the stored column equals the layout file, and that a GPU family
without a layout stores null.
- The layout tests check every layout against its panel 3 configuration: the layout
shows every panel 3 metric and no other, names are unique within panel 3, and the
drawing hints fit each metric's unit.
- On a real gfx950 workload, the stored layout equals the file, and every layout metric
finds a panel 3 definition in the database.

## Open questions

- Optiq needs to read the new column and the layout format documented in
`src/memory_chart/README.md`, and to find metrics by name within panel 3.
- With a panel 3 configuration from `--config-dir`, a layout metric may have no
definition. Readers should show it as missing.
35 changes: 7 additions & 28 deletions projects/rocprofiler-compute/src/memory_chart/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,7 @@ The memory chart (panel 3 of `analyze`) is drawn from a layout file in
chart except the metric values: which boxes it has, where each box goes, which metric
each box and arrow shows, which way each arrow points, and how each part is drawn.

This README is the contract for any tool that draws the chart, such as ROCm Optiq. To
draw the same chart as `analyze`, a tool needs only the layout file, the metric values,
and this page. It never has to work anything out from the order of entries in the file,
from units, or from categories.
This README is the contract for any tool that draws the chart, such as ROCm Optiq.

## Example

Expand Down Expand Up @@ -47,23 +44,6 @@ A small layout with three columns. JSON has no comments; they are here only to e
}
```

## How to draw a layout

1. Draw the boxes that have `"position": "grid"` and no `parent` as columns, left to
right by `column`. In each column, stack them top to bottom by `order`.
2. Draw each box that has a `parent` inside that parent box, top to bottom by `order`.
3. Draw each box with `"position": "above"` or `"below"` outside the columns, above or
below its `host` box, in `order`.
4. Draw a bar above the columns with the two `scope.labels`. The second label starts at
the column of the `scope.split` box.
5. Draw the arrows between their boxes, in the order of the `arrows` list. Put
consecutive arrows with the same `group` under one heading with that name.
6. Show each metric's value and unit, found as described in
[Metric values](#metric-values).

Colors, sizes, number formats, and where an arrow sits along the side of a box are up to
each tool.

## Fields

### Top level
Expand All @@ -87,11 +67,11 @@ All fields are required.
| `column` | yes | The column, counting from 0 on the left. A box inside another box, or above or below one, has that box's column. |
| `order` | yes | The place among the boxes it is stacked with, counting from 0. Boxes are stacked with the other top-level boxes of their column, with the other boxes inside the same parent, or with the other boxes on the same side of the same host. |
| `position` | yes | `grid` for a box in a column or inside another box. `above` or `below` for a small labelled box outside the columns, such as xGMI or PCIe. |
| `style` | yes | `panel` for a box with a border. `list` for a compact list: no blank lines, all values in one color, and no bars. The compute box on the left uses `list`. Boxes above or below the columns are `panel`. |
| `style` | yes | `panel` for a box with a border. `list` for a compact list, used by the compute box on the left. Boxes above or below the columns are `panel`. |
| `parent` | no | The id of the box this box is drawn inside. |
| `host` | for `above` and `below` boxes | The id of the top-level box in a column that this box is drawn above or below. |
| `content` | no | The metrics shown in the box, top to bottom. |
| `stall_level` | no | `GL1`, `GL2`, or `EA`. `analyze` lists the memory bandwidth stalls of that level in the box. Other tools can ignore it; its data is not in the analysis database. |
| `stall_level` | no | `GL1`, `GL2`, or `EA`. `analyze` lists the memory bandwidth stalls of that level in the box. |

A box with boxes inside it shows only its title and those boxes; its metrics and arrows
belong to the inner boxes. Boxes above or below the columns show only their title.
Expand Down Expand Up @@ -142,16 +122,15 @@ always follows them:
box drawn above or below its start box. It never starts or ends at a box that has
boxes inside it.

The tests also check that a layout shows every metric of the GPU's memory chart panel,
and no other metric.

## Metric values

Each `metric` is the name of a metric in the memory chart panel (panel 3). Names are
unique within panel 3. They can repeat in other panels, so look only in panel 3.

In the analysis database (`analyze --output-format db`), each metric of a workload has a
row in `compute_metric_definition`. For a layout metric, use the workload's row whose
In the analysis database (`analyze --output-format db`), each workload's layout file is
stored unchanged in the `memory_chart_layout_extdata` column of `compute_workload`. It
is null when the workload's GPU has no layout. Each metric of a workload has a row in
`compute_metric_definition`. For a layout metric, use the workload's row whose
`name` is the metric name and whose `metric_id` starts with `3.`. That row has the unit.
The value the terminal chart shows is in `compute_workload_metric_value`, in the row
for that definition with `value_name` `Value`. `compute_kernel_metric_value` has the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# SPDX-License-Identifier: MIT

import ast
import json
import re
import warnings
from pathlib import Path
Expand All @@ -12,6 +13,7 @@

import utils.analysis_orm as orm
from config import rocprof_compute_home
from memory_chart.loader import Layouts
from pc_sampling.code_object_analysis import (
CodeObjectSymbol,
InstructionPipelines,
Expand Down Expand Up @@ -297,6 +299,17 @@ def _add_source_lines(
})


def memory_chart_layout_extdata(gpu_arch: str) -> Optional[dict[str, Any]]:
"""The memory chart layout file of *gpu_arch*, unchanged; None if it has none.

src/memory_chart/README.md documents the format.
"""
layout = Layouts.for_arch(gpu_arch)
if layout is None:
return None
return json.loads(layout.path.read_text(encoding="utf-8"))


class db_analysis(OmniAnalyze_Base):
# -----------------------
# Required child methods
Expand Down Expand Up @@ -357,6 +370,9 @@ def run_analysis(self) -> None:
workload_path
),
profiling_config_extdata=self._profiling_config,
memory_chart_layout_extdata=memory_chart_layout_extdata(
sys_info["gpu_arch"]
),
)
Database.get_session().add(workload_obj)
workload_objs.append(workload_obj)
Expand Down
4 changes: 3 additions & 1 deletion projects/rocprofiler-compute/src/utils/analysis_orm.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
from utils.logger import console_debug, console_error, console_warning

PREFIX = "compute_"
SCHEMA_VERSION = "2.3.1"
SCHEMA_VERSION = "2.4.0"


Base = declarative_base()
Expand All @@ -63,6 +63,8 @@ class Workload(Base):
sys_info_extdata = Column(JSON)
roofline_bench_extdata = Column(JSON)
profiling_config_extdata = Column(JSON)
# Memory chart layout file, unchanged; see src/memory_chart/README.md
memory_chart_layout_extdata = Column(JSON)

# Workload can have multiple kernels
kernels = relationship("Kernel", back_populates="workload")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import pytest
from sqlalchemy import text

from memory_chart.loader import Layouts
from pc_sampling import per_kernel_isa_export, source_snapshot_analysis
from pc_sampling.code_object_analysis import CodeObjectInstruction, CodeObjectSymbol
from pc_sampling.pc_sampling_analysis import SOURCE_LINE_MISSING, InstructionLineRecord
Expand Down Expand Up @@ -225,6 +226,14 @@ def make_counter_backed_database_analyzer(
return analyzer


def run_memory_chart_workload(tmp_path, gpu_arch):
"""Run analysis of one counter-backed workload; return its workload row."""
analyzer = make_counter_backed_database_analyzer(str(tmp_path), ["3"], [])
analyzer._runs[str(tmp_path)].sys_info = pd.DataFrame([{"gpu_arch": gpu_arch}])
run_analysis_with_existing_database(analyzer)
return orm.Database.get_session().query(orm.Workload).one()


def run_analysis_with_existing_database(analyzer):
"""Run analysis while preserving the test's existing database session."""
with ExitStack() as patch_stack:
Expand Down Expand Up @@ -3556,3 +3565,24 @@ def evaluate_roofline_metric(name, value, kernel_pmc_df, sys_info):
assert (df[col] == 42.0).all()

assert evaluated_row_counts == [2] * (NUM_KERNELS * len(roofline_metrics))


# =============================================================================
# Memory chart layout (Workload.memory_chart_layout_extdata)
# =============================================================================


def test_run_analysis_stores_the_memory_chart_layout_file_unchanged(
db_session, tmp_path
):
workload = run_memory_chart_workload(tmp_path, "gfx942")
layout_file = Layouts.for_arch("gfx942").path
assert workload.memory_chart_layout_extdata == json.loads(
layout_file.read_text(encoding="utf-8")
)


def test_run_analysis_stores_null_for_an_arch_without_a_layout(db_session, tmp_path):
# Also covers a layout file skipped as broken
workload = run_memory_chart_workload(tmp_path, "gfx1030")
assert workload.memory_chart_layout_extdata is None
Loading