Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 20 additions & 10 deletions projects/rocprofiler-compute/skills/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,10 @@ Every skill directory holds:
├── SKILL.md # frontmatter (name, description) plus the instructions
├── skill-card.md # human-facing card: description, owner, license
└── evals/
├── evals.json # prompts that must and must not route to this skill
├── machine.yml # hardware the behavioral cases need
└── samples/ # copies of the samples the cases use
├── evals.json # minimum set the AMD skills catalog grades
├── extended_evals.json # longer profile cases, run here with --extended
├── machine.yml # hardware the behavioral cases need
└── samples/ # copies of the samples the cases use
```

`SKILL.md` states the decision: which command to run, when, and where the
Expand All @@ -77,10 +78,23 @@ routing needs a model API key, and behavior needs a supported GPU.
```bash
./skills/run_evals.py --list # validate the datasets
./skills/run_evals.py # does the right skill load?
./skills/run_evals.py --mode behavioral # also grade the agent
./skills/run_evals.py --mode behavioral # grade the minimum set
./skills/run_evals.py --extended --mode behavioral # also grade the long profiles
./skills/run_evals.py --skill rocprof-compute-memory # one skill
```

`evals/evals.json` is the minimum set. The AMD skills catalog grades that file
and no other. It keeps at least three prompts that should trigger the skill,
two that should not, and one triggering case with judged behavior. Negative
cases carry only an id, a prompt, and an optional note. The judged cases stay
short: they check that the right workflow is chosen, and they do not profile
a sample.

Cases that build a sample, profile it, and read the report live in
`evals/extended_evals.json`, beside the minimum file and in the same format.
Add `--extended` to run them. The catalog workflow sets `extended: false`, so
moving a case into that file takes it out of the catalog run.

Routing mode denies the agent its tools and only checks which skill it chose,
so it is quick and needs no GPU. Behavioral mode lets the agent work and grades
`logs_contain`, `files_exist`, and the plain-language `expected_behavior` and
Expand All @@ -93,12 +107,8 @@ install script and its docs links. Both drive the `claude` CLI. The script
exits 77 when that CLI is absent, so a caller can treat it as skipped rather
than failed.

Each dataset needs at least three cases that should trigger the skill, two that
should not, and one triggering case with judged behavior. Negative cases carry
only an id, a prompt, and an optional note.

A triggering case that needs an application sets `workspace` to
`evals/samples`, and its prompt names the application. The contents of the
A case that profiles an application lives in `extended_evals.json`. It sets
`workspace` to `evals/samples`, and its prompt names the application. The contents of the
workspace directory are copied to the workspace root when the case runs, as
skillscope does. The AMD skills repo copies only the skill directory, so
`evals/samples/` holds copies of the samples from `sample/`. Each copy and its
Expand Down
Original file line number Diff line number Diff line change
@@ -1,44 +1,6 @@
{
"comment": "Minimum dataset graded by the AMD skills catalog. Cases that profile a sample and read the report live in extended_evals.json.",
"evaluations": [
{
"id": "kb-profile-hip-application",
"skill_should_trigger": true,
"prompt": "profile my HIP application with rocprof-compute and tell me where the time goes\n\nThe application is ./vcopy.cpp.",
"workspace": "evals/samples",
"expected_behavior": [
"asks for the launch command and a workload name before profiling",
"runs rocprof-compute profile --name <name> -- <application>",
"runs analyze with --list-stats to find the hottest kernel"
],
"unexpected_behavior": [
"invents a profiling flag instead of checking --help"
],
"logs_contain": ["rocprof-compute profile", "--list-stats"]
},
{
"id": "kb-which-kernel-is-slow",
"skill_should_trigger": true,
"prompt": "which kernel is the hotspot in ./workloads/my_app and why is it slow?\n\nThe application is ./vmem.hip.",
"workspace": "evals/samples",
"expected_behavior": [
"points --path at the directory containing profiling_config.yaml rather than a parent directory",
"uses --list-stats first, then narrows to one kernel id with -k",
"reads the System Speed-of-Light panel before opening a detailed block"
],
"logs_contain": ["--list-stats", "-k"]
},
{
"id": "kb-expensive-replay-fallback",
"skill_should_trigger": true,
"prompt": "a full rocprof-compute replay of this workload is too expensive, collect something cheaper\n\nThe application is ./vmem.hip.",
"workspace": "evals/samples",
"expected_behavior": [
"offers a single-pass option such as --set, a narrowed -b, or --iteration-multiplexing",
"runs --list-sets before selecting a set",
"warns which analysis panels will be empty afterwards"
],
"logs_contain": ["--list-sets"]
},
{
"id": "kb-unsafe-to-replay",
"skill_should_trigger": true,
Expand All @@ -53,15 +15,16 @@
]
},
{
"id": "kb-kernel-name-filter",
"id": "kb-where-time-goes",
"skill_should_trigger": true,
"prompt": "profile only the kernels whose names end in bound in my ROCm application\n\nThe application is ./occupancy.hip.",
"workspace": "evals/samples",
"expected_behavior": [
"uses rocprof-compute profile -k to filter by kernel name",
"treats the profile-time -k value as a regex matched against kernel names"
],
"logs_contain": ["-k"]
"note": "Catalog trigger check. The profiled version is kb-profile-hip-application in extended_evals.json.",
"prompt": "profile my HIP application with rocprof-compute and tell me where the time goes"
},
{
"id": "kb-cheaper-collection",
"skill_should_trigger": true,
"note": "Catalog trigger check. The profiled version is kb-expensive-replay-fallback in extended_evals.json.",
"prompt": "a full rocprof-compute replay of this workload is too expensive, what cheaper collection should I use?"
},
{
"id": "kb-cuda-application",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
{
"comment": "Profile-and-read cases. The AMD skills catalog does not grade this file. Run it here with ./skills/run_evals.py --extended.",
"evaluations": [
{
"id": "kb-profile-hip-application",
"skill_should_trigger": true,
"prompt": "profile my HIP application with rocprof-compute and tell me where the time goes\n\nThe application is ./vcopy.cpp.",
"workspace": "evals/samples",
"expected_behavior": [
"asks for the launch command and a workload name before profiling",
"runs rocprof-compute profile --name <name> -- <application>",
"runs analyze with --list-stats to find the hottest kernel"
],
"unexpected_behavior": [
"invents a profiling flag instead of checking --help"
],
"logs_contain": [
"rocprof-compute profile",
"--list-stats"
]
},
{
"id": "kb-which-kernel-is-slow",
"skill_should_trigger": true,
"prompt": "which kernel is the hotspot in ./workloads/my_app and why is it slow?\n\nThe application is ./vmem.hip.",
"workspace": "evals/samples",
"expected_behavior": [
"points --path at the directory containing profiling_config.yaml rather than a parent directory",
"uses --list-stats first, then narrows to one kernel id with -k",
"reads the System Speed-of-Light panel before opening a detailed block"
],
"logs_contain": [
"--list-stats",
"-k"
]
},
{
"id": "kb-expensive-replay-fallback",
"skill_should_trigger": true,
"prompt": "a full rocprof-compute replay of this workload is too expensive, collect something cheaper\n\nThe application is ./vmem.hip.",
"workspace": "evals/samples",
"expected_behavior": [
"offers a single-pass option such as --set, a narrowed -b, or --iteration-multiplexing",
"runs --list-sets before selecting a set",
"warns which analysis panels will be empty afterwards"
],
"logs_contain": [
"--list-sets"
]
},
{
"id": "kb-kernel-name-filter",
"skill_should_trigger": true,
"prompt": "profile only the kernels whose names end in bound in my ROCm application\n\nThe application is ./occupancy.hip.",
"workspace": "evals/samples",
"expected_behavior": [
"uses rocprof-compute profile -k to filter by kernel name",
"treats the profile-time -k value as a regex matched against kernel names"
],
"logs_contain": [
"-k"
]
}
]
}
Original file line number Diff line number Diff line change
@@ -1,67 +1,30 @@
{
"comment": "Minimum dataset graded by the AMD skills catalog. Cases that profile a sample and read the report live in extended_evals.json.",
"evaluations": [
{
"id": "mem-access-inefficiency",
"id": "mem-which-blocks",
"skill_should_trigger": true,
"prompt": "find the memory access inefficiencies in this rocprofiler-compute capture\n\nThe application is ./vrandom_access.cpp.",
"workspace": "evals/samples",
"note": "Catalog behavioral check. Asks which blocks to collect and does not stage a sample.",
"prompt": "which rocprof-compute blocks should I collect to inspect cache hit rates and LDS bank conflicts, without a full profile?",
"expected_behavior": [
"reads the memory chart before opening an individual cache level",
"cites hit rates or request counts from the report",
"ties any coalescing or tiling advice to those numbers"
"names the memory chart, or -b memchart, as the place to start",
"collects only the memory blocks the question needs, such as memchart, vl1d, l2, or lds"
],
"unexpected_behavior": [
"recommends an optimization without citing a metric from the report"
],
"logs_contain": ["-b memchart"]
"starts a full multi-pass profile or the roofline benchmarks"
]
},
{
"id": "mem-cache-hit-rates",
"id": "mem-where-hit-rates",
"skill_should_trigger": true,
"prompt": "what are my L2 and vL1D hit rates for this kernel and are they bad?\n\nThe application is ./vsequential_access.cpp.",
"workspace": "evals/samples",
"expected_behavior": [
"opens the l2 and vl1d blocks for a single kernel id",
"judges the rates against what the algorithm should do rather than a fixed cutoff"
],
"unexpected_behavior": [
"applies a remembered threshold such as L2 below 50% being bad, as if the tool defined one"
],
"logs_contain": ["-b"]
"note": "Catalog trigger check. The profiled version is mem-cache-hit-rates in extended_evals.json.",
"prompt": "where do I read L2 and vL1D hit rates for one kernel in rocprof-compute?"
},
{
"id": "mem-lds-bank-conflicts",
"id": "mem-where-bandwidth-goes",
"skill_should_trigger": true,
"prompt": "does my kernel have LDS bank conflicts?\n\nThe application is ./lds.hip.",
"workspace": "evals/samples",
"expected_behavior": [
"opens the lds block for the selected kernel",
"reads the bank conflict counters from the report"
],
"logs_contain": ["lds"]
},
{
"id": "mem-bandwidth-bound",
"skill_should_trigger": true,
"prompt": "this kernel looks memory bound, where is the bandwidth going?\n\nThe application is ./vcopy.cpp.",
"workspace": "evals/samples",
"expected_behavior": [
"walks the hierarchy from the memory chart down to the level responsible",
"distinguishes traffic that caches absorbed from traffic that reached HBM"
],
"logs_contain": ["memchart"]
},
{
"id": "mem-membw-analysis-gfx950",
"skill_should_trigger": true,
"prompt": "run the guided memory bandwidth analysis on my MI350 workload\n\nThe application is ./gl2_backpressure.hip.",
"workspace": "evals/samples",
"expected_behavior": [
"profiles only blocks 3 and 30 with --experimental --membw-analysis",
"notes that the analysis is experimental and shipped for gfx950",
"reports the guidance the tool produced rather than substituting its own cutoffs"
],
"logs_contain": ["--membw-analysis"]
"note": "Catalog trigger check. The profiled version is mem-bandwidth-bound in extended_evals.json.",
"prompt": "this AMD GPU kernel looks memory bound, which rocprof-compute view shows where the bandwidth goes?"
},
{
"id": "mem-host-memory",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
{
"comment": "Profile-and-read cases. The AMD skills catalog does not grade this file. Run it here with ./skills/run_evals.py --extended.",
"evaluations": [
{
"id": "mem-access-inefficiency",
"skill_should_trigger": true,
"prompt": "find the memory access inefficiencies in this rocprofiler-compute capture\n\nThe application is ./vrandom_access.cpp.",
"workspace": "evals/samples",
"expected_behavior": [
"reads the memory chart before opening an individual cache level",
"cites hit rates or request counts from the report",
"ties any coalescing or tiling advice to those numbers"
],
"unexpected_behavior": [
"recommends an optimization without citing a metric from the report"
],
"logs_contain": [
"-b memchart"
]
},
{
"id": "mem-cache-hit-rates",
"skill_should_trigger": true,
"prompt": "what are my L2 and vL1D hit rates for this kernel and are they bad?\n\nThe application is ./vsequential_access.cpp.",
"workspace": "evals/samples",
"expected_behavior": [
"opens the l2 and vl1d blocks for a single kernel id",
"judges the rates against what the algorithm should do rather than a fixed cutoff"
],
"unexpected_behavior": [
"applies a remembered threshold such as L2 below 50% being bad, as if the tool defined one"
],
"logs_contain": [
"-b"
]
},
{
"id": "mem-lds-bank-conflicts",
"skill_should_trigger": true,
"prompt": "does my kernel have LDS bank conflicts?\n\nThe application is ./lds.hip.",
"workspace": "evals/samples",
"expected_behavior": [
"opens the lds block for the selected kernel",
"reads the bank conflict counters from the report"
],
"logs_contain": [
"lds"
]
},
{
"id": "mem-bandwidth-bound",
"skill_should_trigger": true,
"prompt": "this kernel looks memory bound, where is the bandwidth going?\n\nThe application is ./vcopy.cpp.",
"workspace": "evals/samples",
"expected_behavior": [
"walks the hierarchy from the memory chart down to the level responsible",
"distinguishes traffic that caches absorbed from traffic that reached HBM"
],
"logs_contain": [
"memchart"
]
},
{
"id": "mem-membw-analysis-gfx950",
"skill_should_trigger": true,
"prompt": "run the guided memory bandwidth analysis on my MI350 workload\n\nThe application is ./gl2_backpressure.hip.",
"workspace": "evals/samples",
"expected_behavior": [
"profiles only blocks 3 and 30 with --experimental --membw-analysis",
"notes that the analysis is experimental and shipped for gfx950",
"reports the guidance the tool produced rather than substituting its own cutoffs"
],
"logs_contain": [
"--membw-analysis"
]
}
]
}
Loading
Loading