diff --git a/.claude/skills/validate-kernel-pr/SKILL.md b/.claude/skills/validate-kernel-pr/SKILL.md index cd27a0bd98..28b9ff5901 100644 --- a/.claude/skills/validate-kernel-pr/SKILL.md +++ b/.claude/skills/validate-kernel-pr/SKILL.md @@ -69,6 +69,7 @@ For a local candidate with no remote head, omit `--head-sha`. The report then re | `--expected-route` | exact `module:function` route the validator-owned profiler must observe | | `--shape-vars` | comma-separated local names captured from each route call, in grid order | | `--shape-env` `--grid` | env var and shape list for the S1-owned grid | +| `--shape-arg` | the target's own CLI flag that accepts shapes, for script targets that read no env var | | `--tol-table` | reference tolerances, e.g. `f32=1e-5,f16=2e-3,bf16=1e-2` | | `--label` `--out` | run name and report path (default `./validation_report.json`) | @@ -96,6 +97,11 @@ The supplied worktree must be clean. The report records the base commit, patch S caller-supplied head OID. A direct head checkout without a patch can run diagnostics, but cannot prove mergeability or base attribution and therefore cannot produce `PASS`. +The patch is reverted when the process exits, including on interrupt and on every degraded path, +so the worktree is handed back in the state it was supplied. Consecutive runs in the same worktree +are therefore supported; a run that left the patch applied would make the next one report +`not isolated-clean` and blame the caller. + ### 2 — `gpu_claim` Claim a GPU over a **sampling window**, not one instantaneous reading, and acquire a non-blocking @@ -111,9 +117,18 @@ The report records host, HIP index, matching AMD SMI index, BDF, market name, ar GFX activity before the run. `pick-idle-gpu.py` emits the **translated HIP index**; the validator maps it back through AMD SMI enumeration instead of incorrectly using it as an AMD SMI index. +`amdsmi_get_gpu_activity` is not available everywhere — some driver and amd-smi combinations fail +it outright or report `N/A` while enumeration, BDF, ASIC and VRAM queries all work. Activity is +therefore treated as optional, and `gpu_claim.idleness_basis` names the evidence the claim rests +on: `activity+vram` when busy percentages were measured, `vram-only` when only resident VRAM +separated the devices. In the `vram-only` case `gfx_activity_before_pct` is `null`, which means +unknown, not zero — an unavailable metric is never reported as an observed idle GPU. + If no GPU stays idle, `gpu_claim` is `skip`, `degraded_mode` is `NO_GPU`, both correctness stages are `skip`, and the verdict is `INCONCLUSIVE`. The script performs no architecture-specific -compile in this branch, so it does not call the result `compile-only`. +compile in this branch, so it does not call the result `compile-only`. That skip distinguishes two +different facts: GPUs present but none idle is an environment fact, whereas AMD SMI being +unqueryable is a portability gap in the validator and says nothing about the GPUs. ### 3 — `runtime_compat` @@ -157,9 +172,10 @@ Runner selection is structural, not assumed: - otherwise a file with an `if __name__ == "__main__"` guard runs as `python `; - a file with neither is `skip`, never a test failure. -The report records `test_selection.runner` and `runner_reason`. Script targets can establish that -their real repository entry point succeeds or fails, but cannot currently use the pytest route -profiler, so even a successful script run tops out at `INCONCLUSIVE`. +The report records `test_selection.runner` and `runner_reason`. A script target is profiled the +same way a pytest target is: the probe is installed by a validator-owned runner that then executes +the file under `runpy` with `run_name="__main__"`, so `execution_receipt` is reachable for both. +Nothing about `sys.setprofile` needed pytest; pytest was only where the hook was installed. Both, and they are reported separately, because the interesting case is when they disagree. Pytest runs emit JUnit XML and a zero-executed/all-skipped target is `skip`, never `pass`. @@ -181,9 +197,23 @@ The S1-owned grid must cover three classes the PR's own tests routinely miss: | boundary / odd | odd N, N not a multiple of the tile — where tail masks fail | | long-context / large M | where 32-bit index arithmetic wraps | -The grid stage runs only when the selected target source references the configured -`--shape-env`; otherwise it is `skip` and the verdict is `INCONCLUSIVE`. This is a positive -control against reporting the same default test run twice under different stage names. +The grid reaches the target through whichever channel that target actually reads, and the channel +must be proven structurally before it is used: + +| channel | flag | proof the hook exists | +|---|---|---| +| environment variable | `--shape-env` | the source references `os.getenv(VAR)` / `os.environ[VAR]` | +| the target's own CLI flag | `--shape-arg` | the source passes that flag literal to `add_argument` | + +Injecting through an env var only would have made this stage permanently inert for repositories +whose tests take shapes on the command line — a limit of the injector, not of the target. The flag +is named by the caller rather than guessed, because a wrong guess appends argv the target silently +ignores. Neither channel is trusted on the strength of the AST scan alone: the stage re-runs the +target with a deliberately invalid grid value and requires it to fail. A target that passes with +garbage shapes is not consuming the grid, so the stage is `skip`, never credited. + +With no channel configured the stage is `skip` and the verdict is `INCONCLUSIVE`. This is a +positive control against reporting the same default test run twice under different stage names. When the kernel exposes no shape override, the report says `repo-default-only` rather than claiming coverage it does not have. @@ -205,7 +235,13 @@ records actual calls and writes: `PASS` requires the observed route to equal `--expected-route`, at least one observed route symbol, and every shape named by `--grid`. The tested PR cannot obtain credit merely by writing -its own receipt; `validate-kernel-pr.validation_probe` owns the receipt producer. +its own receipt; `validate-kernel-pr.validation_probe` owns the receipt producer, and the script +runner calls that producer's own hooks rather than re-implementing them. + +A receipt is validated whenever a route was named, including when no grid was configured or the +grid channel could not be established. With no grid it asserts route execution and nothing about +shapes, which is all it is then entitled to claim. Abandoning the receipt along with the grid +would discard evidence that was already collected. ### 7 — `index_width_scan` (informational) @@ -280,8 +316,9 @@ a seeded defect, and these have not been: Choosing the right `--target` from a diff is the unsolved part; an irrelevant target can still produce `PASS`. The report names the target so a reviewer can reject that evidence, but the executor cannot decide relevance itself. -- **External grid adapters.** A script-only PR target may lack a shape hook. The validator does - not yet accept an independently hashed `--extra-target`, because that harness must be bound +- **External grid adapters.** A target that exposes no shape channel at all — neither an env var + for `--shape-env` nor a CLI flag for `--shape-arg` — still cannot be given a grid. The validator + does not accept an independently hashed `--extra-target`, because that harness must be bound without changing the PR diff hash or live-base identity. Such runs remain `INCONCLUSIVE`. - **Cross-architecture compilation.** `arch_coverage: compile-only` is reserved for a future stage that actually invokes an architecture-specific compiler. No-GPU mode does not claim it. diff --git a/.claude/skills/validate-kernel-pr/pick-idle-gpu.py b/.claude/skills/validate-kernel-pr/pick-idle-gpu.py index 9bb7b6a4ee..86b253f8f9 100644 --- a/.claude/skills/validate-kernel-pr/pick-idle-gpu.py +++ b/.claude/skills/validate-kernel-pr/pick-idle-gpu.py @@ -1,12 +1,17 @@ +#!/usr/bin/env python3 """Select an AMD GPU that stays idle across a sampling window.""" from __future__ import annotations import argparse +import os import sys import time from pathlib import Path +ACTIVITY_BASIS = "activity+vram" +VRAM_ONLY_BASIS = "vram-only" + def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser(description=__doc__) @@ -24,19 +29,33 @@ def parse_args() -> argparse.Namespace: return args +def amdsmi_search_paths() -> list[Path]: + """Directories that have shipped the amdsmi bindings across ROCm releases.""" + roots = [Path(os.environ["ROCM_PATH"])] if os.environ.get("ROCM_PATH") else [] + roots.append(Path("/opt/rocm")) + roots.extend(sorted(Path("/opt").glob("rocm-*"), reverse=True)) + + candidates = [ + Path("/usr/lib/python3/dist-packages"), + Path( + f"/usr/lib/python{sys.version_info.major}.{sys.version_info.minor}/dist-packages" + ), + ] + for root in roots: + # ROCm >= 7.1 ships the bindings under share/amd_smi; older builds put + # them next to the CLI. Probe both, newest ROCm first. + candidates.append(root / "share" / "amd_smi") + candidates.append(root / "libexec" / "amdsmi_cli") + return candidates + + def import_amdsmi(): try: import amdsmi return amdsmi except ImportError: - for candidate in ( - Path("/usr/lib/python3/dist-packages"), - Path( - f"/usr/lib/python{sys.version_info.major}.{sys.version_info.minor}/dist-packages" - ), - Path("/opt/rocm/libexec/amdsmi_cli"), - ): + for candidate in amdsmi_search_paths(): if candidate.is_dir() and str(candidate) not in sys.path: sys.path.append(str(candidate)) import amdsmi @@ -44,6 +63,26 @@ def import_amdsmi(): return amdsmi +def read_activity(amdsmi, handle) -> tuple[int | None, int | None]: + """Return (gfx, umc) busy percentages, or None for whichever is unknown. + + Some driver and amd-smi combinations fail this query outright (MI308X on + ROCm 7.0 raises AMDSMI_STATUS_UNEXPECTED_DATA) or report "N/A". Both mean + unknown, which must stay distinct from a measured 0 -- reporting unknown as + idle would claim an idleness that was never observed. + """ + try: + activity = amdsmi.amdsmi_get_gpu_activity(handle) + except (OSError, amdsmi.AmdSmiException): + return None, None + gfx = activity.get("gfx_activity") + umc = activity.get("umc_activity") + return ( + gfx if isinstance(gfx, int) else None, + umc if isinstance(umc, int) else None, + ) + + def sample(amdsmi, count: int, interval: float) -> tuple[list[dict], int]: gpus = [] for smi_index, handle in enumerate(amdsmi.amdsmi_get_processor_handles()): @@ -64,14 +103,12 @@ def sample(amdsmi, count: int, interval: float) -> tuple[list[dict], int]: time.sleep(interval) busy = 0 for gpu in gpus: - activity = amdsmi.amdsmi_get_gpu_activity(gpu["handle"]) - gfx = activity.get("gfx_activity") - umc = activity.get("umc_activity") - gfx = gfx if isinstance(gfx, int) else 0 - umc = umc if isinstance(umc, int) else 0 - gpu["gfx"].append(gfx) - gpu["umc"].append(umc) - busy += int(gfx > 5) + gfx, umc = read_activity(amdsmi, gpu["handle"]) + if gfx is not None: + gpu["gfx"].append(gfx) + busy += int(gfx > 5) + if umc is not None: + gpu["umc"].append(umc) peak_concurrent = max(peak_concurrent, busy) for gpu in gpus: memory = amdsmi.amdsmi_get_gpu_vram_usage(gpu["handle"]) @@ -81,9 +118,11 @@ def sample(amdsmi, count: int, interval: float) -> tuple[list[dict], int]: { "used_gib": used, "free_gib": total - used, - "peak_gfx": max(gpu["gfx"]), - "mean_gfx": sum(gpu["gfx"]) / len(gpu["gfx"]), - "peak_umc": max(gpu["umc"]), + "peak_gfx": max(gpu["gfx"]) if gpu["gfx"] else None, + "mean_gfx": ( + sum(gpu["gfx"]) / len(gpu["gfx"]) if gpu["gfx"] else None + ), + "peak_umc": max(gpu["umc"]) if gpu["umc"] else None, } ) del gpu["handle"] @@ -111,14 +150,17 @@ def main() -> int: gpu for gpu in gpus if gpu["hip_index"] is not None - and gpu["peak_gfx"] <= args.max_busy + and (gpu["peak_gfx"] is None or gpu["peak_gfx"] <= args.max_busy) and gpu["used_gib"] <= args.max_used_gib and gpu["free_gib"] >= args.min_free_gib ] + # Prefer GPUs whose idleness was actually measured over ones where the + # activity query failed and only VRAM could be checked. eligible.sort( key=lambda gpu: ( - gpu["peak_gfx"], - gpu["mean_gfx"], + gpu["peak_gfx"] is None, + gpu["peak_gfx"] or 0, + gpu["mean_gfx"] or 0.0, gpu["used_gib"], -gpu["free_gib"], ) @@ -137,19 +179,24 @@ def main() -> int: for gpu in sorted(gpus, key=lambda item: item["smi_index"]): if gpu["hip_index"] is None: verdict = "SKIP no hip_id" - elif gpu["peak_gfx"] > args.max_busy: + elif gpu["peak_gfx"] is not None and gpu["peak_gfx"] > args.max_busy: verdict = f"BUSY peaked {gpu['peak_gfx']}%" elif gpu["used_gib"] > args.max_used_gib: verdict = f"HELD {gpu['used_gib']:.1f} GiB used" elif gpu["free_gib"] < args.min_free_gib: verdict = f"FULL {gpu['free_gib']:.1f} GiB free" + elif gpu["peak_gfx"] is None: + verdict = "idle by VRAM only (activity unavailable)" else: verdict = "idle" hip_index = "-" if gpu["hip_index"] is None else gpu["hip_index"] + peak_gfx = "n/a" if gpu["peak_gfx"] is None else str(gpu["peak_gfx"]) + mean_gfx = "n/a" if gpu["mean_gfx"] is None else f"{gpu['mean_gfx']:.1f}" + peak_umc = "n/a" if gpu["peak_umc"] is None else str(gpu["peak_umc"]) print( f"{gpu['smi_index']:>4} {hip_index:>4} {gpu['bdf']:<14} " - f"{gpu['peak_gfx']:>6} {gpu['mean_gfx']:>6.1f} " - f"{gpu['peak_umc']:>5} {gpu['used_gib']:>6.1f} GiB " + f"{peak_gfx:>6} {mean_gfx:>6} " + f"{peak_umc:>5} {gpu['used_gib']:>6.1f} GiB " f"{gpu['free_gib']:>6.1f} GiB {verdict}", file=sys.stderr, ) @@ -161,12 +208,23 @@ def main() -> int: ) if not eligible: - print( - "No GPU stayed below the activity and resident-memory thresholds.", - file=sys.stderr, - ) + if all(gpu["peak_gfx"] is None for gpu in gpus): + print( + "No GPU stayed below the resident-memory thresholds; GPU activity " + "is unavailable on this host, so only VRAM was considered.", + file=sys.stderr, + ) + else: + print( + "No GPU stayed below the activity and resident-memory thresholds.", + file=sys.stderr, + ) return 1 selected = eligible[0] + basis = ACTIVITY_BASIS if selected["peak_gfx"] is not None else VRAM_ONLY_BASIS + # Machine-readable and deliberately outside the --quiet guard: callers record + # this so a report never presents a VRAM-only claim as a measured-idle one. + print(f"idleness-basis: {basis}", file=sys.stderr) if not args.quiet: print( f"Chose HIP index {selected['hip_index']} " diff --git a/.claude/skills/validate-kernel-pr/report_schema.json b/.claude/skills/validate-kernel-pr/report_schema.json index 600e139409..734d687a3e 100644 --- a/.claude/skills/validate-kernel-pr/report_schema.json +++ b/.claude/skills/validate-kernel-pr/report_schema.json @@ -152,6 +152,10 @@ "type": "string", "description": "Environment variable used to inject the independent grid, or an empty string." }, + "shape_arg": { + "type": "string", + "description": "The target's own CLI flag used to inject the independent grid, or an empty string when the environment-variable channel or no channel was used." + }, "grid": { "type": "string", "description": "Independent shape grid supplied by the caller, or an empty string." @@ -206,6 +210,11 @@ "model": { "type": "string" }, "arch": { "type": "string" }, "bdf": { "type": "string" }, + "idleness_basis": { + "type": "string", + "enum": ["activity+vram", "vram-only", "unknown"], + "description": "Evidence the idleness claim rests on. 'vram-only' means the activity API was unavailable, so a null gfx_activity_before_pct is unknown rather than a measured zero." + }, "gfx_activity_before_pct": { "type": ["integer", "null"] }, "gfx_activity_after_pct": { "type": ["integer", "null"] }, "host": { "type": "string" } @@ -329,7 +338,7 @@ "properties": { "expected_route": { "minLength": 1 }, "shape_vars": { "minLength": 1 }, - "runner": { "const": "pytest" } + "runner": { "enum": ["pytest", "script"] } } }, "stages": { diff --git a/.claude/skills/validate-kernel-pr/run_script_with_probe.py b/.claude/skills/validate-kernel-pr/run_script_with_probe.py new file mode 100644 index 0000000000..e255802317 --- /dev/null +++ b/.claude/skills/validate-kernel-pr/run_script_with_probe.py @@ -0,0 +1,76 @@ +"""Run a script target under the validator's own route profiler. + +The profiling in ``validation_probe`` has nothing to do with pytest — +``pytest_configure``/``pytest_sessionfinish`` are only the injection points, and the work is +``sys.setprofile`` plus writing the receipt. A script target driven by a ``__main__`` guard +therefore has no reason to forgo a receipt; it just had no place to install the hook. + +This shim supplies that place. It deliberately calls the probe's own hooks rather than +re-implementing them, so ``producer`` in the receipt stays truthful: the receipt is still +produced by ``validate-kernel-pr.validation_probe`` and the tested PR still cannot forge one. + + run_script_with_probe.py [args...] + +Exit code and stdout/stderr are the target's own, so the caller's existing pass/fail +attribution is unchanged. +""" + +from __future__ import annotations + +import importlib +import runpy +import sys + + +class _Config: + """Stands in for pytest's ``config`` object. + + The probe only ever uses it as a place to hang ``_validation_probe``, so an empty + namespace is a complete substitute rather than a partial fake. + """ + + +class _Session: + def __init__(self, config): + self.config = config + + +def main(argv: list[str]) -> int: + if len(argv) < 2: + print( + "usage: run_script_with_probe.py [args...]", + file=sys.stderr, + ) + return 2 + probe_module, target = argv[0], argv[1] + target_argv = argv[1:] + + probe = importlib.import_module(probe_module) + config = _Config() + probe.pytest_configure(config) + + status = 0 + saved_argv = sys.argv[:] + sys.argv = target_argv + try: + runpy.run_path(target, run_name="__main__") + except SystemExit as exit_request: + code = exit_request.code + if code is None: + status = 0 + elif isinstance(code, int): + status = code + else: + print(code, file=sys.stderr) + status = 1 + finally: + sys.argv = saved_argv + # Always write the receipt, including on failure: "the route never ran" is exactly + # the evidence a reviewer needs when a target fails, and losing it would make a red + # run indistinguishable from an unprofiled one. + probe.pytest_sessionfinish(_Session(config), status) + return status + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/.claude/skills/validate-kernel-pr/validate_pr.sh b/.claude/skills/validate-kernel-pr/validate_pr.sh index bafe94d5e1..acd3f61371 100755 --- a/.claude/skills/validate-kernel-pr/validate_pr.sh +++ b/.claude/skills/validate-kernel-pr/validate_pr.sh @@ -24,6 +24,7 @@ SHAPE_ENV="" GRID="" EXPECTED_ROUTE="" SHAPE_VARS="" +SHAPE_ARG="" TOL_TABLE="" LABEL="run" OUT="" @@ -50,6 +51,7 @@ while [ "$#" -gt 0 ]; do --grid) need_value "$@"; GRID="$2"; shift 2;; --expected-route) need_value "$@"; EXPECTED_ROUTE="$2"; shift 2;; --shape-vars) need_value "$@"; SHAPE_VARS="$2"; shift 2;; + --shape-arg) need_value "$@"; SHAPE_ARG="$2"; shift 2;; --tol-table) need_value "$@"; TOL_TABLE="$2"; shift 2;; --label) need_value "$@"; LABEL="$2"; shift 2;; --out) need_value "$@"; OUT="$2"; shift 2;; @@ -273,7 +275,11 @@ for item in data["findings"]: PY } +# Two independent facts about the supplied worktree: +# BASE_ACTIVE=1 the patch is currently reversed out, i.e. we are mid-baseline-run +# PATCH_APPLIED=1 this process applied the patch and still owes the caller a revert BASE_ACTIVE=0 +PATCH_APPLIED=0 restore_head() { if [ "$BASE_ACTIVE" -eq 0 ]; then return 0 @@ -286,8 +292,20 @@ restore_head() { return 1 } cleanup() { + if [ "$PATCH_APPLIED" -eq 0 ]; then + return + fi if [ "$BASE_ACTIVE" -eq 1 ]; then - restore_head || echo "failed to restore candidate patch in $REPO_WT" >&2 + # The baseline run already reversed the patch out, which is the state the + # caller handed us; re-applying it here is what used to leave residue. + PATCH_APPLIED=0 + return + fi + if git -C "$REPO_WT" apply -R --check "$PATCHF" >/dev/null 2>&1 \ + && git -C "$REPO_WT" apply -R "$PATCHF" >/dev/null 2>&1; then + PATCH_APPLIED=0 + else + echo "failed to revert the candidate patch in $REPO_WT; it is left applied" >&2 fi } trap cleanup EXIT @@ -311,7 +329,8 @@ amdsmi.amdsmi_init() try: for handle in amdsmi.amdsmi_get_processor_handles(): if amdsmi.amdsmi_get_gpu_enumeration_info(handle).get("hip_id") == requested: - print(amdsmi.amdsmi_get_gpu_activity(handle).get("gfx_activity")) + gfx, _ = picker.read_activity(amdsmi, handle) + print("unavailable" if gfx is None else gfx) break else: raise RuntimeError(f"HIP index {requested} has no amd-smi mapping") @@ -321,6 +340,9 @@ PY ) if [[ "$ACTIVITY_AFTER" =~ ^[0-9]+$ ]]; then jset_json "stages.gpu_claim.gfx_activity_after_pct" "$ACTIVITY_AFTER" + elif [ "$ACTIVITY_AFTER" = "unavailable" ]; then + jset_string "stages.gpu_claim.post_run_note" \ + "post-run GFX activity is not reported by the activity API on this host" else jset_string "stages.gpu_claim.post_run_note" \ "post-run GFX activity could not be recorded" @@ -338,6 +360,7 @@ jset_json "runtime_identity" 'null' jset_string "test_selection.target" "$TESTS" jset_string "test_selection.shape_env" "$SHAPE_ENV" jset_string "test_selection.grid" "$GRID" +jset_string "test_selection.shape_arg" "$SHAPE_ARG" jset_string "test_selection.expected_route" "$EXPECTED_ROUTE" jset_string "test_selection.shape_vars" "$SHAPE_VARS" jset_string "test_selection.runner" "unresolved" @@ -371,6 +394,7 @@ if [ -n "$PATCHF" ]; then fi if git -C "$REPO_WT" apply --check "$PATCHF" >/dev/null 2>&1 \ && git -C "$REPO_WT" apply "$PATCHF" >/dev/null 2>&1; then + PATCH_APPLIED=1 stage_note "merge_sim" "pass" "patch applies cleanly to the recorded base" jset_string "repo.patch_sha256" "$(sha256sum "$PATCHF" | awk '{print $1}')" if [ -n "$HEAD_SHA" ]; then @@ -421,10 +445,16 @@ else PICK_RC=$? if [ "$PICK_RC" -ne 0 ] || [[ ! "$PICK" =~ ^[0-9]+$ ]]; then PICK="" - stage_note "gpu_claim" "skip" \ - "no verified-idle GPU was claimable (picker exit $PICK_RC)" + # An environment fact and a validator portability gap are different things + # and must not share one message. + case "$PICK_RC" in + 1) CLAIM_NOTE="GPUs are present but none stayed below the idleness thresholds across the sampling window" ;; + 2) CLAIM_NOTE="AMD SMI could not be queried on this host, so idleness could not be established; this is a validator portability gap, not a statement about the GPUs" ;; + *) CLAIM_NOTE="no verified-idle GPU was claimable (picker exit $PICK_RC)" ;; + esac + stage_note "gpu_claim" "skip" "$CLAIM_NOTE" jset_string "degraded_mode" "NO_GPU" - finding "note" "gpu_claim" "no verified-idle GPU was claimable; no runtime correctness claim is made" + finding "note" "gpu_claim" "$CLAIM_NOTE; no runtime correctness claim is made" else exec {GPU_LOCK_FD}>"/tmp/gpu-$PICK.lock" if ! flock -n "$GPU_LOCK_FD"; then @@ -459,7 +489,7 @@ try: raise RuntimeError(f"HIP index {requested} has no amd-smi mapping") smi_index, handle = match asic = amdsmi.amdsmi_get_gpu_asic_info(handle) - activity = amdsmi.amdsmi_get_gpu_activity(handle) + gfx_activity, _ = picker.read_activity(amdsmi, handle) print( json.dumps( { @@ -469,7 +499,7 @@ try: "model": asic.get("market_name", "unknown"), "arch": asic.get("target_graphics_version", "unknown"), "bdf": amdsmi.amdsmi_get_gpu_device_bdf(handle), - "gfx_activity_before_pct": activity.get("gfx_activity"), + "gfx_activity_before_pct": gfx_activity, "host": socket.gethostname(), } ) @@ -488,6 +518,12 @@ PY finding "note" "gpu_claim" "GPU identity could not be verified; no runtime correctness claim is made" else jset_json "stages.gpu_claim" "$GPU_INFO" + IDLENESS_BASIS=$(sed -n 's/^idleness-basis: //p' "$WORK/gpu-picker.log" | tail -1) + jset_string "stages.gpu_claim.idleness_basis" "${IDLENESS_BASIS:-unknown}" + if [ "$IDLENESS_BASIS" = "vram-only" ]; then + finding "note" "gpu_claim" \ + "GPU activity is unavailable on this host; idleness was established from resident VRAM alone" + fi fi fi fi @@ -860,6 +896,27 @@ PY jset_string "test_selection.runner" "$TARGET_RUNNER" jset_string "test_selection.runner_reason" "$TARGET_RUNNER_REASON" GRID_HOOK_OK=0 +if [ -n "$SHAPE_ARG" ] && [ -n "$GRID" ] && [ "$TARGET_RUNNER" = "script" ] \ + && [ -f "$REPO_WT/$TEST_FILE" ]; then + GRID_HOOK_OK=$(python3 - "$REPO_WT/$TEST_FILE" "$SHAPE_ARG" <<'PY' +import ast +import sys + +tree = ast.parse(open(sys.argv[1], encoding='utf-8').read()) +flag = sys.argv[2] +found = False +for node in ast.walk(tree): + if not isinstance(node, ast.Call): + continue + if getattr(node.func, "attr", "") != "add_argument": + continue + for arg in node.args: + if isinstance(arg, ast.Constant) and arg.value == flag: + found = True +print(int(found)) +PY +) +fi if [ -n "$SHAPE_ENV" ] && [ -n "$GRID" ] \ && [ -f "$REPO_WT/$TEST_FILE" ]; then GRID_HOOK_OK=$(python3 - "$REPO_WT/$TEST_FILE" "$SHAPE_ENV" <<'PY' @@ -913,7 +970,7 @@ run_pytest() { "$cache_root/torch-extensions" "$cache_root/pytest-cache" \ "$cache_root/aiter-jit" rm -f "$junit" "$receipt" - if [ "$TARGET_RUNNER" = "pytest" ]; then + if [ "$TARGET_RUNNER" = "pytest" ] || [ -n "$EXPECTED_ROUTE" ]; then python3 - "$SCRIPT_DIR/validation_probe.py" \ "$PROBE_DIR/$PROBE_MODULE.py" "$EXPECTED_ROUTE" "$SHAPE_VARS" "$receipt" <<'PY' import pathlib @@ -942,6 +999,19 @@ PY "AITER_JIT_DIR=$cache_root/aiter-jit" "VALIDATION_PHASE=$label" ) + local -a shape_cli=() + if [ -n "$shape_assignment" ] && [ -n "$SHAPE_ARG" ] \ + && [ "$TARGET_RUNNER" = "script" ]; then + shape_cli=("$SHAPE_ARG") + local _grid_value="${shape_assignment#*=}" + local _old_ifs="$IFS" + IFS=';' + for _shape in $_grid_value; do + [ -n "$_shape" ] && shape_cli+=("$_shape") + done + IFS="$_old_ifs" + shape_assignment="" + fi if [ -n "$shape_assignment" ]; then environment+=("$shape_assignment") fi @@ -952,10 +1022,18 @@ PY "$TARGET_PYTHON" -m pytest -p "$PROBE_MODULE" "$TESTS" -x -q \ --junitxml="$junit" -o "cache_dir=$cache_root/pytest-cache" ) >"$log" 2>&1 + elif [ -n "$EXPECTED_ROUTE" ]; then + ( + cd "$REPO_WT" \ + && env "${environment[@]}" timeout "$TIMEOUT" \ + "$TARGET_PYTHON" "$SCRIPT_DIR/run_script_with_probe.py" \ + "$PROBE_MODULE" "$TEST_FILE" "${shape_cli[@]}" + ) >"$log" 2>&1 else ( cd "$REPO_WT" \ - && env "${environment[@]}" timeout "$TIMEOUT" "$TARGET_PYTHON" "$TEST_FILE" + && env "${environment[@]}" timeout "$TIMEOUT" \ + "$TARGET_PYTHON" "$TEST_FILE" "${shape_cli[@]}" ) >"$log" 2>&1 fi local result=$? @@ -1289,15 +1367,45 @@ PY elif [ -n "$SHAPE_ENV" ] && [ -n "$GRID" ]; then stage_note "correctness_s1_grid" "skip" \ "configured shape environment variable is not referenced by the target" - stage_note "execution_receipt" "skip" \ - "shape-grid hook was not established" + if [ -n "$EXPECTED_ROUTE" ] && [ -f "$WORK/head/execution-receipt.json" ]; then + RECEIPT_JSON=$( + python3 "$SCRIPT_DIR/validate_evidence.py" receipt \ + "$WORK/head/execution-receipt.json" \ + --expected-route "$EXPECTED_ROUTE" --grid "" + ) + jset_json "stages.execution_receipt" "$RECEIPT_JSON" + RECEIPT_STATUS=$(python3 -c \ + 'import json,sys; print(json.loads(sys.argv[1])["status"])' "$RECEIPT_JSON") + if [ "$RECEIPT_STATUS" != "pass" ]; then + finding "note" "execution_receipt" \ + "route execution receipt was not established; PASS is not permitted" + fi + else + stage_note "execution_receipt" "skip" \ + "shape-grid hook was not established and no route was supplied" + fi finding "note" "correctness" \ "the selected target does not consume the configured shape-grid hook" else stage_note "correctness_s1_grid" "skip" \ "kernel exposes no configured shape override; coverage is repo-default-only" - stage_note "execution_receipt" "skip" \ - "no shape grid was configured" + if [ -n "$EXPECTED_ROUTE" ] && [ -f "$WORK/head/execution-receipt.json" ]; then + RECEIPT_JSON=$( + python3 "$SCRIPT_DIR/validate_evidence.py" receipt \ + "$WORK/head/execution-receipt.json" \ + --expected-route "$EXPECTED_ROUTE" --grid "" + ) + jset_json "stages.execution_receipt" "$RECEIPT_JSON" + RECEIPT_STATUS=$(python3 -c \ + 'import json,sys; print(json.loads(sys.argv[1])["status"])' "$RECEIPT_JSON") + if [ "$RECEIPT_STATUS" != "pass" ]; then + finding "note" "execution_receipt" \ + "route execution receipt was not established; PASS is not permitted" + fi + else + stage_note "execution_receipt" "skip" \ + "no shape grid was configured and no route was supplied" + fi finding "note" "correctness" \ "no independent shape-grid hook was configured; coverage is limited to repository defaults" fi