assert when found duplicated tuned shape

[yzhou103]

Motivation

assert error when there is duplicate tuned shape in configs

Technical Details

Duplicate shape detection and auto-dedup in config merge

When update_config_files() merges tuned config CSVs from configs/ and model_configs/, it detects and handles
duplicate shape entries as follows:

1). Config loading: All matching CSV files are loaded into source_pairs (a list of (path, DataFrame) tuples) and
concatenated into a single merge_df with ignore_index=True.
2). Dedup key derivation: The dedup keys are derived from the corresponding untuned config file's columns (plus
cu_num if absent, and _tag if present). These keys represent the shape dimensions that uniquely identify a tuning
entry.
3). Duplicate detection: merge_df.duplicated(subset=dedup_keys, keep=False) marks all rows that share the same
shape.
4). Handling duplicates:
- If the merged config has no us column, a RuntimeError is raised asking the user to remove
duplicates manually.
- If us is available, auto-dedup is performed: the merged DataFrame is sorted by us (stable sort) and
drop_duplicates(keep="first") retains only the best-performing (lowest latency) row for each shape. The resulting
index set (best_row_index) is used to filter each source file back to only its winning rows. Updated source files
are saved in-place, and a RuntimeError is raised listing the changes and asking the user to re-run.
5. Source file write-back: Each source file's row range in merge_df is tracked via a running offset. Rows not in
best_row_index are dropped, and if the filtered DataFrame is shorter than the original, the source CSV is
overwritten.
5). re-run test, the duplicated shapes are remove in previous step. re-run the test
2. add untuned shapes for re-tuning
when retune with "-i untuned.csv -o tuned.csv -all " it will retune all shapes in untuned.csv and replace results in tuned.csv

Test Plan

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:sglang	SGLang integration tests
ci:atom	ATOM benchmark (DeepSeek-R1 + GPT-OSS)
ci:vllm	vLLM benchmark
ci:all	All of the above

Add labels via the sidebar or gh pr edit 2376 --add-label <label>

[Copilot]

Pull request overview

This PR tightens config-merge correctness by making duplicate tuned shapes a hard failure during config merges, and adds model-specific untuned shape CSVs to support retuning workflows (with some tuned CSV cleanup to remove duplicate shapes).

Changes:

• Update update_config_files() to detect duplicate shapes during merges and fail fast instead of auto-dropping duplicates.
• Add several new model-specific *_untuned_*.csv shape lists under aiter/configs/model_configs/ to enable retuning coverage for specific models.
• Remove some duplicate tuned GEMM entries from model-specific tuned config CSVs to avoid collisions during merge.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
aiter/jit/core.py	Changes merge-time duplicate handling to error out on duplicate shapes (rather than sorting/deduping).
aiter/configs/model_configs/kimik2_fp4_untuned_fmoe.csv	Adds untuned FMoE shapes for kimik2 fp4 retuning.
aiter/configs/model_configs/kimik2_bf16_untuned_gemm.csv	Adds untuned GEMM shapes for kimik2 bf16 retuning.
aiter/configs/model_configs/kimik2_bf16_tuned_gemm.csv	Removes a tuned GEMM entry to eliminate duplicate-shape overlap during merges.
aiter/configs/model_configs/gptoss_bf16_untuned_gemm.csv	Adds untuned GEMM shapes for gptoss bf16 retuning.
aiter/configs/model_configs/dsv3_fp4_untuned_fmoe.csv	Adds untuned FMoE shapes for dsv3 fp4 retuning.
aiter/configs/model_configs/dsv3_bf16_untuned_gemm.csv	Adds untuned GEMM shapes for dsv3 bf16 retuning.
aiter/configs/model_configs/dsv3_bf16_tuned_gemm.csv	Removes tuned GEMM entries to eliminate duplicate-shape overlap during merges.

Comments suppressed due to low confidence (1)

aiter/jit/core.py:237

• The else branch logs: "Using all columns for deduplication.", but after this change there is no deduplication or duplicate detection performed when the untuned file is missing (and no fallback keys are computed). This makes the log message misleading and can allow duplicate entries to be written without any error. Either implement the intended fallback (e.g., detect/handle duplicates using merge_df.columns or similar) or update the message/behavior so it matches what the code actually does.

        else:
            logger.warning(
                f"Untuned config file not found: {untuned_path}. Using all columns for deduplication."
            )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yzhou103]