Repository navigation
Shard pandas tests across runners in PR and nightly CI #22992
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| #!/usr/bin/env bash | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| # Combine the results of the sharded nightly ("main") pandas-tests jobs. | ||
| # | ||
| # Each shard uploads its partial per-module summary as the GitHub artifact | ||
| # "pandas-test-main-results-<shard_id>". This job downloads them all and merges | ||
| # them into a single main-results.json, which is re-uploaded under that name so | ||
| # that PR runs keep finding it with `gh run download --name main-results.json`. | ||
| # | ||
| # Usage: | ||
| # merge-nightly.sh <num_shards> | ||
| # | ||
| # Unlike the PR-side summary.sh, this script is NOT best effort: main-results.json | ||
| # is the baseline every PR diffs against, and a partial or missing file would show | ||
| # up as spurious "new failures" in those PRs. Failing loudly instead keeps the run | ||
| # from being picked up as the latest successful nightly. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| source rapids-init-pip | ||
|
|
||
| NUM_SHARDS=${1:?usage: merge-nightly.sh <num_shards>} | ||
|
|
||
| rapids-logger "Merging pandas-tests results from ${NUM_SHARDS} shards" | ||
|
|
||
| for ((shard = 0; shard < NUM_SHARDS; shard++)); do | ||
| rapids-logger "Downloading results for shard ${shard}" | ||
| gh run download "${GITHUB_RUN_ID}" \ | ||
| --repo "${GITHUB_REPOSITORY}" \ | ||
| --name "pandas-test-main-results-${shard}" \ | ||
| --dir "shard-${shard}" | ||
| done | ||
|
|
||
| SHARD_RESULTS=() | ||
| for ((shard = 0; shard < NUM_SHARDS; shard++)); do | ||
| SHARD_RESULTS+=("shard-${shard}/main-results.json") | ||
| done | ||
|
|
||
| python ci/cudf_pandas_scripts/pandas-tests/merge-results.py \ | ||
| "${SHARD_RESULTS[@]}" > main-results.json | ||
|
|
||
| rapids-logger "Merged $(wc -l < main-results.json) lines into main-results.json" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """ | ||
| Merge the per-module summaries produced by ``summarize-test-results.py`` for | ||
| several test shards into a single summary. | ||
|
|
||
| Each shard runs a disjoint subset of the suite (see the ``--num-shards`` | ||
| sharding in ``pandas-testing-plugin.py``), so the combined result for a module | ||
| is obtained by summing every numeric field (test counts and the GPU/CPU | ||
| function-call counters) across the shards. The same module may appear in more | ||
| than one shard because sharding is per test, not per module. | ||
|
|
||
| Examples | ||
| -------- | ||
| python merge-results.py shard-0/pr-results.json shard-1/pr-results.json > pr-results.json | ||
| """ | ||
|
|
||
| import json | ||
| import sys | ||
|
|
||
| merged: dict[str, dict] = {} | ||
| for path in sys.argv[1:]: | ||
| with open(path) as f: | ||
| results = json.load(f) | ||
| for module_name, row in results.items(): | ||
| combined = merged.setdefault(module_name, {}) | ||
| for key, value in row.items(): | ||
| if isinstance(value, bool): | ||
| # No boolean fields are expected; keep the first seen value. | ||
| combined.setdefault(key, value) | ||
| elif isinstance(value, (int, float)): | ||
| combined[key] = combined.get(key, 0) + value | ||
| else: | ||
| combined.setdefault(key, value) | ||
|
|
||
| print(json.dumps(merged, indent=4)) | ||
|
There was a problem hiding this comment. 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift Add coverage for the shard aggregation contract. Add unit tests and a unit benchmark for the new result-processing behavior.
As per coding guidelines, “Add unit tests and unit benchmarks.” 🧰 Tools🪛 ast-grep (0.45.2)[warning] 23-23: File path is request-/variable-derived; validate and normalize to prevent path traversal. (open-filename-from-request) [info] 36-36: use jsonify instead of json.dumps for JSON output (use-jsonify) 📍 Affects 2 files
🤖 Prompt for AI AgentsSource: Coding guidelines
Contributor
Author
There was a problem hiding this comment. Added in 691447c — There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== Commit ==='
git show --stat --oneline --decorate --no-renames 691447c5c1 || true
echo '=== merge-results outline ==='
ast-grep outline ci/cudf_pandas_scripts/pandas-tests/merge-results.py --items all || true
echo '=== merge-results implementation ==='
cat -n ci/cudf_pandas_scripts/pandas-tests/merge-results.py
echo '=== merge-results test file ==='
test_file="python/cudf/cudf_pandas_tests/test_pandas_tests_merge_results.py"
if [[ -f "$test_file" ]]; then
cat -n "$test_file"
else
echo "Missing: $test_file"
fi
echo '=== summary.sh test references ==='
rg -n -C 3 --glob '*.py' --glob '*.sh' \
'(summary\.sh|merge_shard_results|pandas-tests-summary|gh\s)' \
python ci tests 2>/dev/null || trueLength of output: 143542
The combined finding also requests tests for You are interacting with an AI system. |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| #!/usr/bin/env bash | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| # Aggregate the results of the sharded pandas-tests PR jobs. | ||
| # | ||
| # Each shard uploads its partial per-module summary as the GitHub artifact | ||
| # "pandas-test-pr-results-<shard_id>". This job downloads them all, merges them | ||
| # into a single pr-results.json, and posts the diff against the latest nightly | ||
| # ("main") results to the job summary. | ||
| # | ||
| # Usage: | ||
| # summary.sh <num_shards> | ||
| # | ||
| # This step is purely informational, so it never fails the workflow; the | ||
| # pass/fail signal for the suite comes from the individual shard jobs. | ||
|
|
||
| # No `set -e`: this step is best effort and must never fail the workflow, so | ||
| # every fallible command is guarded explicitly and the script always exits 0. | ||
| set -uo pipefail | ||
|
|
||
| source rapids-init-pip | ||
|
|
||
| NUM_SHARDS=${1:?usage: summary.sh <num_shards>} | ||
| RAPIDS_FULL_VERSION=$(<./VERSION) | ||
|
|
||
| rapids-logger "Aggregating pandas-tests results from ${NUM_SHARDS} shards" | ||
|
|
||
| # job-summary.py renders markdown tables with pandas; tabulate backs to_markdown. | ||
| if ! rapids-pip-retry install pandas tabulate; then | ||
| rapids-logger "Could not install summary dependencies; skipping summary." | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Download each shard's partial results from the current run. Every shard | ||
| # must be present for the merged total to mean anything; see the check below. | ||
| for ((shard = 0; shard < NUM_SHARDS; shard++)); do | ||
| if ! gh run download "${GITHUB_RUN_ID}" \ | ||
| --repo "${GITHUB_REPOSITORY}" \ | ||
| --name "pandas-test-pr-results-${shard}" \ | ||
| --dir "shard-${shard}"; then | ||
| rapids-logger "Could not download results for shard ${shard}; skipping it." | ||
| fi | ||
| done | ||
|
|
||
| shopt -s nullglob | ||
| SHARD_RESULTS=(shard-*/pr-results.json) | ||
| if [[ ${#SHARD_RESULTS[@]} -ne ${NUM_SHARDS} ]]; then | ||
| # A shard that fails never reaches its upload step, so its results are | ||
| # simply absent. Summarizing anyway would diff part of this PR's suite | ||
| # against the whole nightly baseline and report every test in the missing | ||
| # shard as removed, which is worse than printing nothing at all. | ||
| rapids-logger "Only ${#SHARD_RESULTS[@]} of ${NUM_SHARDS} shard results are available; skipping the summary rather than reporting a partial diff." | ||
| exit 0 | ||
| fi | ||
|
|
||
| rapids-logger "Merging ${#SHARD_RESULTS[@]} shard result file(s)" | ||
| if ! python ci/cudf_pandas_scripts/pandas-tests/merge-results.py \ | ||
|
Contributor
There was a problem hiding this comment. I'm glad we're at least reusing
Contributor
Author
There was a problem hiding this comment. Shared it in 8fa52ae — the download-and-merge moved to |
||
| "${SHARD_RESULTS[@]}" > pr-results.json; then | ||
| rapids-logger "Failed to merge shard results; skipping summary." | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Fetch the latest successful nightly results to diff against. | ||
| MAIN_RUN_ID=$( | ||
| gh run list \ | ||
| -w "Pandas Test Job" \ | ||
| -b "$(<./RAPIDS_BRANCH)" \ | ||
| --repo 'NVIDIA/cudf' \ | ||
| --status success \ | ||
| --limit 7 \ | ||
| --json 'createdAt,databaseId' \ | ||
| --jq 'sort_by(.createdAt) | reverse | .[0] | .databaseId' || true | ||
| ) | ||
|
|
||
| if [[ -z "${MAIN_RUN_ID}" ]]; then | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| rapids-logger "No nightly main results found; skipping diff." | ||
| exit 0 | ||
| fi | ||
|
|
||
| rapids-logger "Fetching latest available results from nightly: ${MAIN_RUN_ID}" | ||
| if ! gh run download \ | ||
| --repo 'NVIDIA/cudf' \ | ||
| --name main-results.json \ | ||
| "${MAIN_RUN_ID}"; then | ||
| rapids-logger "Could not download nightly results; skipping diff." | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Compute the diff and prepare the job summary (best effort). | ||
| if ! python ci/cudf_pandas_scripts/pandas-tests/job-summary.py \ | ||
| main-results.json pr-results.json "${RAPIDS_FULL_VERSION}" >> "$GITHUB_STEP_SUMMARY"; then | ||
| rapids-logger "Failed to render the job summary." | ||
| fi | ||
|
|
||
| exit 0 | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In theory I think we could set
N: 8in the top-levelenv, then use a bash step that generatesseq 0 N-1for theshard_id. I don't know if that's worth the extra effort though, so I'll defer to you if you want to try that.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Went ahead and did this in 8fa52ae — a
pandas-tests-shardsjob now emits both the matrix and thenum-shardsargument from a singleNUM_SHARDS, so the count only lives in one place.