Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/nvidia-4090.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

permissions:
contents: read
pull-requests: write

on:
pull_request:
branches: [ '*' ]
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/nvidia-a100.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

permissions:
contents: read
pull-requests: write

on:
pull_request:
branches: [ '*' ]
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/nvidia-h100.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

permissions:
contents: read
pull-requests: write

on:
pull_request:
branches: [ '*' ]
Expand Down
111 changes: 111 additions & 0 deletions .github/workflows/reusable-ci-benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,11 @@ jobs:
runs-on: ${{ inputs.runner }}
env:
FLA_CI_ENV: 1
# Longer Triton timing windows + pause between commits reduce clock/thermal variance.
FLA_BENCH_WARMUP_MS: 40
FLA_BENCH_REP_MS: 200
FLA_BENCH_OP_WARMUP_ITERS: 6
FLA_BENCH_COOLDOWN_SEC: 3

steps:
- name: Check out repo (full history for cross-commit comparison)
Expand Down Expand Up @@ -112,12 +117,14 @@ jobs:

- name: Run benchmark comparison
if: steps.check_skip.outputs.skip_bench == 'false'
id: benchmark
shell: bash
run: |
$CONDA_BIN_PATH/python scripts/run_benchmark_compare.py \
--base ${{ inputs.base_ref }} \
--head HEAD \
--threshold ${{ inputs.threshold }} \
--no-fail-on-regression \
--output benchmark_results.json

- name: Upload benchmark results
Expand All @@ -128,3 +135,107 @@ jobs:
path: benchmark_results.json
if-no-files-found: ignore
retention-days: 30

- name: Post benchmark results to PR
if: steps.check_skip.outputs.skip_bench == 'false' && github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const path = 'benchmark_results.json';

if (!fs.existsSync(path)) {
console.log('No benchmark results file found');
return;
}

const data = JSON.parse(fs.readFileSync(path, 'utf8'));
const { base_sha, head_sha, machine_info, regressions, speedups, has_regression, has_speedup } = data;

const gpuName = machine_info?.gpu_name || 'Unknown';
const cudaVersion = machine_info?.cuda_version || 'Unknown';
const pytorchVersion = machine_info?.pytorch_version || 'Unknown';
const runnerName = '${{ inputs.runner }}';

// Format the summary table
let summary = '';
if (has_regression || has_speedup) {
summary = '| Op | Mode | B | T | H | D | Base (ms) | Head (ms) | Change |\n';
summary += '|:---|:---:|---:|---:|---:|---:|---:|---:|---:|\n';

// Add regressions
if (regressions && regressions.length > 0) {
for (const r of regressions) {
const change = `+${r.change_pct.toFixed(1)}% 🔴`;
summary += `| ${r.op} | ${r.mode} | ${r.B} | ${r.T} | ${r.H} | ${r.D} | ${r.base_ms.toFixed(3)} | ${r.head_ms.toFixed(3)} | ${change} |\n`;
}
}

// Add speedups
if (speedups && speedups.length > 0) {
for (const s of speedups) {
const change = `${s.change_pct.toFixed(1)}% 🟢`;
summary += `| ${s.op} | ${s.mode} | ${s.B} | ${s.T} | ${s.H} | ${s.D} | ${s.base_ms.toFixed(3)} | ${s.head_ms.toFixed(3)} | ${change} |\n`;
}
}
}

// Build the comment body
const statusEmoji = has_regression ? '⚠️' : '✅';
const statusText = has_regression
? `${regressions.length} regression(s) detected`
: 'No significant performance regressions detected';

let body = `## ${statusEmoji} Benchmark Results (${runnerName.toUpperCase()})\n\n`;
body += `**Status:** ${statusText}\n\n`;
body += `| | |\n`;
body += `|:---|:---|\n`;
body += `| **GPU** | ${gpuName} |\n`;
body += `| **CUDA** | ${cudaVersion} |\n`;
body += `| **PyTorch** | ${pytorchVersion} |\n`;
body += `| **Base** | \`${base_sha}\` |\n`;
body += `| **Head** | \`${head_sha}\` |\n`;
body += `| **Threshold** | ${{ inputs.threshold }}% |\n\n`;

if (has_regression || has_speedup) {
body += `<details>\n<summary>📊 View Details (${(regressions?.length || 0) + (speedups?.length || 0)} significant changes)</summary>\n\n`;
body += summary;
body += '\n</details>\n';
} else {
body += '> All benchmarked operations are within the performance threshold.\n';
}

body += '\n---\n';
body += '*This comment is automatically updated with the latest benchmark results.*';

// Find existing comment
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});

const botComment = comments.find(comment =>
comment.user.type === 'Bot' &&
comment.body.includes(`Benchmark Results (${runnerName.toUpperCase()})`)
);
Comment on lines +212 to +221
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

# First, let's check if the file exists and read the specific lines
head -n 220 .github/workflows/reusable-ci-benchmarks.yml | tail -n 20

Repository: fla-org/flash-linear-attention

Length of output: 811


🌐 Web query:

GitHub API issues listComments pagination default per_page documentation

💡 Result:

The GitHub REST API endpoint for listing comments on an issue (GET /repos/{owner}/{repo}/issues/{issue_number}/comments), often referred to as "listComments" or "list comments on an issue", supports pagination via the standard per_page and page query parameters. The default value for per_page is 30 (maximum 100). This is explicitly documented in the official GitHub REST API documentation for issue comments endpoints. Pagination uses Link headers in responses for navigating pages, as described in the general "Using pagination in the REST API" guide, which notes that endpoints like issues return 30 items by default (e.g., GET /repos/octocat/Spoon-Knife/issues). Example request: GET https://api.github.com/repos/OWNER/REPO/issues/ISSUE_NUMBER/comments?per_page=100&page=1

Citations:


Paginate the comment lookup before deciding to create a new one.

issues.listComments returns 30 items by default. If the benchmark comment appears beyond the first page, this code fails to find it and creates duplicates instead of updating the existing comment.

💡 Suggested fix
-            const { data: comments } = await github.rest.issues.listComments({
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              issue_number: context.issue.number,
-            });
+            const comments = await github.paginate(github.rest.issues.listComments, {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.issue.number,
+              per_page: 100,
+            });
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/reusable-ci-benchmarks.yml around lines 207 - 216, The
lookup for an existing benchmark comment uses github.rest.issues.listComments
which only returns the first page (30 comments), causing duplicates; replace the
single-page call with a paginated fetch (e.g., use github.paginate or iterate
pages of github.rest.issues.listComments with per_page=100) to retrieve all
comments, then search that full list for the bot comment (the logic that assigns
botComment). Keep the same matching criteria (comment.user.type === 'Bot' &&
comment.body.includes(`Benchmark Results (${runnerName.toUpperCase()})`)) and
then proceed to update or create the comment as before.


if (botComment) {
// Update existing comment
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: botComment.id,
body: body,
});
console.log(`Updated existing benchmark comment for ${runnerName}`);
} else {
// Create new comment
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: body,
});
console.log(`Created new benchmark comment for ${runnerName}`);
}
Loading
Loading