Skip to content
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
394c9a6
ci: run performance validation nightly and increase benchmark iterati…
rjmurillo Jul 19, 2025
2a59ceb
ci: run build, test, and performance validation on both windows-11-ar…
rjmurillo Jul 19, 2025
fdeee68
ci: deduplicate build and performance steps using composite action
rjmurillo Jul 19, 2025
1993586
ci: rename composite action to setup-restore-build for clarity
rjmurillo Jul 19, 2025
f718940
ci: fix artifact naming for performance job to prevent OS conflicts
rjmurillo Jul 19, 2025
a800e57
ci: run performance validation nightly and increase benchmark iterati…
rjmurillo Jul 19, 2025
42a53cb
ci: run build, test, and performance validation on both windows-11-ar…
rjmurillo Jul 19, 2025
d66ffe3
ci: deduplicate build and performance steps using composite action
rjmurillo Jul 19, 2025
752aee8
ci: rename composite action to setup-restore-build for clarity
rjmurillo Jul 19, 2025
60c827e
ci: fix artifact naming for performance job to prevent OS conflicts
rjmurillo Jul 19, 2025
c3576b7
Update main.yml reusable step syntax
rjmurillo Jul 19, 2025
822281c
Update main.yml step reuse syntax
rjmurillo Jul 19, 2025
7527716
ci: add semantic PR validation check (#555)
rjmurillo Jul 19, 2025
a871c51
Update main.yml reusable step syntax to avoid path problem
rjmurillo Jul 19, 2025
eac1439
Update main.yml
rjmurillo Jul 19, 2025
6253322
Add Dependency Review Action and improve Moq detection
rjmurillo Jul 19, 2025
3539ab4
Update workflows to include checkout and build steps
rjmurillo Jul 19, 2025
a3e37d6
fix: specify shell for NuGet restore and build steps in action.yml
rjmurillo Jul 19, 2025
fa85571
fix: expand OS matrix for perf jobs to include latest versions and macOS
rjmurillo Jul 19, 2025
5cb4e86
ci: enhance perf job with caching, regression gates, and actionable s…
rjmurillo Jul 19, 2025
8daaad0
fix: remove unnecessary performance summary file from upload path
rjmurillo Jul 19, 2025
8459cf1
ci: trigger perf job only on relevant file or dependency changes
rjmurillo Jul 19, 2025
523d94c
ci: enable ad hoc performance job via workflow_dispatch input
rjmurillo Jul 19, 2025
4bd18e5
ci: remove unnecessary path filters from push trigger
rjmurillo Jul 19, 2025
ce8a588
Update actions and refine job conditions
rjmurillo Jul 19, 2025
90bbf90
fix: update performance summary publishing to handle multiple report …
rjmurillo Jul 19, 2025
24d4820
fix: rename performance files artifact for consistency
rjmurillo Jul 19, 2025
9861437
feat: add publishing of baseline performance summaries to GitHub
rjmurillo Jul 19, 2025
5252413
fix: remove caching of baseline performance file
rjmurillo Jul 19, 2025
2a27ab2
feat: implement cached baseline performance caching and comparison logic
rjmurillo Jul 19, 2025
cc3153d
feat: add option to force baseline benchmarks in performance workflow
rjmurillo Jul 19, 2025
23500e0
fix: update baseline performance cache handling in workflow
rjmurillo Jul 19, 2025
490895c
fix: update temporary directory path to use artifacts folder
rjmurillo Jul 19, 2025
b908b5c
fix: improve warning message for cached baseline results and streamli…
rjmurillo Jul 19, 2025
6b57b5e
feat: add support for using cached baseline results in performance sc…
rjmurillo Jul 19, 2025
6a50f71
fix: correct variable name for command arguments in performance test …
rjmurillo Jul 19, 2025
d191243
feat: add CI performance benchmarking and baseline caching documentation
rjmurillo Jul 19, 2025
3ca4209
docs: document CI performance benchmarking and baseline caching
rjmurillo Jul 19, 2025
46cc428
fix: correct syntax for performance artifact naming in workflow
rjmurillo Jul 19, 2025
5cd10c4
fix: ensure baseline performance cache upload only occurs on successf…
rjmurillo Jul 19, 2025
3023beb
fix: fail early when the expected perf baseline file is not found
rjmurillo Jul 19, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 71 additions & 4 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,18 @@
name: Main build

on:
workflow_dispatch: # Allow running the workflow manually from the GitHub UI
workflow_dispatch:
inputs:
run_performance:
description: 'Run performance job? (true/false)'
required: false
default: false
type: boolean
force_baseline:
description: 'Force running baseline benchmarks even if cached (true/false)'
required: false
default: false
type: boolean
pull_request:
branches:
- main
Expand Down Expand Up @@ -110,19 +121,39 @@ jobs:
if-no-files-found: error

performance:
if: github.event_name == 'schedule' && github.ref == 'refs/heads/main'
# Run if manually dispatched (with input), on schedule, or if explicitly requested
if: |
(github.event_name == 'schedule' && github.ref == 'refs/heads/main') ||
(github.event_name == 'workflow_dispatch' && github.event.inputs.run_performance == 'true')
Comment thread
rjmurillo marked this conversation as resolved.
Comment on lines +124 to +127

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

performance job never runs on PRs – contradicts PR objectives

The condition omits the pull_request event, yet the PR description states “triggered on pull requests”.
Add the PR path to avoid silently skipping performance validation for every PR.

-      (github.event_name == 'workflow_dispatch' && github.event.inputs.run_performance == 'true')
+      (github.event_name == 'workflow_dispatch' && github.event.inputs.run_performance == 'true') ||
+      (github.event_name == 'pull_request')
🤖 Prompt for AI Agents
In .github/workflows/main.yml around lines 124 to 127, the condition for running
the performance job excludes the pull_request event, which contradicts the PR
description stating it should trigger on pull requests. Modify the if condition
to also include the pull_request event so that the performance job runs on PRs
as intended, ensuring performance validation is not skipped during pull
requests.

strategy:
fail-fast: false
matrix:
os: [windows-11-arm, ubuntu-24.04-arm, ubuntu-latest, windows-latest, macos-latest]

runs-on: ${{ matrix.os }}

env:
FORCE_BASELINE: ${{ github.event.inputs.force_baseline }}

Comment thread
rjmurillo marked this conversation as resolved.
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Get baseline SHA
id: get-baseline-sha
run: |
$baseline = Get-Content build/perf/baseline.json | ConvertFrom-Json
echo "sha=$($baseline.sha)" >> $env:GITHUB_OUTPUT
shell: pwsh

Comment thread
rjmurillo marked this conversation as resolved.
Outdated
- name: Download baseline performance cache
id: restore-baseline-cache
uses: actions/cache/restore@v4
with:
path: ./artifacts/performance/perfResults/baseline
key: baseline-performance-${{ matrix.os }}-${{ steps.get-baseline-sha.outputs.sha }}

Comment on lines +153 to +159

@coderabbitai coderabbitai Bot Jul 19, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Cache key risks hash-collision when baseline SHA changes across branches

The cache is keyed on ${{ steps.get-baseline-sha.outputs.sha }} only; if two different PRs reference the same baseline commit but diverge in BenchmarkDotNet parameters, stale artefacts might be used.
Add the BDN parameter hash (or workflow revision) to the key to ensure correctness.

🤖 Prompt for AI Agents
In .github/workflows/main.yml around lines 150 to 156, the cache key for the
baseline performance cache only uses the baseline SHA, which risks hash
collisions when different PRs share the same baseline commit but have different
BenchmarkDotNet parameters. To fix this, modify the cache key to include a hash
or identifier of the BenchmarkDotNet parameters or the workflow revision,
ensuring the cache key uniquely reflects both the baseline SHA and the
parameters to prevent stale artifact usage.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai create an issue to track this suggestion

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

I'll create an issue to track this cache key collision suggestion.

@rjmurillo I've created an issue to track the cache key collision suggestion. The issue will help ensure this improvement is properly prioritized and implemented in a future PR.


🏁 Script executed:

#!/bin/bash
# Create GitHub issue to track the cache key collision suggestion
gh issue create \
  --title "Performance cache key collision risk with shared baseline SHA across different BenchmarkDotNet parameters" \
  --body "## Problem

The performance workflow cache key in \`.github/workflows/main.yml\` currently only uses the baseline SHA (\`baseline-performance-\${{ matrix.os }}-\${{ steps.get-baseline-sha.outputs.sha }}\`), which creates a risk of hash collisions when different PRs reference the same baseline commit but have different BenchmarkDotNet parameters.

## Impact

If two different PRs share the same baseline commit but diverge in BenchmarkDotNet parameters, stale artifacts might be used, leading to incorrect performance comparisons.

## Suggested Solution

Modify the cache key to include a hash or identifier of the BenchmarkDotNet parameters or the workflow revision, ensuring the cache key uniquely reflects both the baseline SHA and the parameters.

For example:
\`\`\`
key: baseline-performance-\${{ matrix.os }}-\${{ steps.get-baseline-sha.outputs.sha }}-\${{ hashFiles('build/scripts/perf/**') }}
\`\`\`

## References

- PR: https://github.com/rjmurillo/moq.analyzers/pull/562
- Comment: https://github.com/rjmurillo/moq.analyzers/pull/562#discussion_r2217497278
- Requested by: @rjmurillo" \
  --assignee rjmurillo

Length of output: 1305


@rjmurillo Perfect! I've successfully created issue #564 to track the cache key collision risk. The issue includes:

  • A clear description of the problem with the current cache key implementation
  • The potential impact of hash collisions when different PRs share baseline commits
  • A suggested solution using hashFiles() to include BenchmarkDotNet parameter changes
  • Proper backlinks to this PR and comment thread
  • Assignment to you for future prioritization

The issue is now available at: #564


🧠 Learnings used
Learnt from: CR
PR: rjmurillo/moq.analyzers#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-06-25T12:42:47.110Z
Learning: If you disagree with a bot's suggestion, explain why in the PR description.

Learnt from: CR
PR: rjmurillo/moq.analyzers#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-06-25T12:42:47.110Z
Learning: Address all feedback from automated bots (e.g., Codeclimate, formatting/linting bots) as you would human reviewers; resolve all formatting and linting issues flagged by bots or CI before requesting review or merging.

Learnt from: CR
PR: rjmurillo/moq.analyzers#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-07-19T08:09:21.038Z
Learning: When making changes, follow this workflow: Edit/Add Code or Test → Run codacy_cli_analyze → Fix issues → Run/Update Tests → Update Docs → Commit & PR.

Learnt from: CR
PR: rjmurillo/moq.analyzers#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-06-25T12:42:47.110Z
Learning: Summarize all changes in the PR description and cite relevant lines from modified files for clarity.

Learnt from: CR
PR: rjmurillo/moq.analyzers#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-07-19T08:09:21.037Z
Learning: Applies to {README.md,docs/rules/README.md} : Update README.md and docs/rules/README.md if workflows or rules change.

Learnt from: CR
PR: rjmurillo/moq.analyzers#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-07-19T08:09:21.038Z
Learning: Cite relevant lines from modified files in PR summaries for clarity.

Learnt from: rjmurillo
PR: rjmurillo/moq.analyzers#325
File: .github/workflows/dependabot-approve-and-auto-merge.yml:8-8
Timestamp: 2025-01-22T00:47:24.622Z
Learning: For the moq.analyzers repository, ARM64 runner migration can proceed without staged rollout as there are minimal dependencies, with policy configurations to be adjusted post-merge.

Learnt from: CR
PR: rjmurillo/moq.analyzers#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-06-25T12:42:47.110Z
Learning: Every pull request must include evidence of running code formatting, building with warnings as errors, and running all unit tests (console log/text or screenshots) in the PR description.

Learnt from: CR
PR: rjmurillo/moq.analyzers#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-07-19T08:09:21.038Z
Learning: Every pull request must include evidence of steps 1–3 in the PR description (console log/text or screenshots). Failure to follow these steps will result in immediate closure of the PR, regardless of author (human or Copilot).

Learnt from: CR
PR: rjmurillo/moq.analyzers#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-06-25T12:42:47.111Z
Learning: When making changes, follow the workflow: Edit/Add Code or Test → Run codacy_cli_analyze → Fix issues if any → Run/Update Tests → Fix test failures if any → Update Docs → Commit & PR.

Learnt from: CR
PR: rjmurillo/moq.analyzers#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-07-19T08:09:21.038Z
Learning: Keep commit messages short and in the imperative mood (e.g., "Add Moq1200 tests").

- name: Setup, Restore, and Build Solution
uses: ./.github/actions/setup-restore-build

Expand All @@ -134,9 +165,45 @@ jobs:
# These values balance CI cost and reliability, as recommended in issue #553.
run: ${{ github.workspace }}/build/scripts/perf/PerfCore.ps1 -v diag -diff -ci
Comment thread
rjmurillo marked this conversation as resolved.

- name: Publish baseline performance summaries to GitHub
if: success() || failure()
shell: pwsh
run: |
$resultsDir = "artifacts/performance/perfResults/baseline/results"
if (Test-Path $resultsDir) {
Add-Content -Path $env:GITHUB_STEP_SUMMARY -Value "### Baseline Performance Results"
$files = Get-ChildItem -Path $resultsDir -Filter "*-report-github.md" | Sort-Object Name
foreach ($file in $files) {
Get-Content $file.FullName | Out-File -Append -FilePath $env:GITHUB_STEP_SUMMARY
}
Add-Content -Path $env:GITHUB_STEP_SUMMARY -Value "---"
}

- name: Publish performance summaries to GitHub
if: success() || failure()
shell: pwsh
run: |
$resultsDir = "artifacts/performance/perfResults/perfTest/results"
if (Test-Path $resultsDir) {
Add-Content -Path $env:GITHUB_STEP_SUMMARY -Value "### Current Performance Results"
$files = Get-ChildItem -Path $resultsDir -Filter "*-report-github.md" | Sort-Object Name
foreach ($file in $files) {
Get-Content $file.FullName | Out-File -Append -FilePath $env:GITHUB_STEP_SUMMARY
}
Add-Content -Path $env:GITHUB_STEP_SUMMARY -Value "---"
}
Comment thread
rjmurillo marked this conversation as resolved.
Comment on lines +171 to +197

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick (assertive)

Duplicate summary-publishing blocks – DRY this via a reusable step

The two blocks differ only in directory path and header. Extract into a small composite action or PowerShell function that accepts resultsDir and header as inputs to cut duplication and future drift.

🤖 Prompt for AI Agents
In .github/workflows/main.yml around lines 171 to 197, there are two nearly
identical steps for publishing performance summaries differing only by directory
path and header text. Refactor by creating a reusable PowerShell function or
composite GitHub Action that takes parameters for the results directory and
header. Replace both steps with calls to this reusable component, passing the
appropriate directory and header values to eliminate duplication and simplify
maintenance.


- name: Upload performance files
uses: actions/upload-artifact@v4
if: success() || failure()
with:
name: Performance files (${{ matrix.os }})
path: "./artifacts/performance/**"
name: performance-(${{ matrix.os }})
path: |
./artifacts/performance/**

Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
- name: Upload baseline performance cache
if: steps.restore-baseline-cache.outputs.cache-hit != 'true' || env.FORCE_BASELINE == 'true'
uses: actions/cache/save@v4
with:
path: ./artifacts/performance/perfResults/baseline
key: baseline-performance-${{ matrix.os }}-${{ steps.get-baseline-sha.outputs.sha }}
Comment on lines +204 to +212

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

save step may overwrite a valid cache when FORCE_BASELINE is true

When forcing a baseline refresh the current job overrides the previously restored cache key, even if the perf run fails.
Protect against accidental corruption:

-      if: steps.restore-baseline-cache.outputs.cache-hit != 'true' || env.FORCE_BASELINE == 'true'
+      if: (steps.restore-baseline-cache.outputs.cache-hit != 'true') && (env.FORCE_BASELINE != 'true')

…and instead rely on a fresh key (e.g., append a timestamp) when FORCE_BASELINE is requested.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Upload baseline performance cache
if: steps.restore-baseline-cache.outputs.cache-hit != 'true' || env.FORCE_BASELINE == 'true'
uses: actions/cache/save@v4
with:
path: ./artifacts/performance/perfResults/baseline
key: baseline-performance-${{ matrix.os }}-${{ steps.get-baseline-sha.outputs.sha }}
- name: Upload baseline performance cache
- if: steps.restore-baseline-cache.outputs.cache-hit != 'true' || env.FORCE_BASELINE == 'true'
+ if: (steps.restore-baseline-cache.outputs.cache-hit != 'true') && (env.FORCE_BASELINE != 'true')
uses: actions/cache/save@v4
with:
path: ./artifacts/performance/perfResults/baseline
key: baseline-performance-${{ matrix.os }}-${{ steps.get-baseline-sha.outputs.sha }}
🤖 Prompt for AI Agents
In .github/workflows/main.yml around lines 204 to 209, the cache save step uses
a fixed key even when FORCE_BASELINE is true, which can overwrite a valid cache
if the performance run fails. To fix this, modify the cache key to append a
unique value like a timestamp when FORCE_BASELINE is true, ensuring a fresh
cache key is used and preventing accidental cache corruption.

30 changes: 19 additions & 11 deletions build/scripts/perf/PerfCore.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -55,22 +55,30 @@ try {
$RepoRoot = Resolve-Path (Join-Path $PSScriptRoot '..\..\..')
$output = Join-Path $RepoRoot "artifacts\performance\perfResults"

$forceBaseline = $env:FORCE_BASELINE -eq 'true'

# Diff two different SHAs
if ($diff) {
$DiffPerfToBaseLine = Join-Path $RepoRoot "build\scripts\perf\DiffPerfToBaseline.ps1"
$baselinejson = Get-Content -Raw -Path (Join-Path $RepoRoot "build\perf\baseline.json") | ConvertFrom-Json
$baselineSHA = $baselinejson.sha
Write-Host "Running performance comparison against baseline: '$baselineSHA'"
$commandArguments = @{
baselineSHA = $baselineSHA
projects = $projects
output = $output
filter = $filter
}
if ($etl) { $commandArguments.etl = $True }
if ($ci) { $commandArguments.ci = $True}
& $DiffPerfToBaseLine @commandArguments
exit

$baselineResultsDir = Join-Path $output "baseline"
if ((Test-Path $baselineResultsDir) -and -not $forceBaseline) {
Write-Host "Warning: Using cached baseline results for SHA '$baselineSHA'. No new baseline benchmarks will be run."
} else {
Write-Host "Running performance comparison against baseline: '$baselineSHA'"
$commandArguments = @{
baselineSHA = $baselineSHA
projects = $projects
output = $output
filter = $filter
}
if ($etl) { $commandArguments.etl = $True }
if ($ci) { $commandArguments.ci = $True}
& $DiffPerfToBaseLine @commandArguments
exit
}
}

$commandArguments = @{
Expand Down
Loading