Add failureLabels and Jira notifications to fips qualifier - #83298
Conversation
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
openshift-merge-bot
Bot
added
the
rehearsals-ack
WalkthroughThe ChangesQualifier failure escalation
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 inconclusive)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core-services/release-controller/release-qualifiers.yaml`:
- Around line 150-155: Update both escalation rules in the alert configuration
by adding an overLastRuns value above each failures threshold, with each value
greater than its corresponding failures count. Preserve the existing overPeriod
settings, and verify behavior for non-consecutive failures and failures outside
each configured time window.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: b130566c-123a-4332-bb16-8edbbf44693a
📒 Files selected for processing (1)
core-services/release-controller/release-qualifiers.yaml
| failures: 4 | ||
| overPeriod: "24h" | ||
| priority: "Normal" | ||
| - name: "alert" | ||
| failures: 8 | ||
| overPeriod: "72h" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n -C 8 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|TimeBoundedFailures' \
--glob '*.go' --glob '*.yaml' .
rg -n -C 4 'failures: 4|failures: 8|overLastRuns|overPeriod' \
core-services/release-controller/release-qualifiers.yamlRepository: openshift/release
Length of output: 31181
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- repository files ---'
git ls-files | rg '(^|/)(release-qualifiers\.yaml|types\.go|.*qualif.*\.go)$' || true
printf '%s\n' '--- local configuration and nearby documentation ---'
sed -n '1,180p' core-services/release-controller/release-qualifiers.yaml
printf '%s\n' '--- upstream evaluator source ---'
for path in \
pkg/releasequalifiers/notifications/jira/types.go \
pkg/releasequalifiers/notifications/jira/jira.go \
pkg/releasequalifiers/notifications/jira/*.go \
pkg/releasequalifiers/*.go
do
url="https://raw.githubusercontent.com/openshift/release-controller/master/$path"
printf '\n### %s\n' "$url"
curl -fsSL "$url" 2>/dev/null | rg -n -C 8 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Failures|failure' || true
doneRepository: openshift/release
Length of output: 10594
🌐 Web query:
openshift release-controller release qualifiers OverPeriod OverLastRuns evaluator failures source
💡 Result:
In the OpenShift release-controller, "release qualifiers" refer to a configuration-driven mechanism used to implement stricter validation of release candidates beyond basic job success or failure [1]. These qualifiers allow the controller to reject or qualify releases based on aggregated failure data and specific failure patterns [1]. The parameters OverPeriod and OverLastRuns are associated with the evaluator logic for these qualifiers. They define the time-based or run-based scope for analyzing job failures [2][1]: 1. OverPeriod: Specifies a time duration (e.g., a rolling window) over which the evaluator aggregates failure statistics to determine if a release should be qualified or rejected. 2. OverLastRuns: Specifies a count of recent Prow job executions (runs) over which the failure rate or failure patterns are evaluated. This evaluation logic is designed to prevent "flaky" or consistently failing releases from progressing by examining performance trends rather than relying solely on the status of a single job execution [3][4]. The release-controller's source code for these features is located in the official openshift/release-controller repository [5]. Specifically, configuration parsing and command-line flags related to release qualifiers are defined in cmd/release-controller/main.go [2], while the runtime logic for processing release tags, audit failures, and synchronization is handled in components such as cmd/release-controller/audit.go and cmd/release-controller/sync_tags.go [6][7]. Technical note: When debugging these failures, developers often review the release payload metadata and the status of blocking/informing jobs tracked by the controller [3][4]. Information regarding specific infrastructure-level failures versus product bugs is often analyzed by examining Prow job logs and the release-controller's own audit logs [8][7].
Citations:
- 1: https://myteam.exceeds.ai/profile/bradmwilliams
- 2: https://github.com/openshift/release-controller/blob/b485b5997bf1/cmd/release-controller/main.go
- 3: https://github.com/openshift-eng/edge-tooling/blob/main/payload-monitor/payload_monitor/collectors/release_controller.py
- 4: https://raw.githubusercontent.com/openshift/release-controller/master/artifacts/release.openshift.io_releasepayloads.yaml
- 5: https://github.com/openshift/release-controller
- 6: https://github.com/openshift/release-controller/blob/b485b5997bf1/cmd/release-controller/sync_tags.go
- 7: https://github.com/openshift/release-controller/blob/b485b5997bf1/cmd/release-controller/audit.go
- 8: https://github.com/openshift-eng/ai-helpers/blob/main/plugins/ci/skills/prow-job-analysis/references/ci-infrastructure-changes.md
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
tree="$(mktemp)"
curl -fsSL 'https://api.github.com/repos/openshift/release-controller/git/trees/master?recursive=1' >"$tree"
printf '%s\n' '--- qualifier-related upstream paths ---'
jq -r '.tree[] | select(.path | test("qualif|jira|notification"; "i")) | .path' "$tree"
printf '%s\n' '--- references in upstream source ---'
paths="$(jq -r '.tree[] | select(.type == "blob" and (.path | test("\\.(go|yaml|yml)$"))) | .path' "$tree")"
while IFS= read -r path; do
content="$(curl -fsSL "https://raw.githubusercontent.com/openshift/release-controller/master/$path" 2>/dev/null || true)"
if printf '%s' "$content" | rg -q 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation'; then
printf '\n### %s\n' "$path"
printf '%s' "$content" | rg -n -C 12 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation'
fi
done <<<"$paths"
rm -f "$tree"Repository: openshift/release
Length of output: 206
🌐 Web query:
site:github.com/openshift/release-controller "OverLastRuns" "OverPeriod"
💡 Result:
Based on the provided search results from the OpenShift release-controller repository, there is no evidence that OverLastRuns or OverPeriod are existing functions, configuration fields, or established parameters within the release-controller codebase. The repository code, including periodic.go, controller.go, and sync_upgrade.go, manages ProwJobs and release image streams using various synchronization loops and configuration structures, but these specific terms do not appear as recognized identifiers or configurable settings in the current implementation [1][2][3].
Citations:
- 1: https://github.com/openshift/release-controller/blob/b485b5997bf1/cmd/release-controller/periodic.go
- 2: https://github.com/openshift/release-controller/blob/b485b5997bf1/cmd/release-controller/controller.go
- 3: https://github.com/openshift/release-controller/blob/b485b5997bf1/cmd/release-controller/sync_upgrade.go
🌐 Web query:
site:github.com/openshift/release-controller/pkg/releasequalifiers "Evaluate" "OverPeriod"
💡 Result:
No exact indexed match was found for:
site:github.com/openshift/release-controller/pkg/releasequalifiers "Evaluate" "OverPeriod"
The releasequalifiers package is documented on Go.dev, but the indexed API does not show an Evaluate or OverPeriod symbol. (pkg.go.dev)
Citations:
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
tmpdir="$(mktemp -d)"
curl -fsSL 'https://codeload.github.com/openshift/release-controller/tar.gz/refs/heads/master' |
tar -xz -C "$tmpdir"
repo="$(find "$tmpdir" -mindepth 1 -maxdepth 1 -type d -name 'release-controller-*' -print -quit)"
printf '%s\n' '--- upstream references ---'
rg -n -C 10 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation|failures' \
"$repo" -g '*.go' -g '*.yaml' -g '*.yml' || true
printf '%s\n' '--- candidate evaluator files ---'
find "$repo" -type f \( -name '*.go' -o -name '*.yaml' -o -name '*.yml' \) -print |
rg 'qualif|jira|notification' || true
rm -rf "$tmpdir"Repository: openshift/release
Length of output: 313
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
tmpdir="$(mktemp -d)"
curl -fsSL 'https://codeload.github.com/openshift/release-controller/tar.gz/refs/heads/master' |
tar -xz -C "$tmpdir"
repo="$(find "$tmpdir" -mindepth 1 -maxdepth 1 -type d -name 'release-controller-*' -print -quit)"
printf '%s\n' '--- upstream package outline ---'
if command -v ast-grep >/dev/null 2>&1; then
ast-grep outline "$repo/pkg/releasequalifiers" || true
fi
printf '%s\n' '--- relevant Go declarations and call sites ---'
rg -n 'type Escalation|func .*Escal|OverLastRuns|OverPeriod|overLastRuns|overPeriod|Failures' \
"$repo/pkg" "$repo/cmd" -g '*.go' || true
rm -rf "$tmpdir"Repository: openshift/release
Length of output: 313
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- remote refs ---'
git ls-remote --heads --tags https://github.com/openshift/release-controller.git 2>&1 | head -40 || true
printf '%s\n' '--- raw endpoint status ---'
for path in \
README.md \
pkg/releasequalifiers/notifications/jira/types.go \
pkg/releasequalifiers/notifications/jira/evaluator.go \
pkg/releasequalifiers/notifications/jira/jira.go \
pkg/releasequalifiers/notifications/jira/notification.go \
pkg/releasequalifiers/evaluator.go \
pkg/releasequalifiers/types.go
do
status="$(curl -sS -o /tmp/release-controller-probe -w '%{http_code}' \
"https://raw.githubusercontent.com/openshift/release-controller/master/$path")"
printf '%s %s\n' "$status" "$path"
if [ "$status" = 200 ]; then
rg -n -C 10 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Failures|failures|Escalation' \
/tmp/release-controller-probe || true
fi
doneRepository: openshift/release
Length of output: 4806
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
tmpdir="$(mktemp -d)"
curl -fsSL 'https://codeload.github.com/openshift/release-controller/tar.gz/refs/heads/main' |
tar -xz -C "$tmpdir"
repo="$(find "$tmpdir" -mindepth 1 -maxdepth 1 -type d -name 'release-controller-*' -print -quit)"
printf '%s\n' '--- relevant source files ---'
rg -l 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation' "$repo" -g '*.go' -g '*.yaml' -g '*.yml' |
sed -n '1,120p'
printf '%s\n' '--- evaluator semantics ---'
rg -n -C 16 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Failures|failures' \
"$repo" -g '*.go' -g '*.yaml' -g '*.yml' || true
rm -rf "$tmpdir"Repository: openshift/release
Length of output: 50374
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
log=/tmp/coderabbit-shell-logs/shell-output-LhJ6GD
printf '%s\n' '--- relevant upstream paths from the previous result ---'
rg -n '^(([^:]+/){0,8}[^:]+):.*(OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation)' "$log" \
-g '!vendor/**' | head -200 || true
printf '%s\n' '--- first source-file section ---'
sed -n '1,350p' "$log" | rg -n -C 12 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation|releasequalifiers|notifications/jira' || trueRepository: openshift/release
Length of output: 50373
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
tmpdir="$(mktemp -d)"
curl -fsSL 'https://codeload.github.com/openshift/release-controller/tar.gz/refs/heads/main' |
tar -xz -C "$tmpdir"
repo="$(find "$tmpdir" -mindepth 1 -maxdepth 1 -type d -name 'release-controller-*' -print -quit)"
printf '%s\n' '--- exact qualifier matches outside vendor ---'
rg -n -C 12 --glob '!vendor/**' --glob '*.go' --glob '*.yaml' --glob '*.yml' \
'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation' "$repo" | head -1000
rm -rf "$tmpdir"Repository: openshift/release
Length of output: 50374
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
tmpdir="$(mktemp -d)"
curl -fsSL 'https://codeload.github.com/openshift/release-controller/tar.gz/refs/heads/main' |
tar -xz -C "$tmpdir"
repo="$(find "$tmpdir" -mindepth 1 -maxdepth 1 -type d -name 'release-controller-*' -print -quit)"
file="$repo/pkg/cmd/release-payload-controller/jira_escalations_controller.go"
printf '%s\n' '--- window selection and failure evaluation ---'
sed -n '450,710p' "$file"
printf '%s\n' '--- tests for default window, period handling, and failures ---'
testfile="$repo/pkg/cmd/release-payload-controller/jira_escalations_controller_test.go"
sed -n '90,160p' "$testfile"
sed -n '560,675p' "$testfile"
rm -rf "$tmpdir"Repository: openshift/release
Length of output: 14073
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
from datetime import datetime, timedelta, timezone
now = datetime.now(timezone.utc)
history = [
("failure", now - timedelta(hours=1)),
("success", now - timedelta(hours=2)),
("failure", now - timedelta(hours=3)),
("failure", now - timedelta(hours=4)),
("success", now - timedelta(hours=5)),
("failure", now - timedelta(hours=6)),
]
failures = 4
over_last_runs = None
period = timedelta(hours=24)
window_size = over_last_runs if over_last_runs is not None else failures
count_window = min(window_size, len(history))
time_window_count = sum(completed > now - period for _, completed in history)
effective_window = min(max(time_window_count, count_window), len(history))
relevant = history[:effective_window]
# Mirrors evaluateFailures: an omitted overLastRuns makes window_size == failures.
trigger = all(state != "success" for state, _ in relevant[:failures])
print({
"window_size": window_size,
"effective_window": effective_window,
"failure_count_in_24h": sum(state != "success" for state, _ in relevant),
"triggered": trigger,
})
assert effective_window == 6
assert sum(state != "success" for state, _ in relevant) == 4
assert trigger is False # the configured rule does not count non-consecutive failures
# A count window greater than the failure threshold selects total-failure mode.
over_last_runs = 10
window_size = over_last_runs
trigger = sum(state != "success" for state, _ in relevant) >= failures
assert trigger is True
print({"with_overLastRuns": over_last_runs, "triggered": trigger})
PYRepository: openshift/release
Length of output: 289
Add overLastRuns to both escalation rules.
Without overLastRuns, the evaluator requires consecutive failures. overPeriod does not change this behavior. Set overLastRuns above each failures threshold and test non-consecutive failures and failures outside each time window.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@core-services/release-controller/release-qualifiers.yaml` around lines 150 -
155, Update both escalation rules in the alert configuration by adding an
overLastRuns value above each failures threshold, with each value greater than
its corresponding failures count. Preserve the existing overPeriod settings, and
verify behavior for non-consecutive failures and failures outside each
configured time window.
Source: MCP tools
|
[REHEARSALNOTIFIER] Note: If this PR includes changes to step registry files ( Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core-services/release-controller/release-qualifiers.yaml`:
- Around line 142-150: Update both escalation blocks in the release qualifiers
configuration to add overLastRuns before failures, with each value greater than
its corresponding failures threshold (4 for warn and 8 for alert). Preserve the
existing overPeriod and priority settings, and add coverage for non-consecutive
failures and failures outside the 24-hour and 72-hour windows.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 59aea5c6-457f-4edc-bd99-8defee3293fa
📒 Files selected for processing (1)
core-services/release-controller/release-qualifiers.yaml
sosiouxme
left a comment
sosiouxme
left a comment
There was a problem hiding this comment.
/lgtm
but
/hold
in case we do actually want a change or comment
openshift-ci
Bot
added
the
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: redhat-chai-bot, sosiouxme The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/unhold |
openshift-ci
Bot
removed
the
do-not-merge/hold
|
@redhat-chai-bot: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@redhat-chai-bot: Updated the
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
…#83298) * Add failureLabels and Jira notifications to fips qualifier Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add failureLabels and Jira notifications to driver-toolkit qualifier --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: redhat-chai-bot <chai-bot@redhat.com>
3 participants
Adds
failureLabels(prevent_rc,prevent_ec) and Jira notification escalations to thefipsqualifier inrelease-qualifiers.yaml.\n\nThis builds on #82837 which added the fips qualifier without failure labels or notifications.\n\n## Changes\n\n- AddedfailureLabels: [prevent_rc, prevent_ec]— prevents promotion to RC/EC when fips-scan fails\n- Addednotifications.jirawith ART project escalations:\n - warn: 4 failures in 24h → Normal priority ticket\n - alert: 8 failures in 72h → High priority ticketAI-generated. Review for accuracy.
@neisw requested via Chai Bot
Summary by CodeRabbit
The
fipsqualifier now blocks RC and EC promotion whenfips-scanfails by applying theprevent_rcandprevent_ecfailure labels.It also sends ART Jira notifications for repeated failures: