Skip to content

Add failureLabels and Jira notifications to fips qualifier #83298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 2 commits into from
Aug 13, 2026
Merged

Add failureLabels and Jira notifications to fips qualifier #83298

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f61d5dd
Add failureLabels and Jira notifications to fips qualifier
redhat-chai-bot Aug 11, 2026
0ae368b
Add failureLabels and Jira notifications to driver-toolkit qualifier
redhat-chai-bot Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions core-services/release-controller/release-qualifiers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,3 +136,21 @@ qualifiers:
summary: "FIPS Payload Validation"
description: "FIPS Payload Validation"
payloadBadgeStatus: "Yes"
failureLabels:
- "prevent_rc"
- "prevent_ec"
notifications:
jira:
project: "ART"
summary: "FIPS qualifier job failures"
description: "Automated ticket for FIPS qualifier failures"
thread: "fips-qualifier"
escalations:
- name: "warn"
failures: 4
overPeriod: "24h"
priority: "Normal"
- name: "alert"
failures: 8
overPeriod: "72h"
Comment on lines +168 to +173

@coderabbitai coderabbitai Bot Aug 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

rg -n -C 8 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|TimeBoundedFailures' \
  --glob '*.go' --glob '*.yaml' .

rg -n -C 4 'failures: 4|failures: 8|overLastRuns|overPeriod' \
  core-services/release-controller/release-qualifiers.yaml

Repository: openshift/release

Length of output: 31181

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- repository files ---'
git ls-files | rg '(^|/)(release-qualifiers\.yaml|types\.go|.*qualif.*\.go)$' || true

printf '%s\n' '--- local configuration and nearby documentation ---'
sed -n '1,180p' core-services/release-controller/release-qualifiers.yaml

printf '%s\n' '--- upstream evaluator source ---'
for path in \
  pkg/releasequalifiers/notifications/jira/types.go \
  pkg/releasequalifiers/notifications/jira/jira.go \
  pkg/releasequalifiers/notifications/jira/*.go \
  pkg/releasequalifiers/*.go
do
  url="https://raw.githubusercontent.com/openshift/release-controller/master/$path"
  printf '\n### %s\n' "$url"
  curl -fsSL "$url" 2>/dev/null | rg -n -C 8 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Failures|failure' || true
done

Repository: openshift/release

Length of output: 10594

🌐 Web query:

openshift release-controller release qualifiers OverPeriod OverLastRuns evaluator failures source

💡 Result:

In the OpenShift release-controller, "release qualifiers" refer to a configuration-driven mechanism used to implement stricter validation of release candidates beyond basic job success or failure [1]. These qualifiers allow the controller to reject or qualify releases based on aggregated failure data and specific failure patterns [1]. The parameters OverPeriod and OverLastRuns are associated with the evaluator logic for these qualifiers. They define the time-based or run-based scope for analyzing job failures [2][1]: 1. OverPeriod: Specifies a time duration (e.g., a rolling window) over which the evaluator aggregates failure statistics to determine if a release should be qualified or rejected. 2. OverLastRuns: Specifies a count of recent Prow job executions (runs) over which the failure rate or failure patterns are evaluated. This evaluation logic is designed to prevent "flaky" or consistently failing releases from progressing by examining performance trends rather than relying solely on the status of a single job execution [3][4]. The release-controller's source code for these features is located in the official openshift/release-controller repository [5]. Specifically, configuration parsing and command-line flags related to release qualifiers are defined in cmd/release-controller/main.go [2], while the runtime logic for processing release tags, audit failures, and synchronization is handled in components such as cmd/release-controller/audit.go and cmd/release-controller/sync_tags.go [6][7]. Technical note: When debugging these failures, developers often review the release payload metadata and the status of blocking/informing jobs tracked by the controller [3][4]. Information regarding specific infrastructure-level failures versus product bugs is often analyzed by examining Prow job logs and the release-controller's own audit logs [8][7].

Citations:

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

tree="$(mktemp)"
curl -fsSL 'https://api.github.com/repos/openshift/release-controller/git/trees/master?recursive=1' >"$tree"

printf '%s\n' '--- qualifier-related upstream paths ---'
jq -r '.tree[] | select(.path | test("qualif|jira|notification"; "i")) | .path' "$tree"

printf '%s\n' '--- references in upstream source ---'
paths="$(jq -r '.tree[] | select(.type == "blob" and (.path | test("\\.(go|yaml|yml)$"))) | .path' "$tree")"
while IFS= read -r path; do
  content="$(curl -fsSL "https://raw.githubusercontent.com/openshift/release-controller/master/$path" 2>/dev/null || true)"
  if printf '%s' "$content" | rg -q 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation'; then
    printf '\n### %s\n' "$path"
    printf '%s' "$content" | rg -n -C 12 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation'
  fi
done <<<"$paths"
rm -f "$tree"

Repository: openshift/release

Length of output: 206

🌐 Web query:

site:github.com/openshift/release-controller "OverLastRuns" "OverPeriod"

💡 Result:

Based on the provided search results from the OpenShift release-controller repository, there is no evidence that OverLastRuns or OverPeriod are existing functions, configuration fields, or established parameters within the release-controller codebase. The repository code, including periodic.go, controller.go, and sync_upgrade.go, manages ProwJobs and release image streams using various synchronization loops and configuration structures, but these specific terms do not appear as recognized identifiers or configurable settings in the current implementation [1][2][3].

Citations:

🌐 Web query:

site:github.com/openshift/release-controller/pkg/releasequalifiers "Evaluate" "OverPeriod"

💡 Result:

No exact indexed match was found for:

site:github.com/openshift/release-controller/pkg/releasequalifiers "Evaluate" "OverPeriod"

The releasequalifiers package is documented on Go.dev, but the indexed API does not show an Evaluate or OverPeriod symbol. (pkg.go.dev)

Citations:

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

tmpdir="$(mktemp -d)"
curl -fsSL 'https://codeload.github.com/openshift/release-controller/tar.gz/refs/heads/master' |
  tar -xz -C "$tmpdir"
repo="$(find "$tmpdir" -mindepth 1 -maxdepth 1 -type d -name 'release-controller-*' -print -quit)"

printf '%s\n' '--- upstream references ---'
rg -n -C 10 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation|failures' \
  "$repo" -g '*.go' -g '*.yaml' -g '*.yml' || true

printf '%s\n' '--- candidate evaluator files ---'
find "$repo" -type f \( -name '*.go' -o -name '*.yaml' -o -name '*.yml' \) -print |
  rg 'qualif|jira|notification' || true

rm -rf "$tmpdir"

Repository: openshift/release

Length of output: 313

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

tmpdir="$(mktemp -d)"
curl -fsSL 'https://codeload.github.com/openshift/release-controller/tar.gz/refs/heads/master' |
  tar -xz -C "$tmpdir"
repo="$(find "$tmpdir" -mindepth 1 -maxdepth 1 -type d -name 'release-controller-*' -print -quit)"

printf '%s\n' '--- upstream package outline ---'
if command -v ast-grep >/dev/null 2>&1; then
  ast-grep outline "$repo/pkg/releasequalifiers" || true
fi

printf '%s\n' '--- relevant Go declarations and call sites ---'
rg -n 'type Escalation|func .*Escal|OverLastRuns|OverPeriod|overLastRuns|overPeriod|Failures' \
  "$repo/pkg" "$repo/cmd" -g '*.go' || true

rm -rf "$tmpdir"

Repository: openshift/release

Length of output: 313

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- remote refs ---'
git ls-remote --heads --tags https://github.com/openshift/release-controller.git 2>&1 | head -40 || true

printf '%s\n' '--- raw endpoint status ---'
for path in \
  README.md \
  pkg/releasequalifiers/notifications/jira/types.go \
  pkg/releasequalifiers/notifications/jira/evaluator.go \
  pkg/releasequalifiers/notifications/jira/jira.go \
  pkg/releasequalifiers/notifications/jira/notification.go \
  pkg/releasequalifiers/evaluator.go \
  pkg/releasequalifiers/types.go
do
  status="$(curl -sS -o /tmp/release-controller-probe -w '%{http_code}' \
    "https://raw.githubusercontent.com/openshift/release-controller/master/$path")"
  printf '%s %s\n' "$status" "$path"
  if [ "$status" = 200 ]; then
    rg -n -C 10 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Failures|failures|Escalation' \
      /tmp/release-controller-probe || true
  fi
done

Repository: openshift/release

Length of output: 4806

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

tmpdir="$(mktemp -d)"
curl -fsSL 'https://codeload.github.com/openshift/release-controller/tar.gz/refs/heads/main' |
  tar -xz -C "$tmpdir"
repo="$(find "$tmpdir" -mindepth 1 -maxdepth 1 -type d -name 'release-controller-*' -print -quit)"

printf '%s\n' '--- relevant source files ---'
rg -l 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation' "$repo" -g '*.go' -g '*.yaml' -g '*.yml' |
  sed -n '1,120p'

printf '%s\n' '--- evaluator semantics ---'
rg -n -C 16 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Failures|failures' \
  "$repo" -g '*.go' -g '*.yaml' -g '*.yml' || true

rm -rf "$tmpdir"

Repository: openshift/release

Length of output: 50374

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

log=/tmp/coderabbit-shell-logs/shell-output-LhJ6GD
printf '%s\n' '--- relevant upstream paths from the previous result ---'
rg -n '^(([^:]+/){0,8}[^:]+):.*(OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation)' "$log" \
  -g '!vendor/**' | head -200 || true

printf '%s\n' '--- first source-file section ---'
sed -n '1,350p' "$log" | rg -n -C 12 'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation|releasequalifiers|notifications/jira' || true

Repository: openshift/release

Length of output: 50373

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

tmpdir="$(mktemp -d)"
curl -fsSL 'https://codeload.github.com/openshift/release-controller/tar.gz/refs/heads/main' |
  tar -xz -C "$tmpdir"
repo="$(find "$tmpdir" -mindepth 1 -maxdepth 1 -type d -name 'release-controller-*' -print -quit)"

printf '%s\n' '--- exact qualifier matches outside vendor ---'
rg -n -C 12 --glob '!vendor/**' --glob '*.go' --glob '*.yaml' --glob '*.yml' \
  'OverLastRuns|OverPeriod|overLastRuns|overPeriod|Escalation' "$repo" | head -1000

rm -rf "$tmpdir"

Repository: openshift/release

Length of output: 50374

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

tmpdir="$(mktemp -d)"
curl -fsSL 'https://codeload.github.com/openshift/release-controller/tar.gz/refs/heads/main' |
  tar -xz -C "$tmpdir"
repo="$(find "$tmpdir" -mindepth 1 -maxdepth 1 -type d -name 'release-controller-*' -print -quit)"
file="$repo/pkg/cmd/release-payload-controller/jira_escalations_controller.go"

printf '%s\n' '--- window selection and failure evaluation ---'
sed -n '450,710p' "$file"

printf '%s\n' '--- tests for default window, period handling, and failures ---'
testfile="$repo/pkg/cmd/release-payload-controller/jira_escalations_controller_test.go"
sed -n '90,160p' "$testfile"
sed -n '560,675p' "$testfile"

rm -rf "$tmpdir"

Repository: openshift/release

Length of output: 14073

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

python3 - <<'PY'
from datetime import datetime, timedelta, timezone

now = datetime.now(timezone.utc)
history = [
    ("failure", now - timedelta(hours=1)),
    ("success",  now - timedelta(hours=2)),
    ("failure", now - timedelta(hours=3)),
    ("failure", now - timedelta(hours=4)),
    ("success",  now - timedelta(hours=5)),
    ("failure", now - timedelta(hours=6)),
]

failures = 4
over_last_runs = None
period = timedelta(hours=24)

window_size = over_last_runs if over_last_runs is not None else failures
count_window = min(window_size, len(history))
time_window_count = sum(completed > now - period for _, completed in history)
effective_window = min(max(time_window_count, count_window), len(history))
relevant = history[:effective_window]

# Mirrors evaluateFailures: an omitted overLastRuns makes window_size == failures.
trigger = all(state != "success" for state, _ in relevant[:failures])

print({
    "window_size": window_size,
    "effective_window": effective_window,
    "failure_count_in_24h": sum(state != "success" for state, _ in relevant),
    "triggered": trigger,
})
assert effective_window == 6
assert sum(state != "success" for state, _ in relevant) == 4
assert trigger is False  # the configured rule does not count non-consecutive failures

# A count window greater than the failure threshold selects total-failure mode.
over_last_runs = 10
window_size = over_last_runs
trigger = sum(state != "success" for state, _ in relevant) >= failures
assert trigger is True
print({"with_overLastRuns": over_last_runs, "triggered": trigger})
PY

Repository: openshift/release

Length of output: 289

Add overLastRuns to both escalation rules.

Without overLastRuns, the evaluator requires consecutive failures. overPeriod does not change this behavior. Set overLastRuns above each failures threshold and test non-consecutive failures and failures outside each time window.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@core-services/release-controller/release-qualifiers.yaml` around lines 150 -
155, Update both escalation rules in the alert configuration by adding an
overLastRuns value above each failures threshold, with each value greater than
its corresponding failures count. Preserve the existing overPeriod settings, and
verify behavior for non-consecutive failures and failures outside each
configured time window.

Source: MCP tools

priority: "High"