Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions plugins/ci/skills/payload-analysis/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,29 @@ unavailable rather than empty evidence.

#### 3.4: Test Failure Details

Only `test_failures.blocking[]` contains failures that can reject the payload. **`test_failures.informing[]` and `test_failures.flakes[]` cannot fail a job or reject a payload** — never score them as candidate causes, never use them to derive a failure mode's originating payload, and never propose a revert for them.

**"Informing job" ≠ "informing test."** These are two completely different
concepts that share a name:

- **Informing job** (`informing_jobs.failed_jobs[]`): a CI *job* that runs
for visibility but does not gate the payload. Its pass/fail status is
job-level. An informing job can still contain blocking tests.
- **Informing test** (`test_failures.informing[]`): an individual *test case*
with `lifecycle="informing"`. It can appear inside any job — blocking or
informing. Its results never count toward `test_failure_count`.

Never combine informing-job counts with informing-test lists. When
reporting informing/flake tests, list individual test names from
`test_failures.informing[]` / `test_failures.flakes[]` — do NOT report
informing *job* failure counts in the same section.

Report informing and flake tests in their own section of the report, under a heading that says so, with this caveat:

> These tests do not by themselves cause job failures or payload rejections. The only potential impact is if the test itself affects cluster health — for example if it breaks an operator or does something otherwise catastrophic.

Keep them visible: informing tests are new tests being stabilized, and a badly-behaved test can occasionally damage the cluster it runs on. Investigate one only when there is evidence of that, and say plainly that it is not a rejection cause.

From `summary.json` → `test_failures.blocking[]`:
- `test_name`, `jobs`, `first_failed_in`, `payloads_failing`
- `failure_message`, `failure_text` (full, not truncated)
Expand Down
42 changes: 41 additions & 1 deletion plugins/ci/skills/payload-snapshot/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,8 @@ Comprehensive stream-level triage data — start here. Contains:
- Chain data: `chain_length`, `baseline_tag`, `hours_since_baseline`
- `blocking_jobs.failed_jobs[]` — detailed objects with `name`, `state`, `prow_url`, `gcs_url`, and relative path `job_json`. May include: `rhcos_version`, `streak` (with `streak_length`, `originating_payload`, `is_new_failure`, `failure_pattern`), `build_log_errors`, `test_failure_count`, and relative paths `junit_results`, `build_log`
- `informing_jobs.failed_jobs[]` — job name strings
- `test_failures.blocking[]` — `test_name`, `jobs`, `first_failed_in`, `payloads_failing`, `failure_message`, `failure_text` (full, not truncated)
- `test_failures.blocking[]` — **gating** failures only: `test_name`, `jobs`, `first_failed_in`, `payloads_failing`, `failure_message`, `failure_text` (full, not truncated). These are the failures that can fail a job and therefore reject the payload.
- `test_failures.informing[]` / `test_failures.flakes[]` — `test_name`, `jobs`. Neither can fail a job. No onset is tracked for them, because an onset implies there is a culprit to find.
- `payloads[]` — per-payload entries with `tag`, `phase`, `source`, `changelog_source`, relative file paths, `prs[]` with component/diff/comments paths, and `rhcos_changes[]` with RPM diffs per RHCOS variant
- `rhcos_rpms[]` — RPMDB metadata for the target payload's RHCOS variants: `tag`, `name`, `pullspec`, `rpmdb` (relative path to rpmdb.sqlite)
- `data_complete` — `true` when all requested data was ultimately collected, including via a fallback after an initial read failed. `false` means some requested data could not be read at all.
Expand All @@ -198,6 +199,45 @@ Comprehensive stream-level triage data — start here. Contains:
- `data_complete` is `false` only when at least one error was **not** recovered.

<a id="data-completeness"></a>
#### Only gating results count as failures

A test result falls into exactly one of three categories, and only the last
can fail a job or reject a payload:

| Category | Rule | Gates? |
|---|---|---|
| flake | the same test, in the same suite, both failed and passed | no |
| informing | the testcase carries `lifecycle="informing"` | no |
| failure | failed everywhere, no `informing` lifecycle | **yes** |

Informing tests are run to stabilize them and are not expected to gate. A
missing `lifecycle` attribute means the test **does** gate — the attribute
exists only to opt a test out.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

`results.json` records all three so nothing is hidden, each entry carrying
`status` (`failed`, `error`, `flake`) and `test_lifecycle` (`blocking`,
`informing`).

`test_failure_count` on a failed job counts **only gating** results — it is
the failure count, and flakes and informing tests are not added to it.
Those two are listed by name under `test_failures.flakes[]` and
`test_failures.informing[]`. Regression onset (`first_failed_in`) is derived
from gating failures alone.

#### "Informing job" vs "informing test" — two unrelated concepts

The word "informing" appears in two places with **completely different
meanings**. Confusing them produces wrong analysis:

| Concept | Where it lives | What it means |
|---|---|---|
| **Informing job** | `informing_jobs.failed_jobs[]` in summary.json | A CI *job* that runs for visibility but does **not** gate the payload. Job-level pass/fail. |
| **Informing test** | `test_failures.informing[]` in summary.json | An individual *test case* whose `lifecycle="informing"` attribute opts it out of gating. Can appear inside **any** job — blocking or informing. |

An informing *test* can run inside a *blocking* job.
An informing *job* can contain *blocking* tests.
They are orthogonal. Never combine them in the same section or count.

#### gcloud credentials are not required

The CI artifact buckets are public. When gcloud has no active account it is
Expand Down
156 changes: 141 additions & 15 deletions plugins/ci/skills/payload-snapshot/scripts/payload_snapshot.py
Original file line number Diff line number Diff line change
Expand Up @@ -1197,12 +1197,12 @@ def _fetch(self) -> Optional[dict]:
return None

gcs_uri = f"gs://{self.job.gcs_bucket_path}/build-log.txt"
errors_before = _unrecovered_error_count()
raw = _run_gcloud_bytes(
["gcloud", "storage", "cat", gcs_uri], timeout=120
)
with _error_scope() as own_errors:
raw = _run_gcloud_bytes(
["gcloud", "storage", "cat", gcs_uri], timeout=120
)
if not raw:
if _unrecovered_error_count() > errors_before:
if any(not e.get("recovered") for e in own_errors):
_record_collection_error(
"build_log_unavailable",
["gcloud", "storage", "cat", gcs_uri],
Expand Down Expand Up @@ -1406,6 +1406,11 @@ def __init__(self, base_dir: str, chain: list[str], target_tag: str):
self.base_dir = base_dir
self.chain = chain
self.target_tag = target_tag
self._skipped = {"flake": 0, "informing": 0}

def non_gating_counts(self) -> dict[str, int]:
"""Counts of target-payload results excluded from onset tracking."""
return dict(self._skipped)

def track(self) -> list[dict]:
"""Analyze failures in the target payload and trace their origins."""
Expand Down Expand Up @@ -1461,6 +1466,17 @@ def _load_all_failures(self, tag_dir: str) -> dict[str, dict]:
name = test.get("name", "")
if not name:
continue
# Only gating results establish a regression onset. A
# flake or an informing test cannot reject a payload, so
# letting one set first_failed_in sends the analysis
# hunting for a culprit that never existed.
if not _is_gating(test):
if tag_dir.endswith(self.target_tag):
if test.get("status") == "flake":
self._skipped["flake"] += 1
else:
self._skipped["informing"] += 1
continue
if name in failures:
if job_name not in failures[name]["jobs"]:
failures[name]["jobs"].append(job_name)
Expand Down Expand Up @@ -1633,9 +1649,15 @@ def generate(self) -> None:
"failed_jobs": failed_informing_names,
},
"test_failures": {
# Gating failures only — these are what can fail a job and
# therefore reject the payload.
"blocking": [
r for r in regressions if r.get("lifecycle") == "blocking"
],
# Neither of the following can fail a job or reject a
# payload. They are surfaced for visibility, not for blame.
"informing": self._collect_non_gating("informing"),
"flakes": self._collect_non_gating("flake"),
},
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
"payloads": payloads,
}
Expand Down Expand Up @@ -1880,15 +1902,23 @@ def _build_failed_job_details(self, lifecycle: str) -> list[dict]:
f"{tag_rel}/jobs/{lifecycle}/{job_name}/junit/results.json"
)
results_data = _read_json(results_path) or []
entry["test_failure_count"] = len(results_data)
# Counts only results that could fail the job. Flakes and
# informing tests are listed by name under `test_failures`
# but are deliberately not counted here — this number is the
# failure count that drives triage.
entry["test_failure_count"] = sum(
1 for t in results_data if _is_gating(t)
)
if _job_junit_state(
job_name, self.target_tag, "junit_partial"
):
# Real results, but not all of them: the count is a
# lower bound, not an authoritative total.
entry["junit_collection_partial"] = True
elif _job_junit_state(
job_name, self.target_tag, "junit_unavailable"
elif any(
_job_junit_state(job_name, self.target_tag, r)
for r in ("junit_unavailable", "junit_missing",
"junit_unparseable")
):
# Data could not be read. Do NOT emit test_failure_count —
# its absence means "unknown", where 0 would mean "clean".
Expand All @@ -1910,6 +1940,45 @@ def _build_failed_job_details(self, lifecycle: str) -> list[dict]:
details.append(entry)
return details

def _collect_non_gating(self, kind: str) -> list[dict]:
"""Collect the target payload's non-gating test results.

``kind`` is "flake" (failed then passed) or "informing" (opted out
of gating via lifecycle). Neither can fail a job, so no regression
onset is computed for them — an onset implies a culprit to find.
"""
found: dict[str, dict] = {}
lifecycle_dir = os.path.join(
self.base_dir, self.target_tag, "jobs", "blocking"
)
if not os.path.isdir(lifecycle_dir):
return []
for job_name in sorted(os.listdir(lifecycle_dir)):
results = _read_json(os.path.join(
lifecycle_dir, job_name, "junit", "results.json"
))
if not results:
continue
for test in results:
name = test.get("name", "")
if not name:
continue
if kind == "flake":
match = test.get("status") == "flake"
else:
match = (
test.get("status") in ("failed", "error")
and test.get("test_lifecycle") == "informing"
)
if not match:
continue
if name in found:
if job_name not in found[name]["jobs"]:
found[name]["jobs"].append(job_name)
else:
found[name] = {"test_name": name, "jobs": [job_name]}
return sorted(found.values(), key=lambda e: e["test_name"])

def _build_payload_entries(self) -> list[dict]:
"""Build the payloads array with all path references."""
payloads = []
Expand Down Expand Up @@ -2345,8 +2414,14 @@ def _track_regressions(self, base_dir: str, chain: list[str]) -> None:

new_count = sum(1 for r in regressions if r["payloads_failing"] == 1)
persistent_count = len(regressions) - new_count
_log(f" Found {len(regressions)} failing tests: "
_log(f" Found {len(regressions)} gating test failures: "
f"{new_count} new, {persistent_count} persistent")
skipped = tracker.non_gating_counts()
if skipped["flake"] or skipped["informing"]:
_log(f" Excluded from regression tracking: "
f"{skipped['flake']} flake(s), "
f"{skipped['informing']} informing failure(s) — neither can "
f"fail a job")

def _collect_build_logs(self, base_dir: str, chain: list[str]) -> None:
"""Download and parse build-log.txt for failed jobs.
Expand Down Expand Up @@ -2639,10 +2714,12 @@ def _check_gh_auth() -> bool:
_ERROR_SCOPES = threading.local()

# stderr fragments that mean "the object does not exist", not "I failed".
# Keep this narrow: only the specific gcloud "URLs matched no objects"
# outcome is benign. Broad tokens like "not found" would swallow 404s
# and bucket-not-found errors that should be recorded.
_GCLOUD_NO_MATCH_PATTERNS = (
"matched no objects",
"matched no such objects",
"not found",
)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

# stderr fragments that indicate a credentials/permission problem.
Expand Down Expand Up @@ -2707,10 +2784,10 @@ def _sanitize_detail(text: str) -> str:
def _classify_gcloud_stderr(stderr: str) -> str:
"""Classify a failed gcloud invocation from its stderr."""
low = (stderr or "").lower()
if any(p in low for p in _GCLOUD_NO_MATCH_PATTERNS):
return "no_match"
if any(p in low for p in _GCLOUD_AUTH_PATTERNS):
return "auth"
if any(p in low for p in _GCLOUD_NO_MATCH_PATTERNS):
return "no_match"
return "command_failed"


Expand Down Expand Up @@ -2786,6 +2863,7 @@ def _invalidate_suspect_junit(base_dir: str) -> int:
return 0

invalidated = 0
invalidated_tags: set[str] = set()
for entry in previous:
if entry.get("recovered"):
continue
Expand All @@ -2807,6 +2885,12 @@ def _invalidate_suspect_junit(base_dir: str) -> int:
if os.path.isdir(junit_dir):
shutil.rmtree(junit_dir, ignore_errors=True)
invalidated += 1
invalidated_tags.add(tag)
for tag in invalidated_tags:
reg_path = os.path.join(base_dir, tag, "regressions.json")
if _is_within(base_dir, reg_path) and os.path.exists(reg_path):
os.remove(reg_path)
_log(f" removed stale regressions.json for {tag}")
if invalidated:
_log(f" re-collecting {invalidated} job(s) whose previous JUnit "
f"collection was incomplete")
Expand Down Expand Up @@ -3138,8 +3222,11 @@ class _TestResult:
"""Parsed test result from JUnit XML."""

name: str
status: str # passed, failed, error, skipped
status: str # passed, failed, error, skipped, flake
suite_name: str = ""
# The testcase's own lifecycle attribute, NOT the job's. Absent means
# the test gates the job; "informing" means it cannot.
test_lifecycle: str = ""
failure_message: str = ""
failure_text: str = ""
error_message: str = ""
Expand Down Expand Up @@ -3213,6 +3300,7 @@ def _parse_junit_xml(
suite_name = suite.get("name", "")
for tc in suite.findall("testcase"):
name = tc.get("name", "")
test_lifecycle = tc.get("lifecycle", "")
failure_el = tc.find("failure")
error_el = tc.find("error")
skipped_el = tc.find("skipped")
Expand Down Expand Up @@ -3250,6 +3338,7 @@ def _parse_junit_xml(
name=name,
status=status,
suite_name=suite_name,
test_lifecycle=test_lifecycle,
failure_message=failure_message,
failure_text=failure_text,
error_message=error_message,
Expand All @@ -3259,19 +3348,56 @@ def _parse_junit_xml(
agg_failures=agg_failures,
agg_skips=agg_skips,
))
return _mark_flakes(results)


def _mark_flakes(results: list[_TestResult]) -> list[_TestResult]:
"""Re-label failures as flakes when the same test also passed.

A test that failed and then passed on retry is a flake. Flakes do not
fail jobs, so counting them as failures overstates what could have
rejected a payload — and lets a flake drive regression onset.

Grouped per (suite, name): the same name in two suites is two tests.
"""
passed: set[tuple[str, str]] = {
(r.suite_name, r.name) for r in results if r.status == "passed"
}
for r in results:
if r.status in ("failed", "error") and (r.suite_name, r.name) in passed:
r.status = "flake"
return results


def _is_gating(entry: dict) -> bool:
"""True when this recorded result could actually fail its job.

Excludes flakes (a later pass cleared them) and tests explicitly opted
out via ``lifecycle="informing"`` — informing tests are run to
stabilize them and are not expected to gate.
"""
return (
entry.get("status") in ("failed", "error")
and entry.get("test_lifecycle") != "informing"
)


def _test_results_to_json(results: list[_TestResult]) -> list[dict]:
"""Convert _TestResult list to JSON-serializable dicts (failures only)."""
"""Convert _TestResult list to JSON dicts (failures, errors and flakes).

Flakes are included so consumers can see them, but each entry carries
``status`` and ``test_lifecycle`` so only genuinely gating results are
counted as failures.
"""
output = []
for r in results:
if r.status not in ("failed", "error"):
if r.status not in ("failed", "error", "flake"):
continue
d: dict = {
"name": r.name,
"status": r.status,
"suite_name": r.suite_name,
"test_lifecycle": r.test_lifecycle or "blocking",
}
if r.failure_message:
d["failure_message"] = r.failure_message
Expand Down
Loading
Loading