openshift-observability-qe-agent: host skills in release repo

[IshwarKanse]

Summary

Introduces governed skill hosting and autonomous test failure triage for the OpenShift Observability QE agent step.

Scope: The step is active on 2 jobs only — Tempo Operator and OpenTelemetry Operator upstream OCP 4.22 presubmits. All other adopting jobs (stage, product, multi-arch, disconnected, older upstream versions) have the step and explicit post: overrides removed; their workflow-defined post phase (deprovision + send-results-to-reportportal) is restored. Expansion to additional jobs is gated on Sippy cross-run pattern awareness to avoid running the agent against already-known failures.

What changed

Skill hosting (commands.sh, ref.yaml, skills/)

Replaces AGENT_SKILL_URL (free-form URL to any openshift/ org repo) with AGENT_SKILL (a short name, e.g. RHOSDT) that resolves to a skill file in this repository under ci-operator/step-registry/openshift-observability/qe-agent/skills/.

Before	After
Skill URL could point to any openshift/ org repo	Skill fetched only from openshift/release — one repo, one OWNERS gate
Skill content changed silently via branch push in a team repo	Skill changes require a PR to openshift/release with OWNERS review
Mutable branch ref allowed unreviewed content to execute	PR merge is the activation gate: rehearsals skip cleanly (file absent on main)
No path traversal protection	AGENT_SKILL validated against ^[A-Za-z0-9_-]+$; base URL is hardcoded
curl -L followed redirects, bypassing the allowlist check	--max-redirs 0 disables redirect following
No content size bound	100 KB byte limit (wc -c) rejects oversized payloads
WebFetch in --allowedTools	Removed; allowed set is Bash,Read,Write,Grep,Glob
Full session tee'd to stdout — appeared in CI build-log and uploaded to GCS	Session output redirected to a temp file only; deleted on exit — never reaches build-log or GCS

Cost control and audit (commands.sh)

• --max-budget-usd 5 caps spend per run. Observed cost in production runs was $3.45–$4.45; the cap prevents runaway sessions from exhausting the Vertex AI cost center budget.
• --verbose enabled — richer tool-call traces are captured in the temp file for audit extraction, without exposing them in the CI build-log.
• qe-agent-usage.json written to ARTIFACT_DIR after every run: token counts (input, output, cache), USD cost, turn count, and wall-clock duration — extracted from the stream-json result record which contains no cluster data.
• qe-agent-commands.log written to ARTIFACT_DIR: every Bash command Claude executed (command strings only, no output) — extracted via jq from tool_use events. Enables post-run review of what Claude actually did on the cluster.

CI config changes (24 configs)

• 2 configs kept with AGENT_SKILL: RHOSDT and - ref: openshift-observability-qe-agent in post::
  • openshift-grafana-tempo-operator-main__upstream-ocp-4.22-amd64.yaml
  • openshift-open-telemetry-opentelemetry-operator-main__upstream-ocp-4.22-amd64.yaml
• 22 configs cleaned: AGENT_SKILL_URL / AGENT_SKILL env var removed, - ref: openshift-observability-qe-agent removed, and the now-redundant explicit post: section removed (workflow-level post phase handles deprovision).

RHOSDT skill (skills/RHOSDT.md)

Initial skill for the Distributed Tracing QE team covering Tempo Operator, OpenTelemetry Operator, Tracing UI (Cypress), and Disconnected test suites. Validated against 4 production CI runs ($3.45–$4.45, 57–83 turns each). Includes:

• Step 0a/0b ordering: cluster stability check (MCP wait-loop with deadline) before environment setup
• Correct find -path usage for nested test directories
• GOPATH/GOBIN/GOCACHE + PATH exports for Tempo stage/downstream test helpers
• TEST_DIR re-declared at the start of each Bash invocation (fresh shell scope)
• OTEL_SELECTOR captured once and reused across all 4 flakiness-loop reruns
• Cypress JUnit reporter flags on first rerun
• Disconnected suite rerun block and diagnostic section
• Step 5 output ordered correctly: analysis → bug report → fix → flakiness summary
• Outer 4-backtick fences on template blocks to prevent inner fences closing them prematurely

Documentation (README.md)

• Cost benefit section: qe-agent runs inside the already-provisioned CI job at $3–5 in AI API spend vs. 1–2 hours of cluster time for a manual debug session.
• Blast radius section: what Claude can/cannot do, worst-case scenarios (cluster mutation bounded by ephemeral test cluster + post-phase deprovision; runaway session bounded by $5 cap + 90-minute timeout + best_effort: true), audit artifact descriptions, private Prow deck requirement.
• Scope guidance: step should target upstream jobs only until Sippy cross-run pattern awareness is implemented to prevent the agent running against already-known failures.
• Session output behaviour: full stream-json (including --verbose traces and cluster data) is captured to a temp file only; only the two derived audit artifacts reach ARTIFACT_DIR.

Adding a new team skill

1. Create ci-operator/step-registry/openshift-observability/qe-agent/skills/<TEAM>.md
2. Add team identifier to skills/OWNERS
3. Open a PR to openshift/release — step OWNERS review and approve
4. Set AGENT_SKILL: <TEAM> in your CI config

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

The PR transitions the qe-agent step from remote AGENT_SKILL_URL references to local AGENT_SKILL lookup by name, implements a comprehensive RHOSDT skill file for distributed-tracing test failure triage and diagnosis, and updates 24 consumer CI configurations to adopt AGENT_SKILL: RHOSDT.

Changes

qe-agent AGENT_SKILL Refactor

Layer / File(s)	Summary
qe-agent script and step registry contract ci-operator/step-registry/openshift-observability/qe-agent/openshift-observability-qe-agent-commands.sh, ci-operator/step-registry/openshift-observability/qe-agent/openshift-observability-qe-agent-ref.yaml	Script validates AGENT_SKILL against an alphanumeric/-/_ pattern, constructs the skill URL from a fixed GitHub raw base plus the skill name, disables redirects, enforces a 100 KB payload limit, and skips the agent if AGENT_SKILL is unset or the file does not exist. Claude invocation removes WebFetch tool access and defaults model to claude-opus-4-6. Ref.yaml env documentation specifies loading a skill by file name from the skills/ directory and documents skip-if-missing behavior.
qe-agent documentation and configuration guidance ci-operator/step-registry/openshift-observability/qe-agent/README.md	No-op condition changed to check AGENT_SKILL instead of AGENT_SKILL_URL, configuration instructions rewritten for AGENT_SKILL-based setup, skill sourcing documented as runtime fetching from openshift/release qe-agent/skills/ directory, regex validation and skip-if-missing behavior documented, and real-world CI example updated to set AGENT_SKILL: RHOSDT.
RHOSDT skill definition and governance ci-operator/step-registry/openshift-observability/qe-agent/skills/RHOSDT.md, ci-operator/step-registry/openshift-observability/qe-agent/skills/OWNERS	Adds RHOSDT.md implementing a complete agentic workflow for distributed-tracing QE test failure triage: reads qe-agent-context.json, re-establishes environment with step-specific adaptations (git clone, idempotent CRD apply, skip CSV patches), gates on MachineConfigPool stability, parses JUnit XML for failure triage with pattern detection and capped investigation, reruns failing tests with cleanup (namespace + cluster-scoped resources), detects flakiness via 4-run confirmation loop, diagnoses root cause (PRODUCT_BUG / TEST_ISSUE / FLAKY / CLUSTER_INSTABILITY) with operator debug logging to rule out reconciliation loops, and exports per-classification artifacts. Adds OWNERS file granting IshwarKanse approval and review authority.
Consumer CI configuration migration to AGENT_SKILL ci-operator/config/openshift/grafana-tempo-operator/*, ci-operator/config/openshift/open-telemetry-opentelemetry-operator/*, ci-operator/config/openshift/distributed-tracing-console-plugin/*, ci-operator/config/openshift/distributed-tracing-qe/*	All 24 consumer CI configuration YAMLs (11 Grafana Tempo variants, 11 OpenTelemetry variants, 1 distributed-tracing-console-plugin, 1 distributed-tracing-qe) replace AGENT_SKILL_URL: https://raw.githubusercontent.com/openshift/distributed-tracing-qe/main/plugins/qe-agent/skills/SKILL.md with AGENT_SKILL: RHOSDT in their steps.env blocks.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• openshift/release#80345: Originally added AGENT_SKILL_URL in the same Tempo stage YAML files that this PR now replaces with AGENT_SKILL: RHOSDT.
• openshift/release#80351: Originally added AGENT_SKILL_URL to the same OpenTelemetry stage job definitions that this PR now migrates to AGENT_SKILL: RHOSDT.

Suggested reviewers

• rubenvp8510
• frzifus

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: moving agent skills from external URLs to being hosted in the release repository.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	This PR does not contain any Ginkgo test definitions. It modifies CI configuration files (YAML), documentation (Markdown), and automation scripts (Shell) for the qe-agent skill system. The custom c...
Test Structure And Quality	✅ Passed	This PR does not contain any Ginkgo test code. The check for Ginkgo test structure and quality is not applicable as the PR consists entirely of CI configuration files (YAML), documentation (Markdow...
Microshift Test Compatibility	✅ Passed	No new Ginkgo e2e tests are added in this PR. Changes are limited to CI configuration (24 YAML files), QE agent implementation (scripts/docs), and a skill documentation file—none of which contain G...
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PR adds no Ginkgo e2e tests; it only updates CI configs, adds QE agent skill documentation and tooling scripts. SNO compatibility check is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	This PR contains no deployment manifests, operator code, or Kubernetes resource definitions with scheduling constraints. It modifies only CI configuration files, documentation, shell scripts, and s...
Ote Binary Stdout Contract	✅ Passed	This PR contains no OTE binaries or Go test code. Changes are limited to YAML CI configuration files, shell scripts for qe-agent step execution, and markdown documentation. The OTE Binary Stdout Co...
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	This PR contains no new Ginkgo e2e tests. It only modifies CI configurations (24 YAML files), documentation (README, skill definitions), shell scripts, and configuration files. The custom check for...
No-Weak-Crypto	✅ Passed	No weak cryptography detected. The PR contains configuration and documentation updates only; the shell script uses only secure HTTPS operations via standard curl, with no MD5, SHA1, DES, RC4, 3DES,...
Container-Privileges	✅ Passed	No container privilege settings found. The PR modifies CI-Operator configs (environment variables only) and step registry files, not Kubernetes container manifests. No privileged, hostPID, hostNetw...
No-Sensitive-Data-In-Logs	✅ Passed	PR implements robust security controls to prevent sensitive data logging: --verbose and WebFetch removed, tee eliminated, skill size/input validated, no credential echoing, only metadata logged.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ci-operator/step-registry/openshift-observability/qe-agent/README.md`:
- Around line 126-130: The fenced code block in the README.md file that displays
the directory tree structure is missing a language identifier on the opening
fence. This triggers the MD040 markdown linting rule. Add a language label (such
as "text") to the opening triple backticks of the directory tree code block to
comply with markdown linting standards. Change the unlabeled fence to specify
the appropriate language, for example changing ``` to ```text.

In `@ci-operator/step-registry/openshift-observability/qe-agent/skills/RHOSDT.md`:
- Around line 628-666: The markdown code block examples in the Product Bug
Report template and Cluster Instability Report template use triple backticks for
the outer fence (```markdown) while containing inner triple backtick code blocks
(```text), which breaks the markdown parsing and triggers MD040 warnings.
Replace the triple backticks of the outer fences with quadruple backticks
(````markdown) for both the Product Bug Report template section and the Cluster
Instability Report template section to properly support nested code fences
without prematurely closing the outer block.
- Around line 95-105: The MCP wait-loop has two issues: first, the condition
logic is inverted with the `until` and `grep -qvE` combination, causing it to
proceed when pools are unhealthy and loop when they are healthy; fix this by
removing the -v flag from the grep to make the loop actually wait while MCPs are
not ready. Second, there is no timeout enforcement despite the comment promising
a 20-minute limit; add a timeout mechanism (such as a counter variable or
timestamp check) that tracks elapsed time and aborts after 1200 seconds (20
minutes), then captures the degraded machineconfigpool status using the same oc
get command and records it to ${ARTIFACT_DIR}/qe-agent-analysis.md before
exiting with an error code.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: baa7154a-cec6-44e5-8219-2c628c1eca8d

📥 Commits

Reviewing files that changed from the base of the PR and between 879dbb8 and 138d951.

📒 Files selected for processing (29)

• ci-operator/config/openshift/distributed-tracing-console-plugin/openshift-distributed-tracing-console-plugin-main__upstream-amd64-aws.yaml
• ci-operator/config/openshift/distributed-tracing-qe/openshift-distributed-tracing-qe-main__ocp-4.16-disconnected.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.12-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.14-arm-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.16-ibm-z-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.17-fips-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.17-ibm-p-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.19-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.20-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.21-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__upstream-ocp-4.12-amd64.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__upstream-ocp-4.21-amd64.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__upstream-ocp-4.22-amd64.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.12-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.14-arm-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.16-ibm-z-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.17-fips-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.17-ibm-p-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.19-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.20-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.21-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__upstream-ocp-4.12-amd64.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__upstream-ocp-4.21-amd64.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__upstream-ocp-4.22-amd64.yaml
• ci-operator/step-registry/openshift-observability/qe-agent/README.md
• ci-operator/step-registry/openshift-observability/qe-agent/openshift-observability-qe-agent-commands.sh
• ci-operator/step-registry/openshift-observability/qe-agent/openshift-observability-qe-agent-ref.yaml
• ci-operator/step-registry/openshift-observability/qe-agent/skills/OWNERS
• ci-operator/step-registry/openshift-observability/qe-agent/skills/RHOSDT.md

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

ci-operator/step-registry/openshift-observability/qe-agent/skills/RHOSDT.md (2)

231-232: ⚡ Quick win

Provide a concrete method to extract OTEL_SELECTOR from the step script.

Line 231 instructs readers to "read the fetched step script" to determine the selector, but doesn't show how. A grep example would make it clearer and reduce guesswork.

🔧 Suggested addition after line 231

 # Read the fetched step script (from Step 0) to check whether --selector is used in the chainsaw invocation.
+# Example: grep -o '\-\-selector [^ ]*' /tmp/fetched-step-script.sh | awk '{print $2}'
 OTEL_SELECTOR=""  # set to "--selector <value>" if the original script uses one, otherwise leave empty

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci-operator/step-registry/openshift-observability/qe-agent/skills/RHOSDT.md`
around lines 231 - 232, The documentation comment before the OTEL_SELECTOR
variable assignment lacks concrete instructions on how to extract the selector
value from the step script. Add a concrete example using grep after line 231 to
demonstrate how to search the fetched step script for the --selector flag
pattern. The example should show readers how to use grep to find and extract the
selector value from the chainsaw invocation, making the process clear and
reducing ambiguity about what "read the fetched step script" means.

---

211-213: ⚡ Quick win

Provide the explicit fallback command for cluster-scoped resource cleanup.

Lines 211–213 suggest falling back to a test-name label if the test-namespace label is absent, but the exact fallback command is not shown. Add the concrete command to help users apply the fallback without guessing.

🔧 Proposed addition after line 212

     -l chainsaw.kyverno.io/test-namespace=chainsaw-<test-name> \
     --ignore-not-found=true
 #    If that label is absent in the output above, fall back to the test-name label instead:
+#    kubectl delete clusterrole,clusterrolebinding \
+#      -l app.kubernetes.io/managed-by=chainsaw \
+#      -l chainsaw.kyverno.io/test-name=<test-name> \
+#      --ignore-not-found=true

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci-operator/step-registry/openshift-observability/qe-agent/skills/RHOSDT.md`
around lines 211 - 213, The documentation in RHOSDT.md around lines 211-213
provides a comment suggesting a fallback to use the
chainsaw.kyverno.io/test-name label instead of the test-namespace label for
cluster-scoped resource cleanup, but only shows the partial label selector
syntax. Add the explicit and complete kubectl command after line 212 that
demonstrates the full fallback cleanup command using the test-name label
selector, formatted consistently with the primary command shown above it, so
users can directly copy and execute it without having to construct it
themselves.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ci-operator/step-registry/openshift-observability/qe-agent/skills/RHOSDT.md`:
- Around line 292-293: The OTEL_SELECTOR variable is being reset to an empty
string at lines 292-293 before the flakiness confirmation loop, which overwrites
the selector value that was captured during the first rerun around line 232.
Remove the line that resets OTEL_SELECTOR to empty before the for loop (the one
iterating through i in 2 3 4), so that the selector determined in the first
rerun is preserved and reused consistently across all subsequent rerun
iterations without being overwritten.

---

Nitpick comments:
In `@ci-operator/step-registry/openshift-observability/qe-agent/skills/RHOSDT.md`:
- Around line 231-232: The documentation comment before the OTEL_SELECTOR
variable assignment lacks concrete instructions on how to extract the selector
value from the step script. Add a concrete example using grep after line 231 to
demonstrate how to search the fetched step script for the --selector flag
pattern. The example should show readers how to use grep to find and extract the
selector value from the chainsaw invocation, making the process clear and
reducing ambiguity about what "read the fetched step script" means.
- Around line 211-213: The documentation in RHOSDT.md around lines 211-213
provides a comment suggesting a fallback to use the
chainsaw.kyverno.io/test-name label instead of the test-namespace label for
cluster-scoped resource cleanup, but only shows the partial label selector
syntax. Add the explicit and complete kubectl command after line 212 that
demonstrates the full fallback cleanup command using the test-name label
selector, formatted consistently with the primary command shown above it, so
users can directly copy and execute it without having to construct it
themselves.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 0568696f-2e76-4e90-89e4-3d8cfa419afa

📥 Commits

Reviewing files that changed from the base of the PR and between b76d987 and 899e3be.

📒 Files selected for processing (29)

• ci-operator/config/openshift/distributed-tracing-console-plugin/openshift-distributed-tracing-console-plugin-main__upstream-amd64-aws.yaml
• ci-operator/config/openshift/distributed-tracing-qe/openshift-distributed-tracing-qe-main__ocp-4.16-disconnected.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.12-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.14-arm-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.16-ibm-z-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.17-fips-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.17-ibm-p-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.19-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.20-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.21-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__upstream-ocp-4.12-amd64.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__upstream-ocp-4.21-amd64.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__upstream-ocp-4.22-amd64.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.12-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.14-arm-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.16-ibm-z-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.17-fips-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.17-ibm-p-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.19-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.20-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.21-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__upstream-ocp-4.12-amd64.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__upstream-ocp-4.21-amd64.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__upstream-ocp-4.22-amd64.yaml
• ci-operator/step-registry/openshift-observability/qe-agent/README.md
• ci-operator/step-registry/openshift-observability/qe-agent/openshift-observability-qe-agent-commands.sh
• ci-operator/step-registry/openshift-observability/qe-agent/openshift-observability-qe-agent-ref.yaml
• ci-operator/step-registry/openshift-observability/qe-agent/skills/OWNERS
• ci-operator/step-registry/openshift-observability/qe-agent/skills/RHOSDT.md

✅ Files skipped from review due to trivial changes (5)

• ci-operator/step-registry/openshift-observability/qe-agent/skills/OWNERS
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__upstream-ocp-4.21-amd64.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.17-fips-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__upstream-ocp-4.12-amd64.yaml
• ci-operator/step-registry/openshift-observability/qe-agent/README.md

🚧 Files skipped from review as they are similar to previous changes (22)

• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__upstream-ocp-4.22-amd64.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.19-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.16-ibm-z-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.17-ibm-p-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__upstream-ocp-4.21-amd64.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.12-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.17-fips-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.12-stage.yaml
• ci-operator/config/openshift/distributed-tracing-qe/openshift-distributed-tracing-qe-main__ocp-4.16-disconnected.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__upstream-ocp-4.12-amd64.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.20-stage.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.16-ibm-z-stage.yaml
• ci-operator/config/openshift/distributed-tracing-console-plugin/openshift-distributed-tracing-console-plugin-main__upstream-amd64-aws.yaml
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.14-arm-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__upstream-ocp-4.22-amd64.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.14-arm-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.17-ibm-p-stage.yaml
• ci-operator/step-registry/openshift-observability/qe-agent/openshift-observability-qe-agent-commands.sh
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/openshift-open-telemetry-opentelemetry-operator-main__opentelemetry-product-ocp-4.21-stage.yaml
• ci-operator/step-registry/openshift-observability/qe-agent/openshift-observability-qe-agent-ref.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.20-stage.yaml
• ci-operator/config/openshift/grafana-tempo-operator/openshift-grafana-tempo-operator-main__tempo-product-ocp-4.21-stage.yaml

[IshwarKanse]

/pj-rehearse ack

[openshift-merge-bot]

@IshwarKanse: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@IshwarKanse: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-distributed-tracing-console-plugin-main-upstream-amd64-aws-e2e	openshift/distributed-tracing-console-plugin	presubmit	Ci-operator config changed
pull-ci-openshift-open-telemetry-opentelemetry-operator-main-upstream-ocp-4.22-amd64-opentelemetry-upstream-tests	openshift/open-telemetry-opentelemetry-operator	presubmit	Ci-operator config changed
pull-ci-openshift-open-telemetry-opentelemetry-operator-main-upstream-ocp-4.12-amd64-opentelemetry-upstream-tests	openshift/open-telemetry-opentelemetry-operator	presubmit	Ci-operator config changed
pull-ci-openshift-open-telemetry-opentelemetry-operator-main-upstream-ocp-4.21-amd64-opentelemetry-upstream-tests	openshift/open-telemetry-opentelemetry-operator	presubmit	Ci-operator config changed
pull-ci-openshift-grafana-tempo-operator-main-upstream-ocp-4.12-amd64-tempo-upstream-tests	openshift/grafana-tempo-operator	presubmit	Ci-operator config changed
pull-ci-openshift-grafana-tempo-operator-main-upstream-ocp-4.22-amd64-tempo-upstream-tests	openshift/grafana-tempo-operator	presubmit	Ci-operator config changed
pull-ci-openshift-grafana-tempo-operator-main-upstream-ocp-4.21-amd64-tempo-upstream-tests	openshift/grafana-tempo-operator	presubmit	Ci-operator config changed
periodic-ci-openshift-open-telemetry-opentelemetry-operator-main-opentelemetry-product-ocp-4.17-ibm-p-stage-opentelemetry-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-grafana-tempo-operator-main-tempo-product-ocp-4.19-stage-tempo-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-open-telemetry-opentelemetry-operator-main-opentelemetry-product-ocp-4.19-stage-opentelemetry-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-grafana-tempo-operator-main-tempo-product-ocp-4.12-stage-tempo-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-distributed-tracing-qe-main-ocp-4.16-disconnected-distributed-tracing-tests-disconnected	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-open-telemetry-opentelemetry-operator-main-opentelemetry-product-ocp-4.16-ibm-z-stage-opentelemetry-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-open-telemetry-opentelemetry-operator-main-opentelemetry-product-ocp-4.17-fips-stage-opentelemetry-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-open-telemetry-opentelemetry-operator-main-opentelemetry-product-ocp-4.20-stage-opentelemetry-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-grafana-tempo-operator-main-tempo-product-ocp-4.20-stage-tempo-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-grafana-tempo-operator-main-tempo-product-ocp-4.17-ibm-p-stage-tempo-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-grafana-tempo-operator-main-tempo-product-ocp-4.21-stage-tempo-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-open-telemetry-opentelemetry-operator-main-opentelemetry-product-ocp-4.14-arm-stage-opentelemetry-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-grafana-tempo-operator-main-tempo-product-ocp-4.17-fips-stage-tempo-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-open-telemetry-opentelemetry-operator-main-opentelemetry-product-ocp-4.12-stage-opentelemetry-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-grafana-tempo-operator-main-tempo-product-ocp-4.16-ibm-z-stage-tempo-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-grafana-tempo-operator-main-tempo-product-ocp-4.14-arm-stage-tempo-stage-tests	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-open-telemetry-opentelemetry-operator-main-opentelemetry-product-ocp-4.21-stage-opentelemetry-stage-tests	N/A	periodic	Ci-operator config changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[IshwarKanse]

/pj-rehearse ack

[openshift-merge-bot]

@IshwarKanse: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci]

@IshwarKanse: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[sthundat]

lgtm

[gangwgr]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gangwgr, IshwarKanse

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/config/openshift/distributed-tracing-console-plugin/OWNERS [IshwarKanse]
• ci-operator/config/openshift/distributed-tracing-qe/OWNERS [IshwarKanse]
• ci-operator/config/openshift/grafana-tempo-operator/OWNERS [IshwarKanse]
• ci-operator/config/openshift/open-telemetry-opentelemetry-operator/OWNERS [IshwarKanse]
• ci-operator/step-registry/openshift-observability/qe-agent/OWNERS [IshwarKanse]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment