Improve the payload agent - #79922
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThis PR refactors the OpenShift Claude payload agent by removing Jira-dependent revert infrastructure, introducing deterministic payload snapshot generation, updating Claude's analysis entrypoint with system prompt changes requiring Skill-based tool loading and structured output validation with retry logic, and configuring a new scheduled test workflow. ChangesClaude Payload Agent Infrastructure Refactor
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 14 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (14 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/openshift/claude/payload/agent/openshift-claude-payload-agent-commands.sh`:
- Around line 243-282: PHASE_ANALYSIS_DURATION is being finalized before the
validation/regeneration loop so retry and timeout time are not included in the
analysis testcase or TOTAL_DURATION and PHASE_COUNT is not adjusted when the
"recovery nudge" testcase is appended; move the finalization of
PHASE_ANALYSIS_DURATION (and the logic that increments
PHASE_COUNT/TOTAL_DURATION) to after the validation/retry loop so all retry
attempts and timeout durations are included, and update the code that appends
the recovery-nudge testcase to increment PHASE_COUNT (and add its duration to
TOTAL_DURATION) when a timeout leads to that testcase being created; locate the
validation/regeneration loop around the python validate calls and the claude
timeout block plus references to PHASE_ANALYSIS_DURATION, PHASE_COUNT,
TOTAL_DURATION and the recovery nudge testcase to apply these changes.
- Around line 243-282: Ensure schema validation failures cause the step to fail:
first check that VALIDATE_YAML and VALIDATE_JSON are non-empty and executable
(test -x or -f) and immediately echo an error and exit 1 if either validator is
missing; inside the retry loop keep the YAML_OK/JSON_OK checks but on the final
attempt where both are not true (the branch currently echoing the warning),
replace the echo+break with an error message and exit 1 so the job fails; also
ensure when python validation commands themselves fail to run (e.g., non-zero
exit) they are treated as validation failures (leave current pattern of setting
YAML_OK/JSON_OK only on success) so the overall behavior enforces the schema
contract.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: af512c6b-7ec4-403c-8a53-6d022b8d9efb
📒 Files selected for processing (2)
ci-operator/step-registry/openshift/claude/payload/agent/openshift-claude-payload-agent-commands.shci-operator/step-registry/openshift/claude/payload/agent/openshift-claude-payload-agent-ref.yaml
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift-eng/ai-helpers/openshift-eng-ai-helpers-main.yaml`:
- Around line 29-31: The job payload-analysis-eval-presubmit's run_if_changed
regex misses files under evals/scripts (so changes to EVAL_SETUP_SCRIPT won't
trigger the job); update the run_if_changed pattern for
payload-analysis-eval-presubmit to include the evals/scripts/ path (e.g., allow
an optional scripts/ segment like
evals/(scripts/)?(eval-payload-analysis|cases/payload-analysis|snapshots)) so
that files such as
plugins/ci/evals/scripts/extract-payload-analysis-snapshots.sh are matched.
In
`@ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.sh`:
- Around line 29-33: The current conditional around EVAL_SETUP_SCRIPT silently
skips when the variable is non-empty but the path is not a file; update the
check in the openshift-claude-agent-eval-commands.sh script so that if
EVAL_SETUP_SCRIPT is set (non-empty) but [[ -f "${EVAL_SETUP_SCRIPT}" ]] is
false the script prints a clear error message and exits non-zero (e.g., use echo
"ERROR: EVAL_SETUP_SCRIPT set but file not found: ${EVAL_SETUP_SCRIPT}" >&2;
exit 1); keep the existing branch that runs bash "${EVAL_SETUP_SCRIPT}" when the
file exists and unchanged behavior when EVAL_SETUP_SCRIPT is empty.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: e0167284-52bd-4985-ac87-d72138859361
⛔ Files ignored due to path filters (2)
ci-operator/jobs/openshift-eng/ai-helpers/openshift-eng-ai-helpers-main-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift-eng/ai-helpers/openshift-eng-ai-helpers-main-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (7)
ci-operator/config/openshift-eng/ai-helpers/openshift-eng-ai-helpers-main.yamlci-operator/step-registry/openshift/claude/agent-eval/OWNERSci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.shci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-ref.metadata.jsonci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-ref.yamlci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-workflow.metadata.jsonci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-workflow.yaml
✅ Files skipped from review due to trivial changes (3)
- ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-ref.metadata.json
- ci-operator/step-registry/openshift/claude/agent-eval/OWNERS
- ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-workflow.metadata.json
| if [[ -n "${EVAL_SETUP_SCRIPT}" ]] && [[ -f "${EVAL_SETUP_SCRIPT}" ]]; then | ||
| echo "" | ||
| echo "=== Running setup script: ${EVAL_SETUP_SCRIPT} ===" | ||
| bash "${EVAL_SETUP_SCRIPT}" | ||
| fi |
There was a problem hiding this comment.
Fail fast when EVAL_SETUP_SCRIPT is set but missing.
When EVAL_SETUP_SCRIPT is non-empty but the path doesn't resolve to a file, the block silently skips setup. For the payload jobs this means the snapshot extraction never runs and the eval proceeds against missing fixtures, yielding misleading results instead of a clear failure.
🛡️ Proposed fix to fail on a misconfigured setup script path
-if [[ -n "${EVAL_SETUP_SCRIPT}" ]] && [[ -f "${EVAL_SETUP_SCRIPT}" ]]; then
- echo ""
- echo "=== Running setup script: ${EVAL_SETUP_SCRIPT} ==="
- bash "${EVAL_SETUP_SCRIPT}"
-fi
+if [[ -n "${EVAL_SETUP_SCRIPT}" ]]; then
+ if [[ ! -f "${EVAL_SETUP_SCRIPT}" ]]; then
+ echo "ERROR: EVAL_SETUP_SCRIPT is set but not found: ${EVAL_SETUP_SCRIPT}" >&2
+ exit 1
+ fi
+ echo ""
+ echo "=== Running setup script: ${EVAL_SETUP_SCRIPT} ==="
+ bash "${EVAL_SETUP_SCRIPT}"
+fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if [[ -n "${EVAL_SETUP_SCRIPT}" ]] && [[ -f "${EVAL_SETUP_SCRIPT}" ]]; then | |
| echo "" | |
| echo "=== Running setup script: ${EVAL_SETUP_SCRIPT} ===" | |
| bash "${EVAL_SETUP_SCRIPT}" | |
| fi | |
| if [[ -n "${EVAL_SETUP_SCRIPT}" ]]; then | |
| if [[ ! -f "${EVAL_SETUP_SCRIPT}" ]]; then | |
| echo "ERROR: EVAL_SETUP_SCRIPT is set but not found: ${EVAL_SETUP_SCRIPT}" >&2 | |
| exit 1 | |
| fi | |
| echo "" | |
| echo "=== Running setup script: ${EVAL_SETUP_SCRIPT} ===" | |
| bash "${EVAL_SETUP_SCRIPT}" | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.sh`
around lines 29 - 33, The current conditional around EVAL_SETUP_SCRIPT silently
skips when the variable is non-empty but the path is not a file; update the
check in the openshift-claude-agent-eval-commands.sh script so that if
EVAL_SETUP_SCRIPT is set (non-empty) but [[ -f "${EVAL_SETUP_SCRIPT}" ]] is
false the script prints a clear error message and exits non-zero (e.g., use echo
"ERROR: EVAL_SETUP_SCRIPT set but file not found: ${EVAL_SETUP_SCRIPT}" >&2;
exit 1); keep the existing branch that runs bash "${EVAL_SETUP_SCRIPT}" when the
file exists and unchanged behavior when EVAL_SETUP_SCRIPT is empty.
f1094fd to
06026b8
Compare
- Run payload_snapshot.py deterministically before invoking Claude, shifting data gathering from token-burning agent orchestration to a zero-cost Python script - Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir - Add schema validation retry loop for YAML/JSON outputs (up to 3 attempts) - Remove Jira integration and revert staging (now handled by Chai Bot) - Add snapshot phase to JUnit tracking Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
06026b8 to
ae2fbd5
Compare
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nting - Fail the step (exit 1) when validators are missing or outputs are still invalid after 3 retries - Move PHASE_ANALYSIS_DURATION finalization after the retry loop so nudge + validation time is included - Increment PHASE_COUNT when recovery nudge testcase is appended Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
a6e2a5b to
ab3861b
Compare
To be removed before merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ab3861b to
9a7588a
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
ci-operator/config/openshift/release/openshift-release-main__temp.yaml (1)
1-5: ⚡ Quick winResource requests are significantly lower than the step definition.
The file-level resource requests (100m CPU, 200Mi memory) are much lower than the step-level requests defined in
openshift-claude-payload-agent-ref.yaml(1000m CPU, 2Gi memory, 12h timeout). While step-level resources should take precedence, this mismatch may cause confusion. Consider either removing the file-level resources block (if the step resources are sufficient) or aligning them with the expected workload profile.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/config/openshift/release/openshift-release-main__temp.yaml` around lines 1 - 5, The file-level resources block currently sets requests.cpu to 100m and requests.memory to 200Mi which conflicts with the step-level resources (1000m CPU, 2Gi memory, 12h timeout) declared for the agent; either remove the top-level resources block or update the resources.'*'.requests values to match the expected workload (cpu: 1000m, memory: 2Gi) so step/resource expectations are consistent with the values used in openshift-claude-payload-agent-ref.yaml; make the change in the resources -> '*' -> requests keys.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ci-operator/config/openshift/release/openshift-release-main__temp.yaml`:
- Around line 1-17: The file name openshift-release-main__temp.yaml violates the
required pattern; rename the file to follow
openshift-release-main__<stream>-<version>.yaml (for example
openshift-release-main__nightly-4.22.yaml or
openshift-release-main__ci-4.22.yaml) and ensure any internal metadata that
references the variant or stream (e.g., zz_generated_metadata.variant or any
consumers expecting the stream) is updated accordingly so the periodic job
(tests: as: claude-payload-agent) is discovered under the correct stream/version
naming convention.
---
Nitpick comments:
In `@ci-operator/config/openshift/release/openshift-release-main__temp.yaml`:
- Around line 1-5: The file-level resources block currently sets requests.cpu to
100m and requests.memory to 200Mi which conflicts with the step-level resources
(1000m CPU, 2Gi memory, 12h timeout) declared for the agent; either remove the
top-level resources block or update the resources.'*'.requests values to match
the expected workload (cpu: 1000m, memory: 2Gi) so step/resource expectations
are consistent with the values used in openshift-claude-payload-agent-ref.yaml;
make the change in the resources -> '*' -> requests keys.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 6e50cc7d-34aa-4a4b-b917-66f3056bd6ca
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/release/openshift-release-main-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (1)
ci-operator/config/openshift/release/openshift-release-main__temp.yaml
Replace hypershift-team-claude-prow with sa-claude-openshift-ci and update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
/pj-rehearse periodic-ci-openshift-release-main-temp-claude-payload-agent |
|
@stbenjam: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
- Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and validate.py instead of searching /home/claude/.claude - Archive snapshot tarball to ARTIFACT_DIR right after creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
/pj-rehearse periodic-ci-openshift-release-main-temp-claude-payload-agent |
|
@stbenjam: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
The snapshot script creates a nested directory structure (e.g. 4.22/nightly/...) not a flat directory named by the payload tag. Archive the entire snapshot directory contents instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…hot-dir - Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py - Archive snapshot tarball to ARTIFACT_DIR immediately after creation - Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir - Move copy_reports trap before snapshot so artifacts are captured on failure - Handle transient release API failures in the poll loop gracefully - Switch test payload to 4.22.0-0.nightly-2026-06-02-214116 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The skill expects --snapshot-dir to be the directory with summary.json, which is nested under version/stream (e.g. snapshot/4.22/nightly/), not the root snapshot directory. Find summary.json after snapshot creation and pass its parent directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
/pj-rehearse periodic-ci-openshift-release-main-temp-claude-payload-agent |
|
@stbenjam: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@stbenjam, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
Testing complete — the payload agent works correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
/hold cancel |
|
[REHEARSALNOTIFIER] Note: If this PR includes changes to step registry files ( Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
1 similar comment
|
[REHEARSALNOTIFIER] Note: If this PR includes changes to step registry files ( Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
/cc |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: petr-muller, stbenjam The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
* Switch payload agent to snapshot-based payload-analysis skill - Run payload_snapshot.py deterministically before invoking Claude, shifting data gathering from token-burning agent orchestration to a zero-cost Python script - Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir - Add schema validation retry loop for YAML/JSON outputs (up to 3 attempts) - Remove Jira integration and revert staging (now handled by Chai Bot) - Add snapshot phase to JUnit tracking Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use validation scripts from ai-helpers for output schema checks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting - Fail the step (exit 1) when validators are missing or outputs are still invalid after 3 retries - Move PHASE_ANALYSIS_DURATION finalization after the retry loop so nudge + validation time is included - Increment PHASE_COUNT when recovery nudge testcase is appended Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add temporary payload agent job for 4.22 nightly 2026-06-01-074812 To be removed before merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Switch payload agent to sa-claude-openshift-ci credential Replace hypershift-team-claude-prow with sa-claude-openshift-ci and update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix script paths and archive snapshot to artifacts - Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and validate.py instead of searching /home/claude/.claude - Archive snapshot tarball to ARTIFACT_DIR right after creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix snapshot archive path The snapshot script creates a nested directory structure (e.g. 4.22/nightly/...) not a flat directory named by the payload tag. Archive the entire snapshot directory contents instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir - Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py - Archive snapshot tarball to ARTIFACT_DIR immediately after creation - Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir - Move copy_reports trap before snapshot so artifacts are captured on failure - Handle transient release API failures in the poll loop gracefully - Switch test payload to 4.22.0-0.nightly-2026-06-02-214116 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix --snapshot-dir to point to directory containing summary.json The skill expects --snapshot-dir to be the directory with summary.json, which is nested under version/stream (e.g. snapshot/4.22/nightly/), not the root snapshot directory. Find summary.json after snapshot creation and pass its parent directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix validation filenames to use PAYLOAD_TAG directly SANITIZED_TAG replaced dots with dashes, but Claude writes output files using the original tag with dots. The validate scripts were looking for the wrong filenames and always failing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Find validation files by prefix instead of exact name Claude may write filenames with slightly different formatting. Find by glob prefix instead of constructing the exact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove temp payload agent job Testing complete — the payload agent works correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Change "Agent:" to "Model:" in Slack messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Switch payload agent to snapshot-based payload-analysis skill - Run payload_snapshot.py deterministically before invoking Claude, shifting data gathering from token-burning agent orchestration to a zero-cost Python script - Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir - Add schema validation retry loop for YAML/JSON outputs (up to 3 attempts) - Remove Jira integration and revert staging (now handled by Chai Bot) - Add snapshot phase to JUnit tracking Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use validation scripts from ai-helpers for output schema checks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting - Fail the step (exit 1) when validators are missing or outputs are still invalid after 3 retries - Move PHASE_ANALYSIS_DURATION finalization after the retry loop so nudge + validation time is included - Increment PHASE_COUNT when recovery nudge testcase is appended Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add temporary payload agent job for 4.22 nightly 2026-06-01-074812 To be removed before merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Switch payload agent to sa-claude-openshift-ci credential Replace hypershift-team-claude-prow with sa-claude-openshift-ci and update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix script paths and archive snapshot to artifacts - Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and validate.py instead of searching /home/claude/.claude - Archive snapshot tarball to ARTIFACT_DIR right after creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix snapshot archive path The snapshot script creates a nested directory structure (e.g. 4.22/nightly/...) not a flat directory named by the payload tag. Archive the entire snapshot directory contents instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir - Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py - Archive snapshot tarball to ARTIFACT_DIR immediately after creation - Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir - Move copy_reports trap before snapshot so artifacts are captured on failure - Handle transient release API failures in the poll loop gracefully - Switch test payload to 4.22.0-0.nightly-2026-06-02-214116 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix --snapshot-dir to point to directory containing summary.json The skill expects --snapshot-dir to be the directory with summary.json, which is nested under version/stream (e.g. snapshot/4.22/nightly/), not the root snapshot directory. Find summary.json after snapshot creation and pass its parent directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix validation filenames to use PAYLOAD_TAG directly SANITIZED_TAG replaced dots with dashes, but Claude writes output files using the original tag with dots. The validate scripts were looking for the wrong filenames and always failing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Find validation files by prefix instead of exact name Claude may write filenames with slightly different formatting. Find by glob prefix instead of constructing the exact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove temp payload agent job Testing complete — the payload agent works correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Change "Agent:" to "Model:" in Slack messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Switch payload agent to snapshot-based payload-analysis skill - Run payload_snapshot.py deterministically before invoking Claude, shifting data gathering from token-burning agent orchestration to a zero-cost Python script - Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir - Add schema validation retry loop for YAML/JSON outputs (up to 3 attempts) - Remove Jira integration and revert staging (now handled by Chai Bot) - Add snapshot phase to JUnit tracking Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use validation scripts from ai-helpers for output schema checks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting - Fail the step (exit 1) when validators are missing or outputs are still invalid after 3 retries - Move PHASE_ANALYSIS_DURATION finalization after the retry loop so nudge + validation time is included - Increment PHASE_COUNT when recovery nudge testcase is appended Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add temporary payload agent job for 4.22 nightly 2026-06-01-074812 To be removed before merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Switch payload agent to sa-claude-openshift-ci credential Replace hypershift-team-claude-prow with sa-claude-openshift-ci and update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix script paths and archive snapshot to artifacts - Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and validate.py instead of searching /home/claude/.claude - Archive snapshot tarball to ARTIFACT_DIR right after creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix snapshot archive path The snapshot script creates a nested directory structure (e.g. 4.22/nightly/...) not a flat directory named by the payload tag. Archive the entire snapshot directory contents instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir - Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py - Archive snapshot tarball to ARTIFACT_DIR immediately after creation - Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir - Move copy_reports trap before snapshot so artifacts are captured on failure - Handle transient release API failures in the poll loop gracefully - Switch test payload to 4.22.0-0.nightly-2026-06-02-214116 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix --snapshot-dir to point to directory containing summary.json The skill expects --snapshot-dir to be the directory with summary.json, which is nested under version/stream (e.g. snapshot/4.22/nightly/), not the root snapshot directory. Find summary.json after snapshot creation and pass its parent directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix validation filenames to use PAYLOAD_TAG directly SANITIZED_TAG replaced dots with dashes, but Claude writes output files using the original tag with dots. The validate scripts were looking for the wrong filenames and always failing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Find validation files by prefix instead of exact name Claude may write filenames with slightly different formatting. Find by glob prefix instead of constructing the exact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove temp payload agent job Testing complete — the payload agent works correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Change "Agent:" to "Model:" in Slack messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Switch payload agent to snapshot-based payload-analysis skill - Run payload_snapshot.py deterministically before invoking Claude, shifting data gathering from token-burning agent orchestration to a zero-cost Python script - Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir - Add schema validation retry loop for YAML/JSON outputs (up to 3 attempts) - Remove Jira integration and revert staging (now handled by Chai Bot) - Add snapshot phase to JUnit tracking Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use validation scripts from ai-helpers for output schema checks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting - Fail the step (exit 1) when validators are missing or outputs are still invalid after 3 retries - Move PHASE_ANALYSIS_DURATION finalization after the retry loop so nudge + validation time is included - Increment PHASE_COUNT when recovery nudge testcase is appended Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add temporary payload agent job for 4.22 nightly 2026-06-01-074812 To be removed before merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Switch payload agent to sa-claude-openshift-ci credential Replace hypershift-team-claude-prow with sa-claude-openshift-ci and update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix script paths and archive snapshot to artifacts - Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and validate.py instead of searching /home/claude/.claude - Archive snapshot tarball to ARTIFACT_DIR right after creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix snapshot archive path The snapshot script creates a nested directory structure (e.g. 4.22/nightly/...) not a flat directory named by the payload tag. Archive the entire snapshot directory contents instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir - Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py - Archive snapshot tarball to ARTIFACT_DIR immediately after creation - Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir - Move copy_reports trap before snapshot so artifacts are captured on failure - Handle transient release API failures in the poll loop gracefully - Switch test payload to 4.22.0-0.nightly-2026-06-02-214116 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix --snapshot-dir to point to directory containing summary.json The skill expects --snapshot-dir to be the directory with summary.json, which is nested under version/stream (e.g. snapshot/4.22/nightly/), not the root snapshot directory. Find summary.json after snapshot creation and pass its parent directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix validation filenames to use PAYLOAD_TAG directly SANITIZED_TAG replaced dots with dashes, but Claude writes output files using the original tag with dots. The validate scripts were looking for the wrong filenames and always failing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Find validation files by prefix instead of exact name Claude may write filenames with slightly different formatting. Find by glob prefix instead of constructing the exact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove temp payload agent job Testing complete — the payload agent works correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Change "Agent:" to "Model:" in Slack messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Switch payload agent to snapshot-based payload-analysis skill - Run payload_snapshot.py deterministically before invoking Claude, shifting data gathering from token-burning agent orchestration to a zero-cost Python script - Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir - Add schema validation retry loop for YAML/JSON outputs (up to 3 attempts) - Remove Jira integration and revert staging (now handled by Chai Bot) - Add snapshot phase to JUnit tracking Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use validation scripts from ai-helpers for output schema checks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting - Fail the step (exit 1) when validators are missing or outputs are still invalid after 3 retries - Move PHASE_ANALYSIS_DURATION finalization after the retry loop so nudge + validation time is included - Increment PHASE_COUNT when recovery nudge testcase is appended Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add temporary payload agent job for 4.22 nightly 2026-06-01-074812 To be removed before merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Switch payload agent to sa-claude-openshift-ci credential Replace hypershift-team-claude-prow with sa-claude-openshift-ci and update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix script paths and archive snapshot to artifacts - Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and validate.py instead of searching /home/claude/.claude - Archive snapshot tarball to ARTIFACT_DIR right after creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix snapshot archive path The snapshot script creates a nested directory structure (e.g. 4.22/nightly/...) not a flat directory named by the payload tag. Archive the entire snapshot directory contents instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir - Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py - Archive snapshot tarball to ARTIFACT_DIR immediately after creation - Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir - Move copy_reports trap before snapshot so artifacts are captured on failure - Handle transient release API failures in the poll loop gracefully - Switch test payload to 4.22.0-0.nightly-2026-06-02-214116 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix --snapshot-dir to point to directory containing summary.json The skill expects --snapshot-dir to be the directory with summary.json, which is nested under version/stream (e.g. snapshot/4.22/nightly/), not the root snapshot directory. Find summary.json after snapshot creation and pass its parent directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix validation filenames to use PAYLOAD_TAG directly SANITIZED_TAG replaced dots with dashes, but Claude writes output files using the original tag with dots. The validate scripts were looking for the wrong filenames and always failing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Find validation files by prefix instead of exact name Claude may write filenames with slightly different formatting. Find by glob prefix instead of constructing the exact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove temp payload agent job Testing complete — the payload agent works correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Change "Agent:" to "Model:" in Slack messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Switch payload agent to snapshot-based payload-analysis skill - Run payload_snapshot.py deterministically before invoking Claude, shifting data gathering from token-burning agent orchestration to a zero-cost Python script - Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir - Add schema validation retry loop for YAML/JSON outputs (up to 3 attempts) - Remove Jira integration and revert staging (now handled by Chai Bot) - Add snapshot phase to JUnit tracking Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use validation scripts from ai-helpers for output schema checks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting - Fail the step (exit 1) when validators are missing or outputs are still invalid after 3 retries - Move PHASE_ANALYSIS_DURATION finalization after the retry loop so nudge + validation time is included - Increment PHASE_COUNT when recovery nudge testcase is appended Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add temporary payload agent job for 4.22 nightly 2026-06-01-074812 To be removed before merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Switch payload agent to sa-claude-openshift-ci credential Replace hypershift-team-claude-prow with sa-claude-openshift-ci and update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix script paths and archive snapshot to artifacts - Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and validate.py instead of searching /home/claude/.claude - Archive snapshot tarball to ARTIFACT_DIR right after creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix snapshot archive path The snapshot script creates a nested directory structure (e.g. 4.22/nightly/...) not a flat directory named by the payload tag. Archive the entire snapshot directory contents instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir - Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py - Archive snapshot tarball to ARTIFACT_DIR immediately after creation - Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir - Move copy_reports trap before snapshot so artifacts are captured on failure - Handle transient release API failures in the poll loop gracefully - Switch test payload to 4.22.0-0.nightly-2026-06-02-214116 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix --snapshot-dir to point to directory containing summary.json The skill expects --snapshot-dir to be the directory with summary.json, which is nested under version/stream (e.g. snapshot/4.22/nightly/), not the root snapshot directory. Find summary.json after snapshot creation and pass its parent directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix validation filenames to use PAYLOAD_TAG directly SANITIZED_TAG replaced dots with dashes, but Claude writes output files using the original tag with dots. The validate scripts were looking for the wrong filenames and always failing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Find validation files by prefix instead of exact name Claude may write filenames with slightly different formatting. Find by glob prefix instead of constructing the exact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove temp payload agent job Testing complete — the payload agent works correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Change "Agent:" to "Model:" in Slack messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Switch payload agent to snapshot-based payload-analysis skill - Run payload_snapshot.py deterministically before invoking Claude, shifting data gathering from token-burning agent orchestration to a zero-cost Python script - Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir - Add schema validation retry loop for YAML/JSON outputs (up to 3 attempts) - Remove Jira integration and revert staging (now handled by Chai Bot) - Add snapshot phase to JUnit tracking Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use validation scripts from ai-helpers for output schema checks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting - Fail the step (exit 1) when validators are missing or outputs are still invalid after 3 retries - Move PHASE_ANALYSIS_DURATION finalization after the retry loop so nudge + validation time is included - Increment PHASE_COUNT when recovery nudge testcase is appended Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add temporary payload agent job for 4.22 nightly 2026-06-01-074812 To be removed before merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Switch payload agent to sa-claude-openshift-ci credential Replace hypershift-team-claude-prow with sa-claude-openshift-ci and update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix script paths and archive snapshot to artifacts - Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and validate.py instead of searching /home/claude/.claude - Archive snapshot tarball to ARTIFACT_DIR right after creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix snapshot archive path The snapshot script creates a nested directory structure (e.g. 4.22/nightly/...) not a flat directory named by the payload tag. Archive the entire snapshot directory contents instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir - Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py - Archive snapshot tarball to ARTIFACT_DIR immediately after creation - Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir - Move copy_reports trap before snapshot so artifacts are captured on failure - Handle transient release API failures in the poll loop gracefully - Switch test payload to 4.22.0-0.nightly-2026-06-02-214116 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix --snapshot-dir to point to directory containing summary.json The skill expects --snapshot-dir to be the directory with summary.json, which is nested under version/stream (e.g. snapshot/4.22/nightly/), not the root snapshot directory. Find summary.json after snapshot creation and pass its parent directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix validation filenames to use PAYLOAD_TAG directly SANITIZED_TAG replaced dots with dashes, but Claude writes output files using the original tag with dots. The validate scripts were looking for the wrong filenames and always failing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Find validation files by prefix instead of exact name Claude may write filenames with slightly different formatting. Find by glob prefix instead of constructing the exact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove temp payload agent job Testing complete — the payload agent works correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Change "Agent:" to "Model:" in Slack messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Switch payload agent to snapshot-based payload-analysis skill - Run payload_snapshot.py deterministically before invoking Claude, shifting data gathering from token-burning agent orchestration to a zero-cost Python script - Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir - Add schema validation retry loop for YAML/JSON outputs (up to 3 attempts) - Remove Jira integration and revert staging (now handled by Chai Bot) - Add snapshot phase to JUnit tracking Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use validation scripts from ai-helpers for output schema checks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting - Fail the step (exit 1) when validators are missing or outputs are still invalid after 3 retries - Move PHASE_ANALYSIS_DURATION finalization after the retry loop so nudge + validation time is included - Increment PHASE_COUNT when recovery nudge testcase is appended Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add temporary payload agent job for 4.22 nightly 2026-06-01-074812 To be removed before merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Switch payload agent to sa-claude-openshift-ci credential Replace hypershift-team-claude-prow with sa-claude-openshift-ci and update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix script paths and archive snapshot to artifacts - Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and validate.py instead of searching /home/claude/.claude - Archive snapshot tarball to ARTIFACT_DIR right after creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix snapshot archive path The snapshot script creates a nested directory structure (e.g. 4.22/nightly/...) not a flat directory named by the payload tag. Archive the entire snapshot directory contents instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir - Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py - Archive snapshot tarball to ARTIFACT_DIR immediately after creation - Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir - Move copy_reports trap before snapshot so artifacts are captured on failure - Handle transient release API failures in the poll loop gracefully - Switch test payload to 4.22.0-0.nightly-2026-06-02-214116 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix --snapshot-dir to point to directory containing summary.json The skill expects --snapshot-dir to be the directory with summary.json, which is nested under version/stream (e.g. snapshot/4.22/nightly/), not the root snapshot directory. Find summary.json after snapshot creation and pass its parent directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix validation filenames to use PAYLOAD_TAG directly SANITIZED_TAG replaced dots with dashes, but Claude writes output files using the original tag with dots. The validate scripts were looking for the wrong filenames and always failing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Find validation files by prefix instead of exact name Claude may write filenames with slightly different formatting. Find by glob prefix instead of constructing the exact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove temp payload agent job Testing complete — the payload agent works correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Change "Agent:" to "Model:" in Slack messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Switch payload agent to snapshot-based payload-analysis skill - Run payload_snapshot.py deterministically before invoking Claude, shifting data gathering from token-burning agent orchestration to a zero-cost Python script - Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir - Add schema validation retry loop for YAML/JSON outputs (up to 3 attempts) - Remove Jira integration and revert staging (now handled by Chai Bot) - Add snapshot phase to JUnit tracking Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use validation scripts from ai-helpers for output schema checks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting - Fail the step (exit 1) when validators are missing or outputs are still invalid after 3 retries - Move PHASE_ANALYSIS_DURATION finalization after the retry loop so nudge + validation time is included - Increment PHASE_COUNT when recovery nudge testcase is appended Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add temporary payload agent job for 4.22 nightly 2026-06-01-074812 To be removed before merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Switch payload agent to sa-claude-openshift-ci credential Replace hypershift-team-claude-prow with sa-claude-openshift-ci and update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix script paths and archive snapshot to artifacts - Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and validate.py instead of searching /home/claude/.claude - Archive snapshot tarball to ARTIFACT_DIR right after creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix snapshot archive path The snapshot script creates a nested directory structure (e.g. 4.22/nightly/...) not a flat directory named by the payload tag. Archive the entire snapshot directory contents instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir - Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py - Archive snapshot tarball to ARTIFACT_DIR immediately after creation - Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir - Move copy_reports trap before snapshot so artifacts are captured on failure - Handle transient release API failures in the poll loop gracefully - Switch test payload to 4.22.0-0.nightly-2026-06-02-214116 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix --snapshot-dir to point to directory containing summary.json The skill expects --snapshot-dir to be the directory with summary.json, which is nested under version/stream (e.g. snapshot/4.22/nightly/), not the root snapshot directory. Find summary.json after snapshot creation and pass its parent directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix validation filenames to use PAYLOAD_TAG directly SANITIZED_TAG replaced dots with dashes, but Claude writes output files using the original tag with dots. The validate scripts were looking for the wrong filenames and always failing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Find validation files by prefix instead of exact name Claude may write filenames with slightly different formatting. Find by glob prefix instead of constructing the exact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove temp payload agent job Testing complete — the payload agent works correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Change "Agent:" to "Model:" in Slack messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Rehearsal: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/79922/rehearse-79922-periodic-ci-openshift-release-main-temp-claude-payload-agent/2062158405392404480
--
Depends on openshift-eng/ai-helpers#511
Summary by CodeRabbit
This PR refactors the OpenShift payload agent to focus exclusively on payload analysis, removing Jira integration and auto-revert functionality that have been delegated to the Chai bot.
Practical Changes to CI Infrastructure:
The payload agent workflow is being simplified and made more deterministic:
ENABLE_PAYLOAD_REVERTconfiguration have been stripped from the step definition since revert coordination is now handled by Chaipayload-results-*.yamlandpayload-analysis-*-autodl.json), with automatic retries (up to 3 times) if outputs are missing or invalidNew Test Infrastructure:
A new scheduled test workflow (
claude-payload-agent) has been added to the temporary release CI configuration, running yearly to evaluate payload analysis capabilities with Opus model. This enables ongoing assessment of the snapshot-based approach.Impact: These changes streamline the payload analysis pipeline by removing cross-system concerns and improving reliability through deterministic snapshots and output validation.