Skip to content

Improve the payload agent - #79922

Merged
openshift-merge-bot[bot] merged 14 commits into
openshift:mainfrom
not-stbenjam:payload-updates
Jun 3, 2026
Merged

openshift-merge-bot[bot] merged 14 commits into
openshift:mainfrom
not-stbenjam:payload-updates

Conversation

@stbenjam

@stbenjam stbenjam commented Jun 1, 2026

Copy link
Copy Markdown
Member
  • Strips out Jira and auto-reverts, this has moved to Chai bot, we only do the analysis here
  • Uses the new snapshot-based approach, with improved results
  • Enforces proper schema generation

Rehearsal: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/79922/rehearse-79922-periodic-ci-openshift-release-main-temp-claude-payload-agent/2062158405392404480

--

Depends on openshift-eng/ai-helpers#511

Summary by CodeRabbit

This PR refactors the OpenShift payload agent to focus exclusively on payload analysis, removing Jira integration and auto-revert functionality that have been delegated to the Chai bot.

Practical Changes to CI Infrastructure:

The payload agent workflow is being simplified and made more deterministic:

  • Snapshot-based analysis: The agent now generates a deterministic payload snapshot before invoking Claude for analysis, replacing the previous direct analysis approach
  • Removed Jira integration: All Jira credential handling (API token, username, base URL) and the ENABLE_PAYLOAD_REVERT configuration have been stripped from the step definition since revert coordination is now handled by Chai
  • Structured output validation: The agent now validates that analysis produces properly formatted structured outputs (payload-results-*.yaml and payload-analysis-*-autodl.json), with automatic retries (up to 3 times) if outputs are missing or invalid
  • Schema enforcement: The system prompt now enforces that Skill tools be explicitly loaded and that CI skills generate canonical structured outputs rather than writing artifacts directly

New Test Infrastructure:

A new scheduled test workflow (claude-payload-agent) has been added to the temporary release CI configuration, running yearly to evaluate payload analysis capabilities with Opus model. This enables ongoing assessment of the snapshot-based approach.

Impact: These changes streamline the payload analysis pipeline by removing cross-system concerns and improving reliability through deterministic snapshots and output validation.

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 1, 2026
@coderabbitai

coderabbitai Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

This PR refactors the OpenShift Claude payload agent by removing Jira-dependent revert infrastructure, introducing deterministic payload snapshot generation, updating Claude's analysis entrypoint with system prompt changes requiring Skill-based tool loading and structured output validation with retry logic, and configuring a new scheduled test workflow.

Changes

Claude Payload Agent Infrastructure Refactor

Layer / File(s) Summary
Remove Jira credential and revert infrastructure
ci-operator/step-registry/openshift/claude/payload/agent/openshift-claude-payload-agent-ref.yaml, ci-operator/step-registry/openshift/claude/payload/agent/openshift-claude-payload-agent-commands.sh
Jira token mount, environment variables, credential-loading logic, automatic-reverts logging, and documentation text describing optional revert staging are all removed.
Payload snapshot generation and Claude analysis entrypoint update
ci-operator/step-registry/openshift/claude/payload/agent/openshift-claude-payload-agent-commands.sh
Deterministic payload snapshot phase locates and runs payload_snapshot.py before Claude analysis. System prompt is updated to require explicit Skill tool loading and structured output generation. Claude invocation switches to payload-analysis entrypoint with --snapshot-dir parameter.
Structured output validation and retry mechanism
ci-operator/step-registry/openshift/claude/payload/agent/openshift-claude-payload-agent-commands.sh
Validation via validate.py runs after Claude with up to three retry attempts by re-invoking Claude if payload-results-*.yaml or payload-analysis-*-autodl.json are missing or invalid. JUnit phase-case construction aligns with new phase sequence.
Phase count, Slack reporting, and CI test configuration
ci-operator/step-registry/openshift/claude/payload/agent/openshift-claude-payload-agent-commands.sh, ci-operator/config/openshift/release/openshift-release-main__temp.yaml
Phase count set to 3 unconditionally (incremented on timeout). Slack summary prompt removes revert PR URL instruction. New temporary OpenShift release YAML configuration adds scheduled test workflow with resource defaults and explicit PAYLOAD_TAG.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested labels

lgtm

Suggested reviewers

  • sosiouxme
  • dgoodwin
🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Title check ❓ Inconclusive The title 'Improve the payload agent' is vague and generic, using non-descriptive language that doesn't convey the specific nature of the changes. Consider a more specific title like 'Remove Jira integration and adopt snapshot-based payload analysis' or 'Refactor payload agent to analysis-only mode with snapshot support'.
✅ Passed checks (14 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR contains no Ginkgo test files or test name definitions. Changes are to shell scripts and YAML configuration files only, so the check for stable/deterministic Ginkgo test names is not applicable.
Test Structure And Quality ✅ Passed The custom check requires reviewing Ginkgo test code, but the PR contains no Go test files—only bash scripts and YAML configuration files. The check is not applicable.
Microshift Test Compatibility ✅ Passed No new Ginkgo e2e tests are added in this PR. Changes are CI/CD configuration and shell scripts only, not test code.
Single Node Openshift (Sno) Test Compatibility ✅ Passed This PR does not add Ginkgo e2e tests. Only CI scripts, step definitions, and config files are modified. SNO check applies only to new test code.
Topology-Aware Scheduling Compatibility ✅ Passed No scheduling constraints (affinity, anti-affinity, topology spread, nodeSelectors, tolerations, PDBs) are introduced. Changes are CI-operator config and shell scripts only.
Ote Binary Stdout Contract ✅ Passed The PR contains only shell scripts and YAML configuration files; no OTE binary or Go test code is modified, so the OTE Binary Stdout Contract check is not applicable.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR does not add Ginkgo e2e tests. All files are CI configuration and shell scripts, not e2e tests. Check not applicable.
No-Weak-Crypto ✅ Passed No weak cryptography (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB), custom crypto implementations, or non-constant-time secret comparisons found in the PR files.
Container-Privileges ✅ Passed No privileged container configurations found. Modified files contain no privileged: true, hostPID/hostNetwork/hostIPC, SYS_ADMIN capabilities, or allowPrivilegeEscalation settings.
No-Sensitive-Data-In-Logs ✅ Passed Script properly loads secrets from files with xtrace disabled, never logs actual token/webhook values, only logs status messages. No sensitive data exposed.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
openshift-ci Bot requested review from dgoodwin and sosiouxme June 1, 2026 14:40
@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 1, 2026
@openshift-merge-bot openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jun 1, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/openshift/claude/payload/agent/openshift-claude-payload-agent-commands.sh`:
- Around line 243-282: PHASE_ANALYSIS_DURATION is being finalized before the
validation/regeneration loop so retry and timeout time are not included in the
analysis testcase or TOTAL_DURATION and PHASE_COUNT is not adjusted when the
"recovery nudge" testcase is appended; move the finalization of
PHASE_ANALYSIS_DURATION (and the logic that increments
PHASE_COUNT/TOTAL_DURATION) to after the validation/retry loop so all retry
attempts and timeout durations are included, and update the code that appends
the recovery-nudge testcase to increment PHASE_COUNT (and add its duration to
TOTAL_DURATION) when a timeout leads to that testcase being created; locate the
validation/regeneration loop around the python validate calls and the claude
timeout block plus references to PHASE_ANALYSIS_DURATION, PHASE_COUNT,
TOTAL_DURATION and the recovery nudge testcase to apply these changes.
- Around line 243-282: Ensure schema validation failures cause the step to fail:
first check that VALIDATE_YAML and VALIDATE_JSON are non-empty and executable
(test -x or -f) and immediately echo an error and exit 1 if either validator is
missing; inside the retry loop keep the YAML_OK/JSON_OK checks but on the final
attempt where both are not true (the branch currently echoing the warning),
replace the echo+break with an error message and exit 1 so the job fails; also
ensure when python validation commands themselves fail to run (e.g., non-zero
exit) they are treated as validation failures (leave current pattern of setting
YAML_OK/JSON_OK only on success) so the overall behavior enforces the schema
contract.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: af512c6b-7ec4-403c-8a53-6d022b8d9efb

📥 Commits

Reviewing files that changed from the base of the PR and between e9cca1a and cde35b1.

📒 Files selected for processing (2)
  • ci-operator/step-registry/openshift/claude/payload/agent/openshift-claude-payload-agent-commands.sh
  • ci-operator/step-registry/openshift/claude/payload/agent/openshift-claude-payload-agent-ref.yaml

@openshift-merge-bot openshift-merge-bot Bot removed the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jun 1, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/config/openshift-eng/ai-helpers/openshift-eng-ai-helpers-main.yaml`:
- Around line 29-31: The job payload-analysis-eval-presubmit's run_if_changed
regex misses files under evals/scripts (so changes to EVAL_SETUP_SCRIPT won't
trigger the job); update the run_if_changed pattern for
payload-analysis-eval-presubmit to include the evals/scripts/ path (e.g., allow
an optional scripts/ segment like
evals/(scripts/)?(eval-payload-analysis|cases/payload-analysis|snapshots)) so
that files such as
plugins/ci/evals/scripts/extract-payload-analysis-snapshots.sh are matched.

In
`@ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.sh`:
- Around line 29-33: The current conditional around EVAL_SETUP_SCRIPT silently
skips when the variable is non-empty but the path is not a file; update the
check in the openshift-claude-agent-eval-commands.sh script so that if
EVAL_SETUP_SCRIPT is set (non-empty) but [[ -f "${EVAL_SETUP_SCRIPT}" ]] is
false the script prints a clear error message and exits non-zero (e.g., use echo
"ERROR: EVAL_SETUP_SCRIPT set but file not found: ${EVAL_SETUP_SCRIPT}" >&2;
exit 1); keep the existing branch that runs bash "${EVAL_SETUP_SCRIPT}" when the
file exists and unchanged behavior when EVAL_SETUP_SCRIPT is empty.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: e0167284-52bd-4985-ac87-d72138859361

📥 Commits

Reviewing files that changed from the base of the PR and between 29dde5a and f1094fd.

⛔ Files ignored due to path filters (2)
  • ci-operator/jobs/openshift-eng/ai-helpers/openshift-eng-ai-helpers-main-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift-eng/ai-helpers/openshift-eng-ai-helpers-main-presubmits.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (7)
  • ci-operator/config/openshift-eng/ai-helpers/openshift-eng-ai-helpers-main.yaml
  • ci-operator/step-registry/openshift/claude/agent-eval/OWNERS
  • ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.sh
  • ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-ref.metadata.json
  • ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-ref.yaml
  • ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-workflow.metadata.json
  • ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-workflow.yaml
✅ Files skipped from review due to trivial changes (3)
  • ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-ref.metadata.json
  • ci-operator/step-registry/openshift/claude/agent-eval/OWNERS
  • ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-workflow.metadata.json

Comment thread ci-operator/config/openshift-eng/ai-helpers/openshift-eng-ai-helpers-main.yaml Outdated
Comment on lines +29 to +33
if [[ -n "${EVAL_SETUP_SCRIPT}" ]] && [[ -f "${EVAL_SETUP_SCRIPT}" ]]; then
echo ""
echo "=== Running setup script: ${EVAL_SETUP_SCRIPT} ==="
bash "${EVAL_SETUP_SCRIPT}"
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Fail fast when EVAL_SETUP_SCRIPT is set but missing.

When EVAL_SETUP_SCRIPT is non-empty but the path doesn't resolve to a file, the block silently skips setup. For the payload jobs this means the snapshot extraction never runs and the eval proceeds against missing fixtures, yielding misleading results instead of a clear failure.

🛡️ Proposed fix to fail on a misconfigured setup script path
-if [[ -n "${EVAL_SETUP_SCRIPT}" ]] && [[ -f "${EVAL_SETUP_SCRIPT}" ]]; then
-    echo ""
-    echo "=== Running setup script: ${EVAL_SETUP_SCRIPT} ==="
-    bash "${EVAL_SETUP_SCRIPT}"
-fi
+if [[ -n "${EVAL_SETUP_SCRIPT}" ]]; then
+    if [[ ! -f "${EVAL_SETUP_SCRIPT}" ]]; then
+        echo "ERROR: EVAL_SETUP_SCRIPT is set but not found: ${EVAL_SETUP_SCRIPT}" >&2
+        exit 1
+    fi
+    echo ""
+    echo "=== Running setup script: ${EVAL_SETUP_SCRIPT} ==="
+    bash "${EVAL_SETUP_SCRIPT}"
+fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if [[ -n "${EVAL_SETUP_SCRIPT}" ]] && [[ -f "${EVAL_SETUP_SCRIPT}" ]]; then
echo ""
echo "=== Running setup script: ${EVAL_SETUP_SCRIPT} ==="
bash "${EVAL_SETUP_SCRIPT}"
fi
if [[ -n "${EVAL_SETUP_SCRIPT}" ]]; then
if [[ ! -f "${EVAL_SETUP_SCRIPT}" ]]; then
echo "ERROR: EVAL_SETUP_SCRIPT is set but not found: ${EVAL_SETUP_SCRIPT}" >&2
exit 1
fi
echo ""
echo "=== Running setup script: ${EVAL_SETUP_SCRIPT} ==="
bash "${EVAL_SETUP_SCRIPT}"
fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.sh`
around lines 29 - 33, The current conditional around EVAL_SETUP_SCRIPT silently
skips when the variable is non-empty but the path is not a file; update the
check in the openshift-claude-agent-eval-commands.sh script so that if
EVAL_SETUP_SCRIPT is set (non-empty) but [[ -f "${EVAL_SETUP_SCRIPT}" ]] is
false the script prints a clear error message and exits non-zero (e.g., use echo
"ERROR: EVAL_SETUP_SCRIPT set but file not found: ${EVAL_SETUP_SCRIPT}" >&2;
exit 1); keep the existing branch that runs bash "${EVAL_SETUP_SCRIPT}" when the
file exists and unchanged behavior when EVAL_SETUP_SCRIPT is empty.

- Run payload_snapshot.py deterministically before invoking Claude,
  shifting data gathering from token-burning agent orchestration to
  a zero-cost Python script
- Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir
- Add schema validation retry loop for YAML/JSON outputs (up to 3
  attempts)
- Remove Jira integration and revert staging (now handled by Chai Bot)
- Add snapshot phase to JUnit tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-merge-bot openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jun 2, 2026
not-stbenjam and others added 2 commits June 2, 2026 16:45
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nting

- Fail the step (exit 1) when validators are missing or outputs are
  still invalid after 3 retries
- Move PHASE_ANALYSIS_DURATION finalization after the retry loop so
  nudge + validation time is included
- Increment PHASE_COUNT when recovery nudge testcase is appended

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
To be removed before merge.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-merge-bot openshift-merge-bot Bot removed the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jun 2, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
ci-operator/config/openshift/release/openshift-release-main__temp.yaml (1)

1-5: ⚡ Quick win

Resource requests are significantly lower than the step definition.

The file-level resource requests (100m CPU, 200Mi memory) are much lower than the step-level requests defined in openshift-claude-payload-agent-ref.yaml (1000m CPU, 2Gi memory, 12h timeout). While step-level resources should take precedence, this mismatch may cause confusion. Consider either removing the file-level resources block (if the step resources are sufficient) or aligning them with the expected workload profile.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci-operator/config/openshift/release/openshift-release-main__temp.yaml`
around lines 1 - 5, The file-level resources block currently sets requests.cpu
to 100m and requests.memory to 200Mi which conflicts with the step-level
resources (1000m CPU, 2Gi memory, 12h timeout) declared for the agent; either
remove the top-level resources block or update the resources.'*'.requests values
to match the expected workload (cpu: 1000m, memory: 2Gi) so step/resource
expectations are consistent with the values used in
openshift-claude-payload-agent-ref.yaml; make the change in the resources -> '*'
-> requests keys.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ci-operator/config/openshift/release/openshift-release-main__temp.yaml`:
- Around line 1-17: The file name openshift-release-main__temp.yaml violates the
required pattern; rename the file to follow
openshift-release-main__<stream>-<version>.yaml (for example
openshift-release-main__nightly-4.22.yaml or
openshift-release-main__ci-4.22.yaml) and ensure any internal metadata that
references the variant or stream (e.g., zz_generated_metadata.variant or any
consumers expecting the stream) is updated accordingly so the periodic job
(tests: as: claude-payload-agent) is discovered under the correct stream/version
naming convention.

---

Nitpick comments:
In `@ci-operator/config/openshift/release/openshift-release-main__temp.yaml`:
- Around line 1-5: The file-level resources block currently sets requests.cpu to
100m and requests.memory to 200Mi which conflicts with the step-level resources
(1000m CPU, 2Gi memory, 12h timeout) declared for the agent; either remove the
top-level resources block or update the resources.'*'.requests values to match
the expected workload (cpu: 1000m, memory: 2Gi) so step/resource expectations
are consistent with the values used in openshift-claude-payload-agent-ref.yaml;
make the change in the resources -> '*' -> requests keys.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 6e50cc7d-34aa-4a4b-b917-66f3056bd6ca

📥 Commits

Reviewing files that changed from the base of the PR and between a6e2a5b and 9a7588a.

⛔ Files ignored due to path filters (1)
  • ci-operator/jobs/openshift/release/openshift-release-main-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (1)
  • ci-operator/config/openshift/release/openshift-release-main__temp.yaml

Comment thread ci-operator/config/openshift/release/openshift-release-main__temp.yaml Outdated
Replace hypershift-team-claude-prow with sa-claude-openshift-ci and
update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@stbenjam

stbenjam commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-release-main-temp-claude-payload-agent

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@stbenjam: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

- Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and
  validate.py instead of searching /home/claude/.claude
- Archive snapshot tarball to ARTIFACT_DIR right after creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@stbenjam

stbenjam commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-release-main-temp-claude-payload-agent

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@stbenjam: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

stbenjam and others added 3 commits June 3, 2026 08:03
The snapshot script creates a nested directory structure (e.g.
4.22/nightly/...) not a flat directory named by the payload tag.
Archive the entire snapshot directory contents instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…hot-dir

- Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py
- Archive snapshot tarball to ARTIFACT_DIR immediately after creation
- Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir
- Move copy_reports trap before snapshot so artifacts are captured on failure
- Handle transient release API failures in the poll loop gracefully
- Switch test payload to 4.22.0-0.nightly-2026-06-02-214116

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The skill expects --snapshot-dir to be the directory with summary.json,
which is nested under version/stream (e.g. snapshot/4.22/nightly/), not
the root snapshot directory. Find summary.json after snapshot creation
and pass its parent directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@stbenjam

stbenjam commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-release-main-temp-claude-payload-agent

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@stbenjam: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 3, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@stbenjam, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

couldn't prepare candidate: couldn't rebase candidate onto bc11c16f916d0002d047ce0e4800b8358b8b654c due to conflicts
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

Testing complete — the payload agent works correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 3, 2026
@stbenjam

stbenjam commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

/hold cancel

@openshift-ci openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 3, 2026
@openshift-merge-bot openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jun 3, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@stbenjam: no rehearsable tests are affected by this change

Note: If this PR includes changes to step registry files (ci-operator/step-registry/) and you expected jobs to be found, try rebasing your PR onto the base branch. This helps pj-rehearse accurately detect changes when the base branch has moved forward.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

1 similar comment
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@stbenjam: no rehearsable tests are affected by this change

Note: If this PR includes changes to step registry files (ci-operator/step-registry/) and you expected jobs to be found, try rebasing your PR onto the base branch. This helps pj-rehearse accurately detect changes when the base branch has moved forward.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@petr-muller

Copy link
Copy Markdown
Member

/cc

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 3, 2026
@openshift-ci

openshift-ci Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: petr-muller, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot
openshift-merge-bot Bot merged commit 295c5f1 into openshift:main Jun 3, 2026
11 checks passed
IshwarKanse pushed a commit to IshwarKanse/release that referenced this pull request Jun 4, 2026
* Switch payload agent to snapshot-based payload-analysis skill

- Run payload_snapshot.py deterministically before invoking Claude,
  shifting data gathering from token-burning agent orchestration to
  a zero-cost Python script
- Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir
- Add schema validation retry loop for YAML/JSON outputs (up to 3
  attempts)
- Remove Jira integration and revert staging (now handled by Chai Bot)
- Add snapshot phase to JUnit tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Use validation scripts from ai-helpers for output schema checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting

- Fail the step (exit 1) when validators are missing or outputs are
  still invalid after 3 retries
- Move PHASE_ANALYSIS_DURATION finalization after the retry loop so
  nudge + validation time is included
- Increment PHASE_COUNT when recovery nudge testcase is appended

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add temporary payload agent job for 4.22 nightly 2026-06-01-074812

To be removed before merge.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Switch payload agent to sa-claude-openshift-ci credential

Replace hypershift-team-claude-prow with sa-claude-openshift-ci and
update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix script paths and archive snapshot to artifacts

- Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and
  validate.py instead of searching /home/claude/.claude
- Archive snapshot tarball to ARTIFACT_DIR right after creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix snapshot archive path

The snapshot script creates a nested directory structure (e.g.
4.22/nightly/...) not a flat directory named by the payload tag.
Archive the entire snapshot directory contents instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir

- Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py
- Archive snapshot tarball to ARTIFACT_DIR immediately after creation
- Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir
- Move copy_reports trap before snapshot so artifacts are captured on failure
- Handle transient release API failures in the poll loop gracefully
- Switch test payload to 4.22.0-0.nightly-2026-06-02-214116

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix --snapshot-dir to point to directory containing summary.json

The skill expects --snapshot-dir to be the directory with summary.json,
which is nested under version/stream (e.g. snapshot/4.22/nightly/), not
the root snapshot directory. Find summary.json after snapshot creation
and pass its parent directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix validation filenames to use PAYLOAD_TAG directly

SANITIZED_TAG replaced dots with dashes, but Claude writes output
files using the original tag with dots. The validate scripts were
looking for the wrong filenames and always failing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Find validation files by prefix instead of exact name

Claude may write filenames with slightly different formatting.
Find by glob prefix instead of constructing the exact name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove temp payload agent job

Testing complete — the payload agent works correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Change "Agent:" to "Model:" in Slack messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
fracappa pushed a commit to fracappa/release that referenced this pull request Jun 4, 2026
* Switch payload agent to snapshot-based payload-analysis skill

- Run payload_snapshot.py deterministically before invoking Claude,
  shifting data gathering from token-burning agent orchestration to
  a zero-cost Python script
- Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir
- Add schema validation retry loop for YAML/JSON outputs (up to 3
  attempts)
- Remove Jira integration and revert staging (now handled by Chai Bot)
- Add snapshot phase to JUnit tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Use validation scripts from ai-helpers for output schema checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting

- Fail the step (exit 1) when validators are missing or outputs are
  still invalid after 3 retries
- Move PHASE_ANALYSIS_DURATION finalization after the retry loop so
  nudge + validation time is included
- Increment PHASE_COUNT when recovery nudge testcase is appended

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add temporary payload agent job for 4.22 nightly 2026-06-01-074812

To be removed before merge.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Switch payload agent to sa-claude-openshift-ci credential

Replace hypershift-team-claude-prow with sa-claude-openshift-ci and
update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix script paths and archive snapshot to artifacts

- Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and
  validate.py instead of searching /home/claude/.claude
- Archive snapshot tarball to ARTIFACT_DIR right after creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix snapshot archive path

The snapshot script creates a nested directory structure (e.g.
4.22/nightly/...) not a flat directory named by the payload tag.
Archive the entire snapshot directory contents instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir

- Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py
- Archive snapshot tarball to ARTIFACT_DIR immediately after creation
- Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir
- Move copy_reports trap before snapshot so artifacts are captured on failure
- Handle transient release API failures in the poll loop gracefully
- Switch test payload to 4.22.0-0.nightly-2026-06-02-214116

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix --snapshot-dir to point to directory containing summary.json

The skill expects --snapshot-dir to be the directory with summary.json,
which is nested under version/stream (e.g. snapshot/4.22/nightly/), not
the root snapshot directory. Find summary.json after snapshot creation
and pass its parent directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix validation filenames to use PAYLOAD_TAG directly

SANITIZED_TAG replaced dots with dashes, but Claude writes output
files using the original tag with dots. The validate scripts were
looking for the wrong filenames and always failing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Find validation files by prefix instead of exact name

Claude may write filenames with slightly different formatting.
Find by glob prefix instead of constructing the exact name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove temp payload agent job

Testing complete — the payload agent works correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Change "Agent:" to "Model:" in Slack messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
TimurMP pushed a commit to TimurMP/release that referenced this pull request Jun 4, 2026
* Switch payload agent to snapshot-based payload-analysis skill

- Run payload_snapshot.py deterministically before invoking Claude,
  shifting data gathering from token-burning agent orchestration to
  a zero-cost Python script
- Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir
- Add schema validation retry loop for YAML/JSON outputs (up to 3
  attempts)
- Remove Jira integration and revert staging (now handled by Chai Bot)
- Add snapshot phase to JUnit tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Use validation scripts from ai-helpers for output schema checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting

- Fail the step (exit 1) when validators are missing or outputs are
  still invalid after 3 retries
- Move PHASE_ANALYSIS_DURATION finalization after the retry loop so
  nudge + validation time is included
- Increment PHASE_COUNT when recovery nudge testcase is appended

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add temporary payload agent job for 4.22 nightly 2026-06-01-074812

To be removed before merge.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Switch payload agent to sa-claude-openshift-ci credential

Replace hypershift-team-claude-prow with sa-claude-openshift-ci and
update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix script paths and archive snapshot to artifacts

- Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and
  validate.py instead of searching /home/claude/.claude
- Archive snapshot tarball to ARTIFACT_DIR right after creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix snapshot archive path

The snapshot script creates a nested directory structure (e.g.
4.22/nightly/...) not a flat directory named by the payload tag.
Archive the entire snapshot directory contents instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir

- Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py
- Archive snapshot tarball to ARTIFACT_DIR immediately after creation
- Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir
- Move copy_reports trap before snapshot so artifacts are captured on failure
- Handle transient release API failures in the poll loop gracefully
- Switch test payload to 4.22.0-0.nightly-2026-06-02-214116

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix --snapshot-dir to point to directory containing summary.json

The skill expects --snapshot-dir to be the directory with summary.json,
which is nested under version/stream (e.g. snapshot/4.22/nightly/), not
the root snapshot directory. Find summary.json after snapshot creation
and pass its parent directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix validation filenames to use PAYLOAD_TAG directly

SANITIZED_TAG replaced dots with dashes, but Claude writes output
files using the original tag with dots. The validate scripts were
looking for the wrong filenames and always failing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Find validation files by prefix instead of exact name

Claude may write filenames with slightly different formatting.
Find by glob prefix instead of constructing the exact name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove temp payload agent job

Testing complete — the payload agent works correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Change "Agent:" to "Model:" in Slack messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
krisnababu pushed a commit to oharan2/release that referenced this pull request Jul 3, 2026
* Switch payload agent to snapshot-based payload-analysis skill

- Run payload_snapshot.py deterministically before invoking Claude,
  shifting data gathering from token-burning agent orchestration to
  a zero-cost Python script
- Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir
- Add schema validation retry loop for YAML/JSON outputs (up to 3
  attempts)
- Remove Jira integration and revert staging (now handled by Chai Bot)
- Add snapshot phase to JUnit tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Use validation scripts from ai-helpers for output schema checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting

- Fail the step (exit 1) when validators are missing or outputs are
  still invalid after 3 retries
- Move PHASE_ANALYSIS_DURATION finalization after the retry loop so
  nudge + validation time is included
- Increment PHASE_COUNT when recovery nudge testcase is appended

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add temporary payload agent job for 4.22 nightly 2026-06-01-074812

To be removed before merge.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Switch payload agent to sa-claude-openshift-ci credential

Replace hypershift-team-claude-prow with sa-claude-openshift-ci and
update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix script paths and archive snapshot to artifacts

- Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and
  validate.py instead of searching /home/claude/.claude
- Archive snapshot tarball to ARTIFACT_DIR right after creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix snapshot archive path

The snapshot script creates a nested directory structure (e.g.
4.22/nightly/...) not a flat directory named by the payload tag.
Archive the entire snapshot directory contents instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir

- Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py
- Archive snapshot tarball to ARTIFACT_DIR immediately after creation
- Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir
- Move copy_reports trap before snapshot so artifacts are captured on failure
- Handle transient release API failures in the poll loop gracefully
- Switch test payload to 4.22.0-0.nightly-2026-06-02-214116

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix --snapshot-dir to point to directory containing summary.json

The skill expects --snapshot-dir to be the directory with summary.json,
which is nested under version/stream (e.g. snapshot/4.22/nightly/), not
the root snapshot directory. Find summary.json after snapshot creation
and pass its parent directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix validation filenames to use PAYLOAD_TAG directly

SANITIZED_TAG replaced dots with dashes, but Claude writes output
files using the original tag with dots. The validate scripts were
looking for the wrong filenames and always failing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Find validation files by prefix instead of exact name

Claude may write filenames with slightly different formatting.
Find by glob prefix instead of constructing the exact name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove temp payload agent job

Testing complete — the payload agent works correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Change "Agent:" to "Model:" in Slack messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
andrej1991 pushed a commit to andrej1991/release that referenced this pull request Jul 6, 2026
* Switch payload agent to snapshot-based payload-analysis skill

- Run payload_snapshot.py deterministically before invoking Claude,
  shifting data gathering from token-burning agent orchestration to
  a zero-cost Python script
- Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir
- Add schema validation retry loop for YAML/JSON outputs (up to 3
  attempts)
- Remove Jira integration and revert staging (now handled by Chai Bot)
- Add snapshot phase to JUnit tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Use validation scripts from ai-helpers for output schema checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting

- Fail the step (exit 1) when validators are missing or outputs are
  still invalid after 3 retries
- Move PHASE_ANALYSIS_DURATION finalization after the retry loop so
  nudge + validation time is included
- Increment PHASE_COUNT when recovery nudge testcase is appended

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add temporary payload agent job for 4.22 nightly 2026-06-01-074812

To be removed before merge.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Switch payload agent to sa-claude-openshift-ci credential

Replace hypershift-team-claude-prow with sa-claude-openshift-ci and
update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix script paths and archive snapshot to artifacts

- Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and
  validate.py instead of searching /home/claude/.claude
- Archive snapshot tarball to ARTIFACT_DIR right after creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix snapshot archive path

The snapshot script creates a nested directory structure (e.g.
4.22/nightly/...) not a flat directory named by the payload tag.
Archive the entire snapshot directory contents instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir

- Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py
- Archive snapshot tarball to ARTIFACT_DIR immediately after creation
- Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir
- Move copy_reports trap before snapshot so artifacts are captured on failure
- Handle transient release API failures in the poll loop gracefully
- Switch test payload to 4.22.0-0.nightly-2026-06-02-214116

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix --snapshot-dir to point to directory containing summary.json

The skill expects --snapshot-dir to be the directory with summary.json,
which is nested under version/stream (e.g. snapshot/4.22/nightly/), not
the root snapshot directory. Find summary.json after snapshot creation
and pass its parent directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix validation filenames to use PAYLOAD_TAG directly

SANITIZED_TAG replaced dots with dashes, but Claude writes output
files using the original tag with dots. The validate scripts were
looking for the wrong filenames and always failing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Find validation files by prefix instead of exact name

Claude may write filenames with slightly different formatting.
Find by glob prefix instead of constructing the exact name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove temp payload agent job

Testing complete — the payload agent works correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Change "Agent:" to "Model:" in Slack messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
TimurMP pushed a commit to TimurMP/release that referenced this pull request Jul 8, 2026
* Switch payload agent to snapshot-based payload-analysis skill

- Run payload_snapshot.py deterministically before invoking Claude,
  shifting data gathering from token-burning agent orchestration to
  a zero-cost Python script
- Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir
- Add schema validation retry loop for YAML/JSON outputs (up to 3
  attempts)
- Remove Jira integration and revert staging (now handled by Chai Bot)
- Add snapshot phase to JUnit tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Use validation scripts from ai-helpers for output schema checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting

- Fail the step (exit 1) when validators are missing or outputs are
  still invalid after 3 retries
- Move PHASE_ANALYSIS_DURATION finalization after the retry loop so
  nudge + validation time is included
- Increment PHASE_COUNT when recovery nudge testcase is appended

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add temporary payload agent job for 4.22 nightly 2026-06-01-074812

To be removed before merge.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Switch payload agent to sa-claude-openshift-ci credential

Replace hypershift-team-claude-prow with sa-claude-openshift-ci and
update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix script paths and archive snapshot to artifacts

- Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and
  validate.py instead of searching /home/claude/.claude
- Archive snapshot tarball to ARTIFACT_DIR right after creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix snapshot archive path

The snapshot script creates a nested directory structure (e.g.
4.22/nightly/...) not a flat directory named by the payload tag.
Archive the entire snapshot directory contents instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir

- Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py
- Archive snapshot tarball to ARTIFACT_DIR immediately after creation
- Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir
- Move copy_reports trap before snapshot so artifacts are captured on failure
- Handle transient release API failures in the poll loop gracefully
- Switch test payload to 4.22.0-0.nightly-2026-06-02-214116

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix --snapshot-dir to point to directory containing summary.json

The skill expects --snapshot-dir to be the directory with summary.json,
which is nested under version/stream (e.g. snapshot/4.22/nightly/), not
the root snapshot directory. Find summary.json after snapshot creation
and pass its parent directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix validation filenames to use PAYLOAD_TAG directly

SANITIZED_TAG replaced dots with dashes, but Claude writes output
files using the original tag with dots. The validate scripts were
looking for the wrong filenames and always failing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Find validation files by prefix instead of exact name

Claude may write filenames with slightly different formatting.
Find by glob prefix instead of constructing the exact name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove temp payload agent job

Testing complete — the payload agent works correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Change "Agent:" to "Model:" in Slack messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
SachinNinganure pushed a commit to SachinNinganure/release that referenced this pull request Jul 20, 2026
* Switch payload agent to snapshot-based payload-analysis skill

- Run payload_snapshot.py deterministically before invoking Claude,
  shifting data gathering from token-burning agent orchestration to
  a zero-cost Python script
- Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir
- Add schema validation retry loop for YAML/JSON outputs (up to 3
  attempts)
- Remove Jira integration and revert staging (now handled by Chai Bot)
- Add snapshot phase to JUnit tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Use validation scripts from ai-helpers for output schema checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting

- Fail the step (exit 1) when validators are missing or outputs are
  still invalid after 3 retries
- Move PHASE_ANALYSIS_DURATION finalization after the retry loop so
  nudge + validation time is included
- Increment PHASE_COUNT when recovery nudge testcase is appended

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add temporary payload agent job for 4.22 nightly 2026-06-01-074812

To be removed before merge.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Switch payload agent to sa-claude-openshift-ci credential

Replace hypershift-team-claude-prow with sa-claude-openshift-ci and
update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix script paths and archive snapshot to artifacts

- Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and
  validate.py instead of searching /home/claude/.claude
- Archive snapshot tarball to ARTIFACT_DIR right after creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix snapshot archive path

The snapshot script creates a nested directory structure (e.g.
4.22/nightly/...) not a flat directory named by the payload tag.
Archive the entire snapshot directory contents instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir

- Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py
- Archive snapshot tarball to ARTIFACT_DIR immediately after creation
- Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir
- Move copy_reports trap before snapshot so artifacts are captured on failure
- Handle transient release API failures in the poll loop gracefully
- Switch test payload to 4.22.0-0.nightly-2026-06-02-214116

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix --snapshot-dir to point to directory containing summary.json

The skill expects --snapshot-dir to be the directory with summary.json,
which is nested under version/stream (e.g. snapshot/4.22/nightly/), not
the root snapshot directory. Find summary.json after snapshot creation
and pass its parent directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix validation filenames to use PAYLOAD_TAG directly

SANITIZED_TAG replaced dots with dashes, but Claude writes output
files using the original tag with dots. The validate scripts were
looking for the wrong filenames and always failing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Find validation files by prefix instead of exact name

Claude may write filenames with slightly different formatting.
Find by glob prefix instead of constructing the exact name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove temp payload agent job

Testing complete — the payload agent works correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Change "Agent:" to "Model:" in Slack messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
malingatembo pushed a commit to malingatembo/release that referenced this pull request Aug 5, 2026
* Switch payload agent to snapshot-based payload-analysis skill

- Run payload_snapshot.py deterministically before invoking Claude,
  shifting data gathering from token-burning agent orchestration to
  a zero-cost Python script
- Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir
- Add schema validation retry loop for YAML/JSON outputs (up to 3
  attempts)
- Remove Jira integration and revert staging (now handled by Chai Bot)
- Add snapshot phase to JUnit tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Use validation scripts from ai-helpers for output schema checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting

- Fail the step (exit 1) when validators are missing or outputs are
  still invalid after 3 retries
- Move PHASE_ANALYSIS_DURATION finalization after the retry loop so
  nudge + validation time is included
- Increment PHASE_COUNT when recovery nudge testcase is appended

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add temporary payload agent job for 4.22 nightly 2026-06-01-074812

To be removed before merge.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Switch payload agent to sa-claude-openshift-ci credential

Replace hypershift-team-claude-prow with sa-claude-openshift-ci and
update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix script paths and archive snapshot to artifacts

- Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and
  validate.py instead of searching /home/claude/.claude
- Archive snapshot tarball to ARTIFACT_DIR right after creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix snapshot archive path

The snapshot script creates a nested directory structure (e.g.
4.22/nightly/...) not a flat directory named by the payload tag.
Archive the entire snapshot directory contents instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir

- Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py
- Archive snapshot tarball to ARTIFACT_DIR immediately after creation
- Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir
- Move copy_reports trap before snapshot so artifacts are captured on failure
- Handle transient release API failures in the poll loop gracefully
- Switch test payload to 4.22.0-0.nightly-2026-06-02-214116

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix --snapshot-dir to point to directory containing summary.json

The skill expects --snapshot-dir to be the directory with summary.json,
which is nested under version/stream (e.g. snapshot/4.22/nightly/), not
the root snapshot directory. Find summary.json after snapshot creation
and pass its parent directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix validation filenames to use PAYLOAD_TAG directly

SANITIZED_TAG replaced dots with dashes, but Claude writes output
files using the original tag with dots. The validate scripts were
looking for the wrong filenames and always failing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Find validation files by prefix instead of exact name

Claude may write filenames with slightly different formatting.
Find by glob prefix instead of constructing the exact name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove temp payload agent job

Testing complete — the payload agent works correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Change "Agent:" to "Model:" in Slack messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
TimurMP pushed a commit to TimurMP/release that referenced this pull request Sep 5, 2026
* Switch payload agent to snapshot-based payload-analysis skill

- Run payload_snapshot.py deterministically before invoking Claude,
  shifting data gathering from token-burning agent orchestration to
  a zero-cost Python script
- Replace /ci:analyze-payload with /ci:payload-analysis --snapshot-dir
- Add schema validation retry loop for YAML/JSON outputs (up to 3
  attempts)
- Remove Jira integration and revert staging (now handled by Chai Bot)
- Add snapshot phase to JUnit tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Use validation scripts from ai-helpers for output schema checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address CodeRabbit feedback: fail on invalid outputs, fix JUnit accounting

- Fail the step (exit 1) when validators are missing or outputs are
  still invalid after 3 retries
- Move PHASE_ANALYSIS_DURATION finalization after the retry loop so
  nudge + validation time is included
- Increment PHASE_COUNT when recovery nudge testcase is appended

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add temporary payload agent job for 4.22 nightly 2026-06-01-074812

To be removed before merge.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Switch payload agent to sa-claude-openshift-ci credential

Replace hypershift-team-claude-prow with sa-claude-openshift-ci and
update GOOGLE_APPLICATION_CREDENTIALS key from claude-prow to token.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix script paths and archive snapshot to artifacts

- Use hardcoded /opt/ai-helpers paths for payload_snapshot.py and
  validate.py instead of searching /home/claude/.claude
- Archive snapshot tarball to ARTIFACT_DIR right after creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix snapshot archive path

The snapshot script creates a nested directory structure (e.g.
4.22/nightly/...) not a flat directory named by the payload tag.
Archive the entire snapshot directory contents instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix payload agent: hardcode script paths, archive snapshot, fix snapshot-dir

- Hardcode /opt/ai-helpers paths for payload_snapshot.py and validate.py
- Archive snapshot tarball to ARTIFACT_DIR immediately after creation
- Fix --snapshot-dir to pass the root snapshot dir, not a tag-named subdir
- Move copy_reports trap before snapshot so artifacts are captured on failure
- Handle transient release API failures in the poll loop gracefully
- Switch test payload to 4.22.0-0.nightly-2026-06-02-214116

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix --snapshot-dir to point to directory containing summary.json

The skill expects --snapshot-dir to be the directory with summary.json,
which is nested under version/stream (e.g. snapshot/4.22/nightly/), not
the root snapshot directory. Find summary.json after snapshot creation
and pass its parent directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix validation filenames to use PAYLOAD_TAG directly

SANITIZED_TAG replaced dots with dashes, but Claude writes output
files using the original tag with dots. The validate scripts were
looking for the wrong filenames and always failing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Find validation files by prefix instead of exact name

Claude may write filenames with slightly different formatting.
Find by glob prefix instead of constructing the exact name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove temp payload agent job

Testing complete — the payload agent works correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Change "Agent:" to "Model:" in Slack messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants