Skip to content

feat: add typed capability effect evidence to agent runners #2755

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stranske merged 13 commits into from
Jul 11, 2026
Merged

feat: add typed capability effect evidence to agent runners #2755

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
05e18fe
feat(runners): expose typed capability effect evidence
stranske Jul 11, 2026
3e166bc
style(runners): apply Black formatting
stranske Jul 11, 2026
2dceec9
fix(runners): address capability evidence review
codex-automation Jul 11, 2026
8eb03f0
fix(runners): stop invalid evidence before execution
codex-automation Jul 11, 2026
42a0866
chore(autofix): formatting/lint
github-actions[bot] Jul 11, 2026
ed0841d
fix(runners): harden evidence value validation
stranske Jul 11, 2026
7a85b6b
chore(codex-keepalive): apply updates (PR #2755)
github-actions[bot] Jul 11, 2026
2de0785
chore(codex-keepalive): apply updates (PR #2755)
github-actions[bot] Jul 11, 2026
949afcd
chore: remove completed-agent telemetry drift
stranske Jul 11, 2026
14794bd
chore(codex-autofix): apply updates (PR #2755)
github-actions[bot] Jul 11, 2026
669e614
chore: finalize telemetry-clean PR head
stranske Jul 11, 2026
e23d104
chore(codex-autofix): apply updates (PR #2755)
github-actions[bot] Jul 11, 2026
1d71a9f
chore(codex-autofix): apply updates (PR #2755)
github-actions[bot] Jul 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 80 additions & 2 deletions .github/workflows/reusable-claude-run.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,40 @@ on:
required: false
default: 'agent-standard'
type: string
capability_id:
description: 'Optional existing capability ID for bounded outcome evidence.'
required: false
default: ''
type: string
effect_fingerprint:
description: 'Optional lowercase sha256 fingerprint of the bounded effect.'
required: false
default: ''
type: string
evidence_artifact_ref:
description: 'Optional durable logical reference to supporting evidence.'
required: false
default: ''
type: string
supervision_mode:
description: >-
Optional supervision mode (shadow, human-reviewed,
human-on-exception, unattended).
required: false
default: ''
type: string
capability_evidence_status:
description: 'Optional evidence status (accepted, rejected, not-evaluated).'
required: false
default: ''
type: string
terminal_disposition:
description: >-
Optional terminal disposition (success, failure, no-change, blocked,
cancelled).
required: false
default: ''
type: string
Comment thread
coderabbitai[bot] marked this conversation as resolved.
secrets:
CLAUDE_AUTH_JSON:
description: >-
Expand Down Expand Up @@ -137,6 +171,24 @@ on:
files-changed:
description: 'Number of files changed by Claude'
value: ${{ jobs.claude.outputs.files-changed }}
capability-id:
description: 'Validated existing capability identifier; empty when evidence is absent.'
value: ${{ jobs.claude.outputs.capability-id }}
effect-fingerprint:
description: 'Validated lowercase sha256 fingerprint of the bounded effect.'
value: ${{ jobs.claude.outputs.effect-fingerprint }}
evidence-artifact-ref:
description: 'Validated durable logical reference to supporting evidence.'
value: ${{ jobs.claude.outputs.evidence-artifact-ref }}
supervision-mode:
description: 'Validated supervision mode for this result.'
value: ${{ jobs.claude.outputs.supervision-mode }}
capability-evidence-status:
description: 'Validated capability evidence status.'
value: ${{ jobs.claude.outputs.capability-evidence-status }}
terminal-disposition:
description: 'Validated terminal disposition for the result.'
value: ${{ jobs.claude.outputs.terminal-disposition }}
llm-analysis-run:
description: 'Whether LLM analysis was performed'
value: ${{ jobs.claude.outputs.llm-analysis-run }}
Expand Down Expand Up @@ -189,6 +241,13 @@ jobs:
changes-made: ${{ steps.commit.outputs.changes-made }}
commit-sha: ${{ steps.commit.outputs.commit-sha }}
files-changed: ${{ steps.commit.outputs.files-changed }}
capability-id: ${{ steps.capability_evidence.outputs.capability-id }}
effect-fingerprint: ${{ steps.capability_evidence.outputs.effect-fingerprint }}
evidence-artifact-ref: ${{ steps.capability_evidence.outputs.evidence-artifact-ref }}
supervision-mode: ${{ steps.capability_evidence.outputs.supervision-mode }}
capability-evidence-status: >-
${{ steps.capability_evidence.outputs.capability-evidence-status }}
terminal-disposition: ${{ steps.capability_evidence.outputs.terminal-disposition }}
error-summary: ${{ steps.run_claude.outputs.error-summary }}
llm-analysis-run: ${{ steps.compat.outputs.llm-analysis-run }}
llm-provider: ${{ steps.compat.outputs.llm-provider }}
Expand Down Expand Up @@ -387,6 +446,25 @@ jobs:
}
fi

- name: Validate optional capability effect evidence
id: capability_evidence
env:
CAPABILITY_ID: ${{ inputs.capability_id }}
EFFECT_FINGERPRINT: ${{ inputs.effect_fingerprint }}
EVIDENCE_ARTIFACT_REF: ${{ inputs.evidence_artifact_ref }}
SUPERVISION_MODE: ${{ inputs.supervision_mode }}
CAPABILITY_EVIDENCE_STATUS: ${{ inputs.capability_evidence_status }}
TERMINAL_DISPOSITION: ${{ inputs.terminal_disposition }}
run: |
PYTHONPATH="${GITHUB_WORKSPACE}/.workflows-lib:${GITHUB_WORKSPACE}" \
python -m scripts.runner_lib normalize-evidence \
--capability-id "${CAPABILITY_ID}" \
--effect-fingerprint "${EFFECT_FINGERPRINT}" \
--evidence-artifact-ref "${EVIDENCE_ARTIFACT_REF}" \
--supervision-mode "${SUPERVISION_MODE}" \
--capability-evidence-status "${CAPABILITY_EVIDENCE_STATUS}" \
--terminal-disposition "${TERMINAL_DISPOSITION}"

Comment thread
coderabbitai[bot] marked this conversation as resolved.
- name: Validate prompt template integrity
id: guard
if: ${{ inputs.mode != 'e2e-test' }}
Expand Down Expand Up @@ -636,7 +714,7 @@ jobs:

- name: Run Claude
id: run_claude
if: always()
if: always() && steps.capability_evidence.outcome != 'failure'
env:
PREFLIGHT_MISSING: ${{ steps.preflight.outputs.missing || 'false' }}
PREFLIGHT_ERROR: ${{ steps.preflight.outputs.error || '' }}
Expand Down Expand Up @@ -782,7 +860,7 @@ jobs:

- name: Commit and push changes
id: commit
if: always()
if: always() && steps.capability_evidence.outcome != 'failure'
env:
PUSH_TOKEN: ${{ steps.run_base.outputs.push_token || '' }}
PUSH_ALLOWED: ${{ steps.run_base.outputs.push_allowed }}
Expand Down
78 changes: 78 additions & 0 deletions .github/workflows/reusable-codex-run.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,40 @@ on:
required: false
default: 'agent-standard'
type: string
capability_id:
description: 'Optional existing capability ID for bounded outcome evidence.'
required: false
default: ''
type: string
effect_fingerprint:
description: 'Optional lowercase sha256 fingerprint of the bounded effect.'
required: false
default: ''
type: string
evidence_artifact_ref:
description: 'Optional durable logical reference to supporting evidence.'
required: false
default: ''
type: string
supervision_mode:
description: >-
Optional supervision mode (shadow, human-reviewed,
human-on-exception, unattended).
required: false
default: ''
type: string
capability_evidence_status:
description: 'Optional evidence status (accepted, rejected, not-evaluated).'
required: false
default: ''
type: string
terminal_disposition:
description: >-
Optional terminal disposition (success, failure, no-change, blocked,
cancelled).
required: false
default: ''
type: string
secrets:
CODEX_AUTH_JSON:
description: 'JSON contents of ~/.codex/auth.json for ChatGPT subscription auth'
Expand Down Expand Up @@ -143,6 +177,24 @@ on:
files-changed:
description: 'Number of files changed by Codex'
value: ${{ jobs.codex.outputs.files-changed }}
capability-id:
description: 'Validated existing capability identifier; empty when evidence is absent.'
value: ${{ jobs.codex.outputs.capability-id }}
effect-fingerprint:
description: 'Validated lowercase sha256 fingerprint of the bounded effect.'
value: ${{ jobs.codex.outputs.effect-fingerprint }}
evidence-artifact-ref:
description: 'Validated durable logical reference to supporting evidence.'
value: ${{ jobs.codex.outputs.evidence-artifact-ref }}
supervision-mode:
description: 'Validated supervision mode for this result.'
value: ${{ jobs.codex.outputs.supervision-mode }}
capability-evidence-status:
description: 'Validated capability evidence status.'
value: ${{ jobs.codex.outputs.capability-evidence-status }}
terminal-disposition:
description: 'Validated terminal disposition for the result.'
value: ${{ jobs.codex.outputs.terminal-disposition }}
worker-profile-id:
description: 'Registry execution profile ID supplied by caller'
value: ${{ jobs.codex.outputs.worker-profile-id }}
Expand Down Expand Up @@ -227,6 +279,13 @@ jobs:
changes-made: ${{ steps.commit.outputs.changes-made }}
commit-sha: ${{ steps.commit.outputs.commit-sha }}
files-changed: ${{ steps.commit.outputs.files-changed }}
capability-id: ${{ steps.capability_evidence.outputs.capability-id }}
effect-fingerprint: ${{ steps.capability_evidence.outputs.effect-fingerprint }}
evidence-artifact-ref: ${{ steps.capability_evidence.outputs.evidence-artifact-ref }}
supervision-mode: ${{ steps.capability_evidence.outputs.supervision-mode }}
capability-evidence-status: >-
${{ steps.capability_evidence.outputs.capability-evidence-status }}
terminal-disposition: ${{ steps.capability_evidence.outputs.terminal-disposition }}
worker-profile-id: ${{ inputs.execution_profile }}
worker-requested-model: ${{ inputs.codex_model }}
worker-selected-model: ${{ steps.run_codex.outputs.model }}
Expand Down Expand Up @@ -443,6 +502,25 @@ jobs:
}
fi

- name: Validate optional capability effect evidence
id: capability_evidence
env:
CAPABILITY_ID: ${{ inputs.capability_id }}
EFFECT_FINGERPRINT: ${{ inputs.effect_fingerprint }}
EVIDENCE_ARTIFACT_REF: ${{ inputs.evidence_artifact_ref }}
SUPERVISION_MODE: ${{ inputs.supervision_mode }}
CAPABILITY_EVIDENCE_STATUS: ${{ inputs.capability_evidence_status }}
TERMINAL_DISPOSITION: ${{ inputs.terminal_disposition }}
run: |
PYTHONPATH="${GITHUB_WORKSPACE}/.workflows-lib:${GITHUB_WORKSPACE}" \
python -m scripts.runner_lib normalize-evidence \
--capability-id "${CAPABILITY_ID}" \
--effect-fingerprint "${EFFECT_FINGERPRINT}" \
--evidence-artifact-ref "${EVIDENCE_ARTIFACT_REF}" \
--supervision-mode "${SUPERVISION_MODE}" \
--capability-evidence-status "${CAPABILITY_EVIDENCE_STATUS}" \
--terminal-disposition "${TERMINAL_DISPOSITION}"

- name: Validate prompt template integrity
id: guard
if: ${{ inputs.mode != 'e2e-test' }}
Expand Down
82 changes: 80 additions & 2 deletions .github/workflows/reusable-cursor-run.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,40 @@ on:
required: false
default: 'agent-standard'
type: string
capability_id:
description: 'Optional existing capability ID for bounded outcome evidence.'
required: false
default: ''
type: string
effect_fingerprint:
description: 'Optional lowercase sha256 fingerprint of the bounded effect.'
required: false
default: ''
type: string
evidence_artifact_ref:
description: 'Optional durable logical reference to supporting evidence.'
required: false
default: ''
type: string
supervision_mode:
description: >-
Optional supervision mode (shadow, human-reviewed,
human-on-exception, unattended).
required: false
default: ''
type: string
capability_evidence_status:
description: 'Optional evidence status (accepted, rejected, not-evaluated).'
required: false
default: ''
type: string
terminal_disposition:
description: >-
Optional terminal disposition (success, failure, no-change, blocked,
cancelled).
required: false
default: ''
type: string
secrets:
CURSOR_API_KEY:
description: >-
Expand Down Expand Up @@ -121,6 +155,24 @@ on:
files-changed:
description: 'Number of files changed by Cursor'
value: ${{ jobs.cursor.outputs.files-changed }}
capability-id:
description: 'Validated existing capability identifier; empty when evidence is absent.'
value: ${{ jobs.cursor.outputs.capability-id }}
effect-fingerprint:
description: 'Validated lowercase sha256 fingerprint of the bounded effect.'
value: ${{ jobs.cursor.outputs.effect-fingerprint }}
evidence-artifact-ref:
description: 'Validated durable logical reference to supporting evidence.'
value: ${{ jobs.cursor.outputs.evidence-artifact-ref }}
supervision-mode:
description: 'Validated supervision mode for this result.'
value: ${{ jobs.cursor.outputs.supervision-mode }}
capability-evidence-status:
description: 'Validated capability evidence status.'
value: ${{ jobs.cursor.outputs.capability-evidence-status }}
terminal-disposition:
description: 'Validated terminal disposition for the result.'
value: ${{ jobs.cursor.outputs.terminal-disposition }}
llm-analysis-run:
description: 'Whether LLM analysis was performed'
value: ${{ jobs.cursor.outputs.llm-analysis-run }}
Expand Down Expand Up @@ -173,6 +225,13 @@ jobs:
changes-made: ${{ steps.commit.outputs.changes-made }}
commit-sha: ${{ steps.commit.outputs.commit-sha }}
files-changed: ${{ steps.commit.outputs.files-changed }}
capability-id: ${{ steps.capability_evidence.outputs.capability-id }}
effect-fingerprint: ${{ steps.capability_evidence.outputs.effect-fingerprint }}
evidence-artifact-ref: ${{ steps.capability_evidence.outputs.evidence-artifact-ref }}
supervision-mode: ${{ steps.capability_evidence.outputs.supervision-mode }}
capability-evidence-status: >-
${{ steps.capability_evidence.outputs.capability-evidence-status }}
terminal-disposition: ${{ steps.capability_evidence.outputs.terminal-disposition }}
error-summary: ${{ steps.run_cursor.outputs.error-summary }}
llm-analysis-run: ${{ steps.compat.outputs.llm-analysis-run }}
llm-provider: ${{ steps.compat.outputs.llm-provider }}
Expand Down Expand Up @@ -404,6 +463,25 @@ jobs:
}
fi

- name: Validate optional capability effect evidence
id: capability_evidence
env:
CAPABILITY_ID: ${{ inputs.capability_id }}
EFFECT_FINGERPRINT: ${{ inputs.effect_fingerprint }}
EVIDENCE_ARTIFACT_REF: ${{ inputs.evidence_artifact_ref }}
SUPERVISION_MODE: ${{ inputs.supervision_mode }}
CAPABILITY_EVIDENCE_STATUS: ${{ inputs.capability_evidence_status }}
TERMINAL_DISPOSITION: ${{ inputs.terminal_disposition }}
run: |
PYTHONPATH="${GITHUB_WORKSPACE}/.workflows-lib:${GITHUB_WORKSPACE}" \
python -m scripts.runner_lib normalize-evidence \
--capability-id "${CAPABILITY_ID}" \
--effect-fingerprint "${EFFECT_FINGERPRINT}" \
--evidence-artifact-ref "${EVIDENCE_ARTIFACT_REF}" \
--supervision-mode "${SUPERVISION_MODE}" \
--capability-evidence-status "${CAPABILITY_EVIDENCE_STATUS}" \
--terminal-disposition "${TERMINAL_DISPOSITION}"

Comment thread
coderabbitai[bot] marked this conversation as resolved.
- name: Validate prompt template integrity
id: guard
if: ${{ inputs.mode != 'e2e-test' }}
Expand Down Expand Up @@ -725,7 +803,7 @@ jobs:

- name: Run Cursor
id: run_cursor
if: always()
if: always() && steps.capability_evidence.outcome != 'failure'
env:
PREFLIGHT_MISSING: ${{ steps.preflight.outputs.missing || 'false' }}
PREFLIGHT_ERROR: ${{ steps.preflight.outputs.error || '' }}
Expand Down Expand Up @@ -890,7 +968,7 @@ jobs:

- name: Commit and push changes
id: commit
if: always()
if: always() && steps.capability_evidence.outcome != 'failure'
env:
PUSH_TOKEN: ${{ steps.auth_token.outputs.push_token || '' }}
PUSH_ALLOWED: ${{ steps.auth_token.outputs.push_allowed }}
Expand Down
Loading
Loading