Skip to content

fix(headless): preserve benchmark evidence provenance - #2547

Merged
Astro-Han merged 10 commits into
apache:mainfrom
Astro-Han:fix/headless-benchmark-evidence-provenance
Aug 9, 2026
Merged

Astro-Han merged 10 commits into
apache:mainfrom
Astro-Han:fix/headless-benchmark-evidence-provenance

Conversation

@Astro-Han

@Astro-Han Astro-Han commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Preserve benchmark evidence provenance from RuntimeEvent export through Harbor artifacts and fixed-prompt reporting.

  • Keep the Python ATIF trajectory validator aligned with Core for source-message digests and nested tool parent references.
  • Record whether persisted token usage is final or checkpoint-derived, preserve that source through Harbor recovery and fixed-prompt WAL events, and exclude provisional or ambiguous legacy usage from exact A/B metering.
  • Mark OpenCode, Codex, Maka deadline hydration, Harbor proxy, and Pier usage at their evidence-owning producer boundary; every possibly billable request must settle before native final evidence remains final, and every proxy-derived aggregate must also contain usage for each parsed stream.
  • Timestamp fixed-prompt terminal events when the runner settles instead of when the task launches.

The new token source field is optional so existing Harbor artifacts remain readable. Exact metering and requireFinalUsage accept only explicit final provenance; legacy events without source remain readable but provisional.

Verification

  • node --test --test-concurrency=4 packages/core/dist/__tests__/runtime-event.test.js packages/headless/dist/__tests__/harbor-adapter.test.js packages/headless/dist/__tests__/cell-output.test.js packages/headless/dist/__tests__/harbor-cell.test.js packages/headless/dist/__tests__/harbor-task-runner.test.js packages/headless/dist/__tests__/fixed-prompt-controller.test.js packages/headless/dist/__tests__/harness-ab-report.test.js — 408 passed before the review follow-up
  • node --test --test-concurrency=3 packages/headless/dist/__tests__/cell-output.test.js packages/headless/dist/__tests__/fixed-prompt-controller.test.js packages/headless/dist/__tests__/harness-ab-report.test.js — 121 passed after the review follow-up
  • node --test --test-concurrency=3 packages/headless/dist/__tests__/provider-auth-proxy.test.js packages/headless/dist/__tests__/harbor-adapter.test.js packages/headless/dist/__tests__/harbor-task-runner.test.js packages/headless/dist/__tests__/pier-task-runner.test.js packages/headless/dist/__tests__/cell-output.test.js packages/headless/dist/__tests__/fixed-prompt-controller.test.js packages/headless/dist/__tests__/harness-ab-report.test.js — 364 passed, 1 environment-dependent skip after the final fresh-eye follow-up
  • npm --workspace @maka/headless run typecheck
  • npx biome check on all files changed by the review follow-up
  • npx biome format and npx biome lint on the complete original TypeScript and JSON change set
  • python3 -m py_compile packages/headless/harbor/maka_trajectory.py

Review focus

This PR intentionally excludes Runtime terminal-ledger changes, error-content folding, QuickJS wiring, benchmark cohort machinery, and experiment-specific reporting.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

@Astro-Han
Astro-Han marked this pull request as ready for review August 9, 2026 06:51
@Astro-Han
Astro-Han merged commit 4dd541d into apache:main Aug 9, 2026
11 checks passed
@Astro-Han
Astro-Han deleted the fix/headless-benchmark-evidence-provenance branch August 9, 2026 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant