test(e2e): capture rebuild-hermes phase timings for perf comparison

[atulya-singh]

Summary

Issue #7144 requires proving a median wall-time improvement of at least 25% for
the rebuild-hermes lanes "on the same runner class across repeated runs"
(acceptance criterion 4). That evidence needs a stable, machine-readable timing
record before any cache or fixture change lands. This PR adds that measurement
capture to both rebuild-hermes lanes without changing any behavior under test.

Each run now emits a rebuild-hermes-timing.json artifact containing:

• the lane (normal or stale-base),
• a runner-class fingerprint (platform, arch, CPU count, CPU model, total
  memory) so before/after runs can be confirmed to share a runner class before
  they are compared,
• per-phase elapsed durations, and
• total wall time.

The existing progress reporter already timestamped every phase transition; this
extends it to also accumulate the durations it was already computing, and adds a
pure summary/serialization module so the artifact shape is unit-testable without
Docker or hosted inference.

Related Issue

Part of #7144 (incremental; does not close the issue). This lands the
measurement foundation for acceptance criterion 4 ("Before/after timings are
collected on the same runner class across repeated runs"). The cache and pinned-
fixture work the issue also tracks remains follow-up, and now has a baseline to
measure against.

Note for reviewers: the issue's "persist reusable BuildKit cache across jobs"
scope item conflicts with the existing requireDockerEngineRebuilds boundary in
tools/e2e/workflow-boundary.mts, which intentionally keeps every rebuild-*
job on the Docker engine cache (no external buildx builder). That trade-off needs
a maintainer design decision and is deliberately out of scope here.

Changes

• test/e2e/live/rebuild-hermes-timing.ts (new): pure describeRunnerClass and
  buildRebuildHermesTimingSummary helpers plus the timing/runner-class types.
  No I/O beyond an injectable os sample; durations normalize to non-negative
  whole milliseconds so repeated runs stay directly comparable.
• test/e2e/live/rebuild-hermes-progress.ts: accumulate each completed phase
  span and expose timeline(), a non-mutating snapshot that closes the
  in-flight phase at read time and freezes at stop(). Existing heartbeat/log
  output is unchanged.
• test/e2e/live/rebuild-hermes.test.ts: write the rebuild-hermes-timing.json
  artifact at the end of the run (before the final credential-scan gate, so the
  timing survives an assertion failure). No phase, assertion, or command changes.
• test/e2e/support/rebuild-hermes-timing.test.ts (new): unit coverage for the
  timeline accumulation (completed/in-flight/post-stop) and the summary builder
  (lane labeling, duration normalization, runner-class fingerprint fallback).

No new configuration, fallback, migration, or compatibility layer is introduced,
and no external buildx builder or cache backend is added, so the
requireDockerEngineRebuilds boundary is preserved.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: internal E2E measurement
  instrumentation with no user-facing behavior change.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification:
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result: npx vitest run --project e2e-support test/e2e/support/rebuild-hermes-timing.test.ts test/e2e/support/rebuild-hermes-progress.test.ts test/e2e/support/e2e-workflow.test.ts → 46 passed; npm run typecheck:cli → clean; npm run test:projects:check → membership exact. The live rebuild-hermes lane runs in the E2E workflow (Docker + hosted inference required, not runnable locally).
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: not applicable (additive E2E instrumentation for two live lanes).
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Atulya Singh atulyarajsingh@gmail.com

Summary by CodeRabbit

• New Features

  • Added richer Hermes rebuild progress reporting, including a phase timeline with per-phase elapsed time and an overall total duration.
  • Added end-to-end timing artifacts (rebuild-hermes-timing.json) that capture lane, runner environment details, per-phase timings, and timestamps using a stable schema.

• Tests

  • Added unit and end-to-end coverage for timeline behavior (phase transitions, stop freezing), correct total/phase timestamps, deterministic runner-class identification, and duration normalization/clamping.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: eaa94bc7-1e60-42e6-924a-a593a9661eaa

📥 Commits

Reviewing files that changed from the base of the PR and between 7e6ad5a and 60a4b4b.

📒 Files selected for processing (2)

• test/e2e/live/rebuild-hermes-progress.ts
• test/e2e/support/rebuild-hermes-timing.test.ts

🚧 Files skipped from review as they are similar to previous changes (2)

• test/e2e/support/rebuild-hermes-timing.test.ts
• test/e2e/live/rebuild-hermes-progress.ts

---

📝 Walkthrough

Walkthrough

Hermes rebuild progress now records completed phases and total elapsed time, exposes timeline snapshots, normalizes timing data with runner metadata, and writes a versioned JSON artifact. New support tests cover timing, stopping, runner classification, and normalization.

Changes

Hermes rebuild timing

Layer / File(s)	Summary
Timing model and normalization test/e2e/live/rebuild-hermes-timing.ts	Defines timing summary contracts, runner-class sampling, lane values, and normalization of phase and total durations.
Progress timeline tracking test/e2e/live/rebuild-hermes-progress.ts	Tracks phase completion and overall elapsed time, freezes state on stop, and exposes timeline().
Artifact emission and validation test/e2e/live/rebuild-hermes.test.ts, test/e2e/support/rebuild-hermes-timing.test.ts	Writes rebuild-hermes-timing.json and tests timeline capture, runner fingerprints, duration normalization, and empty-CPU handling.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant HermesE2ETest
  participant RebuildHermesProgress
  participant RunnerSampler
  participant TimingSummary
  participant Artifacts
  HermesE2ETest->>RebuildHermesProgress: collect timeline
  HermesE2ETest->>RunnerSampler: collect runner metadata
  HermesE2ETest->>TimingSummary: build normalized summary
  TimingSummary->>Artifacts: write rebuild-hermes-timing.json

Loading

Suggested labels: area: e2e, area: observability

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: capturing rebuild-hermes phase timings for performance comparison.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, rebuild-hermes, rebuild-hermes-stale-base

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[apurvvkumaria]

Approved exact head 60a4b4b against base 980e348. Independent review found no correctness or security blockers across all four changed files. Focused e2e-support validation passed 47/47 tests; formatting, test-project membership, and test-title checks passed. All current CI, CodeQL, DCO/commit verification, and automated advisors are green with zero unresolved threads. Credentialed E2E is safely recorded as an approved fork skip bound to this exact head/base in run 29659578683.