feat(evaluator): Update publish_to_intake to record latency + tokens

[shanaiabuggy]

Summary

publish_to_intake previously emitted a stub single-step trajectory with no telemetry, so evaluations published from nemo-evaluator showed no token counts and a 0ms latency. This wires each trial's recorded token usage and runtime into the ATIF payload so Intake's rollup captures them:

• Tokens ride along as ATIF final_metrics, which Intake promotes onto the trajectory's root span (attributes_number) — exactly what the evaluation rollup sums into tokens.
• Latency is captured by giving the trajectory a real duration: a NAT invocation window (started_at → started_at + runtime_sec) on the step, so the root span's end − start is the trial's runtime instead of 0.

Metrics are emitted only when the harness actually recorded them — a trial with no usage/runtime yields no final_metrics and no window, so latency reads as unmeasured rather than a misleading 0ms. Cost is still not captured (there is no per-trial cost source in the trial measurements yet).

Related Issue

Follow-up to the evaluator → Intake publish path: closes the token/latency capture gap the initial stub deferred. (No Linear issue linked; add one if applicable.)

Changes

• intake/publish.py — project each trial's TrialMeasurements (from trial.metadata) once: build ATIF final_metrics from its token fields (_token_final_metrics), and derive ended_at = started_at + runtime_sec when a runtime was recorded; pass both to trial_to_atif_ingest.
• intake/mapping.py — trial_to_atif_ingest gains an optional ended_at; when set, it attaches a NAT invocation window (extra["invocation"] with epoch-second start_timestamp/end_timestamp) to the step. Purely additive — no change when ended_at is absent.
• Idempotency preserved — started_at stays the window start and runtime_sec is a stable recorded value, so the root span's id and start_time don't change on re-publish.
• Tests — token projection (test_publish.py) and invocation-window presence/absence (test_mapping.py).

No API or schema change: AtifFinalMetricsParam and the step extra field already exist in the SDK types, so no OpenAPI/SDK regen.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with documentation updates
• Documentation only
• Contributor tooling or automation
• CI, build, or test infrastructure

Quality Gates

• Tests added or updated for changed behavior
• Documentation not applicable — justification: internal evaluator → Intake mapping; no user-facing surface or schema change.

Verification

• Pull request title follows the repository's Conventional Commit format
• Every commit includes an appropriate Signed-off-by: trailer
• uv run pre-commit run -a passes, or any blocked checks are identified below
• Targeted tests pass, or tests are marked not applicable above
• No secrets, API keys, or credentials are included

Targeted validation (targeted checks run locally, not the full pre-commit sweep):

• uv run --frozen pytest plugins/nemo-evaluator/tests/intake/test_publish.py plugins/nemo-evaluator/tests/intake/test_mapping.py → 35 passed
• uv run --frozen ty check plugins/nemo-evaluator/src/nemo_evaluator/intake/{publish,mapping}.py → clean
• uv run ruff check → clean

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added reporting of prompt, completion, and cached token usage for evaluation trials.
  • Added runtime-duration reporting, including invocation start and end timestamps.
  • Trials without token or runtime measurements continue to omit unavailable fields.

• Tests

  • Added coverage for invocation timing and token-usage metrics.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6b6f5695-cac3-4277-b315-c2f68f3d7a03

📥 Commits

Reviewing files that changed from the base of the PR and between 80c1c46 and dbacf92.

📒 Files selected for processing (4)

• plugins/nemo-evaluator/src/nemo_evaluator/intake/mapping.py
• plugins/nemo-evaluator/src/nemo_evaluator/intake/publish.py
• plugins/nemo-evaluator/tests/intake/test_mapping.py
• plugins/nemo-evaluator/tests/intake/test_publish.py

---

📝 Walkthrough

Walkthrough

Trial publishing now maps token measurements to ATIF final metrics and runtime measurements to invocation end timestamps. The ingestion mapping preserves existing output when no runtime is recorded. Tests cover populated and empty measurements.

Changes

Trial ATIF telemetry

Layer / File(s)	Summary
ATIF invocation timing plugins/nemo-evaluator/src/nemo_evaluator/intake/mapping.py, plugins/nemo-evaluator/tests/intake/test_mapping.py	trial_to_atif_ingest accepts ended_at and records invocation start/end timestamps when provided. Tests cover timestamped and unchanged output.
Trial measurement publishing plugins/nemo-evaluator/src/nemo_evaluator/intake/publish.py, plugins/nemo-evaluator/tests/intake/test_publish.py	Trial publishing maps prompt, completion, and cache-read token counts to ATIF final metrics. It derives ended_at from recorded runtime. Tests cover populated and empty token measurements.

Suggested reviewers: a2bondar

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 36.36% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: recording per-trial latency and token usage in publish_to_intake.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch sbuggy/update-plublish-to-intake

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	31885/40497	78.7%	63.4%
Integration Tests	18530/38423	48.2%	20.9%

[shanaiabuggy]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.