Skip to content

fix(observability): export shared metrics after task completion - #73544

Merged
jquesnelle merged 2 commits into
NousResearch:mainfrom
afourniernv:fix/relay-desktop-outbox-export
Jul 28, 2026
Merged

fix(observability): export shared metrics after task completion#73544
jquesnelle merged 2 commits into
NousResearch:mainfrom
afourniernv:fix/relay-desktop-outbox-export

Conversation

@afourniernv

@afourniernv afourniernv commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Shared-metrics packages were created when a Relay session closed. Desktop sessions are long-lived, so a normal completed desktop task could update SQLite without creating an outbox entry.

This moves the export opportunity to task completion. Package creation is gated by the durable SQLite outbox, so Hermes creates packages at most once per UTC day across tasks, threads, processes, and restarts. Session close and runtime shutdown remain fallback opportunities and use the same gate.

The gate uses the committed package timestamp rather than the file-write timestamp. If writing the outbox file fails, a later task can retry the write without packaging the same counter deltas twice.

Package creation only runs after Relay subscriber flushing succeeds. A failed flush leaves the daily gate open so a later same-day task can flush the complete queue and create that day's package.

Related Issue

Follow-up to #67607.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • Flush Relay subscriber work and attempt a gated export after a task closes.
  • Add a transactional, persistent once-per-UTC-day package gate to SharedMetricsStore.
  • Preserve session-finalization and shutdown exports as idempotent fallbacks.
  • Prevent task, session-close, and shutdown paths from advancing the daily gate after a failed subscriber flush.
  • Add desktop, restart, next-day catch-up, flush-retry, native binding, and concurrent-export coverage.

How to Test

  1. Enable telemetry.shared_metrics.enabled.
  2. Complete a desktop task without closing the desktop session and verify an outbox JSON file exists under telemetry/shared_metrics/outbox.
  3. Complete another task on the same UTC day and verify no second package is created; on the next UTC day, verify pending deltas are packaged.

Validation run:

  • 662 passed across the shared-metrics store, runtime, lifecycle, desktop gateway, lazy-session, and smoke-harness suites.
  • A temporary native-binding harness verified that a failed first flush creates no package row or file, while a later same-day task creates one package containing both tasks. The harness was not added to the branch.
  • Direct Hermes -> local OpenAI-compatible endpoint -> NeMo Relay -> SQLite/outbox smoke passed with all three base metric families and no privacy canaries.
  • Ruff and git diff --check passed.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS arm64

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

A direct native smoke completed one Hermes task without requiring an explicit session close. The SQLite snapshot showed value=1 and packaged_value=1 for hermes.task_run.started, hermes.task_run.finished, and hermes.model_call.count, with one schema-valid outbox package.

The temporary native flush-failure harness then forced the first eligible flush to fail. It verified zero package rows and files after that failure, followed by one same-day package with started=2 and finished=2 after the next task completed.

Signed-off-by: Alex Fournier <afournier@nvidia.com>
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard telemetry Touches outbound telemetry, usage attribution, or analytics — needs opt-in gating before merge labels Jul 28, 2026
Signed-off-by: Alex Fournier <afournier@nvidia.com>
@jquesnelle
jquesnelle enabled auto-merge July 28, 2026 19:22
@jquesnelle
jquesnelle disabled auto-merge July 28, 2026 19:42
@jquesnelle
jquesnelle merged commit 38574a7 into NousResearch:main Jul 28, 2026
36 of 38 checks passed
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…top-outbox-export

fix(observability): export shared metrics after task completion
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have telemetry Touches outbound telemetry, usage attribution, or analytics — needs opt-in gating before merge type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants