feat(observability): add Relay setup and first-use metrics

[afourniernv]

What does this PR do?

Adds consented setup and first-use metrics to Hermes' existing Relay shared-metrics pipeline. Setup runs for profiles that already opted in emit bounded start and terminal lifecycle marks, while the first usable runtime boundary and first successful task are persisted as one-time profile milestones.

All prerequisite observability PRs have merged. The diff against main now contains only the setup and first-use changes in this PR.

Predecessors

1. feat(observability): integrate NeMo Relay runtime and shared metrics #67607 - core Relay runtime, task metrics, aggregation, and local export.
2. feat(observability): report model and provider usage #68881 - bounded model metrics.
3. feat(observability): aggregate bounded tool metrics #68882 - bounded tool lifecycle and approval metrics.
4. feat(observability): aggregate bounded skill metrics #68883 - skill lifecycle, provenance, reuse, and reuse-after-patch metrics.
5. feat(observability): add Relay client resource metrics #68978 - bounded client OS, architecture, and install-method resources.
6. feat(observability): add Relay active install metrics #69416 - pseudonymous profile identity lifecycle and rolling client-active metrics.

All six predecessors are merged.

Related Issue

Builds on merged #69416. Relates to #69526, which tracks the separate product decision about telemetry consent during first-run onboarding.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• Emit allowlisted hermes.setup.started and hermes.setup.finished Relay marks for opted-in setup commands, using only bounded mode, outcome, and failure-stage values.
• Record hermes.client.first_usable once at the first consented session or task runtime boundary.
• Record hermes.client.first_successful_task transactionally with the first accepted successful task terminal.
• Persist both first-use milestones as one-time SQLite latches that survive restart and concurrent processes.
• Own setup's synthetic Relay scope through terminal emission and close it on success, failure, cancellation, disabled telemetry, or mark/session errors.
• Extend the closed package schema, documentation, command/runtime/store tests, and installed-Relay smoke artifact.
• Disable unrelated automatic title generation in the smoke fixture so it continues to assert exactly the managed task's two model calls on current main.

How to Test

1. Run ./.venv/bin/pytest -q tests/hermes_cli/test_relay_shared_metrics.py tests/hermes_cli/test_relay_shared_metrics_runtime.py tests/hermes_cli/test_setup_shared_metrics.py tests/hermes_cli/test_setup_blank_slate.py tests/hermes_cli/test_setup_model_provider.py.
2. Run ./.venv/bin/pytest -q tests/hermes_cli -k setup.
3. Run ./.venv/bin/python scripts/smoke_nemo_relay_shared_metrics.py with the installed NeMo Relay 0.7.1 binding.
4. Run Ruff on the Python files changed by this PR.
5. Run uv lock --check.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS arm64

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Validated after merging current main:

• Focused shared-metrics and setup-command suites: 187 passed across five files.
• Broader tests/hermes_cli -k setup selection: 130 passed, 1 skipped, with no failures.
• Ruff passed on every Python file changed by this PR.
• uv lock --check passed.
• End-to-end installed-Relay 0.7.1 smoke passed with setup started/finished, first usable, and first successful task each exported exactly once alongside the existing model, task, tool, skill, and client-active metrics.
• The smoke validated SQLite durability, immutable schema-valid delta packages, real Relay scope/subscriber delivery, and privacy-canary exclusion.

[afourniernv]

One scope clarification from review: this PR measures setup only for profiles that already have shared metrics enabled. Fresh first-time setup is not captured because Hermes does not currently ask for telemetry consent during onboarding, and this path deliberately creates no identity or persisted telemetry before consent.

I am keeping that privacy boundary in this PR. Until onboarding consent is decided, these metrics should be described as consented setup activity rather than complete first-time setup coverage.

I opened #69526 to track the product decision and implementation. If first-run coverage is required, the proposed shape is to hold the setup-start fact only in memory, emit it after an explicit opt-in, and discard it on opt-out or exit before consent.

[teknium1]

Thanks for the detailed consent boundary and the focused setup instrumentation.

Problems

• hermes_cli/observability/relay_shared_metrics.py:1271 returns when enabled() becomes false. The setup wrapper opens a synthetic scope before executing the wizard (hermes_cli/setup.py:2737), and the telemetry section can turn that setting off. That path therefore skips record_setup_finished(), which is also the only path to close_owned_session(), leaving the opened Relay session unclosed.

Suggested changes

• Make scope closure unconditional for a non-null attempt; gate only terminal metric emission on the final consent state. Add a test for enabled → disabled during hermes setup telemetry that asserts no terminal metric is persisted and the owned session is closed.

Automated hermes-sweeper review.

[afourniernv]

@teknium1 This is caught up with current main and ready for another review. The prior enabled-to-disabled cleanup finding remains fixed and covered by regression tests. Local validation after the merge: 187 focused tests passed, the broader setup selection passed (130 passed, 1 skipped), Ruff and uv lock --check passed, and the installed Relay 0.7.1 end-to-end smoke passed. Fresh CI is running on f9d65253f.

[afourniernv]

Caught up with current main in signed merge commit ca54f33df and requested another review.

Local validation after the merge:

• 187 focused observability and setup tests passed.
• 79 tests across all current test_setup*.py modules passed.
• Ruff passed for the changed Python surface.
• uv lock --check passed against the merged lockfile.

Fresh CI is running on the updated head.

[afourniernv]

@teknium1 Could you take another look when CI completes? The branch is caught up and the prior cleanup finding remains covered.