Skip to content

feat(observability): add Relay active install metrics - #69416

Merged
jquesnelle merged 22 commits into
NousResearch:mainfrom
afourniernv:feat/hermes-relay-install-activation-metrics
Aug 5, 2026
Merged

feat(observability): add Relay active install metrics#69416
jquesnelle merged 22 commits into
NousResearch:mainfrom
afourniernv:feat/hermes-relay-install-activation-metrics

Conversation

@afourniernv

@afourniernv afourniernv commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds the remaining Category 1 install/activation slice on top of the Relay client-resource stack. Consented Hermes session and task starts now emit an empty Relay activation mark, and the profile-scoped subscriber records one pseudonymous client-active counter per rolling 24-hour window.

This is a stacked PR on #68978. Until the earlier PRs land, GitHub shows the cumulative stack against main; the active-install-only diff is available here.

Stack Order

  1. feat(observability): integrate NeMo Relay runtime and shared metrics #67607 - core Relay runtime, task metrics, aggregation, and local export.
  2. feat(observability): report model and provider usage #68881 - bounded model metrics.
  3. feat(observability): aggregate bounded tool metrics #68882 - bounded tool lifecycle and approval metrics.
  4. feat(observability): aggregate bounded skill metrics #68883 - skill lifecycle, provenance, reuse, and reuse-after-patch metrics.
  5. feat(observability): add Relay client resource metrics #68978 - bounded client OS, architecture, and install-method resources.
  6. This PR - pseudonymous profile identity lifecycle and rolling active-install metrics.

Merge and review in this order. This PR is cumulative until its predecessors land on main.

Related Issue

Depends on #68978.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • Emit an empty, allowlisted hermes.client.active Relay mark from consented session and task boundaries.
  • Create a random profile-scoped UUID before the first active metric is persisted.
  • Use one SQLite transaction for the install identity, rolling 24-hour compare-and-set latch, and counter increment so concurrent Hermes processes cannot double-count.
  • Export the active-install metric with no dimensions; bounded client properties remain package resources and session/task identifiers never enter the event or package.
  • Document reset semantics: removing the stopped profile's shared-metrics directory rotates identity while discarding its old aggregates and queued packages together.
  • Extend the closed package schema, real-binding runtime coverage, cross-process tests, profile-isolation tests, and end-to-end smoke artifact.

How to Test

  1. Run scripts/run_tests.sh tests/hermes_cli/test_relay_shared_metrics.py tests/hermes_cli/test_relay_shared_metrics_runtime.py -q.
  2. Run the full stacked regression set documented in the PR validation comment.
  3. Run .venv/bin/python scripts/smoke_nemo_relay_shared_metrics.py with the installed NeMo Relay binding.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS arm64

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

  • Focused store/runtime suites: 244 passed.
  • Full stacked regression set: 1,042 passed across 14 files.
  • End-to-end installed-Relay smoke: passed with one hermes.client.active counter, a stable pseudonymous profile ID across packages, all prior aggregate families, schema validation, and privacy-canary checks.

@afourniernv

Copy link
Copy Markdown
Contributor Author

Validation completed on the Category 1 stack tip:

  • scripts/run_tests.sh tests/hermes_cli/test_relay_shared_metrics.py tests/hermes_cli/test_relay_shared_metrics_runtime.py -q: 244 passed.
  • Full stacked regression set across Relay lifecycle, plugin, task, model, tool, skill, scheduler, store, and runtime coverage: 1,042 passed across 14 files.
  • .venv/bin/python scripts/smoke_nemo_relay_shared_metrics.py: passed with the installed Relay binding. The exported packages contain one payload-free hermes.client.active metric, preserve one anonymous install ID across delta packages, validate against the closed schema, and contain none of the prompt/model/tool/skill privacy canaries.
  • Cross-process activation regression confirms simultaneous starts increment the active-install metric once.
  • Two-profile regression confirms each HERMES_HOME receives a distinct install identity and isolated active counter.

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard provider/nvidia NVIDIA NIM telemetry Touches outbound telemetry, usage attribution, or analytics — needs opt-in gating before merge labels Jul 22, 2026
@afourniernv
afourniernv force-pushed the feat/hermes-relay-install-activation-metrics branch from 4af564f to 5006fb2 Compare July 22, 2026 15:30
@afourniernv

Copy link
Copy Markdown
Contributor Author

Two scope clarifications from review:

  • The activity latch is intentionally a rolling 24-hour sliding window, not calendar-day DAU. The implementation and PR description already use that behavior; I will avoid calling it DAU or daily-active.
  • The stable UUID is stored under one HERMES_HOME, so the current aggregation unit is an opted-in profile, not a physical Hermes installation. Profiles on the same machine receive separate IDs, and packages from one profile remain linkable until that profile's metrics state is reset.

I opened #69528 for the identity-scope decision. My recommendation for this local-first slice is to preserve the profile boundary and rename/document it as a random, resettable pseudonymous profile ID. Moving to a shared installation-level identity would first require explicit cross-profile consent, reset, deletion, and migration semantics.

@afourniernv
afourniernv force-pushed the feat/hermes-relay-install-activation-metrics branch 3 times, most recently from 68b2cdd to e20d4a0 Compare July 29, 2026 01:48
@afourniernv
afourniernv force-pushed the feat/hermes-relay-install-activation-metrics branch 4 times, most recently from 350803c to 21d5627 Compare July 29, 2026 18:51
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv
afourniernv force-pushed the feat/hermes-relay-install-activation-metrics branch from 21d5627 to 46bae75 Compare July 29, 2026 19:21
@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users area/install-update Installer, updater, packaging, wheels, doctor labels Jul 30, 2026
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	hermes_cli/observability/schemas/hermes.shared_metrics.v1.schema.json
#	scripts/smoke_nemo_relay_shared_metrics.py
#	tests/agent/test_skill_commands.py
#	tests/hermes_cli/test_relay_shared_metrics.py
#	tests/hermes_cli/test_relay_shared_metrics_runtime.py
#	tests/tools/test_skill_manager_tool.py
#	tests/tools/test_skill_usage.py
#	tests/tools/test_skills_tool.py
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	docs/observability/relay-shared-metrics.md
#	hermes_cli/observability/shared_metrics.py
#	tests/hermes_cli/test_relay_shared_metrics.py
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	hermes_cli/observability/schemas/hermes.shared_metrics.v1.schema.json
#	hermes_cli/observability/shared_metrics_contract.py
#	hermes_cli/observability/shared_metrics_subscriber.py
#	scripts/smoke_nemo_relay_shared_metrics.py
#	tests/hermes_cli/test_relay_shared_metrics.py
#	tests/hermes_cli/test_relay_shared_metrics_runtime.py
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	tests/tools/test_skills_hub.py
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	tests/run_agent/test_run_agent.py
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv
afourniernv marked this pull request as ready for review August 4, 2026 22:14
@jquesnelle
jquesnelle merged commit 6564f31 into NousResearch:main Aug 5, 2026
38 checks passed
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…lay-install-activation-metrics

feat(observability): add Relay active install metrics
33hodl pushed a commit to 33hodl/hermes-agent that referenced this pull request Aug 12, 2026
…lay-install-activation-metrics

feat(observability): add Relay active install metrics
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/install-update Installer, updater, packaging, wheels, doctor comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have provider/nvidia NVIDIA NIM sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data telemetry Touches outbound telemetry, usage attribution, or analytics — needs opt-in gating before merge type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants