feat(observability): report model and provider usage

[afourniernv]

What does this PR do?

This adds the model-route slice on top of the Hermes-to-Relay shared-metrics pipeline now present on main.

Each logical model call increments hermes.model_route.count using only the terminal model identifier and Hermes provider route, for example nvidia/nemotron-3-ultra through openrouter or zai/glm-5.2 through brev. When the provider returns a valid model ID, that terminal ID wins over the configured fallback.

Hermes does not maintain a model catalog or calculate model family, locality, pricing, latency, token, retry, or outcome dimensions in this metric. Model and provider identifiers are lowercased and structurally bounded; malformed values become unknown. Pricing and classification can be derived by the metrics backend.

The metric and package transition is backward-compatible with local state already created from main:

• hermes.model_call.count keeps its original bounded-family contract and is retained only for draining pending legacy rows.
• New code cannot record the retired metric; it emits hermes.model_route.count instead.
• Package schema hermes.shared_metrics.v1 remains unchanged for committed outbox payloads.
• New packages use hermes.shared_metrics.v2, which accepts both contracts so pending legacy and current counters can be packaged together.
• The Hermes-owned Relay event projection moves to hermes.metrics.event.v2.
• The SQLite layout remains schema 1; no database migration or new dependency is required.

The lifecycle remains correct across retries, provider fallback, and concurrent tasks: one logical request produces one model-route counter, and fallback records the terminal provider/model pair instead of the failed initial route.

Stack Order

1. This PR (feat(observability): report model and provider usage #68881) - model and provider usage.
2. feat(observability): aggregate bounded tool metrics #68882 - bounded tool lifecycle and approval metrics.
3. feat(observability): aggregate bounded skill metrics #68883 - skill lifecycle, provenance, reuse, and reuse-after-patch metrics.
4. feat(observability): add Relay client resource metrics #68978 - Relay client resource dimensions.
5. feat(observability): add Relay active install metrics #69416 - active-install metrics.
6. feat(observability): add Relay setup and first-use metrics #69437 - setup and first-use metrics.

Review and merge in this order. The cumulative follow-up branches need to be restacked onto this compatibility commit.

Related Issue

Follow-up to #67607.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• Add hermes.model_route.count with exactly two dimensions: model and provider.
• Prefer the accepted response model ID when valid, then fall back to the configured model ID.
• Keep identifiers open-ended rather than normalizing through a checked-in catalog.
• Collapse malformed, non-string, or oversized identifiers to unknown.
• Treat attempt errors as provisional so provider fallback remains one logical call.
• Key active calls by task and request ID so reused request IDs cannot cross-contaminate concurrent tasks.
• Preserve the original v1 package schema and legacy metric validator for existing state.
• Add package schema v2 for legacy/current coexistence without changing SQLite.
• Keep prompts, responses, endpoints, errors, session IDs, task IDs, and request IDs out of the shared-metrics event and package.
• Update the native smoke, observability documentation, and lifecycle/privacy/upgrade tests.

How to Test

1. Run HERMES_PYTHON=/path/to/python scripts/run_tests.sh -j 4 tests/hermes_cli/test_relay_shared_metrics.py tests/hermes_cli/test_relay_shared_metrics_runtime.py tests/agent/test_auxiliary_relay.py tests/run_agent/test_run_agent.py tests/scripts/test_smoke_nemo_relay_shared_metrics.py tests/plugins/test_nemo_relay_plugin.py -q.
2. Run python -m ruff check hermes_cli/observability/shared_metrics.py hermes_cli/observability/shared_metrics_contract.py hermes_cli/observability/shared_metrics_subscriber.py hermes_cli/observability/relay_shared_metrics.py scripts/smoke_nemo_relay_shared_metrics.py tests/hermes_cli/test_relay_shared_metrics.py tests/hermes_cli/test_relay_shared_metrics_runtime.py tests/plugins/test_nemo_relay_plugin.py.
3. Run ty check hermes_cli/observability/shared_metrics.py hermes_cli/observability/shared_metrics_contract.py hermes_cli/observability/shared_metrics_subscriber.py hermes_cli/observability/relay_shared_metrics.py.
4. Run python scripts/smoke_nemo_relay_shared_metrics.py.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• I've run the affected test suite
• I've added tests for my changes
• I've tested on my platform: macOS 26.5 arm64

Documentation & Housekeeping

• I've updated relevant documentation
• I've updated cli-config.yaml.example if I added/changed config keys — N/A, no config keys changed
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact — no platform-specific behavior added
• I've updated tool descriptions/schemas if I changed tool behavior — N/A

Screenshots / Logs

Validation on current head:

• Focused model/Relay/agent suite: 638 passed.
• Store compatibility suite: 73 passed, including pending-v1 aggregation and immutable-v1-outbox regressions.
• Ruff, ty, both JSON schemas, and git diff --check: passed.
• Native Hermes → Relay → SQLite → schema-validated v2 outbox smoke: passed.
• The smoke exported {"model":"gpt-relay-smoke-sensitive-model","provider":"custom"} as hermes.model_route.count; prompt and response canaries remained absent.
• Earlier live provider smokes covered Anthropic, OpenAI Chat Completions, OpenAI Responses, and NVIDIA endpoints; the terminal model/provider pair reached Relay, SQLite, and the outbox on each path.

[teknium1]

Thanks for preserving the v1 outbox compatibility path and retaining the opt-in shared-metrics gate.

Problems

• hermes_cli/observability/shared_metrics_contract.py:201 still accepts only metrics-owned hermes.model_call scopes. Auxiliary requests use agent/auxiliary_client.py:2597-2619 → agent/relay_llm.py:760, which creates relay_runtime.LOGICAL_LLM_SCOPE instead. Those events therefore cannot produce hermes.model_route.count, despite the PR’s stated logical-call coverage. The added tests/agent/test_auxiliary_relay.py exercises native Relay fallback but not the shared-metrics store/package path.

Suggested changes

• Add an auxiliary route projection (including fallback) to the shared-metrics lifecycle, with an integration test proving one terminal hermes.model_route.count; alternatively document the narrower primary-agent-only scope.

This is an automated hermes-sweeper review.

[jquesnelle]

@afourniernv could you fix up the small merge conflict here?

[afourniernv]

Caught up to current main and resolved the merge conflict in a97abcd. I propagated the updated head through the remaining metrics stack; all six PRs are mergeable and zero commits behind. Local cumulative validation: 917 passed, with one unrelated macOS /tmp test deselected after reproducing the same failure on current main.

[afourniernv]

@jquesnelle just wanted to check in here and see if this LGTY :) ?