feat(scheduler): add filter_duration_seconds and filter_denials_total…

[Rickydama3]

What type of PR is this?
/kind feature

What this PR does / why we need it:
This PR adds two new Prometheus metrics to the HAMi scheduler to track the latency and outcomes of the filtering phase:

1. hami_scheduler_filter_duration_seconds (Histogram)
2. hami_scheduler_filter_denials_total (Counter)

These metrics are essential for giving operators visibility into scheduling latency and error rates (e.g., debugging contention issues like #1662).

Fix/Implementation:

• Created pkg/scheduler/routes/metrics.go to define the new metrics.
• Added a time.Now() capture in route.go:PredicateRoute right before s.Filter(extenderArgs) is called.
• Interrogated the extenderFilterResult.FailedNodes map to correctly increment the denials counter with the specific node and reason label.
• Exposed the new metrics in the global registry inside cmd/scheduler/metrics.go.

Which issue(s) this PR fixes:
Part of #2126
Fixes #2458

Special notes for your reviewer:
This fulfills a major requirement of the LFX Mentorship observability gap analysis (#2126). The histogram buckets are configured exponentially from 10ms to 10s to accurately capture typical scheduling latency profiles.

Does this PR introduce a user-facing change?:

Feat: add `filter_duration_seconds` histogram and `filter_denials_total` counter to scheduler metrics

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

* **New Features**
  * Added scheduler metrics for filter execution duration.
  * Added denial counters categorized by node and rejection reason.
  * Registered scheduler route metrics with the Prometheus metrics registry.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

[hami-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: leondon3
Once this PR has been reviewed and has the lgtm label, please assign archlitchi for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8a955cb1-aeed-4690-ace5-ec6c899f4dfd

📥 Commits

Reviewing files that changed from the base of the PR and between 39a09fb and e8fba4f.

📒 Files selected for processing (2)

• pkg/scheduler/routes/metrics.go
• pkg/scheduler/routes/route.go

🚧 Files skipped from review as they are similar to previous changes (2)

• pkg/scheduler/routes/route.go
• pkg/scheduler/routes/metrics.go

---

📝 Walkthrough

Walkthrough

The scheduler routes package adds filter duration and denial metrics. Predicate filtering records duration by node and result, and denial counts by node and reason. Scheduler metrics initialization registers these metrics with the Prometheus registry.

Changes

Scheduler filter metrics

Layer / File(s)	Summary
Define and record filter metrics pkg/scheduler/routes/metrics.go, pkg/scheduler/routes/route.go	Adds FilterDuration and FilterDenials. Predicate filtering records filter duration and denial counts with node and result or reason labels.
Register scheduler route metrics cmd/scheduler/metrics.go	Registers scheduler route metrics during metrics initialization.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• Project-HAMi/HAMi#2455: Modifies cmd/scheduler/metrics.go to register scheduler-route metrics and also updates GPU ratio calculations.

Suggested labels: enhancement

Suggested reviewers: fouof

Poem

A rabbit tracks each filter run,
With node labels for everyone.
Denials count by reason clear,
Prometheus records them here.
Scheduler metrics now appear.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the scheduler metrics added by this pull request.
Linked Issues check	✅ Passed	The changes implement the requested filter duration and denial metrics, labels, instrumentation, and registration for issue #2458.
Out of Scope Changes check	✅ Passed	All reported changes directly support scheduler filter observability and the requirements in issue #2458.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/scheduler/routes/metrics.go`:
- Line 30: Update the bucket configuration in the metrics definition to use a
finite range ending at 10 seconds, such as ExponentialBucketsRange with 0.01 as
the minimum and 10 as the maximum across 10 buckets; preserve the existing
metric and bucket count.

In `@pkg/scheduler/routes/route.go`:
- Line 76: Declare err in the surrounding scope before the extender filter call,
then update the assignment in the Filter flow to use that declaration while
preserving the existing error handling.
- Around line 87-89: Update the failed-node metrics handling in the scheduler
route to translate each raw Filter failure reason into a bounded hami.io denial
code before passing it to FilterDenials.WithLabelValues. Preserve the existing
failedNode label and duration observation, and ensure device errors, node-unfit
messages, and node-unregistered messages map to their corresponding stable codes
rather than raw text.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 80df2507-7ab3-4f3c-89df-3df2602aa0dc

📥 Commits

Reviewing files that changed from the base of the PR and between 3616313 and 39a09fb.

📒 Files selected for processing (3)

• cmd/scheduler/metrics.go
• pkg/scheduler/routes/metrics.go
• pkg/scheduler/routes/route.go

[FouoF]

Thanks for the observability proposal. Under the default Chart path, Filter returns NodeNames, but this implementation reads only Nodes.Items, so successful/allowed metrics are not recorded. It also observes the latency of one Filter call once per candidate node and derives rejection reasons from message strings, without behavior tests or hosted CI. These values would be misleading as a public metric contract, so we are closing the current PR. A replacement should begin with exact event/value semantics and exposition tests.